Audio-reactive visualization
Bridge audio analysis and beat-detection state into the render pipeline via DataTexture and BeatDetector polling.
Audio-reactive visualization
An audio-reactive scene is one where visuals move, change color, or trigger effects in response to the music or sound currently playing. ExoJS gives you two building blocks for this: AudioAnalyser for per-frame spectrum data, and BeatDetector for rhythmic timing. You combine them inside update — sample what you need, map it to visual properties, and let the renderer handle the rest.
The pipeline
The flow is the same for any audio-reactive setup:
- Tap a live audio source (an
AudioBussuch asapp.audio.music, or aVoice) with anAudioAnalyserand/or aBeatDetector. - In
update, read spectrum values, beat envelopes, or subdivision phase. - Apply those readings to drawable properties — scale, position, tint, shader uniforms, particle spawns.
Both the analyser and the beat detector connect as parallel taps. They never affect the source’s main audio routing, so you can attach them to anything that’s already playing. They take a live source (bus, voice, AudioNode, or MediaStream) — not a Sound/AudioStream descriptor.
Building an analyser
An AudioAnalyser wraps a Web Audio AnalyserNode and exposes frequency and time-domain data. You point it at a source, then call its getters each frame:
class VisualizerScene extends Scene {
private music!: AudioStream;
private analyser!: AudioAnalyser;
private spectrum!: Uint8Array;
async load() {
this.music = await this.loader.load(Asset.type('music', 'audio/track.ogg'));
}
init() {
this.app.audio.play(this.music, { loop: true });
// Tap the bus the track plays through.
this.analyser = new AudioAnalyser({ fftSize: 1024 });
this.analyser.source = this.app.audio.music;
}
update(delta: Seconds) {
this.spectrum = this.analyser.getSpectrum();
}
}You can also pass source as a constructor option: new AudioAnalyser({ source: app.audio.music, fftSize: 1024 }). Either form works; the setter is useful when you need to switch sources at runtime. To analyse a single track in isolation, keep the Voice from play() and pass that instead of the bus.
Spectrum sampling strategies
The raw FFT returns N bins linearly spaced from 0 Hz to the Nyquist frequency (half the sample rate). For most visualizations this spacing is awkward — bass gets a handful of bins, treble gets hundreds.
AudioAnalyser gives you four ways to read the spectrum, each with a byte and a float variant:
| Method | Output | Use case |
|---|---|---|
getSpectrum() |
Uint8Array (0–255 per bin) |
Direct bar-graph visualizations |
getSpectrumFloat() |
Float32Array (dBFS per bin) |
Precise amplitude measurement |
getSpectrumMel() |
Uint8Array (0–255 per band) |
Perceptually-weighted display bands |
getSpectrumLog() |
Uint8Array (0–255 per band) |
Octave-uniform display (each octave gets equal visual width) |
The mel and log methods accept an optional bands parameter (default 32) and frequency range (fMin/fMax, default 20 Hz to 20 kHz, clamped to Nyquist). The filterbanks are built once per (bands, fMin, fMax, fftSize) combination and cached on the analyser instance — subsequent calls at the same parameters are just a weighted sum.
update(delta: Seconds) {
const mel32 = this.analyser.getSpectrumMel(undefined, { bands: 32 });
// mel32[0] is the lowest mel band, mel32[31] is the highest
const log64 = this.analyser.getSpectrumLog(undefined, { bands: 64 });
// log64[b] covers ~1/64 of the log2(fMax/fMin) octave range
// Quick band energy without building a filterbank
const bass = this.analyser.getBandEnergy(20, 250);
const { low, mid, high } = this.analyser.getLowMidHigh();
const overall = this.analyser.getRms();
}Beat-driven animation: polling vs. events
BeatDetector gives you both event-style signals and per-frame polling getters. Which one you use depends on what kind of visual you are driving.
The event signals — onBeat, onDownbeat, onBarStart, onTempoChange — fire when the worklet processor detects a beat and dispatches a message to the main thread. Use them for one-shot side effects: spawning a particle burst, triggering a screen flash, advancing a sequencer:
class BeatReactiveScene extends Scene {
private detector!: BeatDetector;
private flash = 0;
init() {
this.detector = new BeatDetector({ source: this.app.audio.music });
this.detector.onBeat.add(() => {
// Spawn particles or trigger another one-shot effect here.
this.flash = 0.3;
});
}
}For continuous animation — something that smoothly decays between beats rather than snapping — use the polling getters inside update:
| Getter | Returns | Description |
|---|---|---|
pulse |
number (0–1) |
Decaying envelope. Peaks at 1 on every beat, then halves every pulseHalfLife seconds (default 0.15). |
barPulse |
number (0–1) |
Same shape, but resets only on downbeats and decays per barPulseHalfLife (default 0.3). |
justBeat |
boolean |
true for the visual frame(s) within justBeatWindow seconds of a beat onset (default 0.03). |
secondsSinceLastBeat |
number |
Elapsed time in seconds since the most recent beat. Returns 0 before the detector locks. |
subdivisionPhase(n) |
number (0–1) |
Phase within an N-subdivision of the current beat. subdivisionPhase(4) gives 16th-note phase. |
These are all pure derivations from the detector’s internal state — they do not allocate, do not fire events, and are safe to call every frame:
const beat = this.detector.pulse;
const downbeat = this.detector.barPulse;
// Smoothly scale a sprite on every beat
this.sprite.setScale(1 + beat * 0.25);
// Brighter tint on the downbeat
const greenBlue = Math.round(255 * (1 - 0.4 * downbeat));
this.sprite.tint = new Color(255, greenBlue, greenBlue, 1);
// Flash white exactly on beat
if (this.detector.justBeat) {
this.sprite.tint = Color.white;
}
// Animate something on 16th notes
const sub = this.detector.subdivisionPhase(4);
if (sub < 0.05) this.sixteenthFlash = 0.1;Tune the envelope shapes with the mutable public fields pulseHalfLife, barPulseHalfLife, and justBeatWindow. Smaller values give snappier responses; larger values give longer afterglow.
justBeat can slip through at low frame rates
justBeat is only true for a frame or two per beat (a ~30 ms window), so a frame-rate dip below ~30fps can skip the window entirely. For effects that must never drop a beat — a sound cue, a scored hit — drive them from the onBeat signal, which is dispatched once per beat regardless of frame rate.
Spectrograms with DataTexture
When you want a scrolling spectrogram — a 2D texture that updates each frame with a new column of frequency data — use DataTexture. Its pixels live in a CPU-side typed array that you mutate directly, then upload to the GPU with commit() or commitRect().
DataTexture defaults to nearest-neighbor filtering with clamp-to-edge wrapping, which is what you want for spectrum data where bilinear filtering would corrupt sampled values.
Upload only the column that changed
A scrolling spectrogram rewrites just one column per frame. commitRect(x, y, width, height) re-uploads that single region to the GPU — far cheaper than commit(), which re-uploads the entire texture every frame.
A simple scrolling spectrogram:
init() {
this.analyser = new AudioAnalyser({ fftSize: 512 });
this.analyser.source = this.app.audio.music;
this.specTex = new DataTexture({
width: 256, // scroll history in columns
height: 64, // one row per mel band
format: TextureFormat.R8,
});
this.specSprite = new Sprite(this.specTex);
this.specSprite.setAnchor(0, 1);
this.specSprite.setPosition(0, this.app.height);
this.col = 0;
}
update(delta: Seconds) {
const bands = this.analyser.getSpectrumMel(undefined, { bands: 64 });
const buf = this.specTex.buffer;
// Write one column of spectrum data
for (let row = 0; row < 64; row++) {
buf[row * 256 + this.col] = bands[row];
}
this.specTex.commitRect(this.col, 0, 1, 64);
this.col = (this.col + 1) % 256;
}For ring-buffer patterns where only the newest column changes, commitRect(col, 0, 1, 64) uploads just that one-pixel-wide column — cheaper than uploading the whole texture every frame.
A compact practical scene
Here is a complete scene that combines an analyser-driven bar graph with beat-triggered background color changes:
class AudioReactiveScene extends Scene {
private music!: AudioStream;
private analyser!: AudioAnalyser;
private detector!: BeatDetector;
private bars!: Graphics;
async load() {
this.music = await this.loader.load(Asset.type('music', 'audio/track.ogg'));
}
init() {
const { width, height } = this.app;
this.app.audio.play(this.music, { loop: true });
// Both taps read the music bus the track plays through.
this.analyser = new AudioAnalyser({ source: this.app.audio.music, fftSize: 512 });
this.detector = new BeatDetector({ source: this.app.audio.music });
this.bars = new Graphics();
this.addChild(this.bars);
// The engine clears to this colour before `draw` runs, and
// `app.clearColor` is the backend's live instance - so mutating it is
// all it takes to animate the background.
this.app.clearColor.set(20, 24, 30, 1);
}
update(delta: Seconds) {
const bands = this.analyser.getSpectrumMel(undefined, { bands: 32 });
const { width, height } = this.app;
const barW = width / bands.length;
this.bars.clear();
for (let i = 0; i < bands.length; i++) {
const h = (bands[i] / 255) * height;
const t = i / Math.max(1, bands.length - 1);
this.bars.fillColor = new Color(Math.round(255 * t), Math.round(200 - 120 * t), Math.round(255 - 180 * t), 1);
this.bars.drawRectangle(i * barW, height - h, barW - 1, h);
}
// Pulse the background on beat
this.app.clearColor.set(20, 24, Math.round(30 + this.detector.pulse * 60), 1);
}
draw(context: RenderingContext) {
context.render(this.bars);
}
}
const app = new Application({ scenes: { AudioReactiveScene }, canvas: { width: 800, height: 600, mount: document.body } });
await app.start(AudioReactiveScene);Timing semantics
A few things to keep in mind when working with audio-driven visuals:
- The analyser spectrum reflects the current frame’s audio buffer. You sample it once per
updatecall; theAnalyserNode’ssmoothingTimeConstant(default 0.8) smooths between consecutive analyses. - Beat detector events are dispatched from the worklet processor to the main thread. They arrive asynchronously and may be arbitrarily close to or slightly behind the visual frame; the worklet’s internal temporal resolution is finer than
requestAnimationFramecadence. The polling getters (pulse,justBeat,subdivisionPhase) sample the detector’s most recent cached state on the main thread. justBeatistruefor at most one or two visual frames per beat (with the default 30ms window at 60fps). If your frame rate drops below ~30fps, you may miss ajustBeatwindow. For critical beat-triggered effects, prefer theonBeatsignal — it is dispatched per beat and delivered on the main thread.- Every timestamp the detector reports —
BeatInfo.audioTime,nextBeatTime, thelookaheadentries andanalysisTime— is anAudioContext.currentTimevalue, soAudioOutputClockconverts any of them straight onto theperformance.now()timeline. analysisTimenames the newest audio the current state describes, andanalysisLatencyis how far behind the context clock that state already was when the main thread received it. The figure is measured rather than assumed and covers the analysis hop together with the worklet-to-main-thread delivery, so it is a budget rather than an exact age.
Tempo confidence is not phase confidence
confidence says how sure the detector is of the tempo; phaseConfidence says how well recent onsets support the position of the beat grid. They come apart more often than they look like they should — a passage with a rock-solid pulse played with heavy rubato reads high confidence and low phase confidence, and so does anything the detector has had to free-run through because no onset arrived where it predicted one.
Gate anything that has to land exactly on the beat — a scored hit, a quantised trigger, a flash meant to be felt rather than seen — on phaseConfidence, and keep confidence for decisions about the tempo itself, such as whether to display a BPM readout at all:
import { BeatDetector } from '@codexo/exojs-audio-fx';
const detector = new BeatDetector();
const shouldScoreHits = (): boolean => detector.phaseConfidence > 0.6;
const shouldShowBpm = (): boolean => detector.confidence > 0.5;
Lining audio up with the frame clock
Web Audio schedules in AudioContext.currentTime and your frame loop measures in performance.now(). The two are different clocks with different origins, and currentTime runs ahead of what the listener actually hears by the length of the output path. Subtracting one from the other gives a number that looks plausible and is wrong by tens of milliseconds - enough to visibly desynchronise a rhythm game.
AudioOutputClock correlates them:
import { AudioOutputClock } from '@codexo/exojs';
const clock = new AudioOutputClock();
/** How long from now until the listener hears the sample scheduled for `contextTime`. */
const msUntilHeard = (contextTime: number): number => clock.contextToPerformanceTime(contextTime) - performance.now();
Where the browser supports AudioContext.getOutputTimestamp(), the correlation names the sample the device is playing at this instant, so the converted time already accounts for the output path and needs no latency arithmetic of your own. Where it does not, the clock pairs currentTime with performance.now() and marks the snapshot 'estimated' so you can tell the two cases apart:
import { AudioOutputClock } from '@codexo/exojs';
const snapshot = new AudioOutputClock().snapshot();
if (snapshot.source === 'estimated') {
// No output timestamp here: the correlation leads the true output by roughly
// `outputLatency`, which this environment may not report either.
}
Display latency is deliberately outside this: only your code knows how far ahead of the photons its render loop runs, so the decision of which frame to draw a beat on stays yours.
Examples
A full audio visualisation: frequency-domain bars, time-domain waveform overlay, and per-band energy meters drawn on a 2D canvas then uploaded as a texture.
A sprite pulses on every beat; particles burst on the onBeat signal.
Where to go next
For color-oriented effects driven by audio — palette cycling, noise overlays, shader filters — see Filters. To write a custom shader that samples a spectrogram DataTexture, see Custom mesh shaders. If you came here before reading Beat detection, that chapter covers the full event API, tempo tracking, and frequency band state in detail.

