Guide Audio Audio-reactive visualization

Audio-reactive visualization

Bridge audio analysis and beat-detection state into the render pipeline via DataTexture and BeatDetector polling.

Intermediate ~5 min read

What you'll learn

map audio analysis to visuals
build a responsive audio-reactive scene

Before you start

Beat detection

Audio-reactive visualization

An audio-reactive scene is one where visuals move, change color, or trigger effects in response to the music or sound currently playing. ExoJS gives you two building blocks for this: AudioAnalyser for per-frame spectrum data, and BeatDetector for rhythmic timing. You combine them inside update — sample what you need, map it to visual properties, and let the renderer handle the rest.

The pipeline

The flow is the same for any audio-reactive setup:

Tap an audio source (Sound, Music, AudioBus) with an AudioAnalyser and/or a BeatDetector.
In update, read spectrum values, beat envelopes, or subdivision phase.
Apply those readings to drawable properties — scale, position, tint, shader uniforms, particle spawns.

Both the analyser and the beat detector connect as parallel taps. They never affect the source’s main audio routing, so you can attach them to anything that’s already playing.

Building an analyser

An AudioAnalyser wraps a Web Audio AnalyserNode and exposes frequency and time-domain data. You point it at a source, then call its getters each frame:

import { AudioAnalyser, Music, Scene } from '@codexo/exojs';

class VisualizerScene extends Scene {
    async load(loader) {
        await loader.load(Music, { track: 'audio/track.ogg' });
    }

    init(loader) {
        this.music = loader.get(Music, 'track').setLoop(true).play();
        this.analyser = new AudioAnalyser({ fftSize: 1024 });
        this.analyser.source = this.music;
    }

    update(delta) {
        this.spectrum = this.analyser.getSpectrum();
    }
}

You can also pass source as a constructor option: new AudioAnalyser({ source: this.music, fftSize: 1024 }). Either form works; the setter is useful when you need to switch sources at runtime.

Spectrum sampling strategies

The raw FFT returns N bins linearly spaced from 0 Hz to the Nyquist frequency (half the sample rate). For most visualizations this spacing is awkward — bass gets a handful of bins, treble gets hundreds.

AudioAnalyser gives you four ways to read the spectrum, each with a byte and a float variant:

Method	Output	Use case
`getSpectrum()`	`Uint8Array` (0–255 per bin)	Direct bar-graph visualizations
`getSpectrumFloat()`	`Float32Array` (dBFS per bin)	Precise amplitude measurement
`getSpectrumMel()`	`Uint8Array` (0–255 per band)	Perceptually-weighted display bands
`getSpectrumLog()`	`Uint8Array` (0–255 per band)	Octave-uniform display (each octave gets equal visual width)

The mel and log methods accept an optional bands parameter (default 32) and frequency range (fMin/fMax, default 20 Hz to 20 kHz, clamped to Nyquist). The filterbanks are built once per (bands, fMin, fMax, fftSize) combination and cached on the analyser instance — subsequent calls at the same parameters are just a weighted sum.

update(delta) {
    const mel32 = this.analyser.getSpectrumMel(null, { bands: 32 });
    // mel32[0] is the lowest mel band, mel32[31] is the highest

    const log64 = this.analyser.getSpectrumLog(null, { bands: 64 });
    // log64[b] covers ~1/64 of the log2(fMax/fMin) octave range

    // Quick band energy without building a filterbank
    const bass = this.analyser.getBandEnergy(20, 250);
    const { low, mid, high } = this.analyser.getLowMidHigh();
    const overall = this.analyser.getRms();
}

Beat-driven animation: polling vs. events

BeatDetector gives you both event-style signals and per-frame polling getters. Which one you use depends on what kind of visual you are driving.

The event signals — onBeat, onDownbeat, onBarStart, onTempoChange — fire when the worklet processor detects a beat and dispatches a message to the main thread. Use them for one-shot side effects: spawning a particle burst, triggering a screen flash, advancing a sequencer:

this.detector = new BeatDetector();
this.detector.source = this.music;

this.detector.onBeat.add(() => {
    this.particleBurst.reset();
    this.flash = 0.3;
});

For continuous animation — something that smoothly decays between beats rather than snapping — use the polling getters inside update:

Getter	Returns	Description
`pulse`	`number` (0–1)	Decaying envelope. Peaks at 1 on every beat, then halves every `pulseHalfLife` seconds (default 0.15).
`barPulse`	`number` (0–1)	Same shape, but resets only on downbeats and decays per `barPulseHalfLife` (default 0.3).
`justBeat`	`boolean`	`true` for the visual frame(s) within `justBeatWindow` seconds of a beat onset (default 0.03).
`secondsSinceLastBeat`	`number`	Elapsed time in seconds since the most recent beat. Returns 0 before the detector locks.
`subdivisionPhase(n)`	`number` (0–1)	Phase within an N-subdivision of the current beat. `subdivisionPhase(4)` gives 16th-note phase.

These are all pure derivations from the detector’s internal state — they do not allocate, do not fire events, and are safe to call every frame:

import { Color } from '@codexo/exojs';

update(delta) {
    const beat = this.detector.pulse;
    const downbeat = this.detector.barPulse;

    // Smoothly scale a sprite on every beat
    this.sprite.setScale(1 + beat * 0.25);

    // Brighter tint on the downbeat
    const greenBlue = Math.round(255 * (1 - 0.4 * downbeat));
    this.sprite.tint = new Color(255, greenBlue, greenBlue, 1);

    // Flash white exactly on beat
    if (this.detector.justBeat) {
        this.sprite.tint = Color.white;
    }

    // Animate something on 16th notes
    const sub = this.detector.subdivisionPhase(4);
    if (sub < 0.05) this.sixteenthFlash = 0.1;
}

Tune the envelope shapes with the mutable public fields pulseHalfLife, barPulseHalfLife, and justBeatWindow. Smaller values give snappier responses; larger values give longer afterglow.

Spectrograms with DataTexture

When you want a scrolling spectrogram — a 2D texture that updates each frame with a new column of frequency data — use DataTexture. Its pixels live in a CPU-side typed array that you mutate directly, then upload to the GPU with commit() or commitRect().

DataTexture defaults to nearest-neighbor filtering with clamp-to-edge wrapping, which is what you want for spectrum data where bilinear filtering would corrupt sampled values.

A simple scrolling spectrogram:

import { DataTexture, Sprite } from '@codexo/exojs';

init(loader) {
    this.analyser = new AudioAnalyser({ fftSize: 512 });
    this.analyser.source = this.music;

    this.specTex = new DataTexture({
        width: 256,    // scroll history in columns
        height: 64,    // one row per mel band
        format: 'r8',
    });

    this.specSprite = new Sprite(this.specTex);
    this.specSprite.setAnchor(0, 1);
    this.specSprite.setPosition(0, this.app.canvas.height);

    this.col = 0;
}

update(delta) {
    const bands = this.analyser.getSpectrumMel(null, { bands: 64 });
    const buf = this.specTex.buffer;

    // Write one column of spectrum data
    for (let row = 0; row < 64; row++) {
        buf[row * 256 + this.col] = bands[row];
    }

    this.specTex.commitRect(this.col, 0, 1, 64);
    this.col = (this.col + 1) % 256;
}

For ring-buffer patterns where only the newest column changes, commitRect(col, 0, 1, 64) uploads just that one-pixel-wide column — cheaper than uploading the whole texture every frame.

A compact practical scene

Here is a complete scene that combines an analyser-driven bar graph with beat-triggered background color changes:

import {
    Application, AudioAnalyser, BeatDetector,
    Color, Graphics, Music, Scene,
} from '@codexo/exojs';

const app = new Application({ canvas: { width: 800, height: 600 } });
document.body.append(app.canvas);

class AudioReactiveScene extends Scene {
    async load(loader) {
        await loader.load(Music, { track: 'audio/track.ogg' });
    }

    init(loader) {
        const { width, height } = this.app.canvas;

        this.music = loader.get(Music, 'track').setLoop(true).play();

        this.analyser = new AudioAnalyser({ source: this.music, fftSize: 512 });
        this.detector = new BeatDetector({ source: this.music });

        this.bars = new Graphics();
        this.addChild(this.bars);
        this._bg = new Color(20, 24, 30, 1);
    }

    update(delta) {
        const bands = this.analyser.getSpectrumMel(null, { bands: 32 });
        const { width, height } = this.app.canvas;
        const barW = width / bands.length;

        this.bars.clear();
        for (let i = 0; i < bands.length; i++) {
            const h = (bands[i] / 255) * height;
            const t = i / Math.max(1, bands.length - 1);
            this.bars.fillColor = new Color(
                Math.round(255 * t),
                Math.round(200 - 120 * t),
                Math.round(255 - 180 * t),
                1,
            );
            this.bars.drawRectangle(
                i * barW, height - h,
                barW - 1, h,
            );
        }

        // Pulse the background on beat
        this._bg = new Color(20, 24, Math.round(30 + this.detector.pulse * 60), 1);
    }

    draw(context) {
        context.backend.clear(this._bg);
        context.render(this.bars);
    }
}

app.start(new AudioReactiveScene());

Timing semantics

A few things to keep in mind when working with audio-driven visuals:

The analyser spectrum reflects the current frame’s audio buffer. You sample it once per update call; the AnalyserNode’s smoothingTimeConstant (default 0.8) smooths between consecutive analyses.
Beat detector events are dispatched from the worklet processor to the main thread. They arrive asynchronously and may be arbitrarily close to or slightly behind the visual frame; the worklet’s internal temporal resolution is finer than requestAnimationFrame cadence. The polling getters (pulse, justBeat, subdivisionPhase) sample the detector’s most recent cached state on the main thread.
justBeat is true for at most one or two visual frames per beat (with the default 30ms window at 60fps). If your frame rate drops below ~30fps, you may miss a justBeat window. For critical beat-triggered effects, prefer the onBeat signal — it is dispatched per beat and delivered on the main thread.

Examples

Audio Visualisation Pointer Audio Open in Playground View source

import { Application, AudioAnalyser, BeatDetector, Color, Music, Scene, Sprite, Text, Texture } from '@codexo/exojs';
import { mountControls } from '@examples/runtime';

const app = new Application({
    canvas: {
        width: 1280,
        height: 720,
        mount: document.body,
        sizingMode: 'fit',
    },
    clearColor: Color.black,
});

class AudioVisualisationScene extends Scene {
    private music!: Music;
    private analyser!: AudioAnalyser;
    private detector!: BeatDetector;
    private canvas!: HTMLCanvasElement;
    private context!: CanvasRenderingContext2D;
    private gradientStyle!: CanvasGradient;
    private progressStyle = 'rgba(255, 255, 255, 0.1)';
    private texture!: Texture;
    private screen!: Sprite;
    private hud!: ReturnType<typeof mountControls>;
    private tapPrompt!: Text;

    override async load(loader): Promise<void> {
        await loader.load(Music, { example: assets.demo.audio.musicLoop });
    }

    override init(loader): void {
        const { width, height } = this.app.canvas;

        this.music = loader.get(Music, 'example');

        // One analyser tap for spectrum/waveform, one beat detector for the
        // beat-pulse ring. Both read the same track without altering playback.
        this.analyser = new AudioAnalyser({ source: this.music });
        this.detector = new BeatDetector();
        this.detector.source = this.music;

        this.canvas = document.createElement('canvas');
        this.canvas.style.position = 'absolute';
        this.canvas.style.top = '12.5%';
        this.canvas.style.left = '0';
        this.canvas.width = width;
        this.canvas.height = height;

        this.context = this.canvas.getContext('2d')!;
        this.context.strokeStyle = '#fff';
        this.context.lineWidth = 4;
        this.context.lineCap = 'round';
        this.context.lineJoin = 'round';

        this.gradientStyle = this.context.createLinearGradient(0, 0, 0, this.canvas.height);
        this.gradientStyle.addColorStop(0, '#f70');
        this.gradientStyle.addColorStop(0.5, '#f30');
        this.gradientStyle.addColorStop(1, '#f70');

        this.texture = new Texture(this.canvas);
        this.screen = new Sprite(this.texture);

        this.hud = mountControls({
            title: 'Audio Visualisation',
            controls: [{ keys: 'Click', action: 'play / pause' }],
            status: 'Playing…',
            hint: 'Frequency bars, waveform, and a beat-pulse ring — all driven by live AudioAnalyser + BeatDetector data.',
        });

        // Shown while the browser still blocks audio (`app.audio.locked`); the
        // first click or keypress unlocks it and the queued music starts.
        this.tapPrompt = new Text('Click or press any key to start the music', { fillColor: Color.white, fontSize: 22, align: 'center' })
            .setAnchor(0.5, 0.5)
            .setPosition(width / 2, height - 64);

        this.app.input.onPointerDown.add(() => {
            // The first gesture only unlocks audio (core auto-starts the queued
            // track); subsequent clicks toggle play / pause.
            if (this.app.audio.locked) {
                return;
            }

            this.music.toggle();
            this.hud.setStatus(this.music.paused ? 'Paused' : 'Playing…');
        });

        // Core defers playback until the AudioContext unlocks on the first
        // gesture, then starts automatically — just call play().
        this.music.setLoop(true).setVolume(0.8).play();
    }

    override draw(context): void {
        const canvas = this.canvas;
        const freqData = this.analyser.getSpectrum();
        const timeDomain = this.analyser.getWaveform();
        const width = canvas.width;
        const height = canvas.height;
        const length = freqData.length;
        const barWidth = Math.ceil(width / length);

        this.context.clearRect(0, 0, width, height);

        // Progress bar of the looping track.
        this.context.fillStyle = this.progressStyle;
        this.context.fillRect(0, 0, width * this.music.progress, height);

        // Frequency bars + waveform polyline.
        this.context.fillStyle = this.gradientStyle;
        this.context.beginPath();

        for (let i = 0; i < length; i++) {
            const barHeight = (height * freqData[i]) / 255;
            const lineHeight = (height * timeDomain[i]) / 255;
            const offsetX = (i * barWidth) | 0;

            this.context.fillRect(offsetX, (height / 2 - barHeight / 2) | 0, barWidth, barHeight | 0);
            this.context.lineTo(offsetX, (height * 0.75 - lineHeight / 2) | 0);
        }

        this.context.stroke();

        // Beat-pulse ring: radius and opacity follow the detector's beat
        // envelope, so it expands on every beat and sits perfectly still in
        // silence (pulse is 0 until a tempo locks).
        const pulse = this.detector.pulse;
        const cx = width / 2;
        const cy = height / 2;
        const baseRadius = Math.min(width, height) * 0.18;
        const radius = baseRadius * (1 + pulse * 0.6);

        this.context.beginPath();
        this.context.arc(cx, cy, radius, 0, Math.PI * 2);
        this.context.lineWidth = 6 + pulse * 10;
        this.context.strokeStyle = `rgba(120, 220, 255, ${0.15 + pulse * 0.7})`;
        this.context.stroke();

        // restore the default stroke style for the next frame's waveform.
        this.context.strokeStyle = '#fff';
        this.context.lineWidth = 4;

        this.screen.updateTexture();

        context.backend.clear();
        context.render(this.screen);

        if (this.app.audio.locked) {
            context.render(this.tapPrompt);
        }
    }
}

app.start(new AudioVisualisationScene());

A full audio visualisation: frequency-domain bars, time-domain waveform overlay, and per-band energy meters drawn on a 2D canvas then uploaded as a texture.

Beat Sync Pulse Audio Open in Playground View source

import { Application, BeatDetector, Color, Music, Scene, Sprite, Text, Texture, Vector } from '@codexo/exojs';
import {
    AlphaFadeOverLifetime,
    BurstSpawn,
    ConeDirection,
    Constant,
    particlesExtension,
    ParticleSystem,
} from '@codexo/exojs-particles';
import { mountControlPanel, mountControls } from '@examples/runtime';

const app = new Application({
    canvas: {
        width: 1280,
        height: 720,
        mount: document.body,
        sizingMode: 'fit',
    },
    clearColor: Color.black,
    loader: {
        basePath: 'assets/',
    },
    extensions: [particlesExtension],
});

class BeatSyncPulseScene extends Scene {
    private music!: Music;
    private detector!: BeatDetector;
    private sprite!: Sprite;
    private pulse = 0;
    private intensity = 0.28;
    private beats = 0;
    private particles!: ParticleSystem;
    private burst!: BurstSpawn;
    private hud!: ReturnType<typeof mountControls>;
    private tapPrompt!: Text;

    override async load(loader): Promise<void> {
        await loader.load(Texture, { bunny: 'image/ship-a.png', particle: 'image/particle-light.png' });
        await loader.load(Music, { track: 'audio/demo-loop-main.ogg' });
    }

    override init(loader): void {
        const { width, height } = this.app.canvas;

        this.music = loader.get(Music, 'track');
        this.sprite = new Sprite(loader.get(Texture, 'bunny')).setAnchor(0.5).setPosition(width / 2, height / 2);

        this.hud = mountControls({
            title: 'Beat Sync Pulse',
            hint: 'The ring and particle burst fire on each detected beat.',
        });

        this.particles = new ParticleSystem(loader.get(Texture, 'particle'), { capacity: 3500 });
        this.particles.setPosition(width / 2, height / 2);
        this.burst = new BurstSpawn({
            schedule: [{ time: 0, count: 90 }],
            lifetime: new Constant(0.6),
            position: new Constant(new Vector(0, 0)),
            velocity: ConeDirection.omni(80, 240),
            scale: new Constant(new Vector(0.2, 0.2)),
        });
        this.particles.addSpawnModule(this.burst);
        this.particles.addUpdateModule(new AlphaFadeOverLifetime());

        // Shown while the browser still blocks audio (`app.audio.locked`); the
        // first click or keypress unlocks it and the queued music starts.
        this.tapPrompt = new Text('Click or press any key to start the music', { fillColor: Color.white, fontSize: 22, align: 'center' })
            .setAnchor(0.5, 0.5)
            .setPosition(width / 2, height - 64);

        mountControlPanel({ title: 'Tuning' }).addSlider({
            label: 'Pulse intensity',
            min: 0.1,
            max: 0.5,
            step: 0.01,
            value: this.intensity,
            onChange: value => {
                this.intensity = value;
            },
        });

        this.detector = new BeatDetector();
        this.detector.source = this.music;
        this.detector.onBeat.add(() => {
            this.pulse = this.intensity;
            this.burst.reset();
            this.beats += 1;
            this.hud.setStatus(`Beats detected: ${this.beats}`);
        });

        // Core defers playback until the AudioContext unlocks on the first
        // gesture, then starts automatically — just call play().
        this.music.setLoop(true).setVolume(0.8).play();
    }

    override update(delta): void {
        this.pulse = Math.max(0, this.pulse - delta.seconds * 1.2);
        this.sprite.setScale(1 + this.pulse);
        this.particles.update(delta);
    }

    override draw(context): void {
        context.backend.clear();
        context.render(this.particles);
        context.render(this.sprite);

        if (this.app.audio.locked) {
            context.render(this.tapPrompt);
        }
    }
}

app.start(new BeatSyncPulseScene());

A sprite pulses on every beat; particles burst on the onBeat signal.

Where to go next

For color-oriented effects driven by audio — palette cycling, noise overlays, shader filters — see Filters. To write a custom shader that samples a spectrogram DataTexture, see Custom mesh shaders. If you came here before reading Beat detection, that chapter covers the full event API, tempo tracking, and frequency band state in detail.