Guide

GuideAudioAudio-reactive visualization

Audio-reactive visualization

Bridge audio analysis and beat-detection state into the render pipeline via DataTexture and BeatDetector polling.

Intermediate~8 min read

What you'll learn

  • map audio analysis to visuals
  • build a responsive audio-reactive scene

Before you start

Audio-reactive visualization

An audio-reactive scene is one where visuals move, change color, or trigger effects in response to the music or sound currently playing. ExoJS gives you two building blocks for this: AudioAnalyser for per-frame spectrum data, and BeatDetector for rhythmic timing. You combine them inside update — sample what you need, map it to visual properties, and let the renderer handle the rest.

The pipeline

The flow is the same for any audio-reactive setup:

  1. Tap a live audio source (an AudioBus such as app.audio.music, or a Voice) with an AudioAnalyser and/or a BeatDetector.
  2. In update, read spectrum values, beat envelopes, or subdivision phase.
  3. Apply those readings to drawable properties — scale, position, tint, shader uniforms, particle spawns.

Both the analyser and the beat detector connect as parallel taps. They never affect the source’s main audio routing, so you can attach them to anything that’s already playing. They take a live source (bus, voice, AudioNode, or MediaStream) — not a Sound/AudioStream descriptor.

Building an analyser

An AudioAnalyser wraps a Web Audio AnalyserNode and exposes frequency and time-domain data. You point it at a source, then call its getters each frame:

examples/guides/audio-reactive-visualization/visualizer-scene.ts
class VisualizerScene extends Scene {
  private music!: AudioStream;
  private analyser!: AudioAnalyser;
  private spectrum!: Uint8Array;

  async load() {
    this.music = await this.loader.load(Asset.type('music', 'audio/track.ogg'));
  }

  init() {
    this.app.audio.play(this.music, { loop: true });

    // Tap the bus the track plays through.
    this.analyser = new AudioAnalyser({ fftSize: 1024 });
    this.analyser.source = this.app.audio.music;
  }

  update(delta: Seconds) {
    this.spectrum = this.analyser.getSpectrum();
  }
}

You can also pass source as a constructor option: new AudioAnalyser({ source: app.audio.music, fftSize: 1024 }). Either form works; the setter is useful when you need to switch sources at runtime. To analyse a single track in isolation, keep the Voice from play() and pass that instead of the bus.

Spectrum sampling strategies

The raw FFT returns N bins linearly spaced from 0 Hz to the Nyquist frequency (half the sample rate). For most visualizations this spacing is awkward — bass gets a handful of bins, treble gets hundreds.

AudioAnalyser gives you four ways to read the spectrum, each with a byte and a float variant:

Method Output Use case
getSpectrum() Uint8Array (0–255 per bin) Direct bar-graph visualizations
getSpectrumFloat() Float32Array (dBFS per bin) Precise amplitude measurement
getSpectrumMel() Uint8Array (0–255 per band) Perceptually-weighted display bands
getSpectrumLog() Uint8Array (0–255 per band) Octave-uniform display (each octave gets equal visual width)

The mel and log methods accept an optional bands parameter (default 32) and frequency range (fMin/fMax, default 20 Hz to 20 kHz, clamped to Nyquist). The filterbanks are built once per (bands, fMin, fMax, fftSize) combination and cached on the analyser instance — subsequent calls at the same parameters are just a weighted sum.

examples/guides/audio-reactive-visualization/spectrum-bands.ts
update(delta: Seconds) {
  const mel32 = this.analyser.getSpectrumMel(undefined, { bands: 32 });
  // mel32[0] is the lowest mel band, mel32[31] is the highest

  const log64 = this.analyser.getSpectrumLog(undefined, { bands: 64 });
  // log64[b] covers ~1/64 of the log2(fMax/fMin) octave range

  // Quick band energy without building a filterbank
  const bass = this.analyser.getBandEnergy(20, 250);
  const { low, mid, high } = this.analyser.getLowMidHigh();
  const overall = this.analyser.getRms();
}

Beat-driven animation: polling vs. events

BeatDetector gives you both event-style signals and per-frame polling getters. Which one you use depends on what kind of visual you are driving.

The event signals — onBeat, onDownbeat, onBarStart, onTempoChange — fire when the worklet processor detects a beat and dispatches a message to the main thread. Use them for one-shot side effects: spawning a particle burst, triggering a screen flash, advancing a sequencer:

examples/guides/audio-reactive-visualization/beat-reactive.ts
class BeatReactiveScene extends Scene {
  private detector!: BeatDetector;
  private flash = 0;

  init() {
    this.detector = new BeatDetector({ source: this.app.audio.music });
    this.detector.onBeat.add(() => {
      // Spawn particles or trigger another one-shot effect here.
      this.flash = 0.3;
    });
  }
}

For continuous animation — something that smoothly decays between beats rather than snapping — use the polling getters inside update:

Getter Returns Description
pulse number (0–1) Decaying envelope. Peaks at 1 on every beat, then halves every pulseHalfLife seconds (default 0.15).
barPulse number (0–1) Same shape, but resets only on downbeats and decays per barPulseHalfLife (default 0.3).
justBeat boolean true for the visual frame(s) within justBeatWindow seconds of a beat onset (default 0.03).
secondsSinceLastBeat number Elapsed time in seconds since the most recent beat. Returns 0 before the detector locks.
subdivisionPhase(n) number (0–1) Phase within an N-subdivision of the current beat. subdivisionPhase(4) gives 16th-note phase.

These are all pure derivations from the detector’s internal state — they do not allocate, do not fire events, and are safe to call every frame:

examples/guides/audio-reactive-visualization/beat-visuals.ts
const beat = this.detector.pulse;
const downbeat = this.detector.barPulse;

// Smoothly scale a sprite on every beat
this.sprite.setScale(1 + beat * 0.25);

// Brighter tint on the downbeat
const greenBlue = Math.round(255 * (1 - 0.4 * downbeat));
this.sprite.tint = new Color(255, greenBlue, greenBlue, 1);

// Flash white exactly on beat
if (this.detector.justBeat) {
  this.sprite.tint = Color.white;
}

// Animate something on 16th notes
const sub = this.detector.subdivisionPhase(4);
if (sub < 0.05) this.sixteenthFlash = 0.1;

Tune the envelope shapes with the mutable public fields pulseHalfLife, barPulseHalfLife, and justBeatWindow. Smaller values give snappier responses; larger values give longer afterglow.

Spectrograms with DataTexture

When you want a scrolling spectrogram — a 2D texture that updates each frame with a new column of frequency data — use DataTexture. Its pixels live in a CPU-side typed array that you mutate directly, then upload to the GPU with commit() or commitRect().

DataTexture defaults to nearest-neighbor filtering with clamp-to-edge wrapping, which is what you want for spectrum data where bilinear filtering would corrupt sampled values.

A simple scrolling spectrogram:

examples/guides/audio-reactive-visualization/spectrogram.ts
init() {
  this.analyser = new AudioAnalyser({ fftSize: 512 });
  this.analyser.source = this.app.audio.music;

  this.specTex = new DataTexture({
    width: 256, // scroll history in columns
    height: 64, // one row per mel band
    format: TextureFormat.R8,
  });

  this.specSprite = new Sprite(this.specTex);
  this.specSprite.setAnchor(0, 1);
  this.specSprite.setPosition(0, this.app.height);

  this.col = 0;
}

update(delta: Seconds) {
  const bands = this.analyser.getSpectrumMel(undefined, { bands: 64 });
  const buf = this.specTex.buffer;

  // Write one column of spectrum data
  for (let row = 0; row < 64; row++) {
    buf[row * 256 + this.col] = bands[row];
  }

  this.specTex.commitRect(this.col, 0, 1, 64);
  this.col = (this.col + 1) % 256;
}

For ring-buffer patterns where only the newest column changes, commitRect(col, 0, 1, 64) uploads just that one-pixel-wide column — cheaper than uploading the whole texture every frame.

A compact practical scene

Here is a complete scene that combines an analyser-driven bar graph with beat-triggered background color changes:

examples/guides/audio-reactive-visualization/full-visualizer.ts

class AudioReactiveScene extends Scene {
  private music!: AudioStream;
  private analyser!: AudioAnalyser;
  private detector!: BeatDetector;
  private bars!: Graphics;

  async load() {
    this.music = await this.loader.load(Asset.type('music', 'audio/track.ogg'));
  }

  init() {
    const { width, height } = this.app;

    this.app.audio.play(this.music, { loop: true });

    // Both taps read the music bus the track plays through.
    this.analyser = new AudioAnalyser({ source: this.app.audio.music, fftSize: 512 });
    this.detector = new BeatDetector({ source: this.app.audio.music });

    this.bars = new Graphics();
    this.addChild(this.bars);

    // The engine clears to this colour before `draw` runs, and
    // `app.clearColor` is the backend's live instance - so mutating it is
    // all it takes to animate the background.
    this.app.clearColor.set(20, 24, 30, 1);
  }

  update(delta: Seconds) {
    const bands = this.analyser.getSpectrumMel(undefined, { bands: 32 });
    const { width, height } = this.app;
    const barW = width / bands.length;

    this.bars.clear();
    for (let i = 0; i < bands.length; i++) {
      const h = (bands[i] / 255) * height;
      const t = i / Math.max(1, bands.length - 1);
      this.bars.fillColor = new Color(Math.round(255 * t), Math.round(200 - 120 * t), Math.round(255 - 180 * t), 1);
      this.bars.drawRectangle(i * barW, height - h, barW - 1, h);
    }

    // Pulse the background on beat
    this.app.clearColor.set(20, 24, Math.round(30 + this.detector.pulse * 60), 1);
  }

  draw(context: RenderingContext) {
    context.render(this.bars);
  }
}

const app = new Application({ scenes: { AudioReactiveScene }, canvas: { width: 800, height: 600, mount: document.body } });

await app.start(AudioReactiveScene);

Timing semantics

A few things to keep in mind when working with audio-driven visuals:

  • The analyser spectrum reflects the current frame’s audio buffer. You sample it once per update call; the AnalyserNode’s smoothingTimeConstant (default 0.8) smooths between consecutive analyses.
  • Beat detector events are dispatched from the worklet processor to the main thread. They arrive asynchronously and may be arbitrarily close to or slightly behind the visual frame; the worklet’s internal temporal resolution is finer than requestAnimationFrame cadence. The polling getters (pulse, justBeat, subdivisionPhase) sample the detector’s most recent cached state on the main thread.
  • justBeat is true for at most one or two visual frames per beat (with the default 30ms window at 60fps). If your frame rate drops below ~30fps, you may miss a justBeat window. For critical beat-triggered effects, prefer the onBeat signal — it is dispatched per beat and delivered on the main thread.
  • Every timestamp the detector reports — BeatInfo.audioTime, nextBeatTime, the lookahead entries and analysisTime — is an AudioContext.currentTime value, so AudioOutputClock converts any of them straight onto the performance.now() timeline.
  • analysisTime names the newest audio the current state describes, and analysisLatency is how far behind the context clock that state already was when the main thread received it. The figure is measured rather than assumed and covers the analysis hop together with the worklet-to-main-thread delivery, so it is a budget rather than an exact age.

Tempo confidence is not phase confidence

confidence says how sure the detector is of the tempo; phaseConfidence says how well recent onsets support the position of the beat grid. They come apart more often than they look like they should — a passage with a rock-solid pulse played with heavy rubato reads high confidence and low phase confidence, and so does anything the detector has had to free-run through because no onset arrived where it predicted one.

Gate anything that has to land exactly on the beat — a scored hit, a quantised trigger, a flash meant to be felt rather than seen — on phaseConfidence, and keep confidence for decisions about the tempo itself, such as whether to display a BPM readout at all:

import { BeatDetector } from '@codexo/exojs-audio-fx';

const detector = new BeatDetector();

const shouldScoreHits = (): boolean => detector.phaseConfidence > 0.6;
const shouldShowBpm = (): boolean => detector.confidence > 0.5;

Lining audio up with the frame clock

Web Audio schedules in AudioContext.currentTime and your frame loop measures in performance.now(). The two are different clocks with different origins, and currentTime runs ahead of what the listener actually hears by the length of the output path. Subtracting one from the other gives a number that looks plausible and is wrong by tens of milliseconds - enough to visibly desynchronise a rhythm game.

AudioOutputClock correlates them:

import { AudioOutputClock } from '@codexo/exojs';

const clock = new AudioOutputClock();

/** How long from now until the listener hears the sample scheduled for `contextTime`. */
const msUntilHeard = (contextTime: number): number => clock.contextToPerformanceTime(contextTime) - performance.now();

Where the browser supports AudioContext.getOutputTimestamp(), the correlation names the sample the device is playing at this instant, so the converted time already accounts for the output path and needs no latency arithmetic of your own. Where it does not, the clock pairs currentTime with performance.now() and marks the snapshot 'estimated' so you can tell the two cases apart:

import { AudioOutputClock } from '@codexo/exojs';

const snapshot = new AudioOutputClock().snapshot();

if (snapshot.source === 'estimated') {
    // No output timestamp here: the correlation leads the true output by roughly
    // `outputLatency`, which this environment may not report either.
}

Display latency is deliberately outside this: only your code knows how far ahead of the photons its render loop runs, so the decision of which frame to draw a beat on stays yours.

Examples

Audio VisualisationPointerAudioOpen in PlaygroundView source

Preview is paused until you click Play.

import { Application, Asset, AudioStream, Color, FixedResolutionCanvasSizing, type Pausable, type RenderingContext, Scene, type Seekable, Sprite, Text, Texture, type Voice } from '@codexo/exojs';
import { AudioAnalyser, BeatDetector } from '@codexo/exojs-audio-fx';
import { mountControls } from '@examples/runtime';

class AudioVisualisationScene extends Scene {
  private music!: AudioStream;
  private musicVoice!: Voice & Pausable & Seekable;
  private analyser!: AudioAnalyser;
  private detector!: BeatDetector;
  private canvas!: HTMLCanvasElement;
  private context!: CanvasRenderingContext2D;
  private gradientStyle!: CanvasGradient;
  private progressStyle = 'rgba(255, 255, 255, 0.1)';
  private texture!: Texture;
  private screen!: Sprite;
  private hud!: ReturnType<typeof mountControls>;
  private tapPrompt!: Text;

  override async load(): Promise<void> {
    const app = this.app;
    const { width, height } = app;

    // AudioStream is a non-leaf resource kind (no seamless placeholder), so it
    // is loaded directly through `Asset.type('music', ...)` and awaited rather
    // than fetched synchronously via `get()`.
    this.music = await this.loader.load(Asset.type('music', assets.demo.audio.musicLoop));

    // One analyser tap for spectrum/waveform, one beat detector for the
    // beat-pulse ring. Both read the music bus the stream plays through,
    // without altering playback.
    this.analyser = new AudioAnalyser({ source: app.audio.music });
    this.detector = new BeatDetector();
    this.detector.source = app.audio.music;

    this.canvas = document.createElement('canvas');
    this.canvas.style.position = 'absolute';
    this.canvas.style.top = '12.5%';
    this.canvas.style.left = '0';
    this.canvas.width = width;
    this.canvas.height = height;

    this.context = this.canvas.getContext('2d')!;
    this.context.strokeStyle = '#fff';
    this.context.lineWidth = 4;
    this.context.lineCap = 'round';
    this.context.lineJoin = 'round';

    this.gradientStyle = this.context.createLinearGradient(0, 0, 0, this.canvas.height);
    this.gradientStyle.addColorStop(0, '#f70');
    this.gradientStyle.addColorStop(0.5, '#f30');
    this.gradientStyle.addColorStop(1, '#f70');

    this.texture = new Texture(this.canvas);
    this.screen = new Sprite(this.texture);

    this.hud = mountControls({
      title: 'Audio Visualisation',
      controls: [{ keys: 'Click', action: 'play / pause' }],
      status: 'Playing…',
      hint: 'Frequency bars, waveform, and a beat-pulse ring — all driven by live AudioAnalyser + BeatDetector data.',
    });

    // Shown while the browser still blocks audio (`app.audio.locked`); the
    // first click or keypress unlocks it and the queued music starts.
    this.tapPrompt = new Text('Click or press any key to start the music', { fillColor: Color.white, fontSize: 22, align: 'center' })
      .setAnchor(0.5, 0.5)
      .setPosition(width / 2, height - 64);

    app.input.onPointerDown.add(() => {
      // The first gesture only unlocks audio (core auto-starts the queued
      // track); subsequent clicks toggle play / pause.
      if (app.audio.locked) {
        return;
      }

      if (this.musicVoice.paused) {
        this.musicVoice.resume();
      } else {
        this.musicVoice.pause();
      }
      this.hud.setStatus(this.musicVoice.paused ? 'Paused' : 'Playing…');
    });

    // Core defers playback until the AudioContext unlocks on the first
    // gesture, then starts automatically - play() returns the Voice now.
    this.musicVoice = app.audio.play(this.music, { loop: true, volume: 0.8 }) as Voice & Pausable & Seekable;
  }

  override draw(context: RenderingContext): void {
    const app = this.app;
    const canvas = this.canvas;
    const freqData = this.analyser.getSpectrum();
    const timeDomain = this.analyser.getWaveform();
    const width = canvas.width;
    const height = canvas.height;
    const length = freqData.length;
    const barWidth = Math.ceil(width / length);

    this.context.clearRect(0, 0, width, height);

    // Progress bar of the looping track.
    const progress = this.musicVoice.duration > 0 ? this.musicVoice.time / this.musicVoice.duration : 0;
    this.context.fillStyle = this.progressStyle;
    this.context.fillRect(0, 0, width * progress, height);

    // Frequency bars + waveform polyline.
    this.context.fillStyle = this.gradientStyle;
    this.context.beginPath();

    for (let i = 0; i < length; i++) {
      const barHeight = (height * freqData[i]) / 255;
      const lineHeight = (height * timeDomain[i]) / 255;
      const offsetX = (i * barWidth) | 0;

      this.context.fillRect(offsetX, (height / 2 - barHeight / 2) | 0, barWidth, barHeight | 0);
      this.context.lineTo(offsetX, (height * 0.75 - lineHeight / 2) | 0);
    }

    this.context.stroke();

    // Beat-pulse ring: radius and opacity follow the detector's beat
    // envelope, so it expands on every beat and sits perfectly still in
    // silence (pulse is 0 until a tempo locks).
    const pulse = this.detector.pulse;
    const cx = width / 2;
    const cy = height / 2;
    const baseRadius = Math.min(width, height) * 0.18;
    const radius = baseRadius * (1 + pulse * 0.6);

    this.context.beginPath();
    this.context.arc(cx, cy, radius, 0, Math.PI * 2);
    this.context.lineWidth = 6 + pulse * 10;
    this.context.strokeStyle = `rgba(120, 220, 255, ${0.15 + pulse * 0.7})`;
    this.context.stroke();

    // restore the default stroke style for the next frame's waveform.
    this.context.strokeStyle = '#fff';
    this.context.lineWidth = 4;

    this.screen.updateTexture();

    context.render(this.screen);

    if (app.audio.locked) {
      context.render(this.tapPrompt);
    }
  }
}

const app = new Application({
  scenes: { AudioVisualisationScene },
  canvas: {
    width: 1280,
    height: 720,
    mount: document.body,
    sizing: new FixedResolutionCanvasSizing(),
  },
  clearColor: Color.black,
});

await app.start(AudioVisualisationScene);

A full audio visualisation: frequency-domain bars, time-domain waveform overlay, and per-band energy meters drawn on a 2D canvas then uploaded as a texture.

Beat Sync PulseAudioOpen in PlaygroundView source

Preview is paused until you click Play.

import { Application, Asset, AudioStream, Color, FixedResolutionCanvasSizing, type RenderingContext, Scene, type Seconds, Sprite, Text, Vector } from '@codexo/exojs';
import { BeatDetector } from '@codexo/exojs-audio-fx';
import { AlphaFadeOverLifetime, BurstSpawn, ConeDirection, Constant, particlesExtension, ParticleSystem } from '@codexo/exojs-particles';
import { mountControlPanel, mountControls } from '@examples/runtime';

class BeatSyncPulseScene extends Scene {
  private music!: AudioStream;
  private detector!: BeatDetector;
  private sprite!: Sprite;
  private pulse = 0;
  private intensity = 0.28;
  private beats = 0;
  private particles!: ParticleSystem;
  private burst!: BurstSpawn;
  private hud!: ReturnType<typeof mountControls>;
  private tapPrompt!: Text;

  override async load(): Promise<void> {
    const app = this.app;
    const { width, height } = app;

    // AudioStream has no seamless adapter - await it explicitly.
    const track = await this.loader.load(Asset.type('music', 'audio/demo-loop-main.ogg'));
    this.music = track;
    this.sprite = new Sprite(this.loader.get('image/ship-a.png')).setAnchor(0.5).setPosition(width / 2, height / 2);

    this.hud = mountControls({
      title: 'Beat Sync Pulse',
      hint: 'The ring and particle burst fire on each detected beat.',
    });

    this.particles = new ParticleSystem(this.loader.get('image/particle-light.png'), { capacity: 3500 });
    this.systems.add(this.particles);
    this.particles.setPosition(width / 2, height / 2);
    this.burst = new BurstSpawn({
      schedule: [{ time: 0, count: 90 }],
      lifetime: new Constant(0.6),
      position: new Constant(new Vector(0, 0)),
      velocity: ConeDirection.omni(80, 240),
      scale: new Constant(new Vector(0.2, 0.2)),
    });
    this.particles.addSpawnModule(this.burst);
    this.particles.addUpdateModule(new AlphaFadeOverLifetime());

    // Shown while the browser still blocks audio (`app.audio.locked`); the
    // first click or keypress unlocks it and the queued music starts.
    this.tapPrompt = new Text('Click or press any key to start the music', { fillColor: Color.white, fontSize: 22, align: 'center' })
      .setAnchor(0.5, 0.5)
      .setPosition(width / 2, height - 64);

    mountControlPanel({ title: 'Tuning' }).addSlider({
      label: 'Pulse intensity',
      min: 0.1,
      max: 0.5,
      step: 0.01,
      value: this.intensity,
      onChange: value => {
        this.intensity = value;
      },
    });

    this.detector = new BeatDetector();
    this.detector.source = app.audio.music;
    this.detector.onBeat.add(() => {
      this.pulse = this.intensity;
      this.burst.reset();
      this.beats += 1;
      this.hud.setStatus(`Beats detected: ${this.beats}`);
    });

    // Core defers playback until the AudioContext unlocks on the first
    // gesture, then starts automatically.
    app.audio.play(this.music, { loop: true, volume: 0.8 });
  }

  override update(delta: Seconds): void {
    this.pulse = Math.max(0, this.pulse - delta * 1.2);
    this.sprite.setScale(1 + this.pulse);
  }

  override draw(context: RenderingContext): void {
    const app = this.app;
    context.render(this.particles);
    context.render(this.sprite);

    if (app.audio.locked) {
      context.render(this.tapPrompt);
    }
  }
}

const app = new Application({
  scenes: { BeatSyncPulseScene },
  canvas: {
    width: 1280,
    height: 720,
    mount: document.body,
    sizing: new FixedResolutionCanvasSizing(),
  },
  clearColor: Color.black,
  loader: {
    basePath: 'assets/',
  },
  extensions: [particlesExtension],
});

await app.start(BeatSyncPulseScene);

A sprite pulses on every beat; particles burst on the onBeat signal.

Where to go next

For color-oriented effects driven by audio — palette cycling, noise overlays, shader filters — see Filters. To write a custom shader that samples a spectrogram DataTexture, see Custom mesh shaders. If you came here before reading Beat detection, that chapter covers the full event API, tempo tracking, and frequency band state in detail.