Guide

GuideRecipesAudio reactive scene

Audio reactive scene

Map audio analysis to movement, particles, and camera response.

Intermediate~2 min read

Audio reactive scene

This recipe combines BeatDetector, ParticleSystem, tween-driven camera shake, and audio-triggered visual effects into one scene. It builds on Beat detection for analysis and Audio-reactive visualization for the visual patterns — this chapter is about orchestration, not re-teaching the individual APIs.

Approach

Three systems run in parallel, each driven by a different facet of the audio analysis:

  1. Beat → particle burst: onBeat fires a BurstSpawn and triggers camera shake.
  2. Bar → tween chain: onBarStart triggers a timed sequence of scale/rotation tweens on a central element.
  3. Frequency → continuous motion: Low-band energy from AudioAnalyser.getLowMidHigh() drives the background tint or a sprite’s idle animation.

Setup

One analyser for frequency data, one beat detector for rhythm, one particle system for visual feedback:

examples/guides/audio-reactive-scene/reactive-scene.ts
const app = new Application({ extensions: [particlesExtension] });

class AudioReactiveScene extends Scene {
  private music!: AudioStream;
  private particleTexture!: Texture;
  private analyser!: AudioAnalyser;
  private detector!: BeatDetector;
  private particles!: ParticleSystem;
  private burst!: BurstSpawn;
  private view!: View;

  async load() {
    const [music, particleTexture] = await Promise.all([this.loader.load(Asset.type('music', 'audio/track.ogg')), this.loader.load('image/particle.png')]);
    this.music = music;
    this.particleTexture = particleTexture;
  }

  init() {
    this.app.audio.play(this.music, { loop: true, volume: 0.8 });

    // Analysis - both taps read the music bus the track plays through.
    this.analyser = new AudioAnalyser({ source: this.app.audio.music, fftSize: 512 });
    this.detector = new BeatDetector({ source: this.app.audio.music });

    // The engine clears to `app.clearColor` before `draw` runs, and that is
    // the backend's live instance - mutating it animates the background.
    this.app.clearColor.set(20, 24, 40, 1);

    // Particles - burst on beat
    this.particles = new ParticleSystem(this.particleTexture, { capacity: 5000 });
    this.burst = new BurstSpawn({
      schedule: [{ time: 0, count: 120 }],
      lifetime: new Constant(0.8),
      velocity: ConeDirection.omni(100, 360),
      scale: new Constant(new Vector(0.2, 0.2)),
    });
    this.particles.addSpawnModule(this.burst);
    this.particles.addUpdateModule(new AlphaFadeOverLifetime());

    // Camera shake
    this.view = new View(400, 300, 800, 600);

    // Beat → burst + shake
    this.detector.onBeat.add(() => {
      this.burst.reset();
      this.view.shake(14, Time.seconds(0.2), { frequency: 30, decay: true });
    });
  }

  update(delta: Seconds) {
    this.view.update(delta * 1000);
    this.particles.update(delta);

    // Continuous frequency → background tint
    const { low } = this.analyser.getLowMidHigh();
    this.app.clearColor.set(20 + low * 26, 24 + low * 15, 40 + low * 52, 1);
  }

  draw(context: RenderingContext) {
    context.render(this.particles, { view: this.view });
  }
}

Layering effects

The pattern scales by adding more independent listeners, each reacting to a different audio facet:

examples/guides/audio-reactive-scene/beat-hooks.ts
// Bar start -> tween chain on a central logo
this.detector.onBarStart.add(() => {
  this.app.tweens.create(this.logo.scale).to({ x: 1.3, y: 1.3 }, 0.15).easing(Ease.cubicOut).yoyo().repeat(1).start();
});

// Mid-band -> particle tint cycling
this.detector.onBeat.add((info: BeatInfo) => {
  if (info.beatInBar === 2 || info.beatInBar === 4) {
    this.burst.config.tint = new Constant(info.beatInBar === 2 ? new Color(0xffa500) : new Color(0x87ceeb));
    this.burst.reset();
  }
});

Camera shake from audio

View.shake(intensity, duration, { frequency, decay }) displaces the view center by a decaying sinusoidal offset for duration seconds. decay: true reduces the shake intensity over the duration; frequency controls the oscillation rate. Call view.shake() inside onBeat for a bass hit, or onDownbeat for a stronger effect on the first beat of each bar.

For frequency-driven shake (low-end rumble instead of discrete hits), lerp the view center by the low-band energy in update:

examples/guides/audio-reactive-scene/beat-hooks.ts
override update(delta: Seconds): void {
  const { low } = this.analyser.getLowMidHigh();
  const shakeX = (Math.random() - 0.5) * low * 20;
  const shakeY = (Math.random() - 0.5) * low * 20;
  this.view.setCenter(400 + shakeX, 300 + shakeY);
}

This produces a continuous subtle shake proportional to bass energy, suitable for engine rumble, earthquake effects, or low-frequency ambience.

Orchestration principles

  • One source, many listeners: One BeatDetector can feed particle bursts, camera shake, HUD flash, and tween triggers simultaneously — each subscriber acts independently.
  • Separate analyser for continuous data: AudioAnalyser.getLowMidHigh() gives band energies every frame without event overhead. Use it for smooth, continuous visuals like tint shifts, bar graphs, or procedural animation.
  • Beat detector for discrete events: onBeat/onDownbeat/onBarStart are for one-shot triggers. They fire at discrete moments and carry timing metadata (isDownbeat, beatInBar) for per-beat variation.
  • Keep the per-frame work bounded: A particle system at 120 particles per beat with 120 BPM spawns 240 particles/second — well within a 5000-capacity system. Check system.aliveCount occasionally to confirm you’re not overrunning capacity.

Examples

Audio Reactive ParticlesAudioOpen in PlaygroundView source

Preview is paused until you click Play.

import { Application, Asset, AudioStream, Color, FixedResolutionCanvasSizing, type RenderingContext, Scene, type Seconds, Text, Vector, type Voice } from '@codexo/exojs';
import { AudioAnalyser, BeatDetector } from '@codexo/exojs-audio-fx';
import { AlphaFadeOverLifetime, ConeDirection, Constant, particlesExtension, ParticleSystem, RateSpawn } from '@codexo/exojs-particles';
import { mountControls } from '@examples/runtime';

const colors = [new Color(255, 120, 140), new Color(120, 220, 255), new Color(130, 255, 170), new Color(255, 220, 120)];

class AudioReactiveParticlesScene extends Scene {
  private music!: AudioStream;
  private musicVoice!: Voice;
  private analyser!: AudioAnalyser;
  private detector!: BeatDetector;
  private ps!: ParticleSystem;
  private spawn!: RateSpawn;
  private rate!: Constant<number>;
  private cone!: ConeDirection;
  private hud!: ReturnType<typeof mountControls>;
  private tapPrompt!: Text;

  override async load(): Promise<void> {
    const app = this.app;
    const { width, height } = app;

    // AudioStream is a non-leaf resource kind (no seamless placeholder), so it
    // is loaded directly through `Asset.type('music', ...)` and awaited rather
    // than fetched synchronously via `get()`.
    this.music = await this.loader.load(Asset.type('music', assets.demo.audio.musicLoop));

    // Two parallel taps of the same track: the analyser gives per-band
    // energy (drives emission), the detector gives beats (recolours).
    this.analyser = new AudioAnalyser({ fftSize: 1024, source: app.audio.music });
    this.detector = new BeatDetector();
    this.detector.source = app.audio.music;

    this.ps = new ParticleSystem(this.loader.get(assets.demo.textures.particleLight), { capacity: 6000 });
    this.systems.add(this.ps);
    this.ps.setPosition(width / 2, height / 2);

    // The rate (density) and the cone speed range (spread) are mutated every
    // frame from live audio energy. Starting both near zero means a silent
    // track emits (almost) nothing - the field is genuinely data-driven, not
    // a timed fountain dressed up as "reactive".
    this.rate = new Constant(0);
    this.cone = ConeDirection.omni(20, 40);
    this.spawn = new RateSpawn({
      rate: this.rate,
      lifetime: new Constant(0.9),
      position: new Constant(new Vector(0, 0)),
      velocity: this.cone,
      scale: new Constant(new Vector(0.22, 0.22)),
      tint: new Constant(colors[0]),
    });
    this.ps.addSpawnModule(this.spawn);
    this.ps.addUpdateModule(new AlphaFadeOverLifetime());

    // Beats only recolour the stream; they do not fake emission on their own.
    this.detector.onBeat.add(() => {
      this.spawn.config.tint = new Constant(colors[(Math.random() * colors.length) | 0]);
    });

    this.hud = mountControls({
      title: 'Audio Reactive Particles',
      controls: [{ keys: 'Audio', action: 'bass → density · treble → spread' }],
      status: 'Listening…',
      hint: 'Density follows low-band energy; spread follows high-band energy. Silence = still.',
    });

    // Shown while the browser still blocks audio (`app.audio.locked`); the
    // first click or keypress unlocks it and the queued music starts.
    this.tapPrompt = new Text('Click or press any key to start the music', { fillColor: Color.white, fontSize: 22, align: 'center' })
      .setAnchor(0.5, 0.5)
      .setPosition(width / 2, height - 64);

    // Core defers playback until the AudioContext unlocks on the first
    // gesture, then starts automatically - play() returns the Voice now.
    this.musicVoice = app.audio.play(this.music, { loop: true, volume: 0.8 });
  }

  override update(_delta: Seconds): void {
    // Low band (bass) drives how MANY particles spawn this second.
    const low = this.analyser.getBandEnergy(20, 180);
    // High band (treble) drives how WIDE the velocity cone fans out.
    const high = this.analyser.getBandEnergy(2000, 16000);

    // bass → density: 0 in silence, up to ~1200 particles/s on heavy bass.
    this.rate.value = low * low * 1200;

    // treble → spread: a tight slow core grows into a fast wide burst.
    this.cone.minSpeed = 40 + high * 120;
    this.cone.maxSpeed = 90 + high * 360;

    if (this.musicVoice) {
      this.hud.setStatus(`bass ${(low * 100) | 0}%  treble ${(high * 100) | 0}%`);
    }
  }

  override draw(context: RenderingContext): void {
    const app = this.app;
    context.render(this.ps);

    if (app.audio.locked) {
      context.render(this.tapPrompt);
    }
  }
}

const app = new Application({
  scenes: { AudioReactiveParticlesScene },
  canvas: {
    width: 1280,
    height: 720,
    mount: document.body,
    sizing: new FixedResolutionCanvasSizing(),
  },
  clearColor: Color.black,
  extensions: [particlesExtension],
});

await app.start(AudioReactiveParticlesScene);

Particles burst on every beat with alpha fade — the canonical beat → particles pattern.

Low Band Camera ShakeAudioOpen in PlaygroundView source

Preview is paused until you click Play.

import { Application, Asset, AudioStream, Color, FixedResolutionCanvasSizing, type RenderingContext, Scene, type Seconds, Sprite, Text, Time, View, type Voice } from '@codexo/exojs';
import { AudioAnalyser } from '@codexo/exojs-audio-fx';
import { mountControls } from '@examples/runtime';

class LowBandCameraShakeScene extends Scene {
  private music!: AudioStream;
  private musicVoice!: Voice;
  private analyser!: AudioAnalyser;
  private view!: View;
  private sprite!: Sprite;
  private hud!: ReturnType<typeof mountControls>;
  private tapPrompt!: Text;

  override async load(): Promise<void> {
    const app = this.app;
    const { width, height } = app;

    // AudioStream is a non-leaf resource kind (no seamless placeholder), so it
    // is loaded directly through `Asset.type('music', ...)` and awaited rather
    // than fetched synchronously via `get()`.
    this.music = await this.loader.load(Asset.type('music', assets.demo.audio.musicLoop));
    this.analyser = new AudioAnalyser({ fftSize: 1024, source: app.audio.music });
    this.view = new View(width / 2, height / 2, width, height);
    this.sprite = new Sprite(this.loader.get(assets.demo.textures.shipA))
      .setAnchor(0.5)
      .setScale(3)
      .setPosition(width / 2, height / 2);

    this.hud = mountControls({
      title: 'Low Band Camera Shake',
      controls: [{ keys: 'Audio', action: 'low-band energy → shake' }],
      status: 'Listening…',
      hint: 'Shake amplitude tracks bass energy only — in silence the camera is perfectly still.',
    });

    // Shown while the browser still blocks audio (`app.audio.locked`); the
    // first click or keypress unlocks it and the queued music starts.
    this.tapPrompt = new Text('Click or press any key to start the music', { fillColor: Color.white, fontSize: 22, align: 'center' })
      .setAnchor(0.5, 0.5)
      .setPosition(width / 2, height - 64);

    // Core defers playback until the AudioContext unlocks on the first
    // gesture, then starts automatically - play() returns the Voice now.
    this.musicVoice = app.audio.play(this.music, { loop: true, volume: 0.8 });
  }

  override update(delta: Seconds): void {
    const low = this.analyser.getBandEnergy(20, 180);

    // No constant floor: amplitude is purely low-band energy, so a quiet
    // passage produces zero shake. A small deadzone keeps faint noise still.
    const amplitude = low > 0.04 ? low * 28 : 0;
    this.view.shake(amplitude, Time.seconds(0.09), { decay: true, frequency: 22 });

    // Advance the shake oscillation (the View only animates when updated).
    this.view.update(delta * 1000);

    if (this.musicVoice) {
      this.hud.setStatus(`bass ${(low * 100) | 0}%`);
    }
  }

  override draw(context: RenderingContext): void {
    const app = this.app;
    context.backend.setView(this.view);
    context.render(this.sprite);
    context.backend.setView(null);

    if (app.audio.locked) {
      context.render(this.tapPrompt);
    }
  }
}

const app = new Application({
  scenes: { LowBandCameraShakeScene },
  canvas: {
    width: 1280,
    height: 720,
    mount: document.body,
    sizing: new FixedResolutionCanvasSizing(),
  },
  clearColor: new Color(22, 24, 34),
});

await app.start(LowBandCameraShakeScene);

Bass-frequency energy drives continuous camera shake — low-end rumble as visual motion.

Where to go next

The next recipe, Game feel, covers general feedback techniques — damage flashes, screen shake, audio cues, and tween-driven response that make interaction feel responsive.