The open-audiovisual handbook

Introduction#

01Source: docs/introduction.md

open-audiovisual (OAV) is a JavaScript framework for live audiovisual performance that runs in the browser. It is plain ES modules: no build step, no dependencies, no account. You clone the repository, run one file, and open a page.

It handles the part every show rebuilds from scratch: MIDI plumbing, turning inputs into parameters, a timeline with scenes, a director's console, a performance mode for the performer, and a backstage page for the stage manager. You write the world, the piece itself. The pitch, a live instrument and the twenty-six example works are on the homepage.

Who it is for#

• Performers play the examples with a MIDI keyboard, a microphone, a camera, a phone, or the on-screen piano, and drive a show from the console.
• Coders write a world in about twenty lines and get inputs, mapping, timeline, console, sound and backstage from one createShow() call.
• AI agents read AGENTS.md, connect to the bundled MCP server, and write worlds inside a chassis that already works. See For AI agents.

Four layers and two spines#

L1 INPUT     "what is happening"       midi · audio · chord · pose · keys · drums · phones
               → every input publishes named SIGNALS: midi/cc/74, chord/consonance, audio/rms
L2 MAPPING   "what it means"           routes: signal → param, with curve, range, smoothing, learn
L3 WORLD     "how the system behaves"  your piece; reads PARAMS, never inputs; any renderer
L4 OUTPUT    "how it leaves"           screen · in-browser sound · MIDI out · OSC to UDP

spines       timeline (automation, scenes, cues) · monitor (backstage over WebSocket)
around them  console (the director's desk) · relay, surface, remote (phones as controllers)

Each layer has one job and a written contract (Architecture), so any layer can be replaced without touching the others. Two rules keep it that way; they are explained in Core concepts.

Where it comes from#

The core was extracted from The Last Input (Che-Yu Wu, 2026), a performance for piano and a living digital world at the IRCAM residency × C-LAB Taiwan Sound Lab in Taipei, played on a 49.4-channel speaker dome. The timeline, the chord semantics, many-to-many MIDI learn and OSC batching all ran a real fourteen-scene show before they were generalized here. The rule since then: the framework only grows features that a real show asked for.

The mapping layer's ideas descend from libmapper's research on signal namespaces. The sister project WebToe is a node-based dataflow engine that imports TouchDesigner projects; WebToe is the engine, open-audiovisual is the show.

How to read this handbook#

• New here: Quick start, then Core concepts, then Writing a world.
• Building a show: Inputs, Mapping, Show control, Phones, relay & surfaces.
• Looking something up: Signals reference and Package reference.
• Something is wrong: Troubleshooting.

Everything here is checked against the source in packages/. When this page and the code disagree, the code is right and this page is a bug: please open an issue.

Quick start#

02Source: docs/getting-started.md

Three ways in#

Pick the one that matches what you already have. They get bigger as you go down.

1. One line in a p5.js sketch. Any p5.js sketch (OpenProcessing, the p5 editor, your own page) can be played from a MIDI controller, from a controller drawn on screen, or from a phone, by adding one line at the top. Read p5.js in one line, and play the sketch that carries it: example 15. The same line also plays sound, runs a drum machine you can write as one line of rhythm, listens to the microphone and records a video; each has a ready-made sketch among the examples (16 to 26).
2. One controller in a page. <oav-controller profile="akai-lpd8"> puts a controller you can play into any web page. Read MIDI controllers & embedding, and pick yours in the controller library.
3. A whole show. A World plus one createShow() call assembles inputs, mapping, timeline, console, sound and backstage. Start from example 01 and keep reading below.

Run it#

You need Node.js (any current version; CI runs the tests on Node 22) and a browser. There is nothing to install: no npm install, no bundler.

git clone https://github.com/frank890417/open-audiovisual.git
cd open-audiovisual
node serve.js            # → http://localhost:8080

serve.js is a static file server with caching turned off (stale code on stage is a nightmare) that also hosts the phone relay at ws://<host>:8080/relay. Pass a port to use another one: node serve.js 3000. The same commands exist as npm scripts:

Open http://localhost:8080/examples/01-hello-particles/. Every example runs without hardware.

Which browser#

Not sure what your device can do? Open the device check. It looks at your browser, Web MIDI, microphone, camera and graphics, and says what to do about each. Nothing runs until you press a button, and nothing leaves your device.

Web MIDI is the one that differs most between browsers (sources checked 2026-10-15):

• Chrome and Edge have it. Use them for a show. Since Chrome 124 any use of MIDI asks for permission, where before only system-exclusive access did (Chrome blog).
• Firefox has Web MIDI on desktop from version 108, after you install Firefox's site-permission add-on (Firefox 108 release notes).
• Safari on Mac, iPhone and iPad has no Web MIDI (caniuse). WebKit lists it as not implemented because of fingerprinting concerns (WebKit), and bug 107250 is still open and unassigned. Chrome on an iPhone or iPad is WebKit underneath, so it has none either. On a Mac there is a third-party Safari extension, Safari-WebMIDI; we have not tested it.

Where the browser has no Web MIDI, there are three routes:

• Play on screen. Every controller in the library plays with a finger, and every event still fires.
• Use a phone as the controller. See Phones, relay & surfaces.
• A browser app for the iPad. MIDIWeb Browser is a third-party app for iOS 17.6 and later that gives pages Web MIDI. We have not tested it with open-audiovisual.

On Windows 11 the built-in Windows MIDI Services (released 2026-02-17) give apps shared MIDI ports and a loopback, so loopMIDI is no longer needed to send MIDI from the browser to a DAW (Windows blog); see Play your DAW with it.

Two more rules apply to every browser:

• Camera and microphone only work in a secure context: http://localhost or https://. Open the show on the machine that runs serve.js through localhost, not through its LAN address.
• Phones can join over plain http:// on the local network for touch, keys and the control surface. iOS only gives motion sensors and the camera to https:// pages (see Phones, relay & surfaces).

What you are looking at#

An example page has two parts: the stage on the left (the world renders there) and the side panel on the right. From the top, the side panel holds:

1. The piano, with two checkboxes: keyboard turns on QWERTY playing, simulate performance starts a hands-free player.
2. Transport: play/pause, previous/next scene, reset, the clock, the current scene, and a scrubber with one block per scene.
3. Layers: what every layer is doing right now (events per second per input, active routes, world and FPS, what is leaving the browser).
4. L1 · Input: MIDI devices (untick to mute one) and buttons for sources that need a click: 🎤 mic, 🖐 hands, 🕺 body.
5. L2 · Mapping: every route, with a mute box and ✕ to remove it.
6. L3 · Params: a slider per param, ✕ to clear an override, learn to map.
7. L4 · Output — sound, when the show has sound: enable sound.
8. Signals: a live meter for every signal.

Each panel title collapses its panel.

Play without hardware#

• On-screen piano: click or touch the keys; drag across them for a glissando.
• QWERTY: tick keyboard first (capture is off by default so letters do not steal the console's hotkeys). The keys A W S E D F T G Y H U J K O L P ; play 17 notes a semitone apart, starting at the piano's base note; Z/X shift an octave; hold Shift for a louder note.
• Simulate performance: a hands-free player noodles in a pentatonic scale with steps, leaps, triads and rests, so you can watch the mapping breathe.
• Drum machine (examples 03 and 06): a step sequencer whose signals have the same shape as the microphone's drum detection.
• On-screen MIDI controller (example 09): a picture of a known controller you can play with the mouse (the supported devices are listed on the controllers page); plug the real one in and the picture moves with your hands.

All of these publish the same signals as real hardware, so nothing downstream can tell the difference.

Your first world#

A world is an object with params and four lifecycle functions. Hand it to createShow() and the rest of the show is assembled around it:

import { createShow } from '@openav/show';
import { createCanvas } from '@openav/stage';

const pulse = {
  name: 'pulse',
  // what the piece is performed WITH: sliders, knobs, hands, the timeline
  params: [{ key: 'size', min: 10, max: 200, def: 40 }],
  init({ container, signals }) {
    this.view = createCanvas(container);
    this.r = 0;
    // a struck note is an event, so it may come straight from a signal
    signals.on('midi/note/on', ({ vel }) => { this.r = 120 * vel; });
  },
  update(dt, state) { this.r = Math.max(0, this.r - dt * 90); this.size = state.size; },
  render() {
    const { ctx } = this.view, { w, h } = this.view.fit();
    ctx.fillStyle = '#000'; ctx.fillRect(0, 0, w, h);
    ctx.fillStyle = '#fff';
    ctx.beginPath(); ctx.arc(w / 2, h / 2, this.size + this.r, 0, 7); ctx.fill();
  },
  dispose() { this.view.dispose(); },
};

await createShow({ world: pulse, modules: { sound: true } });

Play a note: the circle jumps and decays. Drag the size slider, or click learn next to it and turn a knob. That is the whole model: params for continuous control, signals for events.

Make a new example#

1. Copy examples/01-hello-particles/ to examples/<nn>-<name>/ (the next free number). Its index.html already has an import map for every @openav/* package; the ?v= suffixes are cache stamps written by tools/stamp-version.mjs before a deploy.
2. Write your world in main.js (Writing a world).
3. Change the <title> in index.html. Add CDN script tags there if you use p5 or another renderer.
4. Open http://localhost:8080/examples/<nn>-<name>/.

An AI agent can do steps 1–2 with the MCP tool scaffold_world (For AI agents).

Check your work#

• npm test passes (it covers the pure logic: chords, timeline, mapping, relay, surfaces, MIDI parsing and profiles, the site pages).
• The page loads with no console errors. One message is expected: if the backstage monitor is not running, the browser reports that ws://localhost:7457 refused the connection. The page retries quietly with a growing delay (up to 30 s). Run npm run monitor to make it go away.
• Press Space: the timeline plays and something changes over time.
• Tick keyboard and press A: the world reacts without any hardware.
• window.openav in the devtools console is the whole show (signals, params, mapper, timeline…), for poking at it by hand.

Core concepts#

03Source: docs/concepts.md

Six words carry the whole framework: signal, param, route, world, timeline, show. This chapter defines each one and the two rules that hold them together.

Signals#

A signal is a named value that says what is happening: a knob's position, the loudness of the room, the height of a wrist, the consonance of the last chord. Every input package publishes signals into one registry (Signals from @openav/core) and does nothing else. Signals are the only currency of input.

There are two kinds:

Names are paths, source first: midi/cc/74, audio/band/low, pose/hand/right/y, phone/ab12/tilt/x, surface/main/hue. Ranges are normalized where that makes performance sense, and y axes are flipped so that 1 means raised, never raw pixels.

A signal is declared with its range so that meters and learn can scale it:

signals.define('breath/pressure', { min: 0, max: 1, source: 'breath' });
signals.set('breath/pressure', 0.37);               // continuous
signals.define('breath/sigh', { kind: 'pulse' });
signals.pulse('breath/sigh', { depth: 0.8 });       // pulse

set() on a name nobody declared defines it on the spot as continuous 0..1. If your values live elsewhere (degrees, MIDI note numbers), declare the range first.

A knob, a raised hand and a chord's consonance are equal citizens in this registry. That equality is what lets you mix them: a hand can do whatever a knob does.

Params#

A param is a value the world listens to: hue, bloom, gravity. The world declares its params; each one is a line of schema:

{ key: 'energy', label: 'Energy', min: 0, max: 1, def: 0.3 }
{ key: 'mode', label: 'Mode', min: 0, max: 3, def: 0, step: 1 }   // stepped: snaps, holds on the timeline
{ key: 'burst', label: 'Burst!', pulse: true }                      // a trigger, not a level

Every frame each param resolves to one number:

timeline automation at time t      (the score; def if the param has no automation)
  ⬑ overridden by → the latest override   (a slider, a mapped knob, a phone fader)

An override is sticky. Once a hand or a route has touched bloom, the timeline stops moving it until the override is cleared (✕ in the console, or params.clearOverride('bloom')). That matches the stage: when a human grabs a control, the score yields.

The full field list is in Writing a world.

Routes and mapping#

A route connects one signal to one param. Routes are plain data:

{ source: 'pose/hand/right/y', target: 'energy', curve: 'smooth', smooth: 0.15 }

The mapper (@openav/mapping) holds the routes and applies them: it reads the signal, clamps it into a window (inMin..inMax), optionally inverts it, bends it with a curve, scales it into the param's range and, if asked, smooths it over time. One signal may drive several params and one param may listen to several signals. Learn binds the next signal that moves, which works for a knob and for a waving hand alike. Routes serialize to JSON, so a wiring is a file you can carry to the next venue. Details: Mapping.

Worlds#

A world is the piece: an object with a name, its params, and init / update / render / dispose. It reads resolved params every frame and may subscribe to signals for discrete events. It brings its own renderer: 2D canvas, p5, three.js, a WebToe patch, DOM, anything. The framework has no opinion. Details: Writing a world.

Timeline and scenes#

The timeline (@openav/timeline) is the score: keyframed automation per param, plus a list of scenes with titles and notes for the performer. It owns time; worlds never read the clock. Scenes may start before zero for a pre-show standby. Details: Show control.

The show#

A show is a world plus one createShow() call (@openav/show). It builds the signals registry, the params, the stage, the timeline, the mapper, the inputs you ask for, sound, the console, the backstage feed and the frame loop, and returns all of them. Every example is a world plus this call, so when the framework grows a module, every show can have it by declaring it.

const show = await createShow({
  world,
  timeline: { total: 120, automation, scenes },
  routes: [{ source: 'audio/rms', target: 'energy', smooth: 0.1 }],
  modules: { keys: { base: 48 }, chord: true, audio: true, sound: true },
});

The two rules#

Two rules make the framework composable. They come in two forms: one says how the system is designed, the other says what that asks of anyone writing code for it.

1. Signals are the only currency of input. So: continuous control goes through params; discrete events may use signals.

A world that reads signals.get('midi/cc/74') inside update() still works, today, with your controller. But it can no longer be driven by the timeline, by a dancer's wrist, by a phone, or by someone else's controller. You have welded the performance to one input. Declare a param instead and let a route connect midi/cc/74 → yourParam. The same world then works for everyone, and the console, the phone surface and learn all see the param.

Events are different. "A note was struck", "a chord resolved", "a kick hit" are moments, not levels. Subscribing to them in init() is correct and encouraged; it is what makes a world react rather than just drift.

2. Worlds never know who is performing them. So: the shell never knows the work.

A world reads params, and whether a value came from the timeline, a MIDI knob, a dancer or a slider is invisible to it. The same holds for the framework: never edit packages/* to make one example look right. A framework change has to make every work better. Switching works means swapping a world and its data; the shell stays.

Architecture#

04Source: docs/architecture.md

open-audiovisual is a data-layer framework. Its entire value is that four layers and two spines stay cleanly separated, so any of them can be replaced without touching the others. This page states each layer's contract precisely.

The frame loop#

One Loop (@openav/core) drives everything. This is the order createShow() runs each frame (packages/show/index.js):

const loop = new Loop((dt) => {
  keys?.update(dt);                              // 0. on-screen performers: SimPlayer, drum sequencer
  drums?.update(dt);
  audio?.update();                               //    microphone analysis → audio/* signals
  onFrame?.(dt, show);                           //    your assembly-level hook
  timeline.advance(dt);                          // 1. time moves
  director?.update(timeline.t, dt, { holding, playing }); //    with a score: segment modules enter / update / exit
  mapper.update(dt);                             // 2. smoothed routes settle
  const state = stage.frame(dt, timeline.state()); // 3. params resolve → world updates & renders
  sound?.update(state);                          // 4. sound/* params reach the synth
  remote?.frame(dt, state);                      //    param values echo back to phones
  consoleUI.render(state);                       // 5. the desk reflects reality
  monitor.frame(snapshotOf({ timeline, params, signals, stage, loop }, state)); // 6. backstage hears about it
});

Signals arrive between frames, whenever a device or a socket fires. An unsmoothed route writes its param immediately; a smoothed route sets a target that mapper.update(dt) glides toward. OSC output is not part of createShow(); if you use it, flush it from onFrame (Show control).

dt is in seconds and clamped to 0.1 (new Loop(fn, { maxDt })), so a stall never makes a world explode. When the browser freezes requestAnimationFrame (the projection window went to the background), a Web Worker heartbeat keeps the loop ticking, about every 66 ms (~15 frames per second): the timeline, mapped signals and the backstage feed must not stop because someone switched windows.

Param resolution order (the heart of the design):

timeline.state(t)          — automation curves per param
  ⬑ rewritten by  →  director.param()   — a segment module, inside its own segment (with a score)
  ⬑ overridden by →  params.overrides   — set by hand sliders, mapped signals

An override is sticky: once a knob touched bloom, the timeline no longer moves it until the override is cleared (✕ in the console). This matches performance reality — when a human grabs a control, the score yields.

L1 · Input — Signals#

Contract: input packages publish named, normalized signals into a Signals registry, and do nothing else. They never touch params or worlds.

• names are path-like: midi/cc/74, chord/consonance, pose/hand/right/y
• continuous signals declare {min, max} and always hold a current value
• pulse signals (kind: 'pulse') fire events; their payload is the event object
• anyone can subscribe: signals.on(name, cb) or signals.onAny(cb)

Adding a new input modality (breath sensor, gamepad, stock ticker, weather API) means writing one class that calls signals.set() / signals.pulse(). Nothing downstream changes. That is the whole point.

The inputs today: @openav/midi (hardware and on-screen controllers), keys, drums, audio (microphone), pose (camera body and hands), leap (Leap Motion / Ultraleap hands) and phones through remote and surface. Hardware a browser cannot open itself reaches it through a small local bridge that speaks a web protocol (leap runs one on ws://127.0.0.1:6437); the browser side is still one class that publishes signals.

Analyzers sit between raw sources and mapping: @openav/chord subscribes to midi/note/on|off and publishes chord/*. It is still an input as far as the mapper is concerned; it just listens to other signals instead of a device.

L2 · Mapping — Mapper#

Contract: the only component that connects signals to params. Routes are data:

{ source: 'pose/hand/right/y', target: 'energy',
  inMin: 0, inMax: 1, curve: 'smooth', invert: false,
  smooth: 0.15, outMin: 0, outMax: 1 }

• many-to-many: one signal may drive several params; one param may listen to several signals (last writer wins). Learn adds, it never replaces.
• learn is generalized MIDI-learn: mapper.learn('bloom') binds the next signal that moves ≥5% of its range — a knob and a waving hand both qualify.
• routes serialize to JSON → mapping profiles are shareable files.

Full reference: Mapping.

L3 · World — Stage#

Contract: a World is { name, params, init, update(dt, state, io), render, dispose }. It reads state (resolved param values). It may subscribe to signals for discrete events (a note-on spawning a particle is an event, not a parameter), but continuous control must flow through params — otherwise your world stops being performable by the timeline or by other inputs.

The Stage owns lifecycle and param aggregation and deliberately does not own a renderer. p5, three.js, 2D canvas, SVG, DOM — the framework has no opinion.

Constitutional rule (inherited from The Last Input): the shell never knows the work. Switching works = swapping World + automation data. The shell stays.

Full reference: Writing a world.

L4 · Output — two branches, video and audio, both optional#

The output layer splits in two. Both branches hang off the same signals → mapping → params spine — sound is performable state, not a side effect.

Video branch

• screen is the default output; performance mode (T) gives the performer a teleprompter while the audience sees the stage window fullscreen (F).
• recording (@openav/record): the work and the performer's camera composited into one canvas of an exact size (vertical 1080×1920 up to 4K) and recorded with the work's own sound by one MediaRecorder, so picture and sound share a clock. MIDI takes record the playing itself and replay it into the show.
• future: NDI gateway (see roadmap; groundwork lives in sister project WebToe).

Audio branch

• MIDI out (external synths/DAWs): midi.send()/noteOn()/cc() — one throat, observable (onSend) so meters can watch everything, with a real panic().
• In-browser synthesis: @openav/sound — a small pluggable engine contract (params, enable, noteOn, noteOff, set, dispose) with a Tone.js engine first (CDN, loaded only on enable) that plays instruments from a registry: a sampled grand piano, mallets, strings, organ, synth presets, one file per sound, picked from the console. Engine params register as sound/* — a knob, a hand, or the timeline plays the filter cutoff exactly the way it plays the visuals. The drum kit in @openav/drums is a second engine behind the same contract (pure WebAudio, no samples needed).
• OSC: browser → HTTP → osc-bridge.js → UDP (Spat, Reaper, TD, lights). Batched per frame; only the latest message per address survives a frame.

Spine · Timeline#

Pure logic, no DOM: params + automation keyframes + scenes → state(t). Scenes may start at negative t (pre-show standby). onSceneChange is the cue hook — fire sound, lights, prompts from it. A scene with hold: true stops playback at its end until the operator releases it.

A show with structure adds the score to this spine (@openav/score): the show written as segment lengths, from which Score computes every start, end, cue and countdown, and Director runs the module that performs each segment (enter once, exit once, a crash switches off only that module). The timeline takes its scenes and total from the score, and the director plugs into it as a param layer:

cut (segment lengths) → Score ─┬─ scenes(), total ──→ Timeline ── state(t) ──→ params
                               └─ Director ── enter / update / exit ──→ segment modules ──→ the world (through `api`)
                                    └─ param() ── timeline.layer ──────────────────────→ Timeline

The score only decides when. What happens still reaches the world as params. Full reference: Score & Director.

Spine · Monitor#

The performance page streams JSON snapshots (throttled, ~15 Hz) over WebSocket to a dependency-free relay (server.js, hand-rolled RFC 6455). Any phone or laptop on the LAN opens the backstage page and sees: clock, scene, world, FPS, overrides, every signal and param, and a "stage feed lost" alarm. The monitor is a first-class citizen because a show without a backstage is a rehearsal.

Around the spines: console and phones#

• Console (@openav/console): the director's desk in the side panel, laid out by layer (Layers → L1 → L2 → L3 → L4 → Signals), plus performance mode. It reads the same objects the loop runs; it owns nothing.
• Phones (@openav/relay, @openav/surface, @openav/remote): phones and iPads join a room on a WebSocket relay and publish phone/* and surface/* signals. To the mapper they are one more input. See Phones, relay & surfaces.

Writing a world#

05Source: docs/writing-a-world.md

A world is the only part of the framework you have to write. Everything else is assembly.

The interface#

export const myWorld = {
  name: 'my-world',

  // params: what the world can be performed WITH.
  // These appear in the console, are automatable by the timeline,
  // and are mappable to any signal.
  params: [
    { key: 'energy', label: 'Energy', min: 0, max: 1, def: 0.3 },
    { key: 'mode',   label: 'Mode',   min: 0, max: 3, def: 0, step: 1 },  // step → snaps, holds on the timeline
    { key: 'reset',  label: 'Reset',  pulse: true },                       // momentary trigger
  ],

  init(ctx) {
    // ctx = { container, signals, params }
    // build your renderer here: p5, three, or the built-in 2D helper:
    this.view = createCanvas(ctx.container);
    // discrete events may come straight from signals:
    this._unsub = ctx.signals.on('chord/event', (a) => this.onChord(a));
    this._unpulse = ctx.params.onPulse('reset', () => this.reset());
  },

  update(dt, state, io) {
    // state = resolved param values. This is your ONLY continuous input.
    // io = { signals } plus any output handles the assembly added (see "Sound output")
  },

  render() { /* draw */ },

  dispose() { this._unsub?.(); this._unpulse?.(); this.view.dispose(); },
};

The contract#

container is an absolutely positioned layer that fills the stage. The stage empties it before every init, so you never clean up DOM someone else left. An exception thrown in update or render is caught and logged ([stage] update: / [stage] render:); the show keeps running.

Params#

If two registered worlds declare the same key, they share one param (the first declaration wins).

The one rule people break#

Continuous control goes through params. Events may use signals.

If your world reads signals.get('midi/cc/74') directly in update(), it still works — today, with your controller. But it can no longer be driven by the timeline, by a dancer, or by someone else's controller profile. You've welded the performance to one input. Route it: declare a param, let the mapper connect midi/cc/74 → yourParam, and the same world works for everyone.

Events are different: "a note was struck", "a chord resolved", "an onset hit" are moments, not levels. Subscribing to those in init() is correct and encouraged — that's what makes worlds reactive rather than just modulated.

Useful event signals: midi/note/on and midi/note/off (any keyboard, the on-screen piano, the simulated performer, a phone), chord/event (one analysis per played gesture), audio/onset, audio/kick, drum/kick. The full list is in the Signals reference.

Scenes and automation#

The timeline is configured next to the world, in the createShow() call:

await createShow({
  world: gardenWorld,
  timeline: {
    total: 120,                                       // seconds
    automation: {
      growth: [[0, 0.6], [40, 1.4], [90, 0.8], [120, 0.2]],   // [t, value], linear between keyframes
      wind:   [[0, 0.05], [60, 0.3], [100, 0.7]],
    },
    scenes: [
      { id: 'dawn',  t: 0,  title: 'Dawn',       note: 'single seeds · listen' },
      { id: 'bloom', t: 40, title: 'Full bloom', note: 'triads — stack real thirds' },
      { id: 'storm', t: 90, title: 'Storm',      note: 'clusters welcome' },
    ],
  },
  modules: { keys: { base: 48 }, chord: true, sound: true },
});

• Keyframes are [seconds, value], sorted by time. Before the first keyframe the value is the first value; after the last, the last value.
• Scenes are sorted by t. title and note are what the performer reads in performance mode (T); the next scene and a countdown show under them.
• A scene with a negative t is a pre-show standby: with t: -30 the clock starts at -0:30.
• Try performing your world with only the timeline. If it is boring, a param is missing.

More in Show control.

Renderers#

The stage does not own a renderer. Build one in init, destroy it in dispose.

2D canvas#

createCanvas(container, { alpha = false }) from @openav/stage returns { canvas, ctx, fit, dispose }. fit() keeps the canvas the size of its container (device pixel ratio capped at 2) and returns { w, h } in CSS pixels; call it at the top of render(). Examples 01, 02, 03, 07, 08 and 09 use it.

p5.js#

Load p5 with a script tag in the example's index.html, then run it in instance mode inside the container. p5 draws on its own loop, so update only stores the state and render stays empty (examples 05 and 06):

<script src="https://cdn.jsdelivr.net/npm/p5@1.9.4/lib/p5.min.js"></script>

const world = {
  name: 'p5-world',
  params: [{ key: 'hue', min: 0, max: 360, def: 200 }],
  init({ container }) {
    const self = this;
    this.s = {};
    this.p5 = new p5((sk) => {
      sk.setup = () => { sk.createCanvas(container.clientWidth, container.clientHeight); sk.colorMode(sk.HSB); };
      sk.draw = () => { sk.background(self.s.hue ?? 200, 60, 20); };
      sk.windowResized = () => sk.resizeCanvas(container.clientWidth, container.clientHeight);
    }, container);
  },
  update(dt, state) { this.s = state; },
  render() { /* p5 runs its own loop */ },
  dispose() { this.p5?.remove(); },
};

three.js#

No bundled example uses three.js yet; this is the pattern. Add three to the example's import map ("three": "https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.js"), keep the renderer on this, and dispose everything:

import * as THREE from 'three';

const orbit = {
  name: 'orbit',
  params: [{ key: 'spin', min: 0, max: 3, def: 1 }],
  init({ container }) {
    this.container = container;
    this.renderer = new THREE.WebGLRenderer({ antialias: true });
    container.appendChild(this.renderer.domElement);
    this.scene = new THREE.Scene();
    this.camera = new THREE.PerspectiveCamera(50, 1, 0.1, 100);
    this.camera.position.z = 4;
    this.mesh = new THREE.Mesh(new THREE.IcosahedronGeometry(1, 2), new THREE.MeshNormalMaterial({ wireframe: true }));
    this.scene.add(this.mesh);
  },
  update(dt, s) { this.mesh.rotation.y += dt * s.spin; },
  render() {
    const w = this.container.clientWidth, h = this.container.clientHeight;
    if (w !== this.w || h !== this.h) {
      this.w = w; this.h = h;
      this.renderer.setSize(w, h); this.camera.aspect = w / h; this.camera.updateProjectionMatrix();
    }
    this.renderer.render(this.scene, this.camera);
  },
  dispose() {
    this.mesh.geometry.dispose(); this.mesh.material.dispose();
    this.renderer.dispose(); this.renderer.domElement.remove();
  },
};

A WebToe patch#

webtoeWorld() from @openav/world-webtoe turns a WebToe node patch into a world. The patch runs in an iframe; every frame in which a param changed, the resolved params are posted into it, and any patch parameter written as ext('name', fallback) listens (example 04):

import { webtoeWorld } from '@openav/world-webtoe';

const world = webtoeWorld({
  name: 'garden-patch',
  project: new URL('./garden.webtoe.json', location.href).href,   // the app fetches it: CORS must allow it
  params: [
    { key: 'speed', min: 0, max: 1, def: 0.5 },
    { key: 'hue', min: 0, max: 360, def: 205 },
  ],
  // extra: (state, out) => { out.hueRad = state.hue * Math.PI / 180; },   // derive more ext() values
});

Options: name (default 'webtoe'), app (default https://webtoe.openaudiovisual.com/), project, params, extra.

DOM, SVG, video#

Anything you append to container is fine. The stage only calls your functions.

Several worlds in one show#

const show = await createShow({ world: calm, worlds: [storm, after], timeline });
// switch on a cue: the old world is disposed, the new one initialized
show.timeline.onSceneChange((i, scene) => {
  if (scene.id === 'storm') show.stage.activate('storm');
});

All worlds' params are registered up front, so the console and the timeline can address any of them. stage.activate(name) does nothing if that world is already active.

Sound output from a world#

update(dt, state, io) receives io. With createShow(), io holds only signals; add output handles once the show exists, and every world sees them:

const show = await createShow({ world });
show.stage.io.midi = show.midi;          // the show's @openav/midi engine (MIDI out)
// show.stage.io.osc = osc;              // an OscOut you created (see Show control → OSC output)

Then, inside the world:

// trigger a sampler note when something happens in the simulation
io.midi?.noteOn(48 + creature.species * 12, 90, 2);   // note, velocity 0..127, channel 2

// stream a position to a spatializer
io.osc?.send(`/source/${i}/xyz`, [x, y, z]);

Often it is simpler to do it at the assembly level instead, as example 05 does: show.signals.on('midi/note/on', ({ note, vel }) => show.midi?.noteOn(note, Math.round(vel * 127), 1)). For instruments inside the page (a grand piano, mallets, strings, organ, synth presets), use modules.sound (Show control).

Checklist before a show#

• every continuous control is a param (try performing your world with ONLY the timeline)
• dispose() releases GPU objects and event subscriptions (worlds get switched live)
• world survives dt spikes (the Loop clamps to 100ms, but don't assume 16ms)
• nothing in the world reads the clock directly — the timeline owns time
• the page loads with zero console errors, and works with only the on-screen piano

Inputs#

06Source: docs/inputs.md

Every input does one thing: publish signals. With createShow() you switch inputs on in modules; each one is also a plain class you can use on its own. The exact names and ranges are in the Signals reference.

Sources that need permission or a user gesture (microphone, camera) are created by createShow() but only start when you click their button in the console's L1 · Input panel: 🎤 mic, 🖐 hands, 🕺 body.

MIDI#

createShow() creates a Midi engine and calls enable() on startup, so the browser may ask for MIDI permission when the page opens. Devices can be plugged in and out during the show; the L1 · Input panel lists them, and unticking one mutes it without unplugging it.

Every incoming message publishes two families of names:

• legacy (what most worlds use, any channel): midi/note/on and midi/note/off pulses, midi/cc/<n> (0..1), midi/bend (−1..1);
• per channel: midi/ch/<ch>/cc/<n>, midi/ch/<ch>/note/<n> (velocity 0..1 while held, 0 on release), midi/ch/<ch>/bend, midi/ch/<ch>/pressure, midi/ch/<ch>/poly/<n>.

A note-on payload is { note, vel, velocity, ch, device }: vel is 0..1, velocity is the raw 1..127, ch is 1..16, device is the port's slug.

With two or more devices connected, each also publishes under its own slug so their controls do not collide: midi/<slug>/cc/<n>, midi/<slug>/note/on, midi/<slug>/note/off, midi/<slug>/bend. The slug comes from the port name (Arturia MiniLab 3 → arturia-minilab-3; a second identical unit gets -2), so a device that drops out and comes back keeps its routes. The legacy names keep publishing too, so a route on midi/cc/74 hears every device.

Clock messages are ignored. Program change, transport and SysEx are not published (SysEx access is never requested).

Using it without createShow():

import { Midi } from '@openav/midi';

const midi = new Midi({ signals });     // options: signals, filterOut = 'IAC', requestAccess
await midi.enable();                    // false if the browser has no Web MIDI or permission was denied
midi.devices();                         // [{ slug, name, listening }]
midi.setListening('nanokontrol2', false);

filterOut is a case-insensitive regular expression on input port names that are ignored. The default, 'IAC', keeps the macOS IAC bus out, because a show that sends MIDI out through IAC would otherwise hear itself. See Troubleshooting if you want to receive from IAC. MIDI output is described in Show control.

Known controllers on screen#

modules: { midi: { controllers: { profile: '<profile id>' } } } adds an on-screen picture of a known controller. Plug that controller in and the picture switches to it and moves with your hands; without one, play the picture with a mouse or a finger. Its controls publish midi/<device>/<control> names, and controllerRoutes() writes the routes that connect them to params. The MIDI controllers & embedding chapter covers profiles, learning unknown devices, LED feedback and embedding; the supported devices are on the controllers page.

On-screen piano, QWERTY and the simulated performer#

@openav/keys is on by default. It mounts a piano above the console with two checkboxes, keyboard (QWERTY capture) and simulate performance. Options (modules.keys): base (lowest note, default 48), octaves (default 2), sim (show the simulate checkbox, default true), capture (show the keyboard checkbox, default true).

It publishes midi/note/on { note, vel, ch: 0 } and midi/note/off { note, ch: 0 }. Note the differences from hardware: ch is 0 and there is no velocity or device field. A world that only reads note and vel works with both.

• QWERTY (after ticking keyboard): A W S E D F T G Y H U J K O L P ; play 17 notes a semitone apart, starting at base; Z/X move an octave (base stays between 24 and 84); Shift adds 20 to the velocity. While capture is on, the piano claims every letter key (preventDefault); Space, arrows and digits pass through to the console.
• The simulated performer (SimPlayer) plays a pentatonic scale an octave above base: mostly steps, sometimes leaps and triads, real rests. From code: show.keys.sim.toggle(true).

import { mountKeys, KeysPiano, SimPlayer, PIANO_KEYMAP } from '@openav/keys';

const keys = mountKeys(el, { signals, base: 48, octaves: 2 });   // → { piano, sim, update(dt), dispose() }
// call keys.update(dt) every frame (it drives the SimPlayer)

Drum machine#

modules: { drums: true } adds a 16-step, four-lane sequencer (kick, snare, hat, clap) with patterns (four on floor, breakbeat, half time, latin, sparse), a tempo slider (60–180 BPM, default 112), swing and humanized timing. Tick drum machine to start it. It plays its own synthesized kit (no samples) and publishes:

• drum/kick, drum/snare, drum/hat, drum/clap: pulses { level };
• drum/<lane>/env: a 0..1 envelope per lane that decays in about 0.12 s, ready to map straight onto a param;
• midi/note/on { note, vel, ch: 10 } with General MIDI notes (kick 36, snare 38, clap 39, closed hat 42).

The shapes match what the microphone's drum detection publishes (audio/kick, audio/kick/env…), so a world mapped to a kick cannot tell a real bass drum from the machine. Example 06 routes both to the same params. Options (modules.drums): engine (another sound engine), autoEnableEngine (default true: the kit's audio engine is switched on by the first hit, no extra click).

Microphone#

modules: { audio: true } creates an AudioAnalyzer; the microphone starts when you click 🎤 mic. (Example 06 writes audio: 'mic'; any truthy value works.) Echo cancellation, noise suppression and auto gain are turned off so the analysis hears the room as it is.

Published every frame: audio/rms (smoothed loudness), audio/peak, audio/band/low (20–250 Hz), audio/band/mid (250 Hz–2 kHz), audio/band/high (2–8 kHz), audio/centroid (brightness), and the pulse audio/onset { rms } with a 100 ms refractory time. Drum detection splits transients by band: audio/kick (20–120 Hz), audio/snare (150–800 Hz), audio/hat (6–14 kHz), each a pulse { level } with a matching …/env envelope.

import { AudioAnalyzer } from '@openav/audio';

const audio = new AudioAnalyzer({ signals, fftSize: 2048, smooth: 0.7 });
await audio.enableMic();            // or audio.enableElement(videoEl), or someNode.connect(audio.input())
// every frame: audio.update()      → also returns { rms, peak, low, mid, high, centroid, onset, kick, snare, hat }

Chords#

modules: { chord: true } (or { chord: { window: 80 } }) adds a ChordDetector and feeds it every midi/note/on and midi/note/off. It groups notes struck within window milliseconds (default 80) into one gesture and publishes, per gesture:

• chord/consonance (−1..1), chord/count (number of notes), chord/root (lowest MIDI note);
• chord/event, a pulse carrying the whole analysis:

This is not a music-theory library; it is a performance-semantics layer built for a piece where consonance makes a garden flourish and clusters make it decay (example 02). detector.analyze(notes, vels) is a pure function you can call yourself.

Body and hands#

modules: { pose: true } adds a PoseTracker (one body), modules: { hands: true } a HandTracker (two hands, 21 points each). The camera starts when you click 🕺 body or 🖐 hands. Both use MediaPipe Tasks Vision (0.10.14), loaded from jsDelivr with its model from Google's storage the first time you enable it; the inference runs on your machine and no video leaves it. The view is mirrored by default (mirror: true), like a mirror on stage, and y is flipped so 1 means raised.

• Body: pose/present, pose/hand/left/x|y, pose/hand/right/x|y (wrists), pose/hand/left/v, pose/hand/right/v (wrist speed), pose/hands/spread, pose/height (nose height: crouch ↔ stand), pose/lean (−1..1).
• Hands: per side, hand/<side>/present, hand/<side>/x|y (palm), hand/<side>/pinch/index and hand/<side>/pinch/middle (thumb tip to finger tip), hand/<side>/spread (index tip to pinky tip). Distances are divided by palm size, so moving toward the camera does not change a pinch. A closed pinch reads near 0: use invert: true on the route when closing should raise the value (example 03).

Both trackers draw their skeleton on a 2D context: tracker.skeleton(ctx, w, h).

Leap Motion#

modules: { leap: true } adds a LeapInput. Browsers cannot read the sensor, so one small bridge per machine does: node packages/leap/bridge/leap-bridge.mjs (needs Ultraleap Hand Tracking installed; --mock gives two animated hands without a sensor; bash packages/leap/bridge/install.sh keeps it running at login on macOS). It serves ws://127.0.0.1:6437 on loopback only. The console's L1 · Input panel shows which link is missing (bridge, tracking service, device) and draws the hands; its simulate button turns the mouse into hands.

Signals: per side, leap/hand/<side>/present, x|y|z (palm in the interaction box, 0..1, y up, z toward you), pinch, grab, roll|pitch|yaw (radians), speed, fingertips leap/hand/<side>/<finger>/x|y|z; leap/hands, leap/both/distance, leap/status; pulses pinch-start|pinch-end, grab-start|grab-end per side and leap/both/pinch-start|pinch-end, with hysteresis (a pinch starts above 0.86 and ends below 0.5). Joints for drawing the hand are on leap.hands. The same bridge answers old leap.js sketches on /v6.json, and serves raw hand lines on /raw (example 12, package README).

Phones and iPads#

modules: { remote: true } lets phones join the show over the relay that serve.js hosts. Their sensors publish phone/<id>/… (tilt, acceleration, rotation, knocks, touch, light), mirrored to phone/any/…; the control surface publishes surface/<page>/<widget>; the phone's piano publishes midi/note/on|off like a keyboard. The show page itself can read its own motion sensors on a phone with localSensors(). Everything is in Phones, relay & surfaces.

Your own input#

An input is anything that calls signals.set() or signals.pulse(). Here is the pointer as an instrument:

const { signals } = show;
signals.define('pointer/x', { source: 'pointer' });     // continuous, 0..1 by default
signals.define('pointer/y', { source: 'pointer' });
signals.define('pointer/tap', { kind: 'pulse', source: 'pointer' });

addEventListener('pointermove', (e) => {
  signals.set('pointer/x', e.clientX / innerWidth);
  signals.set('pointer/y', 1 - e.clientY / innerHeight);  // 1 = top, like every other y
});
addEventListener('pointerdown', (e) => signals.pulse('pointer/tap', { x: e.clientX, y: e.clientY }));

show.mapper.addRoute({ source: 'pointer/x', target: 'hue', smooth: 0.25 });

Name it source first (breath/pressure, weather/wind), declare its range, and everything downstream (meters, learn, routes, the phone surface) works without changes.

Mapping#

07Source: docs/mapping.md

The mapping layer is where "what happened" becomes "what it means". The Mapper (@openav/mapping) is the only component that connects signals to params, and everything it knows is a list of routes.

Routes#

A route is a plain object. Only source and target are required; addRoute() fills in the rest:

show.mapper.addRoute({ source: 'midi/cc/74', target: 'bloom' });
show.mapper.addRoute({ source: 'phone/any/tilt/x', target: 'drift', inMin: -1, inMax: 1, smooth: 0.12 });
show.mapper.addRoute({ source: 'hand/right/pinch/index', target: 'tightness', curve: 'smooth', invert: true, smooth: 0.1 });

In createShow(), the same objects go in routes: [...].

What happens to a value#

When a signal arrives, every enabled route whose source matches runs these steps, in order:

1. Window: x = (value − inMin) / (inMax − inMin), clamped to 0..1.
2. Invert: if invert, x = 1 − x.
3. Curve: x = curve(x).
4. Scale: out = outMin + (outMax − outMin) · x.
5. Write: with smooth: 0, the param is overridden with out (and clamped and snapped to its step). With smooth > 0, out becomes a target that mapper.update(dt) glides toward every frame.

Curves#

An unknown curve name falls back to linear.

Windows and ranges#

inMin/inMax say which part of the signal matters; outMin/outMax say where it lands. Typical uses:

• a bipolar signal: { source: 'midi/bend', inMin: -1, inMax: 1 }, or a phone tilt { source: 'phone/any/tilt/x', inMin: -1, inMax: 1 };
• a quiet signal: { source: 'audio/rms', inMax: 0.3 } uses the whole param range for the levels a voice actually reaches;
• a knob that only sweeps part of a param: { source: 'midi/cc/1', target: 'hue', outMin: 180, outMax: 270 };
• an inverted relation without invert: outMin: 1, outMax: 0.

Smoothing#

smooth is a time constant in seconds: after smooth seconds the param has covered about 63% of the distance to the new value (a one-pole filter, k = 1 − e^(−dt/smooth) per frame). 0.05–0.15 hides the steps of a MIDI knob or a 30 Hz phone stream; 0.2–0.5 makes a hand feel heavy. The glide starts from the param's current value, so the first movement never jumps.

Pulse targets#

When the target is a pulse param, the route fires it instead of writing a value:

• a pulse source (midi/note/on, audio/kick, drum/kick, phone/any/knock…) fires it on every event;
• a continuous source fires it when the normalized value crosses 0.5 upward (a phone-surface button going from 0 to 1, a fader pushed past the middle).

curve, invert, smooth and the output range do not apply to pulse targets.

Many-to-many#

One signal may drive many params (add several routes with the same source), and one param may listen to many signals. When two routes write the same param, the last one to write wins. Learn adds a route; it never replaces one.

The patch bay design plans per-route combine (last, max, add, avg) and label fields; they are not implemented yet.

Overrides and the timeline#

A route writes its param as an override, and overrides are sticky: as soon as a mapped signal moves, the timeline stops driving that param. Clearing the override (✕ next to the param, or show.params.clearOverride(key)) hands it back to the timeline until the signal moves again.

A smoothed route keeps writing its last target every frame once its signal has arrived, so its param stays overridden even after ✕ and even when the route is muted. Remove the route (✕ in L2 · Mapping) to give the param back to the timeline for good.

Learn#

Click learn next to a param in L3 · Params (or call mapper.learn('bloom')), then move something. The mapper binds the first signal that:

• moves by at least 5% of its declared range from where it was when learn started (a knob, a fader, a hand, a phone tilt), or
• fires, if it is a pulse (a key, a pad, a knock).

The new route has the defaults: linear, no smoothing, the param's full range. Edit it afterwards if it needs a curve. Clicking learn again cancels. Learn adds: a param that already has a route keeps it and gets a second one.

To bind a whole list, the L2 · Mapping panel has 🎛 map knobs in order: it arms learn for the first param, you turn one knob, it is bound and the next param is armed. skip, back, stop (or Esc) steer it. In code it is LearnWizard (details).

When one MIDI message publishes several names (midi/cc/74, midi/ch/1/cc/74, and with two devices midi/<slug>/cc/74), learn binds the first one, midi/cc/74, which hears every device. Change the source to the per-device name if you need one device only.

Muting and removing#

The L2 · Mapping panel lists every route as source → target with a checkbox (mute) and ✕ (remove). In code:

const r = show.mapper.routesFor('bloom')[0];
r.enabled = false;                  // mute
show.mapper.removeRoute(r.id);      // remove

Saved profiles#

The mapper saves its routes to localStorage under openav.map.<profile>. createShow() uses the world's name as the profile (or the profile option) and:

1. loads the saved routes if there are any, otherwise adds the declared routes;
2. when a saved profile exists, still adds every declared route whose source → target pair is missing, so new routes in code arrive;
3. saves every 3 seconds.

So learned routes and edits survive a reload. The flip side: if you change the curve or smoothing of a declared route in code, the saved copy of that source → target pair wins. To start clean, pass profile: false (routes come from code on every load, as in example 09) or remove the saved profile:

localStorage.removeItem('openav.map.particles');   // 'particles' = the world's name

To carry a wiring to another machine or another world, use a map file (next section).

Map files: export, import and share#

The L2 · Mapping panel has three buttons under map knobs in order:

• ⬇ export downloads the routes as oav-map-<world>.json.
• ⬆ import picks a map file. You can also drop a .json anywhere on the page. Import replaces the routes; the new set is remembered at once.
• clear removes every route (click twice). On the next load the world's declared routes come back, as on a first visit.

Below the buttons the panel says what the last import did: how many routes came in, and every route it skipped and why. A file that cannot be read says so and changes nothing.

The format#

{
  "format": "openav.map",
  "version": 1,
  "world": "particles",
  "exported": "2026-10-15T08:30:00.000Z",
  "routes": [
    { "signal": "midi/cc/74", "param": "bloom", "curve": "exp",
      "range": { "in": [0, 1], "out": [0, 100] }, "smooth": 0.1, "invert": false },
    { "signal": "audio/rms", "param": "energy", "curve": "linear",
      "range": { "in": [0, 0.3], "out": [0, 1] }, "smooth": 0, "invert": false, "enabled": false }
  ]
}

Routes are named by signal and param, never by id, so a file travels between worlds:

• a route whose param this world does not have is skipped and listed;
• a route whose signal nothing in this show can produce (say leap/palm/x without a Leap Motion module) is skipped and listed. Signals that only appear when the first message arrives (midi/…, surface/…, phone/…, osc/…, key…, take/…) are kept and counted as waiting;
• when the file's world is a different one, the range.out is dropped so each param keeps its own range (the in window, curve, smoothing and invert come along);
• version newer than this build reads is refused with a message. A bare array from mapper.toJSON() (the old way) still imports, as version 0.
• Files over 256 KB or with over 500 routes are refused.

Load on open: ?map=#

https://openaudiovisual.com/examples/01-hello-particles/?map=https://example.com/my-map.json
https://openaudiovisual.com/examples/01-hello-particles/?map=maps/stage-left.json

?map=<url> fetches the file when the page opens and loads it like an import. It accepts a same-origin address (relative paths included) or an absolute https one; http: to another host, data:, javascript:, file: and addresses with a login are refused. A cross-site file must be served with CORS (Access-Control-Allow-Origin: *; raw GitHub and gists do this). The show starts regardless: if the address is refused, the fetch fails or the file is bad, the panel shows the reason and the routes stay as they were. An explicit ?map= beats the remembered wiring, and what it loads is remembered like any import. In code, createShow({ map: 'maps/a.json' }) does the same and map: false ignores ?map=.

Remembered for you#

The mapper saves its routes to localStorage (openav.map.<world>) every 3 seconds and right after an import or clear, one slot per world name, so the next visit starts where you stopped. Every read and write is wrapped: in a private window or with storage blocked nothing breaks, the wiring just is not remembered.

In code:

import { loadMapUrl, parseMap } from '@openav/mapping';

const file = show.mapper.exportMap();                    // the object above
const report = show.mapper.importMap(text);              // text or object; { mode: 'add' } keeps current routes
// → { ok, added, skipped: [{ signal, param, reason: 'param' | 'signal' }], waiting, dropped, crossWorld }
await loadMapUrl(show.mapper, 'maps/a.json');            // what ?map= calls; never throws
parseMap(text);                                          // validate without touching a mapper

Routes written for you#

Several packages generate routes in this same shape:

• autoSurface(world.params) (@openav/surface) builds a phone control panel and the surface/<page>/<widget> → param routes for it, with smooth: 0.04; routesFromLayout(layout, params) does the same for a hand-written layout (Phones, relay & surfaces).
• controllerRoutes(profile, { knob1: 'hue', pad1: 'burst' }) (@openav/midi) writes routes from an on-screen controller's controls (MIDI controllers & embedding).

Mapper API#

import { Mapper } from '@openav/mapping';

const mapper = new Mapper({ signals, params, profile: 'default', onChange, onLearn });
mapper.addRoute(route)        // → the stored route (defaults filled in, id assigned)
mapper.removeRoute(id)
mapper.routesFor('bloom')     // → routes targeting a param
mapper.routes                 // the live array
mapper.learn('bloom')         // arm / cancel learn; mapper.learnTarget is the armed key
mapper.update(dt)             // every frame: advances smoothed routes
mapper.toJSON(); mapper.fromJSON(array)
mapper.exportMap(); mapper.importMap(text|object, { mode })   // the portable file, see above
mapper.clear()                // remove every route (and forget them)
mapper.report                 // what the last import / ?map= said
mapper.save(); mapper.load()  // localStorage 'openav.map.<profile>'; load() → true if found
mapper.dispose()              // stop listening to signals

onChange() runs when routes change (for UIs); onLearn(route, signalName) runs when learn binds.

Show control#

08Source: docs/show-control.md

The parts that turn a sketch into a performance: the one-call assembly, the timeline, the console, performance mode, the backstage monitor, and the ways sound and data leave the browser.

createShow#

createShow() (@openav/show) builds a whole show around a world and returns every part. It is async because it waits for the world's init.

import { createShow } from '@openav/show';

const show = await createShow({
  world: myWorld,
  timeline: { total: 120, automation: { hue: [[0, 205], [60, 320]] }, scenes },
  routes: [{ source: 'audio/rms', target: 'energy', inMax: 0.3, smooth: 0.1 }],
  modules: { keys: { base: 48 }, chord: true, audio: true, sound: true },
  artwork: { title: 'Untitled', artist: 'You', year: 2026 },
  hint: 'play the piano · Space plays the timeline',
});

Without mount, the page body becomes a two-column grid: the stage, and a 340 px side panel with the piano and the console.

It returns { signals, params, stage, timeline, mapper, midi, controllers, midiPanel, keys, drums, sound, audio, hands, pose, chord, remote, score, director, console, app, loop }; parts you did not ask for are null. The same object is window.openav, for the devtools console. The frame order is in Architecture.

Timeline#

const timeline = new Timeline({
  params,
  total: 2400,                       // seconds (default 600 when used directly; createShow uses 120)
  automation: {
    growth: [[0, 0.2], [300, 1.0, 'ease'], [1200, 0.4, 'out']],   // [t, value, curve?] keyframes; no curve = linear
    mode:   [[0, 0], [600, 1], [1800, 2]],          // step params hold (no interp)
  },
  scenes: [
    { id: 'standby', t: -30, title: 'Standby', note: 'house lights · breathe' },
    { id: 'act1',    t: 0,   title: 'Act I',   note: 'single seeds only' },
    { id: 'act2',    t: 600, title: 'Act II',  note: 'full harmony' },
  ],
});
timeline.onSceneChange((i, scene) => { /* fire cues: sound, lights, OSC */ });

• negative scene times = pre-show standby (the clock shows -0:30)
• jumpScene(±1) (← → keys) is your emergency navigation — rehearse with it
• timeline.rate exists for rehearsal speed-through (rate = 4)
• playback stops by itself at total
• a scene with hold: true stops playback at its end until it is released (Space, →): see Hold
• loop: true wraps the show at total and plays on; loop: 'scene' repeats the scene the playhead is in until you move on: see Curves, loop, cues, recording
• reset() goes to start (default 0, so a pre-show standby scene at −30 is reached with the scrubber; pass start: -30, or start: 'first' for "the first scene", to make R return to it)
• with score the scenes, total and start come from a score; then automation is read at score.baseTime(t), so a cut derived with scale: 0.5 stretches it

Curves, loop, cues, recording#

Curves. A keyframe is [t, value, curve?]. The third item says how the segment that ends at that keyframe gets there: 'linear' (the default: leave it out and you get what the timeline always did), 'ease' (slow, fast, slow), 'in' (starts slow), 'out' (starts fast), 'step' (hold the old value until the keyframe, then jump), or your own function of 0..1. CURVES is exported. A param with step still holds, whatever the curve says.

automation: { speed: [[0, 0.3], [16, 1.6, 'in'], [20, 3.4, 'step'], [30, 0.8, 'out']] }

Loop. loop: true wraps at total (back to 0) and keeps playing; 'scene' repeats the scene the playhead is in (next() moves the loop on to the next scene); a scene id loops that scene; { from, to } loops a range in seconds or scene ids. It is a property: timeline.loop = !timeline.loop is the whole L key. onLoop(cb) tells you each lap, and a loop that wraps into another scene is an onSceneChange with cause: 'loop'. A hold inside the range still holds.

Cues. scene.cues is [{ name, t }] (the console scrubber already draws them as ticks). onCue((cue, { cause }) => …) fires one when playback passes it, once, and never when you seek, jump, drag the scrubber or reset: a scrub is not a performance. Each lap of a loop fires them again. fireCue(cue) sets one off by hand. With a score the director fires its own cues; use onCue on plain timelines.

show.timeline.onCue((cue) => show.params.firePulse(cue.name));   // scenes: [{ …, cues: [{ name: 'flash', t: 20 }] }]

Recording. record(key, source) arms a lane. While the timeline plays, source() is read every frame: a number is written into a take, null means nobody is touching and closes it.

timeline.record('hue', () => params.isOverridden('hue') ? params.overrides.hue : null);
timeline.onRecord(({ key }) => params.clearOverride(key));      // the curve takes over again

A take also closes on pause, seek, reset, a hold, the end of the show and the end of each loop lap, while the recorder stays armed until stop() / stopRecording(). The merge rule (mergeTake()): the take is thinned (a point stays when it moved more than 0.6 % of the param's range, or 0.5 s passed), and it replaces the keyframes inside its time span. Keyframes outside stay, so the lane glides from the previous keyframe to the take's first point, and from its last point to the next keyframe. Existing curves outside the take are untouched. With a score the keyframes are written in the base cut's time, like the ones you declared.

A cue list is just a scene-change handler:

show.timeline.onSceneChange((i, scene) => {
  if (scene.id === 'storm') show.stage.activate('storm');   // switch worlds
  show.midi?.cc(20, i, 16);                                  // tell the lighting desk
});

The console#

mountConsole(el, app, { layers, signals }) (@openav/console) builds the director's desk; createShow() mounts it in the side panel. app is the object createShow() returns as show.app (timeline, params, mapper, signals, stage, midi, sound, audio, hands, pose, …); missing parts simply hide their section. Pass { layers: false } or { signals: false } to leave out those panels. It gives you transport + scrubber with scene blocks, the Layers overview, L1 Input, L2 Mapping, L3 Params (sliders with override and learn chips), L4 sound, live signal meters and a MIDI log.

Keyboard: Space play/pause (at a hold: release) · →/← next / previous scene (→ at a hold: release) · R reset · T performance mode · F fullscreen · Esc leave keyboard-piano capture or stop the mapping wizard. Keys typed into an input field, held keys and ⌘/Ctrl/Alt combinations are ignored. While the on-screen piano captures the keyboard the letters belong to the piano (T and F are piano keys), so R, T and F do nothing until you untick it or press Esc; Space and the arrows always work.

With a score the desk also has the Director panel, cue ticks on the scrubber and a prompter with the operator's acts (Score & Director). The L2 Mapping panel has 🎛 map knobs in order, a wizard that arms learn for each param in turn (details).

Override semantics: touch a slider (or a mapped control) and that param leaves the timeline's hands until you clear it (✕). Yellow label = overridden.

Performance mode#

T flips to a fullscreen teleprompter: current scene title + note in stage- readable type, what the operator does (the scene's acts), next scene preview, clock + scene countdown, a HOLD banner while the show waits, and a warning when a segment module was switched off. This is for the performer; the audience-facing window is your stage in fullscreen (F). Two windows of the same page = one desk, one stage.

Backstage monitor#

node packages/monitor/server.js       # on the performance machine (npm run monitor)
# stage manager's phone/laptop: http://<performance-machine-ip>:7457/

Every show page made with createShow() streams snapshots to ws://<the page's host>:7457 about 15 times a second, so run the monitor server on the machine that serves the show. Backstage shows clock, scene, world, FPS, overrides, all signals and params live, and a red "stage feed lost" banner if the stage stops sending for 4 s — which is exactly the moment a stage manager earns their pay. Zero configuration, zero dependencies; both ends reconnect by themselves. GET /health answers { ok, clients }.

When no monitor is running, the show page keeps retrying with a growing delay (about 5 s, growing to 30 s); the browser logs each refused connection. That message is expected.

For a hand-assembled show:

import { MonitorFeed, snapshotOf } from '@openav/monitor';
const monitor = new MonitorFeed({ url: 'ws://192.168.1.20:7457', hz: 15 });   // both optional
monitor.connect();
// every frame:
monitor.frame(snapshotOf({ timeline, params, signals, stage, loop }, state));

OSC output#

node packages/osc/bridges/osc-bridge.js 7456 192.168.1.50 3456
#                       http port ↑    target host ↑   udp port ↑

The defaults are 7456 127.0.0.1 3456. The browser cannot send UDP, so OscOut posts batches to the bridge over HTTP and the bridge sends them as OSC packets.

import { OscOut } from '@openav/osc';

const osc = new OscOut();          // http://127.0.0.1:7456
await osc.enable();                // probes /health, fails gracefully (false)
osc.send('/source/3/xyz', [x, y, z]);
osc.flush();                       // once per frame (batches, dedupes per address)

Numbers go out as OSC floats — what Spat, Reaper, and TouchDesigner expect; anything else is sent as a string. Only the latest message per address survives a frame. If the bridge disappears, OscOut reports it through onStatus(ok, info) and stops trying after 30 failed flushes.

createShow() does not create an OscOut. Add one yourself (and add "@openav/osc": "../../packages/osc/index.js?v=b59edc6" to the page's import map):

const osc = new OscOut();
await osc.enable();
const show = await createShow({ world, onFrame: () => osc.flush() });
show.app.osc = osc;          // the Layers panel shows the OSC rate
show.stage.io.osc = osc;     // worlds can send from update(dt, state, io)

TouchDesigner as the receiver#

OSC goes to TouchDesigner as above. TouchDesigner can also show an open-audiovisual page as a picture with its Web Render TOP, which draws a web page inside TouchDesigner on the same machine (Web Render TOP). We have not checked whether the Web Render TOP supports Web MIDI or WebGPU. Keep the control on OSC.

If the page comes from openaudiovisual.com, Chrome asks before the page may reach the bridge on 127.0.0.1: see Chrome asks about the local network (Chrome blog). Running node serve.js and opening the page from http://localhost:8080 avoids the prompt.

MIDI output and panic#

midi.noteOn/noteOff/cc/send go to the selected output port — and midi.panic() sends all-notes-off + all-sound-off + sustain-off + per-note note-offs on all 16 channels, through the same observable throat as everything else. Bind it to a pad you can find in the dark. Every performer eventually needs it; ours is one call.

show.midi.outputs.map((o) => o.name);   // available output ports
show.midi.selectOutput('IAC Driver Bus 1');
show.midi.noteOn(60, 100, 1);           // note, velocity 0..127, channel 1..16
show.midi.noteOff(60, 1);
show.midi.cc(74, 64, 1);                // controller, value 0..127, channel
show.midi.send([0xb0, 74, 64]);         // raw bytes
show.midi.panic();

The first output port is used until you choose another (enable(preferredOut) matches a name exactly, then as a substring). Everything sent passes through onSend, which feeds the per-channel MIDI OUT meter in the Layers panel.

Hot-plug. A cable pulled mid-show must not leave the DAW silent without a word. When the current output is unplugged, Midi moves to the next best live port: the one you chose with selectOutput() or enable(name), then the first match of the prefer list (new Midi({ prefer: ['IAC'] })), then the first port. When the port you chose is plugged in again, the output goes back to it. An unrelated device appearing never steals the output. Every switch prints one line in the MIDI log (the Signals panel, via midi.onMessage) and one in the browser console, flashes in the MIDI controller panel, and updates the MIDI out → row of the L1 Input panel. In code: midi.onOutput(({ text, reason }) => …).

Echo. What you send into a virtual bus must not come back as input. Inputs whose name matches filterOut (default IAC, the macOS bus) are never listened to, and a message identical to one just sent that arrives on any input within echoWindow milliseconds (default 40; 0 turns it off) is dropped as echo, once per message sent. midi.echoDropped counts them.

To make an external program follow the show, ScoreMidi sends the score's segment changes and cues as notes and CCs (Score to MIDI).

Sound#

modules: { sound: true } adds Sound (@openav/sound) with the Tone.js engine. Browsers only start audio after a click, so the console shows 🔊 enable sound in the L4 · Output — sound panel, with an instrument picker under it. Once enabled, every midi/note/on and midi/note/off plays the chosen instrument, whatever sent it (a keyboard, the QWERTY piano, the simulated performer, a phone). Tone.js (15.0.4) is loaded from jsDelivr only when sound is enabled.

await createShow({ world, modules: { sound: true } });       // the warm pad, as before
await createShow({ world, modules: { sound: 'piano' } });    // start on the grand piano
await createShow({ world, modules: { sound: { instrument: 'synth/pluck', remember: false, picker: false } } });

Instruments#

The piano samples are Salamander Grand Piano by Alexander Holm, licensed CC-BY 3.0. The picker shows that credit next to the piano, and a show that plays it in public credits it too. They ship in packages/sound/samples/salamander/. If they cannot load, the piano plays the electric piano instead of going silent.

Choosing a sound#

Pick from the console, or from code:

await show.sound.setInstrument('organ');   // true once it plays
show.sound.instrument;                     // 'organ'
show.sound.onChange((e) => console.log(e.type, e.id, e.progress));   // instrument · loading · ready · error
show.timeline.onSceneChange((i, scene) => {
  if (scene.id === 'chorale') show.sound.setInstrument('organ');      // a cue may switch instruments
});

Switching while you play releases the notes held on the old instrument, so nothing hangs. A sampler keeps the old sound playing until its files are in, and the picker shows the progress. Each switch also fires the pulse signal sound/instrument ({ id, name }). The picker's last choice is remembered per page and wins over the declared instrument on the next visit; remember: false turns that off.

The instrument is a discrete choice. What you perform continuously stays a param: the engine's params join the show's params under sound/. sound/cutoff (100–8000 Hz, default 2500), sound/space (reverb wet, 0–1, default 0.3) and sound/volume (−36–0 dB, default −8) are shared by every instrument, so a knob, a hand or the score plays the filter the way it plays the visuals. An instrument may suggest starting values (the piano opens the filter to 8000 Hz) and add knobs of its own, which the console shows while it plays: the organ's sound/drawbars and sound/rotor, sound/attack and sound/vibrato on the strings, sound/vowel on the choir, sound/tremolo on the electric piano and the vibraphone. Automation and overrides always win over a suggested default. The sustain pedal (midi/cc/64) holds every instrument's note-offs. Notes on MIDI channel 10 (the drum machine) are left to a drum engine.

Add your own instrument#

A sound you can pick is one file: data plus a create(Tone, ctx) that returns noteOn, noteOff and dispose. Import it before createShow() and every picker lists it; nothing in packages/* changes.

import { registerInstrument } from '@openav/sound';

registerInstrument({
  id: 'kalimba',
  name: { en: 'Kalimba', zh: '拇指琴' },
  category: 'mallets',
  params: [{ key: 'decay', label: 'Decay (s)', min: 0.2, max: 4, def: 1.2 }],   // → sound/decay
  create(Tone, { output }) {
    const synth = new Tone.PolySynth(Tone.FMSynth, { harmonicity: 5.1, modulationIndex: 2,
      envelope: { attack: 0.001, decay: 1.2, sustain: 0, release: 1 } }).connect(output);
    const hz = (n) => 440 * 2 ** ((n - 69) / 12);
    return {
      noteOn: (note, vel, time) => synth.triggerAttack(hz(note), time, vel),
      noteOff: (note, time) => synth.triggerRelease(hz(note), time),
      set: (key, v) => key === 'decay' && synth.set({ envelope: { decay: v } }),
      dispose: () => synth.dispose(),
    };
  },
});

A duplicate id throws unless you pass registerInstrument(def, { replace: true }). Optional fields: defaults (starting cutoff, space, volume), credit, gain (dB), transpose (semitones), fallback (an id to play if loading fails) and, on the returned object, ready (a Promise: the picker shows progress while you call ctx.onProgress(0..1)), set(key, value) and releaseAll(time). Mallet sounds can be pure data with modalInstrument({ id, name, partials, strike }).

A synth preset is data as well, and becomes synth/<id>:

import { registerSynthPreset } from '@openav/sound';

registerSynthPreset({
  id: 'glass', name: { en: 'Glass', zh: '玻璃' },
  voice: 'fm', harmonicity: 3.01, modulationIndex: 14,
  envelope: { attack: 0.002, decay: 1.4, sustain: 0, release: 1.2 },
  effects: [{ type: 'chorus', frequency: 1.5, depth: 0.4, wet: 0.3 }],
});

Your own engine#

An engine is a small object, so you can replace the whole thing:

const sineEngine = () => {
  let ctx = null, out = null;
  const voices = new Map();
  return {
    params: [{ key: 'gain', label: 'Gain', min: 0, max: 1, def: 0.3 }],   // → sound/gain
    async enable() {
      ctx = new AudioContext();
      out = ctx.createGain(); out.gain.value = 0.3; out.connect(ctx.destination);
    },
    noteOn(note, vel) {
      voices.get(note)?.stop();
      const o = ctx.createOscillator(), g = ctx.createGain();
      o.frequency.value = 440 * 2 ** ((note - 69) / 12);
      g.gain.value = vel * 0.3;
      o.connect(g).connect(out); o.start();
      voices.set(note, o);
    },
    noteOff(note) { voices.get(note)?.stop(); voices.delete(note); },
    set(key, value) { if (key === 'gain' && out) out.gain.value = value; },
    dispose() { ctx?.close(); },
  };
};

await createShow({ world, modules: { sound: { engine: sineEngine() } } });

The drum kit from @openav/drums is also an engine: drumEngine({ samples }) maps General MIDI drum notes to synthesized drums (or to your samples, { 36: 'kick.wav', … }) and has one param, kitVolume (−30–0 dB).

Without createShow#

createShow() is a convenience. When you need your own layout or loop, assemble the parts by hand; this is roughly what it does:

import { Signals, Params, Loop } from '@openav/core';
import { Midi } from '@openav/midi';
import { Mapper } from '@openav/mapping';
import { Timeline } from '@openav/timeline';
import { Stage } from '@openav/stage';
import { mountConsole } from '@openav/console';
import { MonitorFeed, snapshotOf } from '@openav/monitor';

const signals = new Signals();
const params = new Params();
const stage = new Stage({ container: document.querySelector('#stage'), params, signals });
stage.register(myWorld);
await stage.activate(myWorld.name);

const timeline = new Timeline({ params, total: 120, automation, scenes });
const mapper = new Mapper({ signals, params, profile: myWorld.name });
if (!mapper.load()) routes.forEach((r) => mapper.addRoute(r));

const midi = new Midi({ signals });
midi.enable();

// mount the console after every world (and sound engine) is registered: it builds one row per param
const app = { timeline, params, mapper, signals, midi, stage };
const desk = mountConsole(document.querySelector('#desk'), app);
const monitor = new MonitorFeed();
monitor.connect();

const loop = app.loop = new Loop((dt) => {
  timeline.advance(dt);
  mapper.update(dt);
  const state = stage.frame(dt, timeline.state());
  desk.render(state);
  monitor.frame(snapshotOf({ timeline, params, signals, stage, loop }, state));
});
loop.start();

Pre-show checklist#

• Chrome, plugged in, screen-sleep off, notifications off (chrome://settings)
• WebMIDI permission granted for this origin (it's per-origin, per-port)
• mapping profile saved (mapper.save()) AND exported to a JSON file
• monitor server running; stage manager's device on the same network
• sound enabled with a click, and the panic pad mapped
• rehearse the failure: kill the tab, restart, be performing again in <30 s

Score & Director#

09Source: docs/score.md

A show that outgrows one afternoon needs a structure that can grow. The score (@openav/score) writes a show down as segments with lengths, and the director runs the code that performs each segment. They are pure logic with no DOM, and createShow() wires them to the timeline, the console and the prompter for you.

The idea: lengths, never seconds#

A timeline written as a table of absolute seconds cannot grow. Make the second scene longer and every later scene, cue, curve and if (t > 466) in your code is now wrong, and you find them one rehearsal at a time. In a score you write only how long each segment is:

const cuts = [
  { id: 'full', title: 'A day', segments: [
    { id: 'standby', title: 'Standby', dur: 6, hold: true, acts: ['Check sound', 'Press Space when the room is ready'] },
    { id: 'dawn',    title: 'Dawn',    dur: 24, cues: { birds: 0.3 } },
    { id: 'noon',    title: 'Noon',    dur: 30, cues: { flare: { s: 8 }, peak: 0.5 }, acts: ['Open the haze slowly'] },
    { id: 'dusk',    title: 'Dusk',    dur: 24, linger: 8 },
    { id: 'night',   title: 'Night',   dur: 36 },
  ] },
  { id: 'short', from: 'full', scale: 0.5 },          // the same show at half the length
];

Every start, end, cue time, countdown and scene is computed from those lengths. Change noon to dur: 40 and everything after it moves by ten seconds while everything before it stays put. A second version of the show is a few lines of data, not a second timeline to keep in step.

A cut#

A cut is one version of the show. It has an id, an optional title, and either segments or a base cut and a scale.

A derived cut scales dur, linger and { s } cues, so a cue stays in the same place in the music. The ratio cues did not need to move.

A cut is checked when the score is built. Duplicate ids, a dur of 0 or below, a from that is not a cut, a scale of 0 or below, a cut derived from itself, a cue outside its segment, acts that are not strings: each throws a ScoreError that names the cut, the segment and the reason, and lists every problem at once. Nothing half-valid can reach the stage.

The Score#

import { Score } from '@openav/score';
const score = new Score({ cuts, cut: 'full' });      // cut defaults to the first

Automation on a derived cut#

Automation curves are written once, against the cut they were derived from, and a derived cut stretches them. timeline.valueAt(key, t) reads the curve at score.baseTime(t), which is t / scale. On the short cut above, t = 20 reads the curve at 40. A cut that is not derived has scale 1 and nothing changes.

If two cuts differ in structure and not just in length, do not share automation between them. Drive the params from the segment modules instead (param()), where ctx.p already scales for free.

Cues#

A cue is a named moment: cues: { peak: 0.5 } in segment noon makes noon.peak, half way through. Three places see it:

• the scrubber draws a tick for it, and a click on the tick jumps there;
• the director calls onCue({ name, cue, T, n, segment }) once each time the playhead passes it while playing, and calls the active module's cue(name, ctx) hook;
• ctx.cue('peak') inside a module tells you where it sits, as a progress you can compare with ctx.p ('noon.peak' works for another segment's cue, ctx.cueT(name) answers in show seconds).

A seek does not pass the cues it jumps over, and playing back over a cue fires it again. A cue sitting exactly on the first instant of the show fires on the first frame.

Hold#

hold: true means: when playback reaches the end of this segment, stop and wait. A standby that waits for the room to be ready is the typical use. Exactly what happens:

• The timeline stops a microsecond before the boundary, so the show is still inside the held segment, and sets timeline.holding = true and timeline.playing = false. The clock freezes there for as long as you like.
• The held segment's module keeps getting update() every frame, with ctx.p === 1, ctx.holding === true and ctx.held, the number of seconds waited so far. Use ctx.held as the clock for anything that should keep breathing while the show is frozen. (This is the choice made: the show stops at the end of the segment rather than looping inside it. Looping would send p back to 0, fire the cues again and break "every module enters once, exits once".)
• Space, →, the ▶ button, timeline.play() and timeline.next() all release it: the next segment starts at once and playback continues. timeline.release() does the same in code.
• A seek (scrubber, goto, ←) or a reset clears the hold, and a seek never stops at a hold: only playback does. The last segment never holds (the show ends there).
• The backstage monitor shows ⏸ HOLD; the desk and the prompter show a pulsing banner.

Segment modules#

A segment module is the code that performs one segment. Its id is the id of the segment (or the segment's module name):

const dusk = {
  id: 'dusk',
  enter(api, ctx)  { api.sky.caption = 'dusk'; },        // the segment began, or a seek landed inside it
  update(api, ctx) { /* every frame, playing or not */ },
  exit(api, ctx)   { api.sky.caption = ''; },            // it ended (after its linger) or a seek left it: give back what you took
  param(key, value, ctx) { return key === 'sun' ? 1 - ctx.p : value; },   // optional: bend a timeline param while this segment is current
  cue(name, ctx)   { /* optional: one of this segment's cues was just passed */ },
  reset(api)       { /* optional: R was pressed; forget everything */ },
};

enter, update and exit get the api object first (yours, see below). Every hook gets a ctx:

• Enter once, exit once. The module of a segment is entered when the show enters it and exited when it leaves it, however it got there. A seek into the middle of a segment calls enter with the right p, so the module must be able to start from the middle.
• linger. After the segment ends its module keeps getting update() (with p > 1 and phase: 'linger') until the linger is over, and only then exit(). The next segment's module is already running meanwhile.
• param(). While a segment (or its linger) is current its module may rewrite any timeline param. Return a number to replace the value, anything else leaves it. The performer's hand still wins: an override is applied after this layer.
• A module that throws is switched off, alone. Each hook runs inside its own try/catch. If one throws, the director calls that module's exit() once (with phase: 'failed'), prints one console.error, and never calls it again. The other modules and the show carry on. The desk shows a red warning with the module's name and the error; R (reset) switches it back on. A frame loop is not allowed to stop because the youngest code in the room failed.

The Director#

import { Director } from '@openav/score';
const director = new Director({ score, modules: [dawn, noon, dusk], api: { sky } });
// every frame, after the timeline advanced:
director.update(timeline.t, dt, { holding: timeline.holding, playing: timeline.playing });
timeline.layer = (key, value, t) => director.param(key, value, t);
timeline.onSeek((t, kind) => (kind === 'reset' ? director.reset() : director.seek(t)));

Use it with createShow#

const show = await createShow({
  world,
  score: { cuts, cut: 'full', modules: [dawn, noon, dusk],
           api: (show) => ({ sky: show.stage.active }),       // what the modules may touch (default: the whole show)
           midi: false },                                      // see "Score to MIDI"
  timeline: { automation: { haze: [[0, 0.1], [60, 0.5]] } },   // optional: written against the base cut
});

With a score, the timeline's scenes, total and start come from it (anything you pass as timeline.scenes or timeline.total is ignored), and the frame loop gains one step after the timeline advances, director.update(...). createShow also returns show.score, show.director and, for rehearsal and scripts, show.show (so window.openav.show):

The desk#

With a score the console adds:

• Scrubber. Segment blocks, a tick for every cue (click to jump) and a dashed end on a segment that holds.
• Director panel. A cut picker (choosing one reloads the page with ?cut=<id>, so nothing is half-switched), the current segment with a progress bar and the time left, the HOLD banner, the next segment and the countdown, the operator's acts, this segment's cues (ticked as they pass), and the red warning for modules that were switched off.
• Prompter (T). Fullscreen, large, high-contrast: the segment title, what you do (acts), seconds left, the next segment, a HOLD banner while the show waits, and a red line for switched-off modules. Click it or press T again to leave. It also works for plain timeline scenes that carry acts.

Keys#

Typing in a field, a held key (auto-repeat) and any key with ⌘, Ctrl or Alt are ignored. While the on-screen piano captures the keyboard (the keyboard box ticked in the piano bar) the letters belong to the piano: R, T and F do nothing, because T and F are piano keys, and an accidental R must not rewind a show you are playing. Space, the arrows and Esc always work, so you can play and run the show from one keyboard. The MIDI-controller key M follows the same rule.

Map the knobs in order#

The L2 Mapping panel has 🎛 map knobs in order. It arms learn for the first param, you turn one knob, it is bound and the next param is armed; skip, back and stop (or Esc) steer it. Learn only ever adds routes (Mapper.learn), so running it again puts a second knob on a param and keeps the first. The order is the declared order of the params, skipping hidden ones; pass mountConsole(el, app, { learnOrder: ['hue', 'size'] }) for your own. The logic is LearnWizard in @openav/mapping:

import { LearnWizard } from '@openav/mapping';
const wiz = new LearnWizard({ mapper, keys: () => params.schema.map((p) => p.key), onChange: (w) => draw(w) });
wiz.start(); wiz.current; wiz.skip(); wiz.back(); wiz.stop();

Score to MIDI#

Software that should follow the show (a DAW launching a clip per segment, a lighting desk jumping scenes) cannot read the score, but it can read MIDI. ScoreMidi (@openav/midi) turns segment changes and cues into notes and CCs. It is off unless you ask:

score: { cuts, midi: { channel: 15,
                       segment: { note: 36, cc: 20 },                       // segment i → note 36 + i, and CC 20 = i
                       cue: { note: 84, notes: { 'dusk.glow': 90 }, cc: 21 } } }  // a cue → note 84 (or its own note), CC 21 = its number

A reset never sends. Everything goes through the Midi engine's own noteOn/cc, so the MIDI OUT meter shows it and panic() silences it. In code: new ScoreMidi({ midi, channel, segment, cue }), then .segment({ index, cause }) and .cue({ name, n }).

Clocks: why there are none#

A show that keeps two time lines (a story time that old logic reads and a show time the audience lives) needs a way to say which engine follows which: weather that never stops, a city that pauses while an inserted segment plays. That is a clock per engine. This framework has one time line, the show's T, so there is nothing for a clock to choose between, and a segment module that wants to pause an engine simply stops advancing it (ctx.t is its own clock, ctx.held another). If a real show brings back the two-time-line case, a clocks table belongs in the cut next to segments. Until then it would be machinery with no user.

Without createShow#

import { Score, Director } from '@openav/score';
import { Timeline } from '@openav/timeline';

const score = new Score({ cuts, cut: new URLSearchParams(location.search).get('cut') || 'full' });
const timeline = new Timeline({ params, score });                 // scenes, total and start come from the score
const director = new Director({ score, modules, api: { world } });
timeline.layer = (key, value, t) => director.param(key, value, t);
timeline.onSeek((t, kind) => (kind === 'reset' ? director.reset() : director.seek(t)));
const desk = mountConsole(deskEl, { timeline, params, mapper, signals, stage, score, director });

const loop = new Loop((dt) => {
  timeline.advance(dt);
  director.update(timeline.t, dt, { holding: timeline.holding, playing: timeline.playing });
  mapper.update(dt);
  desk.render(stage.frame(dt, timeline.state()));
});

Rehearsal checklist#

• show.show.table() reads like the running order you meant
• play it once start to finish at timeline.rate = 4; every cue fired, no module switched off
• open ?cut= for every version and check the desk shows the right total
• press Space on every hold, press → on every hold, press R during a segment
• break a module on purpose (throw in update) and watch the show carry on
• the prompter (T) reads from the position you will stand in

MIDI controllers & embedding#

10Source: docs/controllers.md

open-audiovisual models MIDI controllers as data: one JSON file per device, with the faceplate layout, every control's exact MIDI message, LED feedback, how to recognise its port, and the document each number came from. The same file drives an on-screen controller you can play, follows the real device when you plug it in, and can be dropped into any web page. Every device is on the controllers page.

Play one in a page#

<script type="module" src="https://openaudiovisual.com/packages/midi/element.js"></script>
<oav-controller profile="akai-lpd8" readout></oav-controller>

That is the whole integration. The element works without Web MIDI (Safari, iPad): it plays and fires events; only hardware and MIDI out report that they are unavailable. It never asks for MIDI permission unless you turn on hardware, midi-out or learn. Safari has none because WebKit has not implemented it (bug 107250, still open).

Listen to it#

const el = document.querySelector('oav-controller');
el.addEventListener('control', (e) => {
  const { id, value, signal, message, source } = e.detail;
  // 'k1'  0..1  'midi/lpd8/k1'  { type: 'cc', ch: 1, cc: 70, value: 53 }  'ui' | 'hardware'
});
await el.ready;
el.set('k1', 0.5);           // move a control as a hand would
el.ingest([0xb0, 70, 64]);   // MIDI from anywhere: the faceplate follows

Other events: noteon, noteoff, connect, disconnect, status, profilechange, ready, error. All bubble and cross the shadow root. Every field of every payload is in packages/midi/README.md.

Without the element#

import { createController } from 'https://openaudiovisual.com/packages/midi/index.js';

const ctl = createController('akai-lpd8', { signals });   // signals: publish into a show
ctl.mount(document.querySelector('#pads'));                // optional
ctl.on('control', (e) => console.log(e.signal, e.value));

Inside a show, do not read midi/lpd8/k1 in update(): route it to a param with controllerRoutes(profile, { k1: 'hue' }) (rule 1 in AGENTS.md). Example 09 does this for six devices at once.

In an iframe#

For places that only take an iframe:

<iframe src="https://openaudiovisual.com/embed/controller/?profile=akai-lpd8&readout&frame=pads"
        width="720" height="220" style="border:0" allow="midi" title="AKAI LPD8"></iframe>

The frame posts { source: 'openav', type, frame, detail } to its parent for every event, and accepts { target: 'openav', type: 'set' | 'ingest' | 'press' | 'release' | 'reset' | 'profile' | 'layout' | 'get', … }. Add origin=https://your.site to talk to your page only.

Play your DAW with it#

1. macOS: Audio MIDI Setup → Window → Show MIDI Studio → IAC Driver → “Device is online”. Windows 11: the built-in Windows MIDI Services (released 2026-02-17, rolled out in stages to 24H2 and 25H2) share MIDI ports between apps and include a loopback, and Microsoft's announcement says web pages using Web MIDI can use it (Windows blog). We have not tested it with open-audiovisual. Older Windows: install loopMIDI and add a port; that still works.
2. <oav-controller profile="akai-lpd8" midi-out="IAC"> (on Windows, give the name of the port you made or the one Windows shows).
3. In Ableton Live, turn on Track and Remote for that input; in TouchDesigner, a MIDI In CHOP on it; in Resolume, Preferences → MIDI.

Hands on the screen are sent; your real hardware is not forwarded (the DAW already hears it). The engine never listens to IAC inputs, and the element mutes an input named like its output, so nothing loops.

Telemetry#

So the project can see where its controllers and shows end up, the packages send one anonymous hit per page load to Google Analytics 4 (the openaudiovisual.com property, G-1YG2JHK2WT):

oav_host is the origin of the page it runs on (for the iframe, the page that holds it), never the path, query or title. Nothing anyone plays is sent, no cookies are set, and the client id is random for every page load, so visitors can't be followed. No gtag.js is loaded: a page with its own Google Analytics keeps its dataLayer untouched. Nothing is sent from localhost, LAN addresses, file:// or openaudiovisual.com itself, or when the browser sends Do Not Track / Global Privacy Control.

Turn it off: <oav-controller telemetry="off"> · createController(p, { telemetry: false }) · createShow({ telemetry: false }) · the iframe's ?telemetry=0 · or globalThis.OPENAV_TELEMETRY = false before anything loads. Source: packages/midi/telemetry.js.

Add a controller#

1. Copy the closest file in packages/midi/profiles/, or a generic layout.
2. Write each control's type and message from the manual's MIDI implementation chart.
3. Cite every source; mark anything unconfirmed "verified": false with a note.
4. validateProfile(profile) lists every problem; npm test checks every shipped profile.
5. Try it with <oav-controller profile="./my-device.json">, then register it in profiles/index.js and open a pull request.

No manual: choose a generic layout, press Learn, move your controls in order, and keep controller.exportMapping() as the start of a profile.

p5.js in one line#

11Source: docs/p5.md

Any p5.js sketch (OpenProcessing, the p5 editor, your own page) can be played from a MIDI controller, from a controller drawn on screen, or from a phone, by adding one line at the top:

document.head.prepend(Object.assign(document.createElement('script'), { src: 'https://openaudiovisual.com/p5/oav-p5.js' }));

On OpenProcessing you can instead paste the URL into Sketch → Libraries → + (custom library). Both work in p5 1.x and 2.x global mode, and nothing has to be called in setup(). A sketch that carries this line, playable right now: example 15.

Read it like the mouse#

function draw() {
  background(knob1 * 255);                 // knob 1, 0..1, like mouseX
  circle(width / 2, height / 2, 50 + knob2 * 300);
}

function notePressed(note, velocity) {     // like keyPressed(): any key, or a pad
  circle(random(width), random(height), velocity * 100);
}

Knob 1 is knob 1 on whichever controller played it: the controller profiles say which MIDI message each knob sends (MiniLab 3's knob 1 is CC 74, LPD8's is CC 70). On a controller OAV has no profile for, the first knob you turn becomes knob1, the next knob2. A sketch that already has its own note, velocity or knob1 keeps it.

Note names#

Wherever a note number goes, a note name goes too (0.8):

function draw() {
  if (noteIsDown('C4')) circle(width / 2, height / 2, 100);   // the same as noteIsDown(60)
}

function mousePressed() {
  midi.play('C#4', 0.8);                    // sends note 61, as a key would
}

A name is a letter A to G (either case), an optional # or b (♯ and ♭ too), and the octave number: C4, c#4, Db3, F♯2, C-1 (0) up to G9 (127). C4 is 60, the scientific pitch notation the course modules use; some software (Ableton, Yamaha) calls 60 "C3", so a name copied from there is one octave off. These are also the names Strudel writes (c4 e4 g4, # for sharp, b for flat). Anything else, such as H4 or C-2, stops with a one-line card saying how to write it. notePressed(note, velocity) still hands you the number: noteName(note) writes it back as a name.

With OP Configurator#

OP Configurator 3000 (OPC, built into OpenProcessing) names a variable and its range. With this line on the page the controller is wired to those variables in order, with no mapping step:

OPC.slider('radius', 60, 10, 300);   // knob 1
OPC.slider('speed', 1, 0, 5, 0.1);   // knob 2
OPC.toggle('trails', true);          // pad 1 flips it

Knobs drive the sliders in the order they are declared, the faders continue after the knobs, and pads drive toggles and buttons. OPC's panel moves with the knob, and parameterChanged() fires as usual. OpenProcessing's OPC panel can also learn hardware by itself (its MIDI switch); once that mapping is seen working, hardware is left to it and this line keeps wiring the on-screen controller and the phone, which OpenProcessing cannot see. Avoid OPC names that are also p5 functions (hue, fill): p5 1.x clashes with them.

One knob, one job. If the sketch also reads knob1 itself, knob 1 is the sketch's: the first OPC slider no longer follows it (drag it on OPC's panel), the other knobs stay wired, and a hint says so once. The first line of the 🎹 panel is the wiring, for example knob1 → knob1 (sketch; radius by its slider) · knob2 → speed (OPC).

The microphone#

The same line reads the microphone, and the sketch reads it the same way: as variables.

function draw() {
  background(230, 40, 10);
  fill(200 + bass * 160, 70, 100);                      // bass: the low end, 0..1
  circle(width / 2, height / 2, 50 + micLevel * 300);   // micLevel: how loud, 0..1
}

function beatDetected(level) {                          // like keyPressed(): a beat just landed
  // flash, spawn, jump to the next scene
}

It is off until the sketch asks. Nothing is requested when the page loads. The first time the sketch reads micLevel, bass, mid, treble or beat, or calls startMic(), the browser asks for the microphone; until the answer arrives every value is 0. A sketch that defines beatDetected() and reads nothing starts it on the first click or key. In the 🎹 panel the 🎤 button turns it on and off by hand and shows the level, and it says so when the browser blocked the microphone (click it again after allowing it from the address bar).

• Click first, if the browser wants one. Browsers keep audio quiet until the page has been clicked. This file asks as soon as draw() reads a value and starts the microphone again on the first click or key if the browser held it back, so no userStartAudio() is needed.
• Next to p5.sound. The analysis is @openav/audio (an AnalyserNode), loaded from next to this file the first time it is needed, with its own AudioContext. p5.sound (p5.AudioIn, p5.FFT) can run on the same page; the two microphones do not share anything, and none of these names is a p5 or p5.sound name. It works in p5 1.x and 2.x global mode.
• How loud is loud. micLevel is a loudness in decibels, measured from the room: its bottom follows the room's own noise (it settles in a second or two, and rises at most 1.5 dB a second, so a voice never becomes the new silence), and −10 dBFS, shouting into a laptop, is 1. So 50 + micLevel * 300 visibly grows when someone talks. A quiet or far microphone: micSensitivity(3) (+9.5 dB on micLevel, ×3 on the bands, still capped at 1). The raw numbers are OAVp5.mic.rms (0..1), OAVp5.mic.db and OAVp5.mic.floor (the room, in dBFS). OAVp5.mic.on('beat', fn) is the same as defining beatDetected, and returns a function that stops it.
• Your own names win. A sketch that already has a bass or mid (a let, a function, a library) keeps it, and this file never starts the microphone for it.
• Off switch. ?mic=click (or window.OAV_P5 = { mic: 'click' }) means only 🎤 and startMic() turn it on; mic=off means nothing can start it (the names stay 0). https or localhost is required, as for any microphone page.
• A sketch with a microphone, ready to open: p5/examples/mic/.

Sound: what is played is heard#

function setup() {
  createCanvas(windowWidth, windowHeight);
  startSound('piano');          // every key is heard: a keyboard, the 🎹 on screen, a phone
}

The instruments are those of @openav/sound: piano (the Salamander grand; its 2 MB of samples load the first time, and the electric piano plays meanwhile, or instead if they cannot load), epiano, organ, strings, choir, harp, marimba, vibraphone, xylophone, glockenspiel, music-box and the synth/… presets. In the 🎹 panel, 🔊 turns sound on and off and shows a picker with every instrument and the loading progress.

• Off until asked. Nothing loads before startSound() or 🔊; then @openav/sound from next to this file and Tone.js 15 from jsDelivr (the option tone=<url> loads it from elsewhere).
• The first click. Browsers play audio only after the page has been clicked, and a key on a MIDI keyboard does not count. A sketch that calls startSound() in setup() is heard from the first click or key; until then the 🔊 on the 🎹 button is yellow.
• Pads are drums. Notes on MIDI channel 10 (MiniLab 3 and LPD8 pads, the General MIDI drum channel) play the drum kit, not the piano. The sustain pedal holds notes.
• No code at all. ?oav_sound=piano on the page (any instrument id; on OpenProcessing ?sketch_oav_sound=piano) makes any sketch heard from the first click; sound=off keeps it silent.
• A sketch that plays and draws, ready to open: p5/examples/keys/.

Drum machine#

function setup() {
  startDrums('breakbeat', 110);   // a 16-step grid on screen; click a cell to change the beat
}

function kickDetected(level) {    // the microphone's own callback
  // swell, flash, spawn
}

function drumHit(lane, velocity) {   // every hit: 'kick', 'snare', 'hat' or 'clap'
}

A microphone sketch moves over unchanged. Each hit calls the same functions the microphone does, so a sketch written for the microphone reacts to the machine as it is. Once a sketch has started the machine, defining kickDetected() no longer opens the microphone on the first click (a sketch that reads micLevel still asks, and then both call it).

• Patterns. 'four on floor', breakbeat, 'half time', latin, sparse, your own startDrums({ kick: 'x...x...x...x...', snare: '....x.......x...', hat: 'x.x.x.x.x.x.x.x.' }) (a lane left out is silent), or one line of rhythm (below).
• The card. The lanes are named in the page's language (大鼓 小鼓 鈸 拍手 in a Chinese browser), the bar it is in shows beside the tempo, and the step that sounds is a yellow column (13:1 against the dark card, readable on a projector).
• Where the grid sits. At the top centre: on OpenProcessing between the 🎹 and OPC's icon and clear of the console, and below the 🎹 panel while that is open. – folds it, × stops the machine, and 🥁 in the 🎹 panel opens and closes it with no code at all. 🥁 brings the grid up stopped: tick its play box to hear it (a sketch's own startDrums() plays at once).
• The kit. @openav/drums' synthesized kit (no samples), heard from the first click like sound; the callbacks run from the start either way.

On time, however heavy the sketch. The hits are handed to the kit ahead of time, on its audio clock: a heartbeat in a Web Worker wakes every 25 ms and schedules what falls in the next 0.1 s at its exact time. Measured on the kit's own output with a draw() that keeps the page busy 30 ms a frame, the kick-to-kick interval varies by 0.000 ms (0.5, which played each hit in the frame it was due, varied by 15 ms). The callbacks still come in a frame: the one in which the hit is heard (its time plus the output latency the browser reports), right before draw(), in the same order as before.

• A tab in the background keeps playing and counting: drumBar follows the clock on the wall and the cues fire, with one barStarted a bar. The hits and stepStarted are not told while nobody can see them, and nothing is told twice when the tab comes back, so a sketch that adds a particle per hit does not burst.
• A draw() longer than 0.1 s makes the machine look 0.3 s ahead from then on (🩺 says so); a click on a cell is then heard up to 0.3 s later.
• A Bluetooth speaker is late by 0.15–0.25 s; where the browser reports it the callbacks wait for it, and ?oav_drumlatency=200 (milliseconds) sets it by hand.
• The safety net: ?oav_drums=frame (on OpenProcessing ?sketch_oav_drums=frame) brings back 0.5's machine, moved by the frames.

One line of rhythm#

function setup() {
  startDrums("bd*4, [~ sd]*2, hh*8", 110);   // a kick on every beat, a snare on 2 and 4, eight hats
}

Any text that is not a pattern name is a line of rhythm; names, grids and numbers work exactly as before. The line shows over the grid, and the word that sounds lights up and fades.

The whole list is in @openav/pattern, loaded from next to oav-p5.js with the drums. Drums the grid has no lane for (crash, ride, toms, open hat) get a row of their own; a hit off the sixteen steps (a triplet) is drawn where it falls inside its step. A mistake in startDrums("…") shows a card in the page's language with the character it is at, why, and how to fix it, and the machine plays 'four on floor' meanwhile; 'x...x...' on its own (one lane of the grid) says to put it inside { kick: … }.

A line copied from strudel.cc works too: startDrums('s("bd*4, [~ sd]*2").fast(2)') plays [bd*4, [~ sd]*2]*2. s("…"), sound("…"), with ", ' or ` quotes, several of them ($: s(…)) as layers, .fast(n) and .slow(n); setcpm(110/4) or setcps(…) sets the tempo when startDrums() is given none. .bank("…") is left out (OAV plays its own synthesized kit) and anything else in the chain (.jux(rev), .every(…), .gain(…)) gets a card that says OAV does not do it yet and points at Open in Strudel; the rest plays.

The way of writing one line of rhythm comes from TidalCycles / Strudel (strudel.cc); OAV implements it independently from their public documentation.

The cells and the line edit each other#

Since 0.7 the line over the grid is always there (a preset shows the line its grid is: 'four on floor' is bd*4, [~ sd]*2, hh*8) and both ways work:

• Click a cell: the line is rewritten the way a person would write it, and the machine plays the new line at once. Steady hits become bd*4 / hh*8, one hit per group [~ sd]*2, a Euclidean rhythm bd(3,8) (with its rotation: bd(3,8,2) starts two steps later), anything else beat by beat: bd [~ ~ ~ bd] bd*2 ~. Lines are joined with , . When the line holds more than sixteen steps can (triplets, < >, /2, ?, bd:2), the first click shows a card saying what would go, and the next click rewrites it.
• Type in the line: click it, change it; Enter, or a pause of 400 ms, plays it and the cells follow. A mistake marks the word in red, a card says what is wrong, and the last line that worked keeps playing (not 'four on floor'). Escape drops what is not played yet. Typed keys never reach the sketch's keyPressed().
• ↶ on the card (or ⌘Z / Ctrl+Z after touching the card) goes back one line; OAVp5.drums.history lists them.
• Copy puts startDrums("…", 110) on the clipboard, ready to paste into setup().
• Open in Strudel opens the same line at strudel.cc in a new tab, as s("…") with setcpm(bpm/4) (one OAV bar is one Strudel cycle of four beats; strudel.cc's docs: setcpm(bpm/bpc)). OAVp5.drums.strudelUrl is the link.

function rhythmChanged(src, how) {   // how: 'code' · 'cell' · 'type' · 'undo' · 'menu'
  line = src;                        // e.g. draw the line on the canvas, so a recording shows it
}

OAVp5.drums.src is the line that plays, OAVp5.drums.edit("…") plays a line as if typed, OAVp5.drums.undo() goes back. Keys that swap the rhythm, step back and open Strudel: p5/examples/rhythm-lab/.

The rhythm on the canvas#

The card is part of the page, not the canvas, so a recording never shows it. drawRhythm() draws the rhythm on the canvas:

function draw() {
  background(0);
  drawRhythm(20, 20, 360, 90, 'grid');          // the sixteen steps, a playhead
  drawRhythm(20, 130, 300, 300, 'circle');      // a ring of 16 per drum, a hand turned by drumPhase
  drawRhythm(340, 130, 300, 300, 'spiral', { bars: 4, colors: { kick: '#ff7b54' } });   // the last 4 bars, wound outwards
}

drawRhythm(x, y, w, h, style, options): style is 'grid', 'circle' or 'spiral'; a hit lights up the moment it is heard (the clock drumPhase reads) and a triplet sits where it falls. Options: colors ({ kick: '#f66', … }, any p5 color), color (one for all), off (the empty steps), playhead (a color, or false), bg, bars (spiral). Drawn inside push() / pop() with p5's own shapes, in 2D and WEBGL, p5 1.x and 2.x; in WEBGL (0, 0) is the middle of the canvas: drawRhythm(-width / 2, -height / 2, width, height, 'circle'). With no box it draws a strip along the bottom ('grid') or a square in the middle. A sketch with all three: p5/examples/rhythm-visual/.

Timeline: bars and cues#

The drum machine is also the clock: sixteen steps a bar, counted from startDrums(). Declare what happens at which bar:

function setup() {
  startDrums('breakbeat', 110);
  cue(1, 'calm', 12);             // bar 1, and again every 12 bars
  cue(5, 'bloom', 12);
  cue(9, 'storm', 12);
}

function cueFired(name, bar) {    // or read cueName in draw()
  look = SCENES[name];
}

The same cue() twice adds nothing, so it may also sit in draw(). On one step the cues fire in the order they were declared, then barStarted, stepStarted and that step's hits. A silent clock: startDrums({}). The whole of it, ready to open: p5/examples/drums/; a line of rhythm with drumPhase and beatPhase: p5/examples/rhythm/; the rhythm drawn on the canvas: p5/examples/rhythm-visual/.

Values over time: lag, env, speed, spring, trail#

A knob or micLevel jumps; what looks good is how it moves between the jumps. TouchDesigner does this with CHOPs; the same operators are here, one call each in draw(), nothing to declare, and seconds are seconds (the same curve at 30 and at 144 fps):

circle(width / 2, height / 2, 50 + lag(micLevel, 0.03, 0.4) * 300);   // a VU meter: opens at once, closes slowly
scale(1 + env('kick', 0.01, 0.3));                                     // the drum machine: every kick swells it, gone in 0.3 s
rotate(spring(knob1, 100, 6) * TWO_PI);                                // flick a knob: it overshoots and settles

Seconds, the same as in TouchDesigner. A lag's seconds mean what a Lag CHOP's do: about the time the value takes to follow 90% of a jump, so lag(x, 0.1) here moves like a Lag CHOP set to 0.1 (Lag CHOP). Underneath it is a one-pole lag with a time constant of seconds ÷ 2.3 (ln 10); @openav/mapping's smooth keeps its own 63% time constant. Attack and decay are lengths in both.

Nothing to declare. The first lag() in draw() keeps the first lag's memory, the second the second, every frame; spring(), speed(), trail() … are counted each on their own. Put one inside an if, or in a loop whose length changes, and the order slips: a card says so, and from then on oav-p5 matches each call by the line it is written on. The sure way is a name, a string at the end: lag(p.size, 0.1, 0.5, 'p' + p.id). A named one is the same wherever it is called, draw(), setup() or mousePressed(); an unnamed one outside draw() has nothing to remember by and returns its input (speed 0, trail []), and a card says that too. env(), trigger() and count('kick') are about events and work anywhere. A sketch with its own speed or count keeps it; OAVp5.chop.speed(…) is the same function under another name.

Seeing the values. scope(value, 'label') returns value, so it can wrap an expression: circle(x, y, scope(lag(micLevel, 0.03, 0.4), 'lag') * 300). Each label is one line in a small oscilloscope across the page from the 🎹: the last 4 seconds and the value now; scope(micLevel, 'mic') beside it shows what the lag did. It lives on the page, not on the canvas: never in a recording or a 📷, never in the way of a click; hints=off or scope=off hides it. To draw it in the work: stroke(255); drawTrail(trail(micLevel, 3), 0, height - 100, width, 100) (or drawTrail(micLevel, …), which keeps its own 2 seconds): 0 at the bottom of the box, 1 at the top (lo, hi after h), the sketch's stroke() (white under noStroke()).

The math is @openav/chop, loaded from next to this file the first time one of these is called; until it arrives (a few frames) each returns its input as it is. What each parameter means follows Derivative's TouchDesigner documentation; the code is OAV's own. Raw against lagged, side by side, with the scope: p5/examples/lag/. env, spring, speed, count and slope driving a disc that a bass drum opens, with startDrums() as the only input: p5/examples/chop/.

Recording: a video with its sound#

function keyPressed() {
  if (key === 'r') recordingIsOn ? stopRecording() : startRecording();   // startRecording(10): ten seconds, then it stops
}

function recordingSaved(info) {   // info.name, info.seconds, info.bytes, info.format ('mp4' or 'webm')
}

In the 🎹 panel, ⏺ starts and stops a take with no code; while it runs, the 🎹 button shows a red dot and the time (a click on the dot stops it). 📷 saves the canvas as a PNG.

• What is in the video. The p5 canvas at its own pixel size (a retina screen doubles it), 30 frames a second, and every sound this file makes: startSound(), the drum machine and the pads, the microphone while it is on, and p5.sound 1.x, mixed into one track while the speakers play as before. The 🎹, the drum grid, the cards and OPC's panel are not on the canvas, so they are not in the video.
• The file. MP4 (H.264 + AAC), which Instagram, YouTube and every editor take, wherever the browser can write it: Chrome 126 and later, Edge, Safari. Elsewhere (Firefox, an older Chrome) WebM, and a card says Instagram will not take it. It is named oav-20261015-183005.mp4; on a computer it lands in Downloads, on a phone a Share button on the card saves it to the photo library or sends it to Instagram.
• Checked after every take. A card says what was recorded (format, length, size, with or without sound) and, when something is wrong, what and why: nothing reached the recording although a sound was on (no click on the page yet, nothing played), WebM, an almost empty file, a black video of a canvas that was not black (Safari recording a WebGL canvas, WebKit 229611), an image from another site on the canvas (no CORS), no recorder in this browser.
• The microphone is in the take while it is on: wear headphones, or the speakers are recorded twice. record=nomic leaves it out.
• p5.sound. p5.sound 1.x (the one for p5 1.x) is recorded. The p5.sound made for p5 2.x connects each sound straight to the speakers, with no output a recording can tap: it is not in the take, and a card says so. Switch the sketch to p5 1.11, or use startSound() and startDrums(). Sound the sketch makes with its own Tone.js or Web Audio code is not recorded either.
• Stills. 📷 (or OAVp5.record.shot()) saves the canvas alone as oav-<date>-<time>.png, without the 🎹 or the cards; from code, p5's own saveCanvas('name', 'png') does the same.
• Options. record=nomic (no microphone) or record=off (no ⏺ or 📷; startRecording() does nothing), recfps=60, reclimit=600 (seconds), title=my-piece (in the file name); for one take, startRecording({ seconds, fps, mic: false, name, canvas, bitrate }).
• The whole of it, ready to open: p5/examples/record/.

Three inputs#

All three arrive as the same MIDI bytes, so the sketch cannot tell them apart (midi.last.source can). Safari and iPad have no Web MIDI; the on-screen controller and the phone still work there. A sketch that uses Web MIDI itself (navigator.requestMIDIAccess, WebMidi.js) also hears the screen and the phone, as one more input called OAV Virtual MIDI.

Other programs are a fourth input (0.7). What a program plays into OAV Virtual MIDI's output — Strudel's .midi('OAV Virtual MIDI') — reaches the sketch at the moment it was stamped with (midi.last.source is virtual-out); an external MIDI clock moves drumPhase, drumBar and a drum machine; and link=<room> brings another tab's or another computer's programs into this sketch. All of it: With Strudel.

When something goes wrong#

Nobody watches OpenProcessing's console while playing, so problems are said on the page: a small card beside the 🎹, never over the middle of the canvas, in the browser's language, once per page, with a × to close it. Each card says what happened, why, and what to press, and the console gets the same sentence.

• The microphone: refused (the lock left of the address → allow → click the canvas), refused by the computer's settings, by an embed without allow="microphone", missing, held by another app (Zoom, OBS), an http: page, or the browser still asking.
• MIDI: no Web MIDI (Safari, iPad: use the 🎹 or a phone), or the permission refused. Only for a sketch that plays from MIDI; a sketch that uses none of this never sees a card.
• Your own code: an error inside notePressed(), kickDetected(), drumHit() and the other callbacks shows the message with the file and line, and the sketch keeps running. A sketch that declares its own note or knob1 (let, var, function) keeps it, and the card says what to read instead (midi.last.note, knob(1)).
• Sound and drums: waiting for the first click, the piano still loading (the electric piano meanwhile), an instrument that failed (the fallback plays), Tone.js unreachable, the drum machine playing silently until a click.
• Recording: started (what it hears, how to stop), saved (format, length, size, sound), a take with no sound in it, WebM (not for Instagram), a black take, an almost empty file, a canvas with another site's image, no recorder in this browser.

?oav_hints=off (or window.OAV_P5 = { hints: 'off' }) keeps them in the console only; OAVp5.hints lists what was said.

🩺 Check-up. 🩺 in the 🎹 panel (or OAVp5.check()) shows the version, the page, the browser, the microphone, MIDI, sound, drums, recording (MP4 or WebM here, the last take), the wiring, the sketch's own names and the hints so far, as one card to screenshot, with a copy button and a link to the full device check.

In Hydra#

Hydra has a loader of its own; the same file works there, and Hydra's arrow functions read the variables each frame:

await loadScript('https://openaudiovisual.com/p5/oav-p5.js')
osc(10, 0.1, () => knob1 * 2).out()                     // knob 1 sets the colour offset

osc(10, 0.1, () => 1 + micLevel * 3).out()              // the microphone, asked for when this line first runs

note, velocity, knob1 … knob8, fader1 …, pad1 …, drumStep and the 🎹 on-screen controller are there the same way. Hydra is not p5, so there are no notePressed() callbacks to define; subscribe instead: midi.on('notePressed', (note, velocity) => { … }) or OAVp5.drums.on('hit', (lane, velocity) => { … }) (each returns a function that unsubscribes). Checked against hydra-synth 1.4.0, the library the editor runs (p5/test/hydra-smoke.cjs).

Notes#

• Why prepend. p5's friendly-error checker reads the last <script> on the page as your sketch. Put first, this file is never mistaken for your code.
• Why one asynchronous line is enough. p5 starts global mode on the window's load event, and a script inserted while the page is loading holds load back until it has run, so knob1 already exists in setup().
• When the line does not load (offline, a typo in the URL), code that reads knob1 stops with knob1 is not defined.
• Embedding a sketch elsewhere. OpenProcessing's own pages allow MIDI and the microphone in the sketch frame. The embed code it gives you does not: add allow="midi" to that <iframe> if the embedded sketch should hear a hardware controller, and allow="microphone" if it should hear the microphone.
• Options, MIDI-level access and tests: p5/README.md. Examples: p5/examples/particles/, p5/examples/opc/, p5/examples/mic/, p5/examples/keys/, p5/examples/drums/, p5/examples/rhythm/, p5/examples/rhythm-visual/, p5/examples/rhythm-lab/, p5/examples/record/, p5/examples/lag/, p5/examples/chop/, p5/examples/with-strudel/ and p5/examples/note-names/.

With Strudel#

12Source: docs/strudel.md

Strudel is a live-coding environment for music in the browser, a JavaScript version of TidalCycles. OAV is a separate program. The two run side by side the way two programs on a stage do: Strudel plays, and OAV brings what Strudel does not have — knobs on screen and on a phone, a microphone that is analysed, Web MIDI where the browser has none — and draws what Strudel plays in a p5.js sketch, on the same page or in another tab or on another computer. They meet through MIDI (OAV's virtual port, "OAV Virtual MIDI") and through the page's own variables (knob1, micLevel, drumPhase). OAV never loads or calls any of Strudel's code.

Everything on this page was run on strudel.cc as a user runs it (its editor, Ctrl+Enter, its documented functions): p5/test/strudel-smoke.cjs does it again.

Load OAV in Strudel#

Strudel → settings → prebake, add one line at the end, save:

await import('https://openaudiovisual.com/p5/oav-p5.js')

• Single quotes. Strudel reads text in double quotes (and backticks) as mini-notation: a URL in double quotes is a [mini] parse error.
• Why the prebake. It runs before your code, also after a reload (about 2.5 s after the editor appears), so OAV is on the page before anything opens MIDI. If Strudel's midin() ran before OAV loaded, midin('OAV Virtual MIDI') gets nothing and says nothing: reload. 🩺 then shows "no program has asked OAV Virtual MIDI for MIDI".
• The line can also be the first line of your code. Options go on the line before it: window.OAV_P5 = { link: 'my-room' }.
• Strudel's share button puts the prebake into the link (with "share with patterns" ticked): whoever opens it gets OAV too.

On strudel.cc OAV changes its manners. It does not ask for MIDI when it loads (hardware=click: Strudel's midin() asks). The 🎹 panel is a box in the corner, a third of the window high, so the code stays in view. One card says which names Strudel already has — note, velocity, beat are Strudel's functions, OAV leaves them alone — and what to read instead: OAVp5.last.note, OAVp5.last.value, OAVp5.mic.on('beat', fn). knob1, micLevel, drumPhase and the rest are free names there.

Recipe 1: knobs, a phone and the microphone as Strudel's parameters#

Strudel's signal() turns a function into a parameter; OAV's names are plain values a function can return.

s("bd*4").gain(signal(() => knob1))
s("hh*8").lpf(signal(() => 300 + micLevel * 8000))

• knob1 moves with knob 1 of the 🎹 on screen, of a MIDI controller, or of a phone (🎹 → 📱, scan the code). knob(2), fader1, pad1 are there too. Measured: knob 1 at 0, the kick is silent; at 127, it is heard (RMS 0.58).
• Strudel reads the value when it schedules, 0.2–0.3 s before the sound: a knob is heard that much later.
• micLevel, bass, mid, treble are OAV's analysis of the microphone (Strudel has none); reading one asks for the microphone. Wear headphones: the microphone hears the speakers, and gain(signal(() => micLevel)) feeds back.
• Strudel's own MIDI input works on OAV's port too, also in Safari (no Web MIDI there; OAV's port is virtual):

const cc = await midin('OAV Virtual MIDI')
s("bd*4").gain(cc(74))

Knob 1 of the on-screen MiniLab 3 is CC 74. midikeys('OAV Virtual MIDI') makes the 🎹 keys and the phone Strudel's keyboard.

Recipe 2: Strudel plays, a p5 sketch draws#

On the same page. Send to OAV's port:

note("c3 [e3 g3] <a3 b3> c4").midi('OAV Virtual MIDI')

Strudel hands each note over 0.21–0.32 s early with the moment it should sound; OAV waits for that moment and then calls notePressed(note, velocity) (in Strudel, with no sketch: OAVp5.on('notePressed', fn)), moves knob1 for a CC, plays a pad. Measured on strudel.cc: each note is heard within 5 ms of its timestamp (10 notes, at most 4.5 ms late; p5/test/strudel-smoke.cjs), and drumPhase follows Strudel's clock to 0.1 ms. What arrives on the port is never handed back to OAV's input, so midikeys('OAV Virtual MIDI') played into .midi('OAV Virtual MIDI') is one echo, not a loop.

Strudel's clock. MIDI counts 24 clocks a beat, so a cycle of one 4-beat bar sends 96:

setcps(128 / 60 / 4)
$: s("bd*4, [~ sd]*2, hh*8")
$: midicmd("clock*96, <start stop>/8").midi('OAV Virtual MIDI')

drumPhase, beatPhase, drumBar, drumStep and drumBpm follow it. A drum machine started with startDrums() plays at Strudel's tempo, on Strudel's downbeat, and starts and stops with it (the card says "following an external clock · 128 BPM"). Measured at 128 bpm: drumPhase within 1.2 ms of Strudel's bar (median 0.04 ms), every kick within 6 ms of Strudel's beat (three runs); Strudel's start, which arrives three times with one timestamp, counts once. Without a clock nothing changes; clock=off ignores one.

In another tab, or on another computer. Give both pages the same room. In Strudel's prebake:

window.OAV_P5 = { link: 'my-room' }
await import('https://openaudiovisual.com/p5/oav-p5.js')

and in the p5 sketch (OpenProcessing: before the one line; or ?sketch_oav_link=my-room on the page's address):

window.OAV_P5 = { link: 'my-room' };
document.head.prepend(Object.assign(document.createElement('script'), { src: 'https://openaudiovisual.com/p5/oav-p5.js' }));

Everything Strudel sends to OAV's port — notes, CCs, the clock — reaches every other OAV page in the room, through the same relay the phones use. 🎹 grows a 🔗: green when another page is there; the panel says the room, the pages and the trip. The receiving page plays each note when Strudel plays it, not when it arrives: it lines the two clocks up from the fastest trips it has seen and the round trips both pages measure (packages/relay/link.js explains how), and Strudel's quarter-second head start covers the trip. Measured with two tabs on one Mac over wss://remote.cheyuwu.com (one way ≈ 61 ms): notes within −3 … +7 ms of Strudel's timestamps (two runs). Across two computers add the difference between the two halves of a round trip (usually a few ms). Keep the default relay (wss://): Chrome does not let strudel.cc reach a relay on your own machine or network (ws://127.0.0.1… is blocked by its local-network checks). On a network whose jitter is longer than Strudel's head start, linkdelay= (ms) buffers more. A page never sends on what it received, and a program that plays a linked note back out is caught as an echo. The room is the phones' room too: one QR code plays every page in it.

The sketch side, ready to run: p5/examples/with-strudel/ (?room=… on its address).

Recipe 3: Strudel, Hydra and OAV on one page#

With OAV in the prebake:

await initHydra()
osc(10, 0.1, () => knob1 * 2).rotate(() => drumPhase * 6.28).color(1, () => micLevel, 0.6).out()
setcps(120 / 60 / 4)
$: s("bd*4, [~ sd]*2, hh*8").gain(signal(() => 0.3 + knob1))
$: midicmd("clock*96").midi('OAV Virtual MIDI')

Hydra turns once a bar on Strudel's clock (through drumPhase), knob 1 drives Hydra and Strudel at once, and the microphone colours the picture.

Options for Strudel#

Set before the line (window.OAV_P5 = { … }), or ?oav_key=value on a p5 page's address.

Credits and licences#

Strudel is © its contributors (begun by Felix Roos and Alex McLean), free software under the GNU Affero General Public License v3: strudel.cc, source at codeberg.org/uzu/strudel. TidalCycles, whose way of writing patterns Strudel carries to the browser, is Alex McLean's. Music you make with Strudel is yours (Strudel's FAQ).

OAV is MIT-licensed and a separate program. It contains no Strudel code, does not load @strudel/* at run time and calls none of Strudel's functions: it only looks at the page's address and at which names the page already has, to choose its manners. The two meet the way two programs on a stage meet, over MIDI. OAV's own rhythm strings (startDrums("bd*4, hh*8"), @openav/pattern) are an independent implementation of the TidalCycles / Strudel way of writing rhythm, from their public documentation. Please do not ask Strudel's community for help with OAV: OAV's issues are on GitHub.

Phones, relay & surfaces#

13Source: docs/remote.md

Phones and iPads can be controllers: their motion sensors, a piano, and a TouchOSC-style control panel that grows from the world's params. Three packages do it, each usable alone:

To the mapper, a phone is one more input: it publishes signals, and routes connect them to params.

Set it up#

1. Run node serve.js on the show machine. It serves the pages and hosts the relay at /relay; nothing else is needed.
2. Open a show that declares modules: { remote: true }, for example http://localhost:8080/examples/08-remote-surface/.
3. The join card at the top left of the stage shows the URL a phone should open, with the show machine's LAN address filled in (the page asks serve.js for it at /__info) and the room: http://<lan-ip>:8080/packages/remote/?room=default.
4. Open that URL on a phone on the same network. The card counts connected phones.

The card also draws a QR code of that URL: point a phone's camera at it.

On openaudiovisual.com (and any static host)#

There is no serve.js there, so no relay of its own. A show that declares modules: { remote: true } uses the public relay (wss://remote.cheyuwu.com/relay) instead, and the card says so (public relay). Open example 08 on the computer, scan the QR code with the phone, and the phone's faders move the show.

• The room is random (show- plus 10 characters), made once and kept in this browser's localStorage, so a reload keeps the same room and two visitors never share one. A room name declared in the show (room: 'main') is ignored here for that reason; ?room=class-7 in the page URL always wins, which is how a classroom puts everyone in one show.
• ?relay=wss://your.host/relay uses a relay of your own (the phone link carries it along); ?relay=off turns the remote off for that page.
• The choice is pickRelay() in @openav/remote/relay-pick: this machine or its LAN (localhost, 192.168.x.x, *.local…) keeps the same-origin /relay and works exactly as before; any other host gets the public relay.
• The public relay has no accounts and no authentication. The room name is the only secret, so treat the QR code like one: whoever scans it can drive that show. Don't put it on a slide you share.

The phone app#

/packages/remote/ is a single page with a status header (connection, latency, room, the phone's id) and tabs:

• 感測 (sensors): a tilt ball, acceleration and rotation readouts, a multi-touch pad, knocks (a sharp jolt of the phone), and light from the front camera. iOS needs a tap on 開始 (start) before it gives motion data.
• 琴鍵 (keys): a piano that is one row of about three octaves in landscape, and two stacked rows in portrait (the upper row continues the range). Vertical position on a key sets velocity; hold-to-sustain and octave buttons included.
• 控制台 (control): a control surface. Its layout comes from, in order: the ?surface=<layout.json url> query, the ?meta=<json url> query (its params go through autoSurface), whatever the show publishes over the relay, and finally a small generic panel so the page is never blank.
• MIDI: a virtual MIDI controller, the same one the show displays (MIDI controllers & embedding).

Each phone keeps a short random id in localStorage; it is the <id> in phone/<id>/….

iOS and HTTPS#

iOS Safari only gives motion sensors and the camera to pages served over https://. Touch, keys and the control surface work over plain http:// on the LAN. For tilt and knocks on an iPhone, reach serve.js through an HTTPS tunnel and open the tunnel's address on the phone; the relay follows the page's origin (wss:// on HTTPS pages). Android Chrome behaves the same for the camera.

The show side#

modules: { remote: true } in createShow() (or mountRemoteHost() from @openav/remote/host) does five things, so the world stays ignorant of phones:

1. joins the relay as the room's runner (the show);
2. files every phone signal into signals, and mirrors phone/<id>/… to phone/any/…, which always holds the value from whichever phone sent it last;
3. builds the control surface: world.surface if the world ships a layout, otherwise autoSurface(world.params); and publishes it to every phone, including phones that join later;
4. adds the surface/… → param routes to the mapper (skipping pairs that are already there);
5. sends param values back to the phones about 10 times a second, so faders follow the timeline and other controllers.

Routes for sensors are written at author time with the phone/any/… alias, because nobody knows a stranger's device id in advance (example 08):

await createShow({
  world: bloomWorld,
  routes: [
    { source: 'phone/any/tilt/x', target: 'drift', inMin: -1, inMax: 1, outMin: 0, outMax: 2, smooth: 0.12 },
    { source: 'phone/any/knock', target: 'burst' },
  ],
  modules: { keys: { base: 48 }, remote: { auto: { pairs: [['cx', 'cy']], meters: ['density'] } } },
});

show.remote is the host: { relay, layout, routes, bindings, room, publish(), frame(dt, state), connect(), dispose() }.

Surface widgets#

A surface turns fingers into named, normalized signals. Every widget publishes under surface/<page>/<id>; when the widget's range is not 0..1, the unscaled value also goes out as …/raw.

Common fields: id, type, label, min, max, def, step, unit, color (cyan amber magenta lime violet coral white), and target (a param key, or an array for xy and bank) for hand-written layouts.

The phone keyboard's notes are { note, vel, velocity, ch: 1, device: 'surface' }.

Layout JSON#

A layout is data, like a route: shareable as a file, writable by an agent, publishable by a world.

{ "version": 1, "title": "Bloom",
  "pages": [{
    "id": "main", "title": "Main", "grid": { "cols": 8, "rows": 4 },
    "widgets": [
      { "id": "hue",  "type": "knob",   "label": "Hue", "x": 0, "y": 0, "w": 2, "h": 2, "min": 0, "max": 360, "def": 205, "color": "cyan", "target": "hue" },
      { "id": "pos",  "type": "xy",     "x": 2, "y": 0, "w": 4, "h": 4, "target": ["cx", "cy"] },
      { "id": "go",   "type": "button", "label": "Burst", "x": 6, "y": 0, "w": 2, "h": 1, "target": "burst" }
    ],
    "portrait": { "grid": { "cols": 4, "rows": 8 }, "place": { "pos": { "x": 0, "y": 2, "w": 4, "h": 4 } } }
  }] }

Phones get turned, so a page decides its arrangement like this:

1. if the page has a portrait or landscape variant for the current orientation, use its grid (widgets missing from its place keep their base spot or fill free cells);
2. otherwise, if the base grid already has the right shape, use it;
3. otherwise auto-reflow: the same widgets, repacked row by row into half (portrait) or double (landscape) the columns.

Rows are a count, not pixels, so the grid always fills the screen and nothing ever scrolls. Widgets without x/y are packed into free cells with a default size per type. validateLayout(layout) returns a list of problems (empty = fine); new Surface() throws with that list on a bad layout.

autoSurface#

autoSurface(params, options) turns a world's params into a layout, the routes that drive the params, and the bindings for feedback:

import { autoSurface, feedbackFor } from '@openav/surface';

const { layout, routes, bindings } = autoSurface(world.params, { pairs: [['cx', 'cy']], meters: ['density'] });
routes.forEach((r) => mapper.addRoute(r));   // surface/main/hue → hue, …
feedbackFor(state, bindings);                // [{ name, value }] to echo back to the phone

How each param becomes a widget:

Options: pairs, meters (params to also show read-only), style ('auto', 'fader', 'knob'), perPage (default 8; a page that scrolls is a page that fails), smooth (default 0.04 s on continuous routes, to hide 30 Hz network steps), title. Each param group gets its own page.

Hand-written layouts#

Give a widget a target and routesFromLayout(layout, params) writes its route (the same shape autoSurface produces): xy maps x to the first target and y to the second, bank maps channel i to the i-th target. A world can carry its layout as world.surface; the show side then uses it instead of generating one.

Theme and phone helpers#

Every control reads CSS custom properties on :root (THEME_VARS: --oav-bg, --oav-surface, --oav-accent, --oav-radius, --oav-touch-min (44 px) …). Override them to re-skin the whole surface. The rules: dark, ≥ 44 px touch targets, the live number always visible, pressed = border and glow in the widget's color, navigator.vibrate where it exists.

@openav/surface also exports phone helpers: lockViewport() (no scrolling or zooming), keepAwake() (Wake Lock), toggleFullscreen(), canFullscreen(), haptic(ms).

The relay#

Clients connect to ws://<host>/relay?role=controller|runner|monitor&room=<name>&id=<device>. A room is a name; peers in different rooms never hear each other.

Messages are JSON: {"type":"signal","name":"phone/ab12/tilt/x","value":0.3,"t":…,"pulse"?:true}, {"type":"batch","items":[…]}, ping/pong, status (peer counts), feedback, config. Clients ignore types they do not know. When the runner leaves, the room forgets its config. Messages over 256 KB are ignored, frames over 1 MB close the connection, and a phone that stops answering pings (screen locked, Wi-Fi gone) is dropped within about 40 s.

import { RelayClient, bindSignals } from '@openav/relay';

const show = new RelayClient({ role: 'runner', room: 'main' }).connect();
bindSignals(show, signals);                        // phone signals → Signals, + phone/any/… alias
show.feedback('surface/main/level', 0.4);          // runner → phones
show.config('surface', { layout });                // runner → phones, kept for late joiners

const phone = new RelayClient({ role: 'controller', room: 'main' }).connect();
phone.set('phone/me/tilt/x', 0.3);                 // continuous: batched at 30 Hz, re-sent every second
phone.send('midi/note/on', { note: 60 }, true);    // pulse: immediate, never coalesced

Continuous values are re-sent once a second so a phone that joins late, or a show page reloaded mid-performance, is never stuck on stale defaults. Pulses are never batched or repeated: a note-on delivered twice is a bug. The client reconnects by itself (0.5 s, doubling up to 5 s).

fileSignal(signals, name, value, { pulse }) is the receiving half of the wire format: it declares a name the first time it is heard (signalMeta() knows the ranges of phone/…, surface/… and midi/… names), fills in whichever velocity spelling a note message lacks (vel 0..1 or velocity 1..127), then pulses or sets.

To run the relay without serve.js: node packages/relay/server.js [port] (default 7458, health at /health), or attach it to your own Node server with attachRelay(server, { path: '/relay' }) from @openav/relay/server.

The show page on a phone#

When the show itself runs on a phone, it can read the phone's own motion sensors without a relay. localSensors(signals) publishes the same names under phone/local/…, mirrored to phone/any/…, so routes written for "the latest phone" work either way. Call .start() from a tap (iOS permission rules):

import { localSensors } from '@openav/remote';

const own = localSensors(show.signals);
button.onclick = async () => { await own.start(); if (own.denied) alert('motion access denied'); };

Security#

The relay has no authentication. Anyone on the same network who knows the room name can join as a controller and move your params. For a show, use a closed network or a room name nobody will guess. On the public relay the room is random and lives in the QR code; it is a convenience for trying a show, not a lock.

Signals reference#

14Source: docs/signals.md

Signals are the single currency of the input layer: named, normalized, source-blind. This page lists every signal published by the built-in input packages.

Conventions:

• ranges are 0..1 unless noted
• pulse signals fire events; their value is the event payload
• y axes are flipped where it makes performance sense (1 = raised), never raw pixels
• <n> is a number, <ch> a MIDI channel 1..16, <id> a device id, <slug> a device name made URL-safe

@openav/midi#

Every hardware message publishes the legacy names and the per-channel names.

With two or more MIDI inputs connected, each device also publishes under its slug (from the port name: Arturia MiniLab 3 → arturia-minilab-3, duplicates get -2, -3):

On-screen controllers (modules.midi.controllers) publish one name per control, midi/<device>/<control> (for example midi/minilab3/knob1), plus /hit pulses for pads and /raw values. They are listed in MIDI controllers & embedding.

@openav/keys#

The on-screen piano, QWERTY playing and the simulated performer publish the legacy note names with a smaller payload:

@openav/chord#

chordType: single · major · minor · sus2 · sus4 · dim · aug · maj7 · min7 · dom7 · halfdim7 · maj9 · min9 · dom9 · six · min6 · cluster (very dissonant pile) · chord (anything else). Inversions resolve to their root-position name via pitch-class rotation.

dissonanceLevel: 0 none · 1 mild (storm warning — foreshadow) · 2 severe (true cluster — commit to the decay language). Born from a real audience note at the IRCAM × C-LAB performance in Taipei: "dissonance detection wasn't strict enough."

@openav/audio#

@openav/pose#

Mirrored by default (webcam-as-mirror). y flipped: 1 = raised.

@openav/pose — HandTracker (21-landmark fine control)#

Per-hand pinch distances = four precise continuous controllers across two hands. Normalized by palm size, so distance to the camera doesn't change your pinch.

(hand/right/* mirrors the set. Both trackers expose skeleton(ctx, w, h) overlays.)

@openav/leap#

Leap Motion / Ultraleap through the machine's bridge. Positions are the palm or fingertip inside the interaction box, normalized and clamped (x right, y up, z toward the performer). <side> is left or right, <finger> one of thumb index middle ring pinky.

@openav/drums (simulator — analyzer-shaped)#

The drum machine publishes the SAME shapes the audio analyzer publishes, so a world mapped to a kick cannot tell mic from machine (that's what a simulator is for). Sequencer hits also travel as midi/note/on (ch 10, GM notes).

@openav/sound (output events)#

The in-page instruments listen to midi/note/on|off and the sustain pedal midi/cc/64, and announce one event of their own.

@openav/record (take playback)#

A TakePlayer publishes a recorded take's events under their original names (midi/note/on, midi/ch/<ch>/note/<n>, midi/cc/<n>…), exactly as the keyboard did, plus two signals of its own.

@openav/remote · @openav/surface#

Phone sensors (/packages/remote/ → 感測). <id> is the device id; phone/any/… carries the value from whichever phone sent it last, so a World's routes can be written before any phone exists.

Surface widgets (normalized 0..1; …/raw carries the unscaled value when the range is not 0..1):

Receiving end: every relay message is filed by fileSignal(signals, name, value, {pulse}) (@openav/relay; bindSignals uses it, so do hosts with their own socket such as the lab's lab.js). It declares the name on first sight (signalMeta), and fills in whichever velocity spelling a midi/…/note/on|off sender left out (vel 0..1 ⇄ velocity 1..127).

Naming your own#

Path-style, source-first: breath/pressure, phone/{id}/gyro/x, weather/wind. Declare ranges with signals.define(name, {min, max}) so meters and norm() work; then just signals.set(name, v). Anything that changes can perform.

Package reference#

15Source: docs/packages.md

Twenty-seven packages live in packages/<name>/. Each is plain ES modules with no dependencies; in a page you import them as @openav/<name> through an import map (copy the one in examples/01-hello-particles/index.html). Nothing is on npm yet.

Subpath entries (@openav/remote/host, @openav/relay/server, @openav/surface/layout, @openav/surface/auto) need their own import-map line in a browser, for example "@openav/remote/host": "../../packages/remote/host.js?v=b59edc6".

@openav/core#

Glue · the shared language: Signals, Params, Loop, Bus.

import { Signals, Params, Loop } from '@openav/core';
const signals = new Signals();
signals.define('breath/pressure', { min: 0, max: 1 });
signals.on('breath/pressure', (v) => console.log(v));
signals.set('breath/pressure', 0.4);

@openav/show#

Glue · createShow(options): one call that assembles a whole show around a world and returns every part. Options, modules and the return value are in Show control.

import { createShow } from '@openav/show';
const show = await createShow({ world, modules: { keys: { base: 48 }, sound: true } });

@openav/midi#

L1 input and L4 output · Web MIDI in and out, many devices at once, and on-screen controllers that mirror real hardware.

import { Midi } from '@openav/midi';
const midi = new Midi({ signals });
await midi.enable();
midi.noteOn(60, 100, 1);

Signals: Signals reference. Controllers, profiles and the shim: MIDI controllers & embedding.

@openav/audio#

L1 input · microphone (or any audio node) analysis.

import { AudioAnalyzer } from '@openav/audio';
const audio = new AudioAnalyzer({ signals });
await audio.enableMic();          // from a click
// every frame: audio.update();

@openav/chord#

L1 input (an analyzer) · groups struck notes into gestures and publishes consonance, triads and clusters.

import { ChordDetector } from '@openav/chord';
const chord = new ChordDetector({ signals });
signals.on('midi/note/on', ({ note, vel }) => chord.noteOn(note, vel));
signals.on('midi/note/off', ({ note }) => chord.noteOff(note));
chord.analyze([60, 64, 67]).chordType;   // 'major'

@openav/pose#

L1 input · body and hand tracking from the camera with MediaPipe, on the device.

import { HandTracker } from '@openav/pose';
const hands = new HandTracker({ signals });
button.onclick = () => hands.enable();

@openav/leap#

L1 input · Leap Motion / Ultraleap hands through the machine's bridge (node packages/leap/bridge/leap-bridge.mjs, ws://127.0.0.1:6437). The same bridge speaks the old LeapJS protocol, so 2014–2021 leap.js sketches run unchanged.

import { LeapInput } from '@openav/leap';
const leap = new LeapInput({ signals }).connect();   // ws://127.0.0.1:6437/v6.json
leap.onFrame(() => leap.skeleton(ctx, w, h));          // joints in mm: leap.hands
// no sensor: leap.simulate() — mouse = right palm, press = pinch

@openav/keys#

L1 input · the on-screen piano, QWERTY playing and a simulated performer, all publishing midi/note/on|off.

import { mountKeys } from '@openav/keys';
const keys = mountKeys(document.querySelector('#keys'), { signals, base: 48 });
// every frame: keys.update(dt);

@openav/drums#

L1 input and L4 output · a synthesized drum kit, a 16-step sequencer, and its grid UI; publishes drum/* in the same shapes as the microphone's drum detection.

import { mountDrums } from '@openav/drums';
const drums = mountDrums(document.querySelector('#drums'), { signals });
// every frame: drums.update(dt);

@openav/mapping#

L2 mapping · routes from signals to params, with curves, smoothing, learn and saved profiles.

import { Mapper } from '@openav/mapping';
const mapper = new Mapper({ signals, params });
mapper.addRoute({ source: 'midi/cc/74', target: 'bloom', curve: 'exp', smooth: 0.1 });
// every frame: mapper.update(dt);

Full reference: Mapping.

@openav/chop#

L2 mapping · values over time, the way TouchDesigner's CHOPs shape them: Lag, Trigger (envelope), Speed, Spring, Trail, Slope, Count. Pure logic, no clock of its own: every step is told the seconds that went by and is solved exactly over them, so 30 fps and 144 fps draw the same curve. What the parameters mean follows Derivative's TouchDesigner documentation; the code is OAV's own. In a p5 sketch, oav-p5 gives these one call each: p5 one line.

import { Lag, envelope } from '@openav/chop';
const vu = new Lag({ up: 0.03, down: 0.4 });       // opens at once, closes slowly
// every frame: r = 50 + vu.step(level, dt) * 300; pump = envelope(kickTimes, now, 0.01, 0.3);

@openav/stage#

L3 world · the world shell: lifecycle and param aggregation, no renderer.

import { Stage, createCanvas } from '@openav/stage';
const stage = new Stage({ container, params, signals });
stage.register(world);
await stage.activate(world.name);
// every frame: const state = stage.frame(dt, timeline.state());

@openav/world-webtoe#

L3 world · performs a WebToe node patch as a world; params reach the patch's ext('name', fallback) bindings.

See Writing a world.

@openav/sound#

L4 output · instruments in the page behind a small engine contract: a Salamander grand piano, mallets, strings, organ, choir, harp and synth presets, chosen from a picker; engine params become sound/* params.

Engine contract: { params, enable(), noteOn(note, vel01), noteOff(note), set(key, value), dispose() }. The instrument list and how to add one: Sound.

import { Sound, toneEngine } from '@openav/sound';
const sound = new Sound({ signals, params, engine: toneEngine({ instrument: 'piano' }) });
button.onclick = () => sound.enable();
// every frame: sound.update(state);

@openav/osc#

L4 output · OSC from the browser through a small Node bridge to UDP.

See Show control.

@openav/out#

L4 output · page parameters to physical hardware through one local bridge (node packages/out/bridge/out-bridge.mjs --config out.json, ws://127.0.0.1:6450/out): WLED light strips (DDP), serial motor boards, OSC. A watchdog sends every channel its configured safe value when the page goes quiet or the last client leaves.

See the package README for the config file and the watchdog rules.

@openav/record#

L4 output (video branch) · the performance video: the work and the performer's camera composited into one canvas of an exact size (vertical 1080×1920, 4:5, 2.7K, 4K, landscape, square) and recorded with the work's own sound by one MediaRecorder, so picture and sound are in sync by construction; plus the sound as its own file, an event log, and MIDI takes (record what you play, play it back into the show, .mid in and out).

Layouts: stack (work on top at full width, camera below), pip (camera inset in a corner), side (work left, camera right), work. 60 fps up to about 1080×1920, 30 fps for 2.7K and 4K; webcams stop at 1080p, so a 4K frame enlarges the camera.

import { Compositor, AudioTap, Recorder, openCamera } from '@openav/record';
const comp = new Compositor({ size: 'vertical-1080p', layout: 'stack', getWorkCanvas: () => canvas });
comp.start();
await comp.setCamera(await openCamera());
const tap = new AudioTap({ context, sources: [workOutput] });
const rec = new Recorder({ canvas: comp.canvas, fps: 60, audioTracks: tap.tracks }).start();
// later: const { blob, mimeType } = await rec.stop();

Package README with the MIDI takes: packages/record. Example: 10-record.

@openav/fab#

L4 output (physical branch) · can it be made? A manufacturability report for any mesh (watertight, volume, overhang, thinnest wall, grams of PETG / PLA, one Chinese warning per problem) and STL / OBJ / GLB files; STEP and DXF when the shape is a smlibwasm Part or Shape. smlibwasm and manifold-3d are not bundled: smlib() / manifold() import() them from a URL on first use.