# open-audiovisual — full text for language models > Every document of open-audiovisual in one file: README, AGENTS.md and every chapter of the handbook (docs/). > Generated by tools/build-home.mjs from the repository (version 0.1.0); the short index is /llms.txt (https://openaudiovisual.com/llms.txt, Traditional Chinese: https://openaudiovisual.com/zh/llms.txt). > Site: https://openaudiovisual.com/ (English) · https://openaudiovisual.com/zh/ (繁體中文) > Handbook: https://openaudiovisual.com/docs/ · MIDI controller library: https://openaudiovisual.com/controllers/ · WebToe, the sister engine: https://openaudiovisual.com/webtoe/ > Examples: https://openaudiovisual.com/examples/ > p5.js in one line: https://openaudiovisual.com/p5/ (oav-p5 0.8.4) > Changelog: https://openaudiovisual.com/changelog/ > Repository: https://github.com/frank890417/open-audiovisual (MIT) > Today: 27 packages, 26 examples, 27 MIDI controller profiles (21 devices, 6 generic layouts). ## MIDI controllers at a glance One page and one JSON profile each; the page lists every control with its exact MIDI message. - AKAI LPD8 (mk2): 16 controls · https://openaudiovisual.com/controllers/akai-lpd8/ · https://openaudiovisual.com/packages/midi/profiles/akai-lpd8.json - AKAI APC mini mk2: 90 controls · https://openaudiovisual.com/controllers/akai-apc-mini-mk2/ · https://openaudiovisual.com/packages/midi/profiles/akai-apc-mini-mk2.json - AKAI MIDImix: 52 controls · https://openaudiovisual.com/controllers/akai-midimix/ · https://openaudiovisual.com/packages/midi/profiles/akai-midimix.json - AKAI MPK mini mk3: 43 controls · https://openaudiovisual.com/controllers/akai-mpk-mini-mk3/ · https://openaudiovisual.com/packages/midi/profiles/akai-mpk-mini-mk3.json - AKAI APC40 mkII: 139 controls · https://openaudiovisual.com/controllers/akai-apc40-mk2/ · https://openaudiovisual.com/packages/midi/profiles/akai-apc40-mk2.json - Arturia MiniLab 3: 48 controls · https://openaudiovisual.com/controllers/arturia-minilab3/ · https://openaudiovisual.com/packages/midi/profiles/arturia-minilab3.json - Arturia KeyStep 37: 44 controls · https://openaudiovisual.com/controllers/arturia-keystep37/ · https://openaudiovisual.com/packages/midi/profiles/arturia-keystep37.json - Arturia BeatStep: 32 controls · https://openaudiovisual.com/controllers/arturia-beatstep/ · https://openaudiovisual.com/packages/midi/profiles/arturia-beatstep.json - Behringer X-TOUCH MINI: 33 controls · https://openaudiovisual.com/controllers/behringer-x-touch-mini/ · https://openaudiovisual.com/packages/midi/profiles/behringer-x-touch-mini.json - DJ TechTools Midi Fighter Twister: 38 controls · https://openaudiovisual.com/controllers/djtechtools-midi-fighter-twister/ · https://openaudiovisual.com/packages/midi/profiles/djtechtools-midi-fighter-twister.json - Korg nanoKONTROL2: 51 controls · https://openaudiovisual.com/controllers/korg-nanokontrol2/ · https://openaudiovisual.com/packages/midi/profiles/korg-nanokontrol2.json - Korg nanoKEY2: 28 controls · https://openaudiovisual.com/controllers/korg-nanokey2/ · https://openaudiovisual.com/packages/midi/profiles/korg-nanokey2.json - Novation Launchpad Mini MK3: 80 controls · https://openaudiovisual.com/controllers/novation-launchpad-mini-mk3/ · https://openaudiovisual.com/packages/midi/profiles/novation-launchpad-mini-mk3.json - Novation Launch Control XL (MK2): 56 controls · https://openaudiovisual.com/controllers/novation-launch-control-xl-mk2/ · https://openaudiovisual.com/packages/midi/profiles/novation-launch-control-xl-mk2.json - Novation Launch Control XL 3: 48 controls · https://openaudiovisual.com/controllers/novation-launch-control-xl-3/ · https://openaudiovisual.com/packages/midi/profiles/novation-launch-control-xl-3.json - Novation Launchkey Mini MK3: 51 controls · https://openaudiovisual.com/controllers/novation-launchkey-mini-mk3/ · https://openaudiovisual.com/packages/midi/profiles/novation-launchkey-mini-mk3.json - Novation Launchkey 25 MK4: 52 controls · https://openaudiovisual.com/controllers/novation-launchkey-25-mk4/ · https://openaudiovisual.com/packages/midi/profiles/novation-launchkey-25-mk4.json - Novation Launchkey 37 MK4: 64 controls · https://openaudiovisual.com/controllers/novation-launchkey-37-mk4/ · https://openaudiovisual.com/packages/midi/profiles/novation-launchkey-37-mk4.json - Novation Launchpad Pro MK3: 105 controls · https://openaudiovisual.com/controllers/novation-launchpad-pro-mk3/ · https://openaudiovisual.com/packages/midi/profiles/novation-launchpad-pro-mk3.json - ODD Ball: 10 controls · https://openaudiovisual.com/controllers/odd-ball/ · https://openaudiovisual.com/packages/midi/profiles/odd-ball.json - teenage engineering OP-1 field: 61 controls · https://openaudiovisual.com/controllers/teenage-engineering-op-1-field/ · https://openaudiovisual.com/packages/midi/profiles/teenage-engineering-op-1-field.json - Generic 8 knobs + 8 pads: 16 controls · https://openaudiovisual.com/controllers/generic-8k8p/ · https://openaudiovisual.com/packages/midi/profiles/generic-8k8p.json - Generic 8 faders: 8 controls · https://openaudiovisual.com/controllers/generic-8f/ · https://openaudiovisual.com/packages/midi/profiles/generic-8f.json - Generic 16 pads: 16 controls · https://openaudiovisual.com/controllers/generic-16p/ · https://openaudiovisual.com/packages/midi/profiles/generic-16p.json - Generic 25-key keyboard: 28 controls · https://openaudiovisual.com/controllers/generic-keys25/ · https://openaudiovisual.com/packages/midi/profiles/generic-keys25.json - Generic 49-key keyboard: 52 controls · https://openaudiovisual.com/controllers/generic-keys49/ · https://openaudiovisual.com/packages/midi/profiles/generic-keys49.json - Generic 61-key keyboard: 64 controls · https://openaudiovisual.com/controllers/generic-keys61/ · https://openaudiovisual.com/packages/midi/profiles/generic-keys61.json --- [open-audiovisual: a web-native framework for audiovisual performance. signals → mapping → params → world. MIDI, audio, chords, hands, phones.](https://openaudiovisual.com) **A web-native framework for audiovisual performance.** MIDI, microphones, chords, hands and phones become named signals; a mapping turns them into parameters; your world draws and sounds them. [tests](https://github.com/frank890417/open-audiovisual/actions/workflows/checks.yml) [MIT license](LICENSE) zero dependencies zero build Web MIDI [21 MIDI controllers](https://openaudiovisual.com/controllers/) [docs](https://openaudiovisual.com/docs/) [繁體中文文件](https://openaudiovisual.com/zh/docs/) [**Website**](https://openaudiovisual.com) · [**Docs**](https://openaudiovisual.com/docs/) · [**繁體中文**](https://openaudiovisual.com/zh/docs/) · [**Controllers**](https://openaudiovisual.com/controllers/) · [**Examples**](#-examples) · [**p5.js in one line**](#-p5js-in-one-line) > Hydra gives you improvisation. cables gives you a platform. Tone gives you sound. > **open-audiovisual gives you a *show*.** Everything TouchDesigner-class tools do on a desktop, the browser can now do with a URL: WebGL/WebGPU rendering, Web MIDI, realtime audio analysis, on-device pose tracking. What's been missing is the *chassis*, the part everyone rebuilds for every performance: MIDI plumbing, signal-to-parameter mapping, a timeline with scenes and cues, a performance mode, a backstage view. This repo is that chassis. ## ⚡ Quick start **Zero build step. Zero dependencies.** Clone it, run one file, you're on stage: ```bash git clone https://github.com/frank890417/open-audiovisual.git cd open-audiovisual node serve.js # → http://localhost:8080 (Chrome or Edge: Web MIDI + camera) ``` Then open and read its `main.js`. Every example runs without hardware: an on-screen piano, QWERTY keys and a simulated performer stand in for instruments. Optional companions (each ~100 dependency-free lines, run in their own terminal): ```bash node packages/monitor/server.js # backstage monitor for the stage manager (:7457) node packages/osc/bridges/osc-bridge.js # OSC → UDP bridge (Spat, Reaper, TD, lights…) ``` ## ✨ What's in the box MIDI controllers 21 real devices modeled as data: every knob, pad and key with its exact message. Play them on screen, plug the real one in. | Audio analysis Microphone or any Web Audio node: loudness, bands, brightness, onsets, kick / snare / hat. In a p5 sketch: `micLevel`. | Chord semantics Notes struck together become one gesture: consonance, triad, cluster. Harmony decides bloom or decay. Hands & body 21-point hand and body tracking in the browser, on the device. Leap Motion through one local bridge. | Phone & iPad remote Tilt, knock, a keyboard, and a TouchOSC-style control surface grown from your world's params. | Mapping & LEARN Signal to param routes with curve, range and smoothing. Many-to-many. Press learn, move anything, it is wired. Worlds Your piece reads params, never inputs. Draw with canvas, p5, three.js or a WebToe node patch. | Timeline & Score Automation, scenes and cues; a show written as segment lengths, with cuts, holds and a director. | Console The director's desk: transport, sliders, learn, meters, and a performance mode on T. Backstage monitor Clock, scene, every signal and FPS on any phone or laptop on the network. | Performance video The work, your hands and the work's own sound in one synced vertical or 4K file. Record MIDI takes too. In a p5 sketch: `startRecording()`. | Agent-native AGENTS.md, a `/create-world` skill and an MCP server: describe a show, an agent writes the world. ## 🏗 Architecture Four layers: L1 input (midi, audio, chord, pose, leap, keys, drums, phones) publishes signals; L2 mapping routes signals to params; L3 world reads params and draws; L4 output sends video and audio out of the browser. Timeline, score, monitor, console and relay run alongside all four. Two rules make the whole thing composable: 1. **Signals are the only currency of input.** A knob, a raised hand, and a chord's consonance are equal citizens with names. That equality *is* input fusion. 2. **Worlds never know who is performing them.** A world reads params. Whether a param came from the timeline, a MIDI knob, a dancer's wrist, or a slider is invisible to it, so the same world can be played by a pianist, a dancer, or an automation curve. **The same diagram as text (for terminals, screen readers and AI agents)** ``` ┌────────────────────────────────────────────────────────────┐ │ L1 INPUT — "what is happening" │ │ @openav/midi @openav/audio @openav/chord @openav/pose│ │ @openav/leap (Leap Motion, via a local bridge) · phones │ │ → all publish named, normalized SIGNALS │ │ midi/cc/74 · chord/consonance · audio/rms · pose/hand/…│ ├────────────────────────────────────────────────────────────┤ │ L2 MAPPING — "what it means" @openav/mapping │ │ signal → param routes: curve · range · smooth · │ │ many-to-many · live LEARN (works for knobs AND hands) │ ├────────────────────────────────────────────────────────────┤ │ L3 WORLD — "how the system behaves" @openav/stage │ │ a World sees params, never inputs. p5 / three / canvas / │ │ anything. Swap works; keep the shell. │ ├────────────────────────────────────────────────────────────┤ │ L4 OUTPUT — "how it leaves the browser" (video ∥ audio) │ │ video: screen · @openav/record · future NDI │ │ audio: @openav/midi out · @openav/sound (Tone.js engine) │ │ · @openav/osc → UDP bridge │ ├────────────────────────────────────────────────────────────┤ │ ⏱ @openav/timeline — param automation + scenes/cues │ │ 📜 @openav/score — segment lengths → cuts, cues, holds; │ │ the director runs a module per segment │ │ 📟 @openav/monitor — backstage: clock, scene, signals, FPS │ │ 🎛 @openav/console — desk · Layers live view · perf mode │ │ 📱 @openav/relay · surface · remote — phones/iPads as │ │ controllers: rooms, TouchOSC-style widgets, autoSurface │ └────────────────────────────────────────────────────────────┘ ``` ## 🖼 Examples Twenty-six works, all playable in the browser. In any example: Space plays the timeline (at a hold: releases it), ←/→ jump scenes, T is performance mode, F is fullscreen. All examples, by route: [openaudiovisual.com/examples/](https://openaudiovisual.com/examples/) **① First open**: Read the whole framework in one file, then add one line to a p5 sketch you already have. [Bursts of thin green and red particle trails falling on black.](https://openaudiovisual.com/examples/01-hello-particles/) 01 · Hello particles The smallest complete app. Read main.js top to bottom and you know the framework. [play](https://openaudiovisual.com/examples/01-hello-particles/) · [source](examples/01-hello-particles/) | [A flower of layered green and yellow petals with coloured circles around it and four tall red, green and blue beams.](https://openaudiovisual.com/examples/15-p5-one-line/) 15 · p5.js in one line A plain p5 sketch whose first line loads oav-p5.js, then plays from a MIDI controller, an on-screen controller or a phone. [play](https://openaudiovisual.com/examples/15-p5-one-line/) · [source](examples/15-p5-one-line/) **② Anything can be an instrument**: A controller, a chord, a microphone, bare hands, a Leap Motion, a phone, Strudel in another tab. [Streaks of green, purple and red particle dots bursting over black, above an on-screen MiniLab 3 controller with its keys and sound buttons.](https://openaudiovisual.com/examples/16-p5-sound/) 16 · p5 keys with sound startSound('piano') in setup(): every key from a MIDI keyboard, the 🎹 on screen or a phone is heard, and bursts into particles. [play](https://openaudiovisual.com/examples/16-p5-sound/) · [source](examples/16-p5-sound/) | [A large pink circle with a thin grey ring around it on dark blue.](https://openaudiovisual.com/examples/17-p5-microphone/) 17 · p5 microphone micLevel, bass and beat read the microphone the way p5 reads mouseX: a circle that grows when you speak, and no p5.sound. [play](https://openaudiovisual.com/examples/17-p5-microphone/) · [source](examples/17-p5-microphone/) | [A small green circle (micLevel) and a larger yellow one (lagged) over two traces: the green drops at once, the yellow falls slowly. A small scope sits in the corner.](https://openaudiovisual.com/examples/22-p5-lag/) 22 · p5 lag: smooth a jumpy value lag(micLevel, 0.03, 0.4) smooths a jumpy value the way TouchDesigner's Lag CHOP does, in the same seconds; scope() plots both in a corner of the page. [play](https://openaudiovisual.com/examples/22-p5-lag/) · [source](examples/22-p5-lag/) [A spiral of pink and blue dots above an on-screen MIDI keyboard controller.](https://openaudiovisual.com/examples/09-midi-controllers/) 09 · MIDI controllers On-screen controllers that move with the real ones you plug in. [play](https://openaudiovisual.com/examples/09-midi-controllers/) · [source](examples/09-midi-controllers/) | [Slender branching plants in a night garden, with green leaves and small blue, red and pink flowers.](https://openaudiovisual.com/examples/02-chord-garden/) 02 · Chord garden Triads bloom, clusters decay. The world listens to chords, never to keys. [play](https://openaudiovisual.com/examples/02-chord-garden/) · [source](examples/02-chord-garden/) | [A sphere packed with colourful cylinders seen end-on, like umbrellas.](https://openaudiovisual.com/examples/06-cylinder-earth/) 06 · Cylinder earth A 2020 voice-controlled WebGL sketch; mic bands or the built-in drum machine play it. [play](https://openaudiovisual.com/examples/06-cylinder-earth/) · [source](examples/06-cylinder-earth/) [A blue and white mandala of fine layered lines.](https://openaudiovisual.com/examples/03-pose-field/) 03 · Hand instrument Fingers as an instrument: pinch, fists, four pattern families. [play](https://openaudiovisual.com/examples/03-pose-field/) · [source](examples/03-pose-field/) | [A violet hand skeleton above green and blue ink strokes that fall like rain on black.](https://openaudiovisual.com/examples/12-leap-hands/) 12 · Leap hands A Leap Motion as an instrument; the mouse is a simulated hand when you have none. [play](https://openaudiovisual.com/examples/12-leap-hands/) · [source](examples/12-leap-hands/) | [Blue and yellow streaks exploding outward from the lower centre.](https://openaudiovisual.com/examples/08-remote-surface/) 08 · Remote surface The phone is the instrument panel; the surface grows from the world's params. [play](https://openaudiovisual.com/examples/08-remote-surface/) · [source](examples/08-remote-surface/) [Soft coloured circles rising across the lower half, a yellow hand pointing up inside a thin ring, and the room oav-strudel-demo written at the bottom.](https://openaudiovisual.com/examples/23-p5-with-strudel/) 23 · p5 with Strudel Strudel plays in another tab and this sketch draws: every note Strudel sends to OAV Virtual MIDI calls notePressed(), and its clock turns drumPhase. [play](https://openaudiovisual.com/examples/23-p5-with-strudel/) · [source](examples/23-p5-with-strudel/) | [A yellow circle, square and triangle lit on black, labelled C4 = 60, E4 = 64 and G4 = 67, with G4 and 392.0 Hz written under them.](https://openaudiovisual.com/examples/24-p5-note-names/) 24 · p5 note names noteIsDown('C4') works like noteIsDown(60): write note names (C4, c#4, Db3) wherever a note number goes; noteNumber(), noteName() and noteFreq() convert. [play](https://openaudiovisual.com/examples/24-p5-note-names/) · [source](examples/24-p5-note-names/) | [A teal disc with a white hand and a thin ring around it, four dots under its centre and a small scope in the corner showing a bouncing line; env, spring and count values are written at the bottom.](https://openaudiovisual.com/examples/25-p5-chop/) 25 · p5 numbers that move env('kick'), spring(), speed() and count('kick', 4) shape a number the way TouchDesigner's Trigger, Spring, Speed and Count CHOPs do: a bass drum opens a disc, a spring bounces it back, a hand turns, every fourth kick changes the colour. [play](https://openaudiovisual.com/examples/25-p5-chop/) · [source](examples/25-p5-chop/) [A circle of sixteen steps with a white hand on the left; on the right the rhythm line bd ~ [~ bd] ~, ~ cp ~ cp, hh(5,8), 'changed by: type', and the three earlier lines; the keys 1 to 4, Z and S explained at the bottom.](https://openaudiovisual.com/examples/26-p5-rhythm-lab/) 26 · p5 rhythm lab OAVp5.drums.edit('bd(3,8), …') swaps the rhythm like typing it, OAVp5.drums.undo() goes back a line, OAVp5.drums.strudelUrl opens the line in Strudel; rhythmChanged() says what the line is now. [play](https://openaudiovisual.com/examples/26-p5-rhythm-lab/) · [source](examples/26-p5-rhythm-lab/) **③ Bring another picture engine**: Bring your own: a node patch, a 2023 p5 sketch, a 2021 particle system. [A node editor with noise, level and output nodes beside a grey cloud texture.](https://openaudiovisual.com/examples/04-webtoe-stage/) 04 · WebToe stage A WebToe node patch performed as a World through ext() bindings. [play](https://openaudiovisual.com/examples/04-webtoe-stage/) · [source](examples/04-webtoe-stage/) | [A glowing yellow ring of jagged petals with small circles inside.](https://openaudiovisual.com/examples/05-prebiotic-flake/) 05 · Prebiotic flake A 2023 MIDI sketch replayed through the framework, now with a Tone.js voice. [play](https://openaudiovisual.com/examples/05-prebiotic-flake/) · [source](examples/05-prebiotic-flake/) | [Cyan and violet fireworks bursting over drooping coloured trails.](https://openaudiovisual.com/examples/07-firework-festival/) 07 · Firework festival Close a finger pinch and a firework launches from that hand (2021). [play](https://openaudiovisual.com/examples/07-firework-festival/) · [source](examples/07-firework-festival/) **④ Run a show**: The show, not just the piece: a timeline you can see, a drum machine that counts the bars, and a score with a director. [A hexagonal ring of orange and yellow dots above a timeline with five curve lanes, scene blocks and a playhead.](https://openaudiovisual.com/examples/14-timeline/) 14 · Timeline Automation you can see: five lanes of keyframe curves drive a ring of light, scenes cut it, cues fire one-shot flashes. [play](https://openaudiovisual.com/examples/14-timeline/) · [source](examples/14-timeline/) | [A yellow and orange flower of long petals below a 16-step drum grid with kick, snare, hat and clap rows.](https://openaudiovisual.com/examples/18-p5-drums/) 18 · p5 drums and bars startDrums('breakbeat', 110) puts a 16-step drum machine on the page; kickDetected() and cueFired() turn its hits and its bars into the picture. [play](https://openaudiovisual.com/examples/18-p5-drums/) · [source](examples/18-p5-drums/) | [A blue circle with a yellow hand sweeping round it, below a drum card that shows the line bd*4, [~ sd]*2, hh*8 and its 16 steps.](https://openaudiovisual.com/examples/19-p5-rhythm/) 19 · p5 one line of rhythm startDrums("bd*4, [~ sd]*2, hh*8") writes the drums as one line of rhythm; drumPhase and beatPhase turn a hand once a bar and swell a circle on every beat. [play](https://openaudiovisual.com/examples/19-p5-rhythm/) · [source](examples/19-p5-rhythm/) [One rhythm drawn three ways on the canvas: a ring of coloured steps with a white hand, a spiral of the last four bars, and a 16-step grid with a playhead, under the drum card.](https://openaudiovisual.com/examples/20-p5-rhythm-visual/) 20 · p5 rhythm you can see drawRhythm() draws the rhythm on the canvas as a grid, a circle or a spiral, so a recording has it; the cells and the line on the card edit each other. [play](https://openaudiovisual.com/examples/20-p5-rhythm-visual/) · [source](examples/20-p5-rhythm-visual/) | [A pale moon in a blue and teal dusk sky, with the word dusk in the corner.](https://openaudiovisual.com/examples/13-score-director/) 13 · Score & director A day of sky as segment lengths: two cuts, a hold, cues, a segment that can crash. [play](https://openaudiovisual.com/examples/13-score-director/) · [source](examples/13-score-director/) **⑤ Take it out of the browser**: One synced video, from one line in a p5 sketch, with or without a camera. [A pink dot in the middle of glowing pink and yellow rings spreading out, below the drum card playing four on the floor.](https://openaudiovisual.com/examples/21-p5-record/) 21 · p5 record a video startRecording() saves the canvas and the sketch's own sound as one video, MP4 where the browser can write it; ⏺ and 📷 in the 🎹 panel need no code. [play](https://openaudiovisual.com/examples/21-p5-record/) · [source](examples/21-p5-record/) | [Overlapping pink, yellow and green rings spreading over a dark field, each one a struck note.](https://openaudiovisual.com/examples/10-record/) 10 · Record The work plus your hands plus its own sound, in one synced video. MIDI takes too. [play](https://openaudiovisual.com/examples/10-record/) · [source](examples/10-record/) | [Overlapping pink, purple and yellow rings beside an on-screen MiniLab 3 with the last note written under it.](https://openaudiovisual.com/examples/11-controller-video/) 11 · Controller video The performance video without a camera: the controller you play, drawn live. [play](https://openaudiovisual.com/examples/11-controller-video/) · [source](examples/11-controller-video/) All examples run without MIDI hardware: every stage ships an on-screen piano (`@openav/keys`: QWERTY capture, Z/X octave) and a **simulated performer** that plays the piece hands-free. 01 and 02 run fully offline. Examples are assembled with **`createShow()`** (`@openav/show`), a World plus one declarative call; the layer-aligned side panel, inputs, sound and backstage all come from the factory, so every work evolves together when the framework grows. **Artwork attribution**: examples 05–07 integrate original artworks by **Che-Yu Wu 吳哲宇** (2020–2023), used here strictly as demos. The framework is MIT; the artworks remain © the artist, all rights reserved. ## 🎹 MIDI controller library [The controllers page: MIDI controllers, modeled. 16 devices, 6 generic layouts, 743 controls, with an on-screen Arturia MiniLab 3 you can play.](https://openaudiovisual.com/controllers/) Controllers are **data**: one JSON file per device with the faceplate, every control's exact MIDI message, LED feedback, how to recognise its port, and the document each number came from. The same file drives an on-screen controller you can play, follows the real device when you plug it in, and drops into any web page. **21 devices, 1141 controls** (plus 6 generic layouts), each with its sources. | maker | devices (spec sheet with the full MIDI map) | |---|---| | AKAI Professional | [**LPD8 (mk2)**](https://openaudiovisual.com/controllers/akai-lpd8/) · [**APC mini mk2**](https://openaudiovisual.com/controllers/akai-apc-mini-mk2/) · [**MIDImix**](https://openaudiovisual.com/controllers/akai-midimix/) · [**MPK mini mk3**](https://openaudiovisual.com/controllers/akai-mpk-mini-mk3/) · [**APC40 mkII**](https://openaudiovisual.com/controllers/akai-apc40-mk2/) | | Arturia | [**MiniLab 3**](https://openaudiovisual.com/controllers/arturia-minilab3/) · [**KeyStep 37**](https://openaudiovisual.com/controllers/arturia-keystep37/) · [**BeatStep**](https://openaudiovisual.com/controllers/arturia-beatstep/) | | Behringer | [**X-TOUCH MINI**](https://openaudiovisual.com/controllers/behringer-x-touch-mini/) | | DJ TechTools | [**TechTools Midi Fighter Twister**](https://openaudiovisual.com/controllers/djtechtools-midi-fighter-twister/) | | KORG | [**nanoKONTROL2**](https://openaudiovisual.com/controllers/korg-nanokontrol2/) · [**nanoKEY2**](https://openaudiovisual.com/controllers/korg-nanokey2/) | | Novation | [**Launchpad Mini MK3**](https://openaudiovisual.com/controllers/novation-launchpad-mini-mk3/) · [**Launch Control XL (MK2)**](https://openaudiovisual.com/controllers/novation-launch-control-xl-mk2/) · [**Launch Control XL 3**](https://openaudiovisual.com/controllers/novation-launch-control-xl-3/) · [**Launchkey Mini MK3**](https://openaudiovisual.com/controllers/novation-launchkey-mini-mk3/) · [**Launchkey 25 MK4**](https://openaudiovisual.com/controllers/novation-launchkey-25-mk4/) · [**Launchkey 37 MK4**](https://openaudiovisual.com/controllers/novation-launchkey-37-mk4/) · [**Launchpad Pro MK3**](https://openaudiovisual.com/controllers/novation-launchpad-pro-mk3/) | | ODD | [**ODD Ball**](https://openaudiovisual.com/controllers/odd-ball/) | | teenage engineering | [**OP-1 field**](https://openaudiovisual.com/controllers/teenage-engineering-op-1-field/) | Embed one anywhere, one line: ```html ``` **Is your controller missing?** [Request it](https://github.com/frank890417/open-audiovisual/issues/new?title=MIDI%20controller%20request%3A%20) (a manual or a MIDI implementation chart helps), or add it yourself: 1. Copy the closest file in [`packages/midi/profiles/`](packages/midi/profiles/) (or a generic layout) and write each control's type and message from the manual. 2. Cite every source; mark anything unconfirmed `"verified": false` with a note. 3. `validateProfile(profile)` lists every problem, `npm test` checks every shipped profile; try it with ``. 4. Register it in `profiles/index.js` and open a pull request. Full steps: [docs/controllers.md](docs/controllers.md#add-a-controller). ## 🎨 p5.js in one line Any p5.js sketch (OpenProcessing, the p5 editor, your own page) can be played from a MIDI controller, a controller drawn on screen, or a phone. Add one line at the top: ```js document.head.prepend(Object.assign(document.createElement('script'), { src: 'https://openaudiovisual.com/p5/oav-p5.js' })); ``` Then read the controller the way p5 reads the mouse: ```js function draw() { background(knob1 * 255); // knob 1, 0..1, like mouseX circle(width / 2, height / 2, 50 + knob2 * 300); } function notePressed(note, velocity) { // like keyPressed(): any key, or a pad circle(random(width), random(height), velocity * 100); } ``` No `setup()` call, p5 1.x and 2.x. On OpenProcessing it also wires to OP Configurator sliders in order. The same line does more, each one a call or a name in the p5 style (oav-p5 0.8.4, [changelog](https://openaudiovisual.com/changelog/)): | write | and | play it | |---|---|---| | `startSound('piano')` | every key from a keyboard, the 🎹 on screen or a phone is heard ([sound](https://openaudiovisual.com/docs/#p5-sound-what-is-played-is-heard)) | [16](https://openaudiovisual.com/examples/16-p5-sound/) | | `micLevel`, `bass`, `beatDetected()` | the microphone, read like `mouseX`, no p5.sound ([microphone](https://openaudiovisual.com/docs/#p5-the-microphone)) | [17](https://openaudiovisual.com/examples/17-p5-microphone/) | | `startDrums("bd*4, [~ sd]*2, hh*8")`, `cue(5, 'bloom')` | a drum machine written as one line of rhythm (click its grid and the line rewrites itself), and a timeline of bars ([drum machine](https://openaudiovisual.com/docs/#p5-drum-machine)) | [18](https://openaudiovisual.com/examples/18-p5-drums/) · [19](https://openaudiovisual.com/examples/19-p5-rhythm/) · [20](https://openaudiovisual.com/examples/20-p5-rhythm-visual/) | | `startRecording()` | the canvas and the sketch's own sound in one video; ⏺ in the 🎹 panel needs no code ([recording](https://openaudiovisual.com/docs/#p5-recording-a-video-with-its-sound)) | [21](https://openaudiovisual.com/examples/21-p5-record/) | | `lag(micLevel, 0.03, 0.4)` | a jumpy value smoothed like TouchDesigner's Lag CHOP, in its seconds; `env()`, `spring()`, `scope()` too ([values over time](https://openaudiovisual.com/docs/#p5-values-over-time-lag-env-speed-spring-trail)) | [22](https://openaudiovisual.com/examples/22-p5-lag/) | | Strudel | load the line in strudel.cc: knobs, a phone and the microphone become Strudel's parameters, and its notes and clock reach a sketch ([With Strudel](https://openaudiovisual.com/docs/#strudel)) | [23](https://openaudiovisual.com/examples/23-p5-with-strudel/) | Details: [openaudiovisual.com/docs/#p5](https://openaudiovisual.com/docs/#p5) · [docs/p5.md](docs/p5.md) · [`p5/`](p5/). ## 🧩 A world in 20 lines ```js export const myWorld = { name: 'pulse', params: [{ key: 'rate', min: 0, max: 10, def: 2, label: 'Rate' }], init({ container, signals }) { this.view = createCanvas(container); this.r = 0; signals.on('midi/note/on', ({ vel }) => { this.r = 100 * vel; }); }, update(dt, state) { this.r = Math.max(0, this.r - dt * 60); this.rate = state.rate; }, render() { const { ctx } = this.view, { w, h } = this.view.fit(); ctx.fillStyle = '#000'; ctx.fillRect(0, 0, w, h); ctx.fillStyle = '#fff'; ctx.beginPath(); ctx.arc(w / 2, h / 2, 20 + this.r, 0, 7); ctx.fill(); }, dispose() { this.view.dispose(); }, }; ``` Register it on a `Stage`, feed it a `Timeline`, mount the `Console`: see [`examples/01-hello-particles/main.js`](examples/01-hello-particles/main.js) for the ~40 lines of assembly, and [docs/writing-a-world.md](docs/writing-a-world.md) for the full contract. ## 📱 Phone / iPad remote `node serve.js`, open an example that declares `modules: { remote: true }` (try [08](examples/08-remote-surface/)), and scan the QR code on the join card with your phone (or type its URL; same wifi). On [openaudiovisual.com](https://openaudiovisual.com/examples/08-remote-surface/) there is no `serve.js`: the example uses the public relay and a random room, and the QR code works the same. Tabs: 感測 sensors (tilt · accel · touch · knock · light), 琴鍵 keys (one row landscape, two stacked rows portrait), 控制台 control surface (faders, knobs, XY, pads… generated from the World's `params`, or from a layout JSON), and MIDI. Details: [docs/remote.md](docs/remote.md). ## 🤖 Agent-native Worlds are visual code, and coding agents are good at code, so this repo ships agent-ready: [`AGENTS.md`](AGENTS.md) gives any AI coding agent the layer contracts and conventions, and the bundled `/create-world` skill ([.claude/skills/create-world](.claude/skills/create-world/SKILL.md)) scaffolds a runnable work from a natural-language description ("a jellyfish world that blooms on consonant chords"). Describe the performance; the agent writes the World; the chassis does the rest. An **MCP server** ships in the box ([`.mcp.json`](.mcp.json), zero-dependency stdio): `list_examples` · `read_doc` · `scaffold_world` · `run_checks` · `list_controllers` · `read_profile` · `validate_profile` · `scaffold_profile`. Connect any MCP-capable agent and it can inspect, scaffold and verify a performance. Machine-readable entry points: [`llms.txt`](llms.txt) · [`llms-full.txt`](llms-full.txt). (Positioning kin: [open-slide](https://github.com/1weiho/open-slide), which does this for slide decks.) ## 📦 Packages Twenty-seven plain-ES-module packages in [`packages/`](packages/), no dependencies. Import what you need through an import map, or let `createShow()` assemble them. Reference: [docs/packages.md](docs/packages.md). | layer | package | role | |---|---|---| | L1 input | [`@openav/midi`](packages/midi/) | Web MIDI in and out, many devices, on-screen controller profiles | | | [`@openav/audio`](packages/audio/) | Microphone analysis: loudness, bands, brightness, onsets, drum hits | | | [`@openav/chord`](packages/chord/) | Chord detection for performance: consonance, triads, clusters | | | [`@openav/pose`](packages/pose/) | Body and 21-point hand tracking from the camera, on the device | | | [`@openav/keys`](packages/keys/) | On-screen piano, QWERTY playing and a simulated performer | | | [`@openav/drums`](packages/drums/) | Synth kit and step sequencer that publish analyzer-shaped signals | | | [`@openav/leap`](packages/leap/) | Leap Motion hands through a small local bridge, as 21-point hand signals | | L2 mapping | [`@openav/mapping`](packages/mapping/) | Signal to param routes: curves, smoothing, many-to-many, learn | | | [`@openav/chop`](packages/chop/) | Values over time, as TouchDesigner's CHOPs shape them: lag, envelope, speed, spring, trail, slope, count; the same curve at any frame rate | | L3 world | [`@openav/stage`](packages/stage/) | The world shell: lifecycle and params, no renderer opinion | | | [`@openav/world-webtoe`](packages/world-webtoe/) | Perform a WebToe node patch as a world | | L4 output | [`@openav/sound`](packages/sound/) | Instruments in the page (Salamander piano, mallets, strings, organ, synth presets), one file to add your own | | | [`@openav/osc`](packages/osc/) | OSC out of the browser through a small Node bridge to UDP | | | [`@openav/out`](packages/out/) | Page parameters to WLED lights, serial motor boards and OSC through one local bridge, with a watchdog that falls back to safe values | | | [`@openav/record`](packages/record/) | Performance video: the work above the webcam in one file with its own sound; MIDI takes recorded and replayed | | | [`@openav/fab`](packages/fab/) | Can it be made? A 3D-print check for any mesh (watertight, overhang, thinnest wall, grams) and STL, OBJ, GLB, STEP files | | spines | [`@openav/timeline`](packages/timeline/) | Param automation, scenes and cues: the clock of the show | | | [`@openav/score`](packages/score/) | The show as segment lengths: cuts, cues, holds, and a director that keeps it running when a module fails | | | [`@openav/pattern`](packages/pattern/) | Rhythm in one line: TidalCycles/Strudel-style mini-notation read into the hits of each bar (Euclidean rhythms, one-per-bar alternation, seeded chance) | | | [`@openav/console`](packages/console/) | The director's desk and performance mode | | | [`@openav/monitor`](packages/monitor/) | Backstage view for any device on the network | | phones and tablets | [`@openav/relay`](packages/relay/) | Zero-dependency room relay for phones and iPads | | | [`@openav/surface`](packages/surface/) | TouchOSC-style widgets and autoSurface(params) | | | [`@openav/remote`](packages/remote/) | The phone and iPad app: sensors, keyboard, control surface | | glue and agents | [`@openav/core`](packages/core/) | Signals, Params and Loop: what every layer speaks | | | [`@openav/show`](packages/show/) | createShow(): one call assembles the whole shell | | | [`@openav/mcp`](packages/mcp/) | MCP server for coding agents: list, read, scaffold, check | ## 📚 Docs **The handbook: [openaudiovisual.com/docs](https://openaudiovisual.com/docs/)** ([繁體中文](https://openaudiovisual.com/zh/docs/)): every chapter on one page, generated from [`docs/`](docs/). **The MIDI controller library: [openaudiovisual.com/controllers](https://openaudiovisual.com/controllers/)**: every modeled controller, playable, with its MIDI map and sources. | start here | go deeper | |---|---| | [Introduction](docs/introduction.md) · [Getting started](docs/getting-started.md) · [Core concepts](docs/concepts.md) | [Architecture](docs/architecture.md): the layer contracts, in detail | | [Writing a world](docs/writing-a-world.md) · [Signals reference](docs/signals.md) · [Mapping](docs/mapping.md) | [Show control](docs/show-control.md): timeline, scenes, performance mode, monitor, OSC | | [Inputs](docs/inputs.md) · [Controllers](docs/controllers.md) · [p5.js in one line](docs/p5.md) | [Score & Director](docs/score.md): a show as segment lengths | | [Phones & remote](docs/remote.md) · [For AI agents](docs/agents.md) · [Troubleshooting](docs/troubleshooting.md) | [Roadmap](docs/roadmap.md) · [Packages](docs/packages.md) · [Lab integration](docs/lab-integration.md) | ## 🔭 Sister project: WebToe [The WebToe editor: operators wired together with a live preview on every node.](https://openaudiovisual.com/webtoe/) [**WebToe**](https://github.com/frank890417/WebToe) is a web-native, node-based dataflow engine (TouchDesigner-style, imports real `.toe` projects). The two are halves of one stack: **WebToe is the engine, open-audiovisual is the show**. A WebToe network can become an openav World, and openav's inputs (MIDI, chords, pose) map naturally onto CHOP channels. The adapter is [`@openav/world-webtoe`](packages/world-webtoe/) (example 04): it embeds the [WebToe editor](https://webtoe.openaudiovisual.com/) and posts params into the patch, where any parameter written as `ext('name', fallback)` follows. More: [openaudiovisual.com/webtoe](https://openaudiovisual.com/webtoe/). ## 🌱 Design lineage The core of this framework was extracted from **The Last Input** (Che-Yu Wu, 2026, C-LAB Taiwan Sound Lab / IRCAM residency, performed on a 49.4-channel speaker dome). The timeline, chord semantics, many-to-many MIDI-learn, and OSC batching all drove a real 14-scene show before they were generalized here. The rule since then: the framework only grows features a real show asked for. The mapping-layer philosophy descends from [libmapper](http://libmapper.github.io/)'s signal-namespace research. ## 🛰 Telemetry The controller element, `createController()`, the iframe embed and `createShow()` send one anonymous usage hit per page load (kind, controller profile, the page's origin, version; no cookies, nothing anyone plays). Nothing is sent from localhost, LAN addresses or `file://`. Off with `telemetry="off"` / `{ telemetry: false }` / `?telemetry=0` / `globalThis.OPENAV_TELEMETRY = false`. Details: [packages/midi/README.md](packages/midi/README.md#telemetry). ## 🗺 Status & roadmap Framework version `0.1.0` (nothing is on npm yet); oav-p5, the one line for p5.js, is versioned on its own: 0.8.4 ([changelog](https://openaudiovisual.com/changelog/)). The chassis is complete and in use: the four layers with the L4 video/audio split, timeline, **Score & Director**, console, backstage monitor, phones as controllers, **performance video** recording and MIDI takes, **Leap Motion** hands, a **21-device MIDI controller library**, **oav-p5** (sound, a drum machine with one line of rhythm, recording, TouchDesigner-style lag, Strudel side by side), an MCP server, and a handbook site. 27 packages · 26 examples · 27 MIDI controller profiles (21 devices, 6 generic layouts, 1141 controls) · 797 tests Planned ([full roadmap](docs/roadmap.md)): - Replay the full signal stream (hands, body, audio analysis) into any world - The score on an external MIDI clock (clock in already moves oav-p5's drums), clock and transport out, a score editor on the desk - Audience-device input: many phones as aggregate signals - npm publish of `@openav/*`, a WebGPU world adapter - An NDI gateway; several machines sharing one signal space over WebRTC; agents that play alongside people ## 🤝 Contributing - **A controller you want:** [open an issue](https://github.com/frank890417/open-audiovisual/issues/new?title=MIDI%20controller%20request%3A%20). **A controller profile you made:** open a pull request ([how](docs/controllers.md#add-a-controller)). - Read [`AGENTS.md`](AGENTS.md) first, human or agent: vanilla ESM, no build step, no runtime dependencies (a hard constraint), and the two rules above. Framework changes must make every example better; never edit `packages/*` to make one example look right. - `npm test` (`node --test tests/*.test.js`) must stay green; add cases for any logic you touch. Thanks to **[Miffy Chen](https://github.com/miffycs)**, whose pull request [#2](https://github.com/frank890417/open-audiovisual/pull/2) added the teenage engineering **OP-1 field** profile (38 controls, every one checked on a real unit over USB), a few opt-in faceplate options for devices that don't look like the default, and the Behringer line in the controllers gallery description. ## License MIT: build shows, teach classes, fork worlds. Attribution appreciated, not required. Examples 05–07 are artworks by Che-Yu Wu, © the artist, all rights reserved (see above). --- # AGENTS.md — for AI coding agents working in this repo > Worlds are visual code. Agents are good at code. This framework is the bridge: > a human describes a performance ("a jellyfish world that blooms on consonant > chords"), you write a World inside a chassis that already handles MIDI, mapping, > timeline, console, and backstage. (Positioning inspired by > [open-slide](https://github.com/1weiho/open-slide), which does this for slides.) ## What this is A zero-build, zero-dependency ESM framework for audiovisual performance. Four layers + two spines — the contracts are in [docs/architecture.md](docs/architecture.md): - **Input** (`packages/midi`, `audio`, `chord`, `pose`, `leap`) → publishes named **signals** - **Mapping** (`packages/mapping`) → routes signals to **params** (curves, smoothing, live learn) - **World** (`packages/stage`) → algorithmic system that reads params, renders anything - **Output** (`packages/osc`, MIDI out) + **Timeline** + **Score & Director** (`packages/score`: a show as segment lengths) + **Console** + **Monitor** Full handbook (every chapter, verified against the code): https://openaudiovisual.com/docs/ — sources in `docs/`, one file for language models at `/llms-full.txt`. ## The two rules you must not break 1. **Continuous control goes through params; discrete events may use signals.** A world that reads `signals.get('midi/cc/74')` in `update()` is welded to one controller and can no longer be driven by the timeline or a dancer. Declare a param, let the mapper route to it. 2. **The shell never knows the work.** Never edit `packages/*` to make one example look right. Framework changes must make every example better. ## MCP: the fast path This repo ships an MCP server (`.mcp.json` is pre-wired — Claude Code picks it up automatically; other agents: `node packages/mcp/server.js` over stdio). Tools: `list_examples` · `read_doc` (start with `agents`, then `writing-a-world`) · `scaffold_world` (donor-based) · `run_checks` · controller profiles: `list_controllers` · `read_profile` · `validate_profile` · `scaffold_profile`. The manual path below works without it. ## How to create a new work (the common request) 1. Copy `examples/01-hello-particles/` to `examples/-/`. 2. Write the World object in `main.js` — the full contract is [docs/writing-a-world.md](docs/writing-a-world.md); it fits in ~20 lines minimum. 3. Declare params (what the piece is performed WITH), scenes and automation (what the timeline does), and any `signals.on(...)` event reactions. 4. Update the import map in `index.html` only if you use extra packages (`@openav/chord`, `@openav/pose`…). A show with structure (cuts, cues, a hold, segment modules) declares `score: { cuts, modules }` in `createShow()`: [docs/score.md](docs/score.md), example `13-score-director`. Signal names: [docs/signals.md](docs/signals.md). 5. Verify: `node serve.js` → open `http://localhost:8080/examples/-/` — QWERTY keys A–L are the no-hardware MIDI fallback (tick *keyboard* in the console first); Space plays the timeline. ## Documenting a public name Every public name has an entry in the API reference, , and the build keeps it that way: a name without an entry, an entry for a name that no longer exists, an example that does not parse, and an example that does not run are all red. Public names are: every global, callback, `OAVp5.*` member and loader option of `p5/oav-p5.js`, every export in a package's `package.json` `"exports"`, and the tools of `packages/mcp`. **Two layers, one source of truth.** - What the *code* says is never typed: the name, its kind (variable, function, event, class, constant, option), the signature and default values, a class's public members, the source file and line. `tools/reference/scan.mjs` reads them on every build. Change a signature and the page changes with it; rename or remove a name and the build fails until you touch its entry. - What a *person* writes lives beside the code: `p5/reference/.mjs` for oav-p5, `packages//reference.mjs` for a package. Plain data, `export default { name: { … } }`, English and Chinese side by side. `p5/reference/mic.mjs` (`micLevel`) is the model entry. **The loop when you add (or rename, or remove) a name:** ```bash node tools/build-reference.mjs --missing # a paste-ready stub for every name without an entry # write the entry (and delete the entry of a name you removed) node tools/build-reference.mjs # regenerates docs/reference/, zh/docs/reference/ and reference.json # (tools/release.sh --dry-run regenerates EVERYTHING that reads it: llms-full.txt, the search index, the handbook, and runs all tests) node --test tests/reference.test.js # the gates, with the message that says what to fix PW= node p5/test/reference-smoke.cjs # every example and claim in a real headless Chromium (CI runs it too) ``` Never edit `docs/reference/index.html`, `zh/docs/reference/index.html` or `docs/reference/reference.json` by hand: they are generated and `--check` (inside `tools/release.sh`) rejects a hand edit. `tools/reference/undocumented.json` is the ledger of names still waiting for an entry; it may only shrink (`node tools/build-reference.mjs --prune-ledger` drops the ones that have an entry now). Do not add to it: write the entry. **What an entry holds** (the checks are in `tools/reference/model.mjs`, `checkEntry`): | field | | |---|---| | `group` | a category id from `tools/reference/categories.mjs` | | `summary` `{ en, zh }` | one sentence that stands alone: it is the line in lists, search results and "related" cards | | `description` `[{ en, zh }]` | what it is, how to use it, its unit and range, when it updates, what to do first, what to watch out for | | `value` | variables: `{ type, range, en, zh }`; options: `{ type, options, default, en, zh }` | | `params` `[{ name, type, optional, default, en, zh }]` | must match the names the code declares, in order (a parameter read from `arguments` says `viaArguments: true`) | | `returns`, `needs`, `notes`, `related`, `keywords`, `tbd` | `needs` = what must happen before it has a value; `tbd` = what you could not confirm (the page shows "To be confirmed": never guess) | | `examples` `[{ code, caption: { en, zh }, run, needs }]` | at least one for an oav-p5 name | | `claims` `[{ before, expr, equals \| min, max, moves }]` | what the entry says about a value, checked in a running sketch | **Write it like p5.js does.** The first sentence defines the thing. Then how to use it, the unit and range, when the value changes, what must happen before it has a value (the microphone permission, a click), and the one thing that surprises people. Examples go from the smallest that shows the point (about 8 lines), to a realistic use, to the gotcha; each is a complete sketch with `setup()` and `draw()` that runs when pasted into OpenProcessing, **without** the loader line (the page adds it when you copy or run), with a one-sentence caption saying what you will see. An example about a loading option starts with `window.OAV_P5 = { … };`: that line has to run before oav-p5 loads, so the page puts the loader line right after it. Code comments stay short and English, so both languages share one program. Write only what the code does; what you could not read out of it goes in `tbd`. The Chinese is Traditional Chinese as written in Taiwan, with no full-width semicolon `；` and no dash `——` (the check fails on them: use `。` or `，`). ## Verifying changes - `npm test` — pure-logic tests (chord analysis, timeline, mapping). Add cases for any logic you touch; browser-only code is exercised via the examples. - Manual smoke: every example must load with zero console errors (the `ws://…7457` monitor reconnect warning is expected when no relay runs). - `window.openav` is exposed in every example for devtools/scripted poking. - Shipping: `tools/release.sh "message"` is the only way to deploy (a push to `main` is a deploy). It builds, tests, commits, stamps, pushes and checks the live site in a fixed order; see [RELEASING.md](RELEASING.md) for why. ## Conventions - Vanilla ESM + JSDoc. No TypeScript, no build step, no runtime dependencies — a ` ``` ```js const world = { name: 'p5-world', params: [{ key: 'hue', min: 0, max: 360, def: 200 }], init({ container }) { const self = this; this.s = {}; this.p5 = new p5((sk) => { sk.setup = () => { sk.createCanvas(container.clientWidth, container.clientHeight); sk.colorMode(sk.HSB); }; sk.draw = () => { sk.background(self.s.hue ?? 200, 60, 20); }; sk.windowResized = () => sk.resizeCanvas(container.clientWidth, container.clientHeight); }, container); }, update(dt, state) { this.s = state; }, render() { /* p5 runs its own loop */ }, dispose() { this.p5?.remove(); }, }; ``` ### three.js No bundled example uses three.js yet; this is the pattern. Add three to the example's import map (`"three": "https://cdn.jsdelivr.net/npm/three@0.160.0/build/three.module.js"`), keep the renderer on `this`, and dispose everything: ```js import * as THREE from 'three'; const orbit = { name: 'orbit', params: [{ key: 'spin', min: 0, max: 3, def: 1 }], init({ container }) { this.container = container; this.renderer = new THREE.WebGLRenderer({ antialias: true }); container.appendChild(this.renderer.domElement); this.scene = new THREE.Scene(); this.camera = new THREE.PerspectiveCamera(50, 1, 0.1, 100); this.camera.position.z = 4; this.mesh = new THREE.Mesh(new THREE.IcosahedronGeometry(1, 2), new THREE.MeshNormalMaterial({ wireframe: true })); this.scene.add(this.mesh); }, update(dt, s) { this.mesh.rotation.y += dt * s.spin; }, render() { const w = this.container.clientWidth, h = this.container.clientHeight; if (w !== this.w || h !== this.h) { this.w = w; this.h = h; this.renderer.setSize(w, h); this.camera.aspect = w / h; this.camera.updateProjectionMatrix(); } this.renderer.render(this.scene, this.camera); }, dispose() { this.mesh.geometry.dispose(); this.mesh.material.dispose(); this.renderer.dispose(); this.renderer.domElement.remove(); }, }; ``` ### A WebToe patch `webtoeWorld()` from `@openav/world-webtoe` turns a [WebToe](https://github.com/frank890417/WebToe) node patch into a world. The patch runs in an iframe; every frame in which a param changed, the resolved params are posted into it, and any patch parameter written as `ext('name', fallback)` listens (example 04): ```js import { webtoeWorld } from '@openav/world-webtoe'; const world = webtoeWorld({ name: 'garden-patch', project: new URL('./garden.webtoe.json', location.href).href, // the app fetches it: CORS must allow it params: [ { key: 'speed', min: 0, max: 1, def: 0.5 }, { key: 'hue', min: 0, max: 360, def: 205 }, ], // extra: (state, out) => { out.hueRad = state.hue * Math.PI / 180; }, // derive more ext() values }); ``` Options: `name` (default `'webtoe'`), `app` (default `https://webtoe.openaudiovisual.com/`), `project`, `params`, `extra`. ### DOM, SVG, video Anything you append to `container` is fine. The stage only calls your functions. ## Several worlds in one show ```js const show = await createShow({ world: calm, worlds: [storm, after], timeline }); // switch on a cue: the old world is disposed, the new one initialized show.timeline.onSceneChange((i, scene) => { if (scene.id === 'storm') show.stage.activate('storm'); }); ``` All worlds' params are registered up front, so the console and the timeline can address any of them. `stage.activate(name)` does nothing if that world is already active. ## Sound output from a world `update(dt, state, io)` receives `io`. With `createShow()`, `io` holds only `signals`; add output handles once the show exists, and every world sees them: ```js const show = await createShow({ world }); show.stage.io.midi = show.midi; // the show's @openav/midi engine (MIDI out) // show.stage.io.osc = osc; // an OscOut you created (see Show control → OSC output) ``` Then, inside the world: ```js // trigger a sampler note when something happens in the simulation io.midi?.noteOn(48 + creature.species * 12, 90, 2); // note, velocity 0..127, channel 2 // stream a position to a spatializer io.osc?.send(`/source/${i}/xyz`, [x, y, z]); ``` Often it is simpler to do it at the assembly level instead, as example 05 does: `show.signals.on('midi/note/on', ({ note, vel }) => show.midi?.noteOn(note, Math.round(vel * 127), 1))`. For instruments inside the page (a grand piano, mallets, strings, organ, synth presets), use `modules.sound` ([Show control](show-control.md#sound)). ## Checklist before a show - [ ] every continuous control is a param (try performing your world with ONLY the timeline) - [ ] `dispose()` releases GPU objects and event subscriptions (worlds get switched live) - [ ] world survives `dt` spikes (the Loop clamps to 100ms, but don't assume 16ms) - [ ] nothing in the world reads the clock directly — the timeline owns time - [ ] the page loads with zero console errors, and works with only the on-screen piano --- # Inputs Every input does one thing: publish signals. With `createShow()` you switch inputs on in `modules`; each one is also a plain class you can use on its own. The exact names and ranges are in the [Signals reference](signals.md). | input | package | `modules` key | default | publishes | |---|---|---|---|---| | MIDI hardware | `@openav/midi` | `midi` | on (`false` turns it off) | `midi/…` | | on-screen MIDI controllers | `@openav/midi` | `midi: { controllers }` | off | `midi//` | | on-screen piano, QWERTY, simulated performer | `@openav/keys` | `keys` | on (`false` removes it) | `midi/note/on`, `midi/note/off` | | drum machine | `@openav/drums` | `drums` | off | `drum/…`, `midi/note/on` on channel 10 | | microphone | `@openav/audio` | `audio` | off | `audio/…` | | chord analysis | `@openav/chord` | `chord` | off | `chord/…` | | body (camera) | `@openav/pose` | `pose` | off | `pose/…` | | hands (camera) | `@openav/pose` | `hands` | off | `hand/…` | | hands (Leap Motion) | `@openav/leap` | `leap` | off | `leap/…` | | phones and iPads | `@openav/remote` | `remote` | off | `phone/…`, `surface/…`, `midi/note/…` | Sources that need permission or a user gesture (microphone, camera) are created by `createShow()` but only start when you click their button in the console's **L1 · Input** panel: 🎤 mic, 🖐 hands, 🕺 body. ## MIDI `createShow()` creates a `Midi` engine and calls `enable()` on startup, so the browser may ask for MIDI permission when the page opens. Devices can be plugged in and out during the show; the **L1 · Input** panel lists them, and unticking one mutes it without unplugging it. Every incoming message publishes two families of names: - **legacy** (what most worlds use, any channel): `midi/note/on` and `midi/note/off` pulses, `midi/cc/` (0..1), `midi/bend` (−1..1); - **per channel**: `midi/ch//cc/`, `midi/ch//note/` (velocity 0..1 while held, 0 on release), `midi/ch//bend`, `midi/ch//pressure`, `midi/ch//poly/`. A note-on payload is `{ note, vel, velocity, ch, device }`: `vel` is 0..1, `velocity` is the raw 1..127, `ch` is 1..16, `device` is the port's slug. With **two or more devices** connected, each also publishes under its own slug so their controls do not collide: `midi//cc/`, `midi//note/on`, `midi//note/off`, `midi//bend`. The slug comes from the port name (`Arturia MiniLab 3` → `arturia-minilab-3`; a second identical unit gets `-2`), so a device that drops out and comes back keeps its routes. The legacy names keep publishing too, so a route on `midi/cc/74` hears every device. Clock messages are ignored. Program change, transport and SysEx are not published (SysEx access is never requested). Using it without `createShow()`: ```js import { Midi } from '@openav/midi'; const midi = new Midi({ signals }); // options: signals, filterOut = 'IAC', requestAccess await midi.enable(); // false if the browser has no Web MIDI or permission was denied midi.devices(); // [{ slug, name, listening }] midi.setListening('nanokontrol2', false); ``` `filterOut` is a case-insensitive regular expression on input port names that are ignored. The default, `'IAC'`, keeps the macOS IAC bus out, because a show that sends MIDI out through IAC would otherwise hear itself. See [Troubleshooting](troubleshooting.md#a-daw-on-the-iac-bus-is-not-heard) if you want to receive from IAC. MIDI output is described in [Show control](show-control.md#midi-output-and-panic). ### Known controllers on screen `modules: { midi: { controllers: { profile: '' } } }` adds an on-screen picture of a known controller. Plug that controller in and the picture switches to it and moves with your hands; without one, play the picture with a mouse or a finger. Its controls publish `midi//` names, and `controllerRoutes()` writes the routes that connect them to params. The [MIDI controllers & embedding](controllers.md) chapter covers profiles, learning unknown devices, LED feedback and embedding; the supported devices are on the [controllers page](https://openaudiovisual.com/controllers/). ## On-screen piano, QWERTY and the simulated performer `@openav/keys` is on by default. It mounts a piano above the console with two checkboxes, *keyboard* (QWERTY capture) and *simulate performance*. Options (`modules.keys`): `base` (lowest note, default 48), `octaves` (default 2), `sim` (show the simulate checkbox, default `true`), `capture` (show the keyboard checkbox, default `true`). It publishes `midi/note/on` `{ note, vel, ch: 0 }` and `midi/note/off` `{ note, ch: 0 }`. Note the differences from hardware: `ch` is 0 and there is no `velocity` or `device` field. A world that only reads `note` and `vel` works with both. - **QWERTY** (after ticking *keyboard*): `A W S E D F T G Y H U J K O L P ;` play 17 notes a semitone apart, starting at `base`; Z/X move an octave (base stays between 24 and 84); Shift adds 20 to the velocity. While capture is on, the piano claims every letter key (`preventDefault`); Space, arrows and digits pass through to the console. - **The simulated performer** (`SimPlayer`) plays a pentatonic scale an octave above `base`: mostly steps, sometimes leaps and triads, real rests. From code: `show.keys.sim.toggle(true)`. ```js import { mountKeys, KeysPiano, SimPlayer, PIANO_KEYMAP } from '@openav/keys'; const keys = mountKeys(el, { signals, base: 48, octaves: 2 }); // → { piano, sim, update(dt), dispose() } // call keys.update(dt) every frame (it drives the SimPlayer) ``` ## Drum machine `modules: { drums: true }` adds a 16-step, four-lane sequencer (kick, snare, hat, clap) with patterns (*four on floor*, *breakbeat*, *half time*, *latin*, *sparse*), a tempo slider (60–180 BPM, default 112), swing and humanized timing. Tick *drum machine* to start it. It plays its own synthesized kit (no samples) and publishes: - `drum/kick`, `drum/snare`, `drum/hat`, `drum/clap`: pulses `{ level }`; - `drum//env`: a 0..1 envelope per lane that decays in about 0.12 s, ready to map straight onto a param; - `midi/note/on` `{ note, vel, ch: 10 }` with General MIDI notes (kick 36, snare 38, clap 39, closed hat 42). The shapes match what the microphone's drum detection publishes (`audio/kick`, `audio/kick/env`…), so a world mapped to a kick cannot tell a real bass drum from the machine. Example 06 routes both to the same params. Options (`modules.drums`): `engine` (another sound engine), `autoEnableEngine` (default `true`: the kit's audio engine is switched on by the first hit, no extra click). ## Microphone `modules: { audio: true }` creates an `AudioAnalyzer`; the microphone starts when you click 🎤 mic. (Example 06 writes `audio: 'mic'`; any truthy value works.) Echo cancellation, noise suppression and auto gain are turned off so the analysis hears the room as it is. Published every frame: `audio/rms` (smoothed loudness), `audio/peak`, `audio/band/low` (20–250 Hz), `audio/band/mid` (250 Hz–2 kHz), `audio/band/high` (2–8 kHz), `audio/centroid` (brightness), and the pulse `audio/onset` `{ rms }` with a 100 ms refractory time. Drum detection splits transients by band: `audio/kick` (20–120 Hz), `audio/snare` (150–800 Hz), `audio/hat` (6–14 kHz), each a pulse `{ level }` with a matching `…/env` envelope. > [!TIP] > Real-world levels rarely reach 1. Speech often sits around 0.05–0.2 RMS and > band energies are averages across many FFT bins. Narrow the route's window, > for example `{ source: 'audio/rms', target: 'energy', inMax: 0.3 }`. ```js import { AudioAnalyzer } from '@openav/audio'; const audio = new AudioAnalyzer({ signals, fftSize: 2048, smooth: 0.7 }); await audio.enableMic(); // or audio.enableElement(videoEl), or someNode.connect(audio.input()) // every frame: audio.update() → also returns { rms, peak, low, mid, high, centroid, onset, kick, snare, hat } ``` ## Chords `modules: { chord: true }` (or `{ chord: { window: 80 } }`) adds a `ChordDetector` and feeds it every `midi/note/on` and `midi/note/off`. It groups notes struck within `window` milliseconds (default 80) into one gesture and publishes, per gesture: - `chord/consonance` (−1..1), `chord/count` (number of notes), `chord/root` (lowest MIDI note); - `chord/event`, a pulse carrying the whole analysis: | field | meaning | |---|---| | `notes`, `count`, `root`, `vel`, `pcs` | sorted notes, how many, the lowest, mean velocity (0..1), pitch classes | | `consonance` | mean of pairwise interval weights, −1..1 (a single note is 1) | | `isConsonant`, `isDissonant` | two or more notes and consonance > 0.25 / < −0.15 | | `isTriad` | the chord type is major, minor, sus2, sus4, dim or aug | | `thirdsFraction` | share of adjacent gaps that are 3 or 4 semitones ("genuinely stacked thirds") | | `dissonanceLevel` | 0 none · 1 mild (a storm warning) · 2 severe (a true cluster) | | `chordType` | `single`, `major`, `minor`, `sus2`, `sus4`, `dim`, `aug`, `maj7`, `min7`, `dom7`, `halfdim7`, `maj9`, `min9`, `dom9`, `six`, `min6`, `cluster`, or `chord` | This is not a music-theory library; it is a performance-semantics layer built for a piece where consonance makes a garden flourish and clusters make it decay (example 02). `detector.analyze(notes, vels)` is a pure function you can call yourself. ## Body and hands `modules: { pose: true }` adds a `PoseTracker` (one body), `modules: { hands: true }` a `HandTracker` (two hands, 21 points each). The camera starts when you click 🕺 body or 🖐 hands. Both use MediaPipe Tasks Vision (0.10.14), loaded from jsDelivr with its model from Google's storage the first time you enable it; the inference runs on your machine and no video leaves it. The view is mirrored by default (`mirror: true`), like a mirror on stage, and y is flipped so 1 means *raised*. - **Body**: `pose/present`, `pose/hand/left/x|y`, `pose/hand/right/x|y` (wrists), `pose/hand/left/v`, `pose/hand/right/v` (wrist speed), `pose/hands/spread`, `pose/height` (nose height: crouch ↔ stand), `pose/lean` (−1..1). - **Hands**: per side, `hand//present`, `hand//x|y` (palm), `hand//pinch/index` and `hand//pinch/middle` (thumb tip to finger tip), `hand//spread` (index tip to pinky tip). Distances are divided by palm size, so moving toward the camera does not change a pinch. A closed pinch reads near 0: use `invert: true` on the route when closing should raise the value (example 03). Both trackers draw their skeleton on a 2D context: `tracker.skeleton(ctx, w, h)`. ## Leap Motion `modules: { leap: true }` adds a `LeapInput`. Browsers cannot read the sensor, so one small bridge per machine does: `node packages/leap/bridge/leap-bridge.mjs` (needs Ultraleap Hand Tracking installed; `--mock` gives two animated hands without a sensor; `bash packages/leap/bridge/install.sh` keeps it running at login on macOS). It serves `ws://127.0.0.1:6437` on loopback only. The console's **L1 · Input** panel shows which link is missing (bridge, tracking service, device) and draws the hands; its *simulate* button turns the mouse into hands. Signals: per side, `leap/hand//present`, `x|y|z` (palm in the interaction box, 0..1, y up, z toward you), `pinch`, `grab`, `roll|pitch|yaw` (radians), `speed`, fingertips `leap/hand///x|y|z`; `leap/hands`, `leap/both/distance`, `leap/status`; pulses `pinch-start|pinch-end`, `grab-start|grab-end` per side and `leap/both/pinch-start|pinch-end`, with hysteresis (a pinch starts above 0.86 and ends below 0.5). Joints for drawing the hand are on `leap.hands`. The same bridge answers old `leap.js` sketches on `/v6.json`, and serves raw hand lines on `/raw` (example 12, package README). ## Phones and iPads `modules: { remote: true }` lets phones join the show over the relay that `serve.js` hosts. Their sensors publish `phone//…` (tilt, acceleration, rotation, knocks, touch, light), mirrored to `phone/any/…`; the control surface publishes `surface//`; the phone's piano publishes `midi/note/on|off` like a keyboard. The show page itself can read its own motion sensors on a phone with `localSensors()`. Everything is in [Phones, relay & surfaces](remote.md). ## Your own input An input is anything that calls `signals.set()` or `signals.pulse()`. Here is the pointer as an instrument: ```js const { signals } = show; signals.define('pointer/x', { source: 'pointer' }); // continuous, 0..1 by default signals.define('pointer/y', { source: 'pointer' }); signals.define('pointer/tap', { kind: 'pulse', source: 'pointer' }); addEventListener('pointermove', (e) => { signals.set('pointer/x', e.clientX / innerWidth); signals.set('pointer/y', 1 - e.clientY / innerHeight); // 1 = top, like every other y }); addEventListener('pointerdown', (e) => signals.pulse('pointer/tap', { x: e.clientX, y: e.clientY })); show.mapper.addRoute({ source: 'pointer/x', target: 'hue', smooth: 0.25 }); ``` Name it source first (`breath/pressure`, `weather/wind`), declare its range, and everything downstream (meters, *learn*, routes, the phone surface) works without changes. --- # Mapping The mapping layer is where "what happened" becomes "what it means". The `Mapper` (`@openav/mapping`) is the only component that connects signals to params, and everything it knows is a list of routes. ## Routes A route is a plain object. Only `source` and `target` are required; `addRoute()` fills in the rest: | field | default | meaning | |---|---|---| | `source` | — | signal name, e.g. `midi/cc/74`, `hand/right/pinch/index`, `phone/any/tilt/x` | | `target` | — | param key, e.g. `bloom`, `sound/cutoff` | | `inMin`, `inMax` | `0`, `1` | the window of the signal that is used; values outside it are clamped | | `outMin`, `outMax` | the param's `min`, `max` | the range written into the param | | `curve` | `'linear'` | `linear`, `exp`, `log` or `smooth` | | `invert` | `false` | flip the normalized value (1 − x) before the curve | | `smooth` | `0` | glide time in seconds (0 = instant) | | `enabled` | (absent = on) | `false` mutes the route | | `id` | assigned | a number, used to remove the route | ```js show.mapper.addRoute({ source: 'midi/cc/74', target: 'bloom' }); show.mapper.addRoute({ source: 'phone/any/tilt/x', target: 'drift', inMin: -1, inMax: 1, smooth: 0.12 }); show.mapper.addRoute({ source: 'hand/right/pinch/index', target: 'tightness', curve: 'smooth', invert: true, smooth: 0.1 }); ``` In `createShow()`, the same objects go in `routes: [...]`. ## What happens to a value When a signal arrives, every enabled route whose `source` matches runs these steps, in order: 1. **Window**: `x = (value − inMin) / (inMax − inMin)`, clamped to 0..1. 2. **Invert**: if `invert`, `x = 1 − x`. 3. **Curve**: `x = curve(x)`. 4. **Scale**: `out = outMin + (outMax − outMin) · x`. 5. **Write**: with `smooth: 0`, the param is overridden with `out` (and clamped and snapped to its `step`). With `smooth > 0`, `out` becomes a target that `mapper.update(dt)` glides toward every frame. ## Curves | curve | formula | feel | |---|---|---| | `linear` | x | as the hand moves | | `exp` | x² | slow start, fast end: fine control near the bottom | | `log` | √x | fast start, slow end: fine control near the top | | `smooth` | x²(3 − 2x) | gentle at both ends (smoothstep) | An unknown curve name falls back to linear. ## Windows and ranges `inMin`/`inMax` say which part of the signal matters; `outMin`/`outMax` say where it lands. Typical uses: - a bipolar signal: `{ source: 'midi/bend', inMin: -1, inMax: 1 }`, or a phone tilt `{ source: 'phone/any/tilt/x', inMin: -1, inMax: 1 }`; - a quiet signal: `{ source: 'audio/rms', inMax: 0.3 }` uses the whole param range for the levels a voice actually reaches; - a knob that only sweeps part of a param: `{ source: 'midi/cc/1', target: 'hue', outMin: 180, outMax: 270 }`; - an inverted relation without `invert`: `outMin: 1, outMax: 0`. > [!WARNING] > `outMin`/`outMax` default to the target param's range *at the moment the route > is added*. If the param does not exist yet, they default to 0..1. In > `createShow()`, routes are added before the sound engine registers its > `sound/*` params, so a declared route to `sound/cutoff` needs explicit > `outMin: 100, outMax: 8000`. ## Smoothing `smooth` is a time constant in seconds: after `smooth` seconds the param has covered about 63% of the distance to the new value (a one-pole filter, `k = 1 − e^(−dt/smooth)` per frame). 0.05–0.15 hides the steps of a MIDI knob or a 30 Hz phone stream; 0.2–0.5 makes a hand feel heavy. The glide starts from the param's current value, so the first movement never jumps. ## Pulse targets When the target is a `pulse` param, the route fires it instead of writing a value: - a **pulse** source (`midi/note/on`, `audio/kick`, `drum/kick`, `phone/any/knock`…) fires it on every event; - a **continuous** source fires it when the normalized value crosses 0.5 upward (a phone-surface button going from 0 to 1, a fader pushed past the middle). `curve`, `invert`, `smooth` and the output range do not apply to pulse targets. ## Many-to-many One signal may drive many params (add several routes with the same `source`), and one param may listen to many signals. When two routes write the same param, the last one to write wins. *Learn* adds a route; it never replaces one. The [patch bay design](design/patchbay.md) plans per-route `combine` (`last`, `max`, `add`, `avg`) and `label` fields; they are not implemented yet. ## Overrides and the timeline A route writes its param as an **override**, and overrides are sticky: as soon as a mapped signal moves, the timeline stops driving that param. Clearing the override (✕ next to the param, or `show.params.clearOverride(key)`) hands it back to the timeline until the signal moves again. A *smoothed* route keeps writing its last target every frame once its signal has arrived, so its param stays overridden even after ✕ and even when the route is muted. Remove the route (✕ in **L2 · Mapping**) to give the param back to the timeline for good. ## Learn Click *learn* next to a param in **L3 · Params** (or call `mapper.learn('bloom')`), then move something. The mapper binds the first signal that: - **moves** by at least 5% of its declared range from where it was when learn started (a knob, a fader, a hand, a phone tilt), or - **fires**, if it is a pulse (a key, a pad, a knock). The new route has the defaults: linear, no smoothing, the param's full range. Edit it afterwards if it needs a curve. Clicking *learn* again cancels. Learn *adds*: a param that already has a route keeps it and gets a second one. To bind a whole list, the **L2 · Mapping** panel has **🎛 map knobs in order**: it arms learn for the first param, you turn one knob, it is bound and the next param is armed. *skip*, *back*, *stop* (or Esc) steer it. In code it is `LearnWizard` ([details](score.md#map-the-knobs-in-order)). > [!CAUTION] > Any pulse binds immediately. Untick *simulate performance* and stop the drum > machine before you learn, or the next note they play becomes the route. When one MIDI message publishes several names (`midi/cc/74`, `midi/ch/1/cc/74`, and with two devices `midi//cc/74`), learn binds the first one, `midi/cc/74`, which hears every device. Change the `source` to the per-device name if you need one device only. ## Muting and removing The **L2 · Mapping** panel lists every route as `source → target` with a checkbox (mute) and ✕ (remove). In code: ```js const r = show.mapper.routesFor('bloom')[0]; r.enabled = false; // mute show.mapper.removeRoute(r.id); // remove ``` ## Saved profiles The mapper saves its routes to `localStorage` under `openav.map.`. `createShow()` uses the world's name as the profile (or the `profile` option) and: 1. loads the saved routes if there are any, otherwise adds the declared `routes`; 2. when a saved profile exists, still adds every declared route whose `source → target` pair is missing, so new routes in code arrive; 3. saves every 3 seconds. So learned routes and edits survive a reload. The flip side: if you change the curve or smoothing of a declared route in code, the saved copy of that `source → target` pair wins. To start clean, pass `profile: false` (routes come from code on every load, as in example 09) or remove the saved profile: ```js localStorage.removeItem('openav.map.particles'); // 'particles' = the world's name ``` To carry a wiring to another machine or another world, use a map file ([next section](#map-files-export-import-and-share)). ## Map files: export, import and share The **L2 · Mapping** panel has three buttons under *map knobs in order*: - **⬇ export** downloads the routes as `oav-map-.json`. - **⬆ import** picks a map file. You can also **drop a `.json` anywhere on the page**. Import *replaces* the routes; the new set is remembered at once. - **clear** removes every route (click twice). On the next load the world's declared routes come back, as on a first visit. Below the buttons the panel says what the last import did: how many routes came in, and every route it **skipped** and why. A file that cannot be read says so and changes nothing. ### The format ```json { "format": "openav.map", "version": 1, "world": "particles", "exported": "2026-10-15T08:30:00.000Z", "routes": [ { "signal": "midi/cc/74", "param": "bloom", "curve": "exp", "range": { "in": [0, 1], "out": [0, 100] }, "smooth": 0.1, "invert": false }, { "signal": "audio/rms", "param": "energy", "curve": "linear", "range": { "in": [0, 0.3], "out": [0, 1] }, "smooth": 0, "invert": false, "enabled": false } ] } ``` Routes are named by `signal` and `param`, never by id, so a file travels between worlds: - a route whose **param** this world does not have is skipped and listed; - a route whose **signal** nothing in this show can produce (say `leap/palm/x` without a Leap Motion module) is skipped and listed. Signals that only appear when the first message arrives (`midi/…`, `surface/…`, `phone/…`, `osc/…`, `key…`, `take/…`) are kept and counted as *waiting*; - when the file's `world` is a different one, the `range.out` is dropped so each param keeps its own range (the `in` window, curve, smoothing and invert come along); - `version` newer than this build reads is refused with a message. A bare array from `mapper.toJSON()` (the old way) still imports, as version 0. - Files over 256 KB or with over 500 routes are refused. ### Load on open: `?map=` ``` https://openaudiovisual.com/examples/01-hello-particles/?map=https://example.com/my-map.json https://openaudiovisual.com/examples/01-hello-particles/?map=maps/stage-left.json ``` `?map=` fetches the file when the page opens and loads it like an import. It accepts a **same-origin** address (relative paths included) or an absolute **https** one; `http:` to another host, `data:`, `javascript:`, `file:` and addresses with a login are refused. A cross-site file must be served with CORS (`Access-Control-Allow-Origin: *`; raw GitHub and gists do this). The show starts regardless: if the address is refused, the fetch fails or the file is bad, the panel shows the reason and the routes stay as they were. An explicit `?map=` beats the remembered wiring, and what it loads is remembered like any import. In code, `createShow({ map: 'maps/a.json' })` does the same and `map: false` ignores `?map=`. ### Remembered for you The mapper saves its routes to `localStorage` (`openav.map.`) every 3 seconds and right after an import or clear, one slot per world name, so the next visit starts where you stopped. Every read and write is wrapped: in a private window or with storage blocked nothing breaks, the wiring just is not remembered. In code: ```js import { loadMapUrl, parseMap } from '@openav/mapping'; const file = show.mapper.exportMap(); // the object above const report = show.mapper.importMap(text); // text or object; { mode: 'add' } keeps current routes // → { ok, added, skipped: [{ signal, param, reason: 'param' | 'signal' }], waiting, dropped, crossWorld } await loadMapUrl(show.mapper, 'maps/a.json'); // what ?map= calls; never throws parseMap(text); // validate without touching a mapper ``` ## Routes written for you Several packages generate routes in this same shape: - `autoSurface(world.params)` (`@openav/surface`) builds a phone control panel and the `surface// → param` routes for it, with `smooth: 0.04`; `routesFromLayout(layout, params)` does the same for a hand-written layout ([Phones, relay & surfaces](remote.md#autosurface)). - `controllerRoutes(profile, { knob1: 'hue', pad1: 'burst' })` (`@openav/midi`) writes routes from an on-screen controller's controls ([MIDI controllers & embedding](controllers.md)). ## Mapper API ```js import { Mapper } from '@openav/mapping'; const mapper = new Mapper({ signals, params, profile: 'default', onChange, onLearn }); mapper.addRoute(route) // → the stored route (defaults filled in, id assigned) mapper.removeRoute(id) mapper.routesFor('bloom') // → routes targeting a param mapper.routes // the live array mapper.learn('bloom') // arm / cancel learn; mapper.learnTarget is the armed key mapper.update(dt) // every frame: advances smoothed routes mapper.toJSON(); mapper.fromJSON(array) mapper.exportMap(); mapper.importMap(text|object, { mode }) // the portable file, see above mapper.clear() // remove every route (and forget them) mapper.report // what the last import / ?map= said mapper.save(); mapper.load() // localStorage 'openav.map.'; load() → true if found mapper.dispose() // stop listening to signals ``` `onChange()` runs when routes change (for UIs); `onLearn(route, signalName)` runs when learn binds. --- # Show control The parts that turn a sketch into a performance: the one-call assembly, the timeline, the console, performance mode, the backstage monitor, and the ways sound and data leave the browser. ## createShow `createShow()` (`@openav/show`) builds a whole show around a world and returns every part. It is `async` because it waits for the world's `init`. ```js import { createShow } from '@openav/show'; const show = await createShow({ world: myWorld, timeline: { total: 120, automation: { hue: [[0, 205], [60, 320]] }, scenes }, routes: [{ source: 'audio/rms', target: 'energy', inMax: 0.3, smooth: 0.1 }], modules: { keys: { base: 48 }, chord: true, audio: true, sound: true }, artwork: { title: 'Untitled', artist: 'You', year: 2026 }, hint: 'play the piano · Space plays the timeline', }); ``` | option | default | meaning | |---|---|---| | `world` | `null` | the world to perform; it is activated first | | `worlds` | `[]` | more worlds to register (switch with `show.stage.activate(name)`) | | `timeline` | `{ total: 120 }` | `{ total, automation, scenes }` for the [timeline](#timeline) | | `routes` | `[]` | routes to add to the mapper ([Mapping](mapping.md)) | | `modules` | `{}` | inputs and outputs to switch on (below) | | `artwork` | `null` | `{ title, artist, year, note }`: a credit line on the stage and a `` | | `hint` | `''` | a line of HTML at the bottom left of the stage | | `map` | `?map=` in the page address | a map file to load on start: a same-origin or https URL; `false` ignores `?map=` ([Map files](mapping.md#map-files-export-import-and-share)) | | `profile` | the world's name | the mapper's saved-profile name; `false` = ignore saved routes ([Saved profiles](mapping.md#saved-profiles)) | | `onFrame` | `null` | `(dt, show) => {}` every frame, before the timeline advances | | `mount` | generated | `{ stage, side }` elements or selectors to render into, instead of a full-page layout | | `telemetry` | `true` | one anonymous `oav_show_start` hit per page (see the controllers chapter, Telemetry); `false` sends nothing | | `score` | `null` | `{ cuts, cut, modules, api, midi, … }`: the show's structure as segment lengths, performed by segment modules; `?cut=` in the URL picks the version ([Score & Director](score.md#use-it-with-createshow)) | Without `mount`, the page body becomes a two-column grid: the stage, and a 340 px side panel with the piano and the console. | `modules` key | values | what it adds | |---|---|---| | `midi` | on by default; `false` | the `Midi` engine (input and output) | | `midi.controllers` | `true` or `{ profile, profiles, open }` | an on-screen MIDI controller docked on the stage, a 🎹 button and the M key; `?profile=` in the URL picks the device ([controllers](controllers.md)) | | `keys` | on by default; `false`; `{ base, octaves, sim, capture }` | the on-screen piano, QWERTY and the simulated performer | | `drums` | `true`; `{ engine, autoEnableEngine }` | the drum machine | | `audio` | any truthy value | the microphone analyzer (starts from 🎤 mic) | | `chord` | `true`; `{ window }` | chord analysis of every played note | | `hands` | `true` | the hand tracker (starts from 🖐 hands) | | `pose` | `true` | the body tracker (starts from 🕺 body) | | `remote` | `true`; `{ room, auto, surface, feedbackHz, url }` | phones and iPads as controllers ([Phones](remote.md#the-show-side)) | | `sound` | `true`; an instrument id (`'piano'`); `{ instrument, remember, picker, engine, baseUrl }` | instruments in the page with a picker, Tone.js ([Sound](#sound)) | It returns `{ signals, params, stage, timeline, mapper, midi, controllers, midiPanel, keys, drums, sound, audio, hands, pose, chord, remote, score, director, console, app, loop }`; parts you did not ask for are `null`. The same object is `window.openav`, for the devtools console. The frame order is in [Architecture](architecture.md#the-frame-loop). ## Timeline ```js const timeline = new Timeline({ params, total: 2400, // seconds (default 600 when used directly; createShow uses 120) automation: { growth: [[0, 0.2], [300, 1.0, 'ease'], [1200, 0.4, 'out']], // [t, value, curve?] keyframes; no curve = linear mode: [[0, 0], [600, 1], [1800, 2]], // step params hold (no interp) }, scenes: [ { id: 'standby', t: -30, title: 'Standby', note: 'house lights · breathe' }, { id: 'act1', t: 0, title: 'Act I', note: 'single seeds only' }, { id: 'act2', t: 600, title: 'Act II', note: 'full harmony' }, ], }); timeline.onSceneChange((i, scene) => { /* fire cues: sound, lights, OSC */ }); ``` - negative scene times = pre-show standby (the clock shows -0:30) - `jumpScene(±1)` (← → keys) is your emergency navigation — rehearse with it - `timeline.rate` exists for rehearsal speed-through (`rate = 4`) - playback stops by itself at `total` - a scene with `hold: true` stops playback at its end until it is released (Space, →): see [Hold](score.md#hold) - `loop: true` wraps the show at `total` and plays on; `loop: 'scene'` repeats the scene the playhead is in until you move on: see [Curves, loop, cues, recording](#curves-loop-cues-recording) - `reset()` goes to `start` (default 0, so a pre-show standby scene at −30 is reached with the scrubber; pass `start: -30`, or `start: 'first'` for "the first scene", to make R return to it) - with `score` the scenes, total and start come from a [score](score.md); then automation is read at `score.baseTime(t)`, so a cut derived with `scale: 0.5` stretches it | member | what it does | |---|---| | `play()`, `pause()`, `toggle()` | transport. At a hold, `play()` and `toggle()` release it | | `seek(t)` | jump to `t` seconds (clamped between the first scene's negative time, or 0, and `total`) | | `jumpScene(d)` | go to the start of the scene `d` steps away | | `next()`, `prev()` | → and ←: `next()` releases a hold (and plays on), otherwise it is `jumpScene(1)` | | `release()` | let a hold go: the next scene starts and playback continues. `false` when nothing was holding | | `reset()` | stop, clear a hold and rewind to `start` (default 0) | | `advance(dt)` | per frame; moves `t` by `dt × rate` while playing | | `state(t)`, `valueAt(key, t)` | the automation values at a time (no overrides) | | `sceneIndexAt(t)`, `currentScene(t)`, `sceneEnd(i)` | scene lookup | | `onSceneChange(cb)` | `cb(index, scene, { cause })` whenever playback or a seek lands in a different scene; `cause` is `'play'`, `'release'`, `'seek'`, `'jump'`, `'reset'` or `'loop'` | | `onCue(cb)`, `cues()`, `fireCue(cue)` | scene cues fired by playback, the list of them, and the manual trigger | | `loop`, `onLoop(cb)`, `loopRange()`, `loops` | wrap the show, a scene or a range | | `record(key, source)`, `stopRecording()`, `onRecord(cb)`, `mergeTake(key, take)` | write a value source into a lane | | `onSeek(cb)` | `cb(t, kind)` when the playhead is moved by hand (`'seek'`, `'jump'`, `'reset'`), never while playing | | `layer` | `(key, value, t) => value`: rewrite a param before overrides (the director's `param()` plugs in here) | | `t`, `playing`, `holding`, `rate`, `total`, `start`, `scenes`, `automation`, `score` | state you can read (and set) | ### Curves, loop, cues, recording **Curves.** A keyframe is `[t, value, curve?]`. The third item says how the segment that *ends* at that keyframe gets there: `'linear'` (the default: leave it out and you get what the timeline always did), `'ease'` (slow, fast, slow), `'in'` (starts slow), `'out'` (starts fast), `'step'` (hold the old value until the keyframe, then jump), or your own function of 0..1. `CURVES` is exported. A param with `step` still holds, whatever the curve says. ```js automation: { speed: [[0, 0.3], [16, 1.6, 'in'], [20, 3.4, 'step'], [30, 0.8, 'out']] } ``` **Loop.** `loop: true` wraps at `total` (back to 0) and keeps playing; `'scene'` repeats the scene the playhead is in (`next()` moves the loop on to the next scene); a scene id loops that scene; `{ from, to }` loops a range in seconds or scene ids. It is a property: `timeline.loop = !timeline.loop` is the whole L key. `onLoop(cb)` tells you each lap, and a loop that wraps into another scene is an `onSceneChange` with `cause: 'loop'`. A `hold` inside the range still holds. **Cues.** `scene.cues` is `[{ name, t }]` (the console scrubber already draws them as ticks). `onCue((cue, { cause }) => …)` fires one when *playback* passes it, once, and never when you seek, jump, drag the scrubber or reset: a scrub is not a performance. Each lap of a loop fires them again. `fireCue(cue)` sets one off by hand. With a [score](score.md) the director fires its own cues; use `onCue` on plain timelines. ```js show.timeline.onCue((cue) => show.params.firePulse(cue.name)); // scenes: [{ …, cues: [{ name: 'flash', t: 20 }] }] ``` **Recording.** `record(key, source)` arms a lane. While the timeline plays, `source()` is read every frame: a number is written into a take, `null` means nobody is touching and closes it. ```js timeline.record('hue', () => params.isOverridden('hue') ? params.overrides.hue : null); timeline.onRecord(({ key }) => params.clearOverride(key)); // the curve takes over again ``` A take also closes on pause, seek, reset, a hold, the end of the show and the end of each loop lap, while the recorder stays armed until `stop()` / `stopRecording()`. The merge rule (`mergeTake()`): the take is thinned (a point stays when it moved more than 0.6 % of the param's range, or 0.5 s passed), and it **replaces the keyframes inside its time span**. Keyframes outside stay, so the lane glides from the previous keyframe to the take's first point, and from its last point to the next keyframe. Existing curves outside the take are untouched. With a score the keyframes are written in the base cut's time, like the ones you declared. A cue list is just a scene-change handler: ```js show.timeline.onSceneChange((i, scene) => { if (scene.id === 'storm') show.stage.activate('storm'); // switch worlds show.midi?.cc(20, i, 16); // tell the lighting desk }); ``` ## The console `mountConsole(el, app, { layers, signals })` (`@openav/console`) builds the director's desk; `createShow()` mounts it in the side panel. `app` is the object `createShow()` returns as `show.app` (`timeline`, `params`, `mapper`, `signals`, `stage`, `midi`, `sound`, `audio`, `hands`, `pose`, …); missing parts simply hide their section. Pass `{ layers: false }` or `{ signals: false }` to leave out those panels. It gives you transport + scrubber with scene blocks, the Layers overview, L1 Input, L2 Mapping, L3 Params (sliders with override and learn chips), L4 sound, live signal meters and a MIDI log. Keyboard: **Space** play/pause (at a hold: release) · **→/←** next / previous scene (→ at a hold: release) · **R** reset · **T** performance mode · **F** fullscreen · **Esc** leave keyboard-piano capture or stop the mapping wizard. Keys typed into an input field, held keys and ⌘/Ctrl/Alt combinations are ignored. While the on-screen piano captures the keyboard the *letters* belong to the piano (T and F are piano keys), so R, T and F do nothing until you untick it or press Esc; Space and the arrows always work. With a score the desk also has the Director panel, cue ticks on the scrubber and a prompter with the operator's acts ([Score & Director](score.md#the-desk)). The L2 Mapping panel has **🎛 map knobs in order**, a wizard that arms learn for each param in turn ([details](score.md#map-the-knobs-in-order)). Override semantics: touch a slider (or a mapped control) and that param leaves the timeline's hands until you clear it (✕). Yellow label = overridden. ## Performance mode **T** flips to a fullscreen teleprompter: current scene title + note in stage- readable type, what the operator does (the scene's `acts`), next scene preview, clock + scene countdown, a HOLD banner while the show waits, and a warning when a segment module was switched off. This is for the *performer*; the audience-facing window is your stage in fullscreen (**F**). Two windows of the same page = one desk, one stage. ## Backstage monitor ```bash node packages/monitor/server.js # on the performance machine (npm run monitor) # stage manager's phone/laptop: http://:7457/ ``` Every show page made with `createShow()` streams snapshots to `ws://:7457` about 15 times a second, so run the monitor server on the machine that serves the show. Backstage shows clock, scene, world, FPS, overrides, all signals and params live, and a red "stage feed lost" banner if the stage stops sending for 4 s — which is exactly the moment a stage manager earns their pay. Zero configuration, zero dependencies; both ends reconnect by themselves. `GET /health` answers `{ ok, clients }`. When no monitor is running, the show page keeps retrying with a growing delay (about 5 s, growing to 30 s); the browser logs each refused connection. That message is expected. For a hand-assembled show: ```js import { MonitorFeed, snapshotOf } from '@openav/monitor'; const monitor = new MonitorFeed({ url: 'ws://192.168.1.20:7457', hz: 15 }); // both optional monitor.connect(); // every frame: monitor.frame(snapshotOf({ timeline, params, signals, stage, loop }, state)); ``` ## OSC output ```bash node packages/osc/bridges/osc-bridge.js 7456 192.168.1.50 3456 # http port ↑ target host ↑ udp port ↑ ``` The defaults are `7456 127.0.0.1 3456`. The browser cannot send UDP, so `OscOut` posts batches to the bridge over HTTP and the bridge sends them as OSC packets. ```js import { OscOut } from '@openav/osc'; const osc = new OscOut(); // http://127.0.0.1:7456 await osc.enable(); // probes /health, fails gracefully (false) osc.send('/source/3/xyz', [x, y, z]); osc.flush(); // once per frame (batches, dedupes per address) ``` Numbers go out as OSC floats — what Spat, Reaper, and TouchDesigner expect; anything else is sent as a string. Only the latest message per address survives a frame. If the bridge disappears, `OscOut` reports it through `onStatus(ok, info)` and stops trying after 30 failed flushes. `createShow()` does not create an `OscOut`. Add one yourself (and add `"@openav/osc": "../../packages/osc/index.js"` to the page's import map): ```js const osc = new OscOut(); await osc.enable(); const show = await createShow({ world, onFrame: () => osc.flush() }); show.app.osc = osc; // the Layers panel shows the OSC rate show.stage.io.osc = osc; // worlds can send from update(dt, state, io) ``` ### TouchDesigner as the receiver OSC goes to TouchDesigner as above. TouchDesigner can also show an open-audiovisual page as a picture with its Web Render TOP, which draws a web page inside TouchDesigner on the same machine ([Web Render TOP](https://docs.derivative.ca/Web_Render_TOP)). We have not checked whether the Web Render TOP supports Web MIDI or WebGPU. Keep the control on OSC. If the page comes from openaudiovisual.com, Chrome asks before the page may reach the bridge on `127.0.0.1`: see [Chrome asks about the local network](troubleshooting.md#chrome-asks-about-the-local-network-or-about-apps-on-this-device) ([Chrome blog](https://developer.chrome.com/blog/local-network-access)). Running `node serve.js` and opening the page from `http://localhost:8080` avoids the prompt. ## MIDI output and panic `midi.noteOn/noteOff/cc/send` go to the selected output port — and `midi.panic()` sends all-notes-off + all-sound-off + sustain-off + per-note note-offs on all 16 channels, through the same observable throat as everything else. Bind it to a pad you can find in the dark. Every performer eventually needs it; ours is one call. ```js show.midi.outputs.map((o) => o.name); // available output ports show.midi.selectOutput('IAC Driver Bus 1'); show.midi.noteOn(60, 100, 1); // note, velocity 0..127, channel 1..16 show.midi.noteOff(60, 1); show.midi.cc(74, 64, 1); // controller, value 0..127, channel show.midi.send([0xb0, 74, 64]); // raw bytes show.midi.panic(); ``` The first output port is used until you choose another (`enable(preferredOut)` matches a name exactly, then as a substring). Everything sent passes through `onSend`, which feeds the per-channel MIDI OUT meter in the Layers panel. **Hot-plug.** A cable pulled mid-show must not leave the DAW silent without a word. When the current output is unplugged, `Midi` moves to the next best live port: the one you chose with `selectOutput()` or `enable(name)`, then the first match of the `prefer` list (`new Midi({ prefer: ['IAC'] })`), then the first port. When the port you chose is plugged in again, the output goes back to it. An unrelated device appearing never steals the output. Every switch prints one line in the MIDI log (the Signals panel, via `midi.onMessage`) and one in the browser console, flashes in the MIDI controller panel, and updates the `MIDI out →` row of the L1 Input panel. In code: `midi.onOutput(({ text, reason }) => …)`. **Echo.** What you send into a virtual bus must not come back as input. Inputs whose name matches `filterOut` (default `IAC`, the macOS bus) are never listened to, and a message identical to one just sent that arrives on any input within `echoWindow` milliseconds (default 40; `0` turns it off) is dropped as echo, once per message sent. `midi.echoDropped` counts them. To make an external program follow the show, `ScoreMidi` sends the score's segment changes and cues as notes and CCs ([Score to MIDI](score.md#score-to-midi)). ## Sound `modules: { sound: true }` adds `Sound` (`@openav/sound`) with the Tone.js engine. Browsers only start audio after a click, so the console shows **🔊 enable sound** in the *L4 · Output — sound* panel, with an instrument picker under it. Once enabled, every `midi/note/on` and `midi/note/off` plays the chosen instrument, whatever sent it (a keyboard, the QWERTY piano, the simulated performer, a phone). Tone.js (15.0.4) is loaded from jsDelivr only when sound is enabled. ```js await createShow({ world, modules: { sound: true } }); // the warm pad, as before await createShow({ world, modules: { sound: 'piano' } }); // start on the grand piano await createShow({ world, modules: { sound: { instrument: 'synth/pluck', remember: false, picker: false } } }); ``` ### Instruments | id | sound | made of | |---|---|---| | `piano` | grand piano | the Salamander Grand Piano: 30 samples (A, C, D♯, F♯ of every octave), 2.0 MB, loaded when picked | | `epiano` | electric piano | two FM pairs and a stereo tremolo | | `xylophone`, `marimba`, `vibraphone`, `glockenspiel`, `music-box` | mallets | modal synthesis: a few decaying partials at each bar's own ratios, plus a mallet click | | `strings` | string ensemble | detuned saws, bowed attack, chorus, vibrato | | `harp` | plucked string | Karplus–Strong | | `organ` | drawbar organ | nine drawbars as one additive waveform, rotary tremolo | | `choir` | choir pad | saws through vowel formants | | `synth/pad` | warm pad | the original voice, still the default | | `synth/saw-lead`, `synth/square-lead`, `synth/pluck`, `synth/supersaw`, `synth/bass`, `synth/acid`, `synth/brass`, `synth/bell`, `synth/sub`, `synth/chip` | synth presets | one poly synth, presets as data | The piano samples are *Salamander Grand Piano* by Alexander Holm, licensed [CC-BY 3.0](https://creativecommons.org/licenses/by/3.0/). The picker shows that credit next to the piano, and a show that plays it in public credits it too. They ship in `packages/sound/samples/salamander/`. If they cannot load, the piano plays the electric piano instead of going silent. ### Choosing a sound Pick from the console, or from code: ```js await show.sound.setInstrument('organ'); // true once it plays show.sound.instrument; // 'organ' show.sound.onChange((e) => console.log(e.type, e.id, e.progress)); // instrument · loading · ready · error show.timeline.onSceneChange((i, scene) => { if (scene.id === 'chorale') show.sound.setInstrument('organ'); // a cue may switch instruments }); ``` Switching while you play releases the notes held on the old instrument, so nothing hangs. A sampler keeps the old sound playing until its files are in, and the picker shows the progress. Each switch also fires the pulse signal `sound/instrument` (`{ id, name }`). The picker's last choice is remembered per page and wins over the declared instrument on the next visit; `remember: false` turns that off. The instrument is a discrete choice. What you perform continuously stays a param: the engine's params join the show's params under `sound/`. `sound/cutoff` (100–8000 Hz, default 2500), `sound/space` (reverb wet, 0–1, default 0.3) and `sound/volume` (−36–0 dB, default −8) are shared by every instrument, so a knob, a hand or the score plays the filter the way it plays the visuals. An instrument may suggest starting values (the piano opens the filter to 8000 Hz) and add knobs of its own, which the console shows while it plays: the organ's `sound/drawbars` and `sound/rotor`, `sound/attack` and `sound/vibrato` on the strings, `sound/vowel` on the choir, `sound/tremolo` on the electric piano and the vibraphone. Automation and overrides always win over a suggested default. The sustain pedal (`midi/cc/64`) holds every instrument's note-offs. Notes on MIDI channel 10 (the drum machine) are left to a drum engine. ### Add your own instrument A sound you can pick is one file: data plus a `create(Tone, ctx)` that returns `noteOn`, `noteOff` and `dispose`. Import it before `createShow()` and every picker lists it; nothing in `packages/*` changes. ```js import { registerInstrument } from '@openav/sound'; registerInstrument({ id: 'kalimba', name: { en: 'Kalimba', zh: '拇指琴' }, category: 'mallets', params: [{ key: 'decay', label: 'Decay (s)', min: 0.2, max: 4, def: 1.2 }], // → sound/decay create(Tone, { output }) { const synth = new Tone.PolySynth(Tone.FMSynth, { harmonicity: 5.1, modulationIndex: 2, envelope: { attack: 0.001, decay: 1.2, sustain: 0, release: 1 } }).connect(output); const hz = (n) => 440 * 2 ** ((n - 69) / 12); return { noteOn: (note, vel, time) => synth.triggerAttack(hz(note), time, vel), noteOff: (note, time) => synth.triggerRelease(hz(note), time), set: (key, v) => key === 'decay' && synth.set({ envelope: { decay: v } }), dispose: () => synth.dispose(), }; }, }); ``` A duplicate id throws unless you pass `registerInstrument(def, { replace: true })`. Optional fields: `defaults` (starting `cutoff`, `space`, `volume`), `credit`, `gain` (dB), `transpose` (semitones), `fallback` (an id to play if loading fails) and, on the returned object, `ready` (a Promise: the picker shows progress while you call `ctx.onProgress(0..1)`), `set(key, value)` and `releaseAll(time)`. Mallet sounds can be pure data with `modalInstrument({ id, name, partials, strike })`. A synth preset is data as well, and becomes `synth/`: ```js import { registerSynthPreset } from '@openav/sound'; registerSynthPreset({ id: 'glass', name: { en: 'Glass', zh: '玻璃' }, voice: 'fm', harmonicity: 3.01, modulationIndex: 14, envelope: { attack: 0.002, decay: 1.4, sustain: 0, release: 1.2 }, effects: [{ type: 'chorus', frequency: 1.5, depth: 0.4, wet: 0.3 }], }); ``` ### Your own engine An engine is a small object, so you can replace the whole thing: ```js const sineEngine = () => { let ctx = null, out = null; const voices = new Map(); return { params: [{ key: 'gain', label: 'Gain', min: 0, max: 1, def: 0.3 }], // → sound/gain async enable() { ctx = new AudioContext(); out = ctx.createGain(); out.gain.value = 0.3; out.connect(ctx.destination); }, noteOn(note, vel) { voices.get(note)?.stop(); const o = ctx.createOscillator(), g = ctx.createGain(); o.frequency.value = 440 * 2 ** ((note - 69) / 12); g.gain.value = vel * 0.3; o.connect(g).connect(out); o.start(); voices.set(note, o); }, noteOff(note) { voices.get(note)?.stop(); voices.delete(note); }, set(key, value) { if (key === 'gain' && out) out.gain.value = value; }, dispose() { ctx?.close(); }, }; }; await createShow({ world, modules: { sound: { engine: sineEngine() } } }); ``` The drum kit from `@openav/drums` is also an engine: `drumEngine({ samples })` maps General MIDI drum notes to synthesized drums (or to your samples, `{ 36: 'kick.wav', … }`) and has one param, `kitVolume` (−30–0 dB). ## Without createShow `createShow()` is a convenience. When you need your own layout or loop, assemble the parts by hand; this is roughly what it does: ```js import { Signals, Params, Loop } from '@openav/core'; import { Midi } from '@openav/midi'; import { Mapper } from '@openav/mapping'; import { Timeline } from '@openav/timeline'; import { Stage } from '@openav/stage'; import { mountConsole } from '@openav/console'; import { MonitorFeed, snapshotOf } from '@openav/monitor'; const signals = new Signals(); const params = new Params(); const stage = new Stage({ container: document.querySelector('#stage'), params, signals }); stage.register(myWorld); await stage.activate(myWorld.name); const timeline = new Timeline({ params, total: 120, automation, scenes }); const mapper = new Mapper({ signals, params, profile: myWorld.name }); if (!mapper.load()) routes.forEach((r) => mapper.addRoute(r)); const midi = new Midi({ signals }); midi.enable(); // mount the console after every world (and sound engine) is registered: it builds one row per param const app = { timeline, params, mapper, signals, midi, stage }; const desk = mountConsole(document.querySelector('#desk'), app); const monitor = new MonitorFeed(); monitor.connect(); const loop = app.loop = new Loop((dt) => { timeline.advance(dt); mapper.update(dt); const state = stage.frame(dt, timeline.state()); desk.render(state); monitor.frame(snapshotOf({ timeline, params, signals, stage, loop }, state)); }); loop.start(); ``` ## Pre-show checklist - [ ] Chrome, plugged in, screen-sleep off, notifications off (`chrome://settings`) - [ ] WebMIDI permission granted for this origin (it's per-origin, per-port) - [ ] mapping profile saved (mapper.save()) AND exported to a JSON file - [ ] monitor server running; stage manager's device on the same network - [ ] sound enabled with a click, and the panic pad mapped - [ ] rehearse the failure: kill the tab, restart, be performing again in <30 s --- # Score & Director A show that outgrows one afternoon needs a structure that can grow. The **score** (`@openav/score`) writes a show down as segments with *lengths*, and the **director** runs the code that performs each segment. They are pure logic with no DOM, and `createShow()` wires them to the [timeline](show-control.md#timeline), the console and the prompter for you. ## The idea: lengths, never seconds A timeline written as a table of absolute seconds cannot grow. Make the second scene longer and every later scene, cue, curve and `if (t > 466)` in your code is now wrong, and you find them one rehearsal at a time. In a score you write only how *long* each segment is: ```js const cuts = [ { id: 'full', title: 'A day', segments: [ { id: 'standby', title: 'Standby', dur: 6, hold: true, acts: ['Check sound', 'Press Space when the room is ready'] }, { id: 'dawn', title: 'Dawn', dur: 24, cues: { birds: 0.3 } }, { id: 'noon', title: 'Noon', dur: 30, cues: { flare: { s: 8 }, peak: 0.5 }, acts: ['Open the haze slowly'] }, { id: 'dusk', title: 'Dusk', dur: 24, linger: 8 }, { id: 'night', title: 'Night', dur: 36 }, ] }, { id: 'short', from: 'full', scale: 0.5 }, // the same show at half the length ]; ``` Every start, end, cue time, countdown and scene is computed from those lengths. Change `noon` to `dur: 40` and everything after it moves by ten seconds while everything before it stays put. A second version of the show is a few lines of data, not a second timeline to keep in step. ## A cut A cut is one version of the show. It has an `id`, an optional `title`, and either segments or a base cut and a scale. | cut field | meaning | |---|---| | `id` | unique among the cuts; no `.` | | `title` | shown in the cut picker | | `segments` | the list below, in playing order | | `from`, `scale` | a derived cut: the cut named `from` with every length times `scale` (`scale` > 0). It can be derived again; the scales multiply | | `start` | the id of the segment the show opens on and `reset` returns to (default: the first) | | segment field | meaning | |---|---| | `id` | unique within the cut; no `.`. It is also the id of the module that performs it | | `title`, `note` | what the desk and the prompter show | | `dur` | length in seconds, greater than 0. The only time a segment has | | `cues` | named moments inside the segment: a ratio `0 ≤ r < 1` (a fraction of the segment) or `{ s: seconds }` (seconds into it) | | `acts` | what the human operator does in this segment: an array of strings, shown on the desk and, large, on the prompter | | `linger` | seconds the segment's module keeps running after the segment ends (a decay, a tail). Default 0 | | `hold` | playback stops at the end of this segment and waits for the operator ([Hold](#hold)) | | `module` | the name of the module that performs it, when that is not the segment's own id | A derived cut scales `dur`, `linger` and `{ s }` cues, so a cue stays in the same place in the music. The ratio cues did not need to move. A cut is checked when the score is built. Duplicate ids, a `dur` of 0 or below, a `from` that is not a cut, a `scale` of 0 or below, a cut derived from itself, a cue outside its segment, `acts` that are not strings: each throws a `ScoreError` that names the cut, the segment and the reason, and lists every problem at once. Nothing half-valid can reach the stage. ## The Score ```js import { Score } from '@openav/score'; const score = new Score({ cuts, cut: 'full' }); // cut defaults to the first ``` | member | what it gives | |---|---| | `table()` | one row per segment: `{ index, id, title, start, end, dur, linger, hold }` | | `total` | the show's length: the sum of the segment lengths | | `segmentAt(T)`, `indexAt(T)` | the segment playing at show time `T`. A segment owns `[start, end)`; before 0 answers the first, after the end the last | | `cueTime('dusk')` | a moment in show seconds: `'dusk'` is the segment's start, `'dusk.end'` its end, `'dusk.glow'` a cue. `NaN` for anything unknown | | `cuesBetween(T0, T1)` | the cues a playhead passed from `T0` to `T1`: those with `T0 < time ≤ T1`, in time order. Nothing when `T1 ≤ T0` | | `cues()` | every cue of the cut in time order, each with its number `n` | | `scenes()` | the segments as a `scenes` array for the [timeline](show-control.md#timeline) (with `acts`, `hold` and `cues`) | | `validate()` | throws a `ScoreError` when anything is wrong; returns `true` otherwise (the constructor already calls it) | | `cut`, `title`, `segments`, `startT` | the active cut, and where `reset` goes | | `cuts` | every cut as `{ id, title, total, from, scale, base }`, for a version picker | | `scale`, `base`, `baseTime(T)` | the cut's total scale against the cut it was derived from, that cut's id, and `T` expressed on it | ### Automation on a derived cut Automation curves are written once, against the cut they were derived from, and a derived cut stretches them. `timeline.valueAt(key, t)` reads the curve at `score.baseTime(t)`, which is `t / scale`. On the short cut above, `t = 20` reads the curve at 40. A cut that is not derived has scale 1 and nothing changes. If two cuts differ in structure and not just in length, do not share automation between them. Drive the params from the segment modules instead ([`param()`](#segment-modules)), where `ctx.p` already scales for free. ## Cues A cue is a named moment: `cues: { peak: 0.5 }` in segment `noon` makes `noon.peak`, half way through. Three places see it: - the scrubber draws a tick for it, and a click on the tick jumps there; - the director calls `onCue({ name, cue, T, n, segment })` once each time the playhead *passes* it while playing, and calls the active module's `cue(name, ctx)` hook; - `ctx.cue('peak')` inside a module tells you where it sits, as a progress you can compare with `ctx.p` (`'noon.peak'` works for another segment's cue, `ctx.cueT(name)` answers in show seconds). A seek does not pass the cues it jumps over, and playing back over a cue fires it again. A cue sitting exactly on the first instant of the show fires on the first frame. ## Hold `hold: true` means: *when playback reaches the end of this segment, stop and wait.* A standby that waits for the room to be ready is the typical use. Exactly what happens: - The timeline stops a microsecond before the boundary, so the show is still *inside* the held segment, and sets `timeline.holding = true` and `timeline.playing = false`. The clock freezes there for as long as you like. - The held segment's module keeps getting `update()` every frame, with `ctx.p === 1`, `ctx.holding === true` and `ctx.held`, the number of seconds waited so far. Use `ctx.held` as the clock for anything that should keep breathing while the show is frozen. (This is the choice made: the show *stops* at the end of the segment rather than looping inside it. Looping would send `p` back to 0, fire the cues again and break "every module enters once, exits once".) - **Space**, **→**, the ▶ button, `timeline.play()` and `timeline.next()` all release it: the next segment starts at once and playback continues. `timeline.release()` does the same in code. - A seek (scrubber, `goto`, ←) or a reset clears the hold, and a seek never *stops* at a hold: only playback does. The last segment never holds (the show ends there). - The backstage monitor shows `⏸ HOLD`; the desk and the prompter show a pulsing banner. ## Segment modules A segment module is the code that performs one segment. Its `id` is the id of the segment (or the segment's `module` name): ```js const dusk = { id: 'dusk', enter(api, ctx) { api.sky.caption = 'dusk'; }, // the segment began, or a seek landed inside it update(api, ctx) { /* every frame, playing or not */ }, exit(api, ctx) { api.sky.caption = ''; }, // it ended (after its linger) or a seek left it: give back what you took param(key, value, ctx) { return key === 'sun' ? 1 - ctx.p : value; }, // optional: bend a timeline param while this segment is current cue(name, ctx) { /* optional: one of this segment's cues was just passed */ }, reset(api) { /* optional: R was pressed; forget everything */ }, }; ``` `enter`, `update` and `exit` get the `api` object first (yours, see below). Every hook gets a `ctx`: | `ctx` | meaning | |---|---| | `p` | progress through the segment, 0 to 1. Above 1 during `linger`; exactly 1 while holding | | `t` | seconds into the segment (`p × dur`) | | `dt` | seconds since the previous frame | | `dur` | the segment's length in seconds | | `T` | show time in seconds | | `segment` | the segment: `id`, `title`, `start`, `end`, `dur`, `cues`, `acts`, `hold`, `linger`, `index` | | `cue(name)`, `cueT(name)` | where a cue sits, as a progress and in show seconds (`NaN` when unknown) | | `phase` | `'active'`, `'linger'`, or `'failed'` (only inside `exit()` after a crash) | | `holding`, `held` | waiting at a hold, and for how many seconds | | `playing` | whether the timeline is running (`update` is called every frame regardless) | | `api` | the same `api` object, so `param()` and `cue()` can reach the show too | - **Enter once, exit once.** The module of a segment is entered when the show enters it and exited when it leaves it, however it got there. A seek into the middle of a segment calls `enter` with the right `p`, so the module must be able to start from the middle. - **`linger`.** After the segment ends its module keeps getting `update()` (with `p > 1` and `phase: 'linger'`) until the linger is over, and only then `exit()`. The next segment's module is already running meanwhile. - **`param()`.** While a segment (or its linger) is current its module may rewrite any timeline param. Return a number to replace the value, anything else leaves it. The performer's hand still wins: an override is applied *after* this layer. - **A module that throws is switched off, alone.** Each hook runs inside its own `try/catch`. If one throws, the director calls that module's `exit()` once (with `phase: 'failed'`), prints one `console.error`, and never calls it again. The other modules and the show carry on. The desk shows a red warning with the module's name and the error; **R** (reset) switches it back on. A frame loop is not allowed to stop because the youngest code in the room failed. ## The Director ```js import { Director } from '@openav/score'; const director = new Director({ score, modules: [dawn, noon, dusk], api: { sky } }); // every frame, after the timeline advanced: director.update(timeline.t, dt, { holding: timeline.holding, playing: timeline.playing }); timeline.layer = (key, value, t) => director.param(key, value, t); timeline.onSeek((t, kind) => (kind === 'reset' ? director.reset() : director.seek(t))); ``` | option | meaning | |---|---| | `score` | the `Score` | | `modules` | an array of modules (matched by `id`) or an object keyed by segment id | | `api` | the first argument of `enter`/`update`/`exit` (also `ctx.api`). A function is called once, the first time it is needed | | `onStatus(status)` | called when the segment, the holding state or the list of switched-off modules changes, not every frame | | `onCue(cue)` | called for every cue the playhead passes | | `maxStep` | a jump bigger than this many seconds between two `update()` calls counts as a seek, so cues in between are not fired (default 2). Raise it if you rehearse at a very high `timeline.rate` | | member | what it does | |---|---| | `update(T, dt, { holding, playing })` | once per frame: enter, update and exit the modules, fire the cues passed | | `param(key, value, T)` | the timeline's param layer | | `seek(T)` | the playhead jumped: modules whose segment no longer covers `T` exit now, the next `update()` enters the new ones with the right `p` | | `reset()` | every active module exits, every module hears `reset(api)`, switched-off modules come back | | `status()` | `{ cut, T, total, segment: { index, id, title, note, start, end, dur, t, p, remaining, hold, acts, cues }, next: { id, title, start, in } or null, holding, held, active, failed, errors, missing }` | | `register(module)`, `holding`, `failed` | add a module later; read the state | ## Use it with createShow ```js const show = await createShow({ world, score: { cuts, cut: 'full', modules: [dawn, noon, dusk], api: (show) => ({ sky: show.stage.active }), // what the modules may touch (default: the whole show) midi: false }, // see "Score to MIDI" timeline: { automation: { haze: [[0, 0.1], [60, 0.5]] } }, // optional: written against the base cut }); ``` | `score` option | meaning | |---|---| | `cuts`, `cut` | the cuts, and the default one. **`?cut=` in the URL wins** over `cut`; an unknown id is warned about and ignored | | `modules` | the segment modules | | `api` | an object, or a function of the show; default is the show itself | | `onStatus`, `onCue`, `maxStep` | passed to the director | | `midi` | `false` (default), `true` or `{ channel, segment, cue, … }`: [Score to MIDI](#score-to-midi) | With a score, the timeline's scenes, total and start come from it (anything you pass as `timeline.scenes` or `timeline.total` is ignored), and the frame loop gains one step after the timeline advances, `director.update(...)`. `createShow` also returns `show.score`, `show.director` and, for rehearsal and scripts, `show.show` (so `window.openav.show`): | `window.openav.show` | | |---|---| | `table()` | prints the segment table with `console.table` and returns the rows | | `goto('dusk')`, `goto('dusk.glow')`, `goto(2)` | jump to a segment, a cue or a segment by number | | `seek(T)` | jump to show seconds | | `cue('dusk.glow')` | the cue's show seconds (it jumps nowhere) | | `status()` | the director's status | | `next()`, `prev()` | the same as → and ← | | `cuts` | the list of cuts | ## The desk With a score the console adds: - **Scrubber.** Segment blocks, a tick for every cue (click to jump) and a dashed end on a segment that holds. - **Director panel.** A cut picker (choosing one reloads the page with `?cut=`, so nothing is half-switched), the current segment with a progress bar and the time left, the HOLD banner, the next segment and the countdown, the operator's **acts**, this segment's cues (ticked as they pass), and the red warning for modules that were switched off. - **Prompter** (**T**). Fullscreen, large, high-contrast: the segment title, what you do (`acts`), seconds left, the next segment, a HOLD banner while the show waits, and a red line for switched-off modules. Click it or press **T** again to leave. It also works for plain timeline scenes that carry `acts`. ### Keys | key | does | |---|---| | Space | play / pause. At a HOLD: release it | | → | next segment. At a HOLD: release it | | ← | previous segment | | R | back to the start; stopped; switched-off modules come back | | T | prompter | | F | fullscreen | | Esc | stop the knob-mapping wizard, or leave keyboard-piano capture | Typing in a field, a held key (auto-repeat) and any key with ⌘, Ctrl or Alt are ignored. **While the on-screen piano captures the keyboard** (the *keyboard* box ticked in the piano bar) the *letters* belong to the piano: R, T and F do nothing, because T and F are piano keys, and an accidental R must not rewind a show you are playing. Space, the arrows and Esc always work, so you can play and run the show from one keyboard. The MIDI-controller key **M** follows the same rule. ### Map the knobs in order The L2 Mapping panel has **🎛 map knobs in order**. It arms learn for the first param, you turn one knob, it is bound and the next param is armed; *skip*, *back* and *stop* (or Esc) steer it. Learn only ever **adds** routes (`Mapper.learn`), so running it again puts a second knob on a param and keeps the first. The order is the declared order of the params, skipping hidden ones; pass `mountConsole(el, app, { learnOrder: ['hue', 'size'] })` for your own. The logic is `LearnWizard` in `@openav/mapping`: ```js import { LearnWizard } from '@openav/mapping'; const wiz = new LearnWizard({ mapper, keys: () => params.schema.map((p) => p.key), onChange: (w) => draw(w) }); wiz.start(); wiz.current; wiz.skip(); wiz.back(); wiz.stop(); ``` ## Score to MIDI Software that should follow the show (a DAW launching a clip per segment, a lighting desk jumping scenes) cannot read the score, but it can read MIDI. `ScoreMidi` (`@openav/midi`) turns segment changes and cues into notes and CCs. It is **off** unless you ask: ```js score: { cuts, midi: { channel: 15, segment: { note: 36, cc: 20 }, // segment i → note 36 + i, and CC 20 = i cue: { note: 84, notes: { 'dusk.glow': 90 }, cc: 21 } } } // a cue → note 84 (or its own note), CC 21 = its number ``` | option | default | meaning | |---|---|---| | `channel` | 15 | MIDI channel 1 to 16 | | `segment` | `{ note: 36 }` | `note`: base note, the note sent is `note + index`. `cc`: a CC whose value is the index. `velocity`. `false` turns segment markers off | | `cue` | `{ note: 84 }` | `note` for every cue, `notes: { 'seg.name': note }` for particular ones, `cc` whose value is the cue's number in the cut. `false` turns cue markers off | | `length` | 0.15 | seconds before a marker note's note-off | | `scrub` | `false` | also mark when the playhead is moved by hand. By default markers go out only while the show *plays* (or a hold is released), so dragging the scrubber does not launch clips | A reset never sends. Everything goes through the `Midi` engine's own `noteOn`/`cc`, so the MIDI OUT meter shows it and `panic()` silences it. In code: `new ScoreMidi({ midi, channel, segment, cue })`, then `.segment({ index, cause })` and `.cue({ name, n })`. ## Clocks: why there are none A show that keeps two time lines (a *story* time that old logic reads and a *show* time the audience lives) needs a way to say which engine follows which: weather that never stops, a city that pauses while an inserted segment plays. That is a clock per engine. This framework has one time line, the show's `T`, so there is nothing for a clock to choose between, and a segment module that wants to pause an engine simply stops advancing it (`ctx.t` is its own clock, `ctx.held` another). If a real show brings back the two-time-line case, a `clocks` table belongs in the cut next to `segments`. Until then it would be machinery with no user. ## Without createShow ```js import { Score, Director } from '@openav/score'; import { Timeline } from '@openav/timeline'; const score = new Score({ cuts, cut: new URLSearchParams(location.search).get('cut') || 'full' }); const timeline = new Timeline({ params, score }); // scenes, total and start come from the score const director = new Director({ score, modules, api: { world } }); timeline.layer = (key, value, t) => director.param(key, value, t); timeline.onSeek((t, kind) => (kind === 'reset' ? director.reset() : director.seek(t))); const desk = mountConsole(deskEl, { timeline, params, mapper, signals, stage, score, director }); const loop = new Loop((dt) => { timeline.advance(dt); director.update(timeline.t, dt, { holding: timeline.holding, playing: timeline.playing }); mapper.update(dt); desk.render(stage.frame(dt, timeline.state())); }); ``` ## Rehearsal checklist - [ ] `show.show.table()` reads like the running order you meant - [ ] play it once start to finish at `timeline.rate = 4`; every cue fired, no module switched off - [ ] open `?cut=` for every version and check the desk shows the right total - [ ] press Space on every hold, press → on every hold, press R during a segment - [ ] break a module on purpose (throw in `update`) and watch the show carry on - [ ] the prompter (T) reads from the position you will stand in --- # MIDI controllers & embedding open-audiovisual models MIDI controllers as data: one JSON file per device, with the faceplate layout, every control's exact MIDI message, LED feedback, how to recognise its port, and the document each number came from. The same file drives an on-screen controller you can play, follows the real device when you plug it in, and can be dropped into any web page. Every device is on the [controllers page](https://openaudiovisual.com/controllers/). ## Play one in a page ```html ``` That is the whole integration. The element works without Web MIDI (Safari, iPad): it plays and fires events; only hardware and MIDI out report that they are unavailable. It never asks for MIDI permission unless you turn on `hardware`, `midi-out` or learn. Safari has none because WebKit has not implemented it ([bug 107250](https://bugs.webkit.org/show_bug.cgi?id=107250), still open). | Attribute | What it does | |---|---| | `profile` | which device: an id, a short name, or a URL to your own profile JSON | | `layout` | `auto`, `face` (as it looks) or `stack` (re-gridded for a phone held upright) | | `hardware` | `on`, or `ask` for a button: the real device moves the on-screen one | | `midi-out` | a port name (`IAC`, `loopMIDI`), `/regex/`, or `ask`: what you play is sent there | | `channel` | 1–16, when your unit is set to another channel | | `picker`, `readout`, `learn`, `follow` | device picker, last-message line, learn button, show whatever known device you plug in | ## Listen to it ```js const el = document.querySelector('oav-controller'); el.addEventListener('control', (e) => { const { id, value, signal, message, source } = e.detail; // 'k1' 0..1 'midi/lpd8/k1' { type: 'cc', ch: 1, cc: 70, value: 53 } 'ui' | 'hardware' }); await el.ready; el.set('k1', 0.5); // move a control as a hand would el.ingest([0xb0, 70, 64]); // MIDI from anywhere: the faceplate follows ``` Other events: `noteon`, `noteoff`, `connect`, `disconnect`, `status`, `profilechange`, `ready`, `error`. All bubble and cross the shadow root. Every field of every payload is in [packages/midi/README.md](../packages/midi/README.md). ## Without the element ```js import { createController } from 'https://openaudiovisual.com/packages/midi/index.js'; const ctl = createController('akai-lpd8', { signals }); // signals: publish into a show ctl.mount(document.querySelector('#pads')); // optional ctl.on('control', (e) => console.log(e.signal, e.value)); ``` Inside a show, do not read `midi/lpd8/k1` in `update()`: route it to a param with `controllerRoutes(profile, { k1: 'hue' })` (rule 1 in [AGENTS.md](../AGENTS.md)). Example 09 does this for six devices at once. ## In an iframe For places that only take an iframe: ```html ``` The frame posts `{ source: 'openav', type, frame, detail }` to its parent for every event, and accepts `{ target: 'openav', type: 'set' | 'ingest' | 'press' | 'release' | 'reset' | 'profile' | 'layout' | 'get', … }`. Add `origin=https://your.site` to talk to your page only. ## Play your DAW with it 1. macOS: Audio MIDI Setup → Window → Show MIDI Studio → IAC Driver → “Device is online”. Windows 11: the built-in Windows MIDI Services (released 2026-02-17, rolled out in stages to 24H2 and 25H2) share MIDI ports between apps and include a loopback, and Microsoft's announcement says web pages using Web MIDI can use it ([Windows blog](https://blogs.windows.com/windowsexperience/2026/02/17/making-music-with-midi-just-got-a-real-boost-in-windows-11/)). We have not tested it with open-audiovisual. Older Windows: install loopMIDI and add a port; that still works. 2. `` (on Windows, give the name of the port you made or the one Windows shows). 3. In Ableton Live, turn on Track and Remote for that input; in TouchDesigner, a MIDI In CHOP on it; in Resolume, Preferences → MIDI. Hands on the screen are sent; your real hardware is not forwarded (the DAW already hears it). The engine never listens to IAC inputs, and the element mutes an input named like its output, so nothing loops. ## Telemetry So the project can see where its controllers and shows end up, the packages send **one anonymous hit per page load** to Google Analytics 4 (the openaudiovisual.com property, `G-1YG2JHK2WT`): | event | when | params | |---|---|---| | `oav_embed_load` | an ``, `createController()` or the iframe starts | `oav_kind` (element · headless · iframe), `oav_profile`, `oav_host`, `oav_version` | | `oav_hardware_connect` | a real controller is plugged in and matched | `oav_kind`, `oav_profile`, `oav_host`, `oav_version` | | `oav_show_start` | `createShow()` starts | `oav_kind` (show), `oav_host`, `oav_version` | `oav_host` is the **origin** of the page it runs on (for the iframe, the page that holds it), never the path, query or title. Nothing anyone plays is sent, no cookies are set, and the client id is random for every page load, so visitors can't be followed. No gtag.js is loaded: a page with its own Google Analytics keeps its dataLayer untouched. Nothing is sent from localhost, LAN addresses, `file://` or openaudiovisual.com itself, or when the browser sends Do Not Track / Global Privacy Control. Turn it off: `` · `createController(p, { telemetry: false })` · `createShow({ telemetry: false })` · the iframe's `?telemetry=0` · or `globalThis.OPENAV_TELEMETRY = false` before anything loads. Source: `packages/midi/telemetry.js`. ## Add a controller 1. Copy the closest file in `packages/midi/profiles/`, or a generic layout. 2. Write each control's type and message from the manual's MIDI implementation chart. 3. Cite every source; mark anything unconfirmed `"verified": false` with a note. 4. `validateProfile(profile)` lists every problem; `npm test` checks every shipped profile. 5. Try it with ``, then register it in `profiles/index.js` and open a pull request. No manual: choose a generic layout, press Learn, move your controls in order, and keep `controller.exportMapping()` as the start of a profile. --- # p5.js in one line Any p5.js sketch (OpenProcessing, the p5 editor, your own page) can be played from a MIDI controller, from a controller drawn on screen, or from a phone, by adding one line at the top: ```js document.head.prepend(Object.assign(document.createElement('script'), { src: 'https://openaudiovisual.com/p5/oav-p5.js' })); ``` On OpenProcessing you can instead paste the URL into Sketch → Libraries → **+** (custom library). Both work in p5 1.x and 2.x global mode, and nothing has to be called in `setup()`. A sketch that carries this line, playable right now: [example 15](https://openaudiovisual.com/examples/15-p5-one-line/). ## Read it like the mouse ```js function draw() { background(knob1 * 255); // knob 1, 0..1, like mouseX circle(width / 2, height / 2, 50 + knob2 * 300); } function notePressed(note, velocity) { // like keyPressed(): any key, or a pad circle(random(width), random(height), velocity * 100); } ``` | read | value | |---|---| | `knob1` … `knob8`, `knob(n)` | 0..1, numbered as printed on the controller | | `fader1` … `fader8`, `fader(n)` | 0..1 | | `pad1` … `pad16`, `pad(n)` | the velocity while the pad is held, 0 when released | | `note`, `velocity` | the last key (MIDI number) and how hard (0..1) | | `noteIsPressed`, `noteIsDown(60)` | like `keyIsPressed` and `keyIsDown()` | | define | called when | |---|---| | `notePressed(note, velocity)`, `noteReleased(note)` | a key goes down / up (a pad too, unless you define `padPressed`) | | `knobMoved(n, value)`, `faderMoved(n, value)` | a knob or fader moves | | `padPressed(n, velocity)`, `padReleased(n)` | a pad is hit / let go | Knob 1 is knob 1 on whichever controller played it: the [controller profiles](controllers.md) say which MIDI message each knob sends (MiniLab 3's knob 1 is CC 74, LPD8's is CC 70). On a controller OAV has no profile for, the first knob you turn becomes `knob1`, the next `knob2`. A sketch that already has its own `note`, `velocity` or `knob1` keeps it. ## Note names Wherever a note number goes, a note name goes too (0.8): ```js function draw() { if (noteIsDown('C4')) circle(width / 2, height / 2, 100); // the same as noteIsDown(60) } function mousePressed() { midi.play('C#4', 0.8); // sends note 61, as a key would } ``` | write | you get | |---|---| | `noteNumber('C4')` | `60`; a number comes back as it is, held to 0..127 | | `noteName(61)` | `'C#4'`; `noteName(61, { flats: true })` is `'Db4'`, `noteName(61, { octave: false })` is `'C#'` | | `noteFreq('A4')`, `noteFreq(69)` | `440` (Hz), for a sketch that makes its own sound; `noteFreq(69.5)` is a quarter tone up | | `noteIsDown('C4')`, `midi.isDown('C4')`, `midi.play('C#4', 0.8)` | the same as with `60`, `61` | A name is a letter A to G (either case), an optional `#` or `b` (`♯` and `♭` too), and the octave number: `C4`, `c#4`, `Db3`, `F♯2`, `C-1` (0) up to `G9` (127). **C4 is 60**, the scientific pitch notation the course modules use; some software (Ableton, Yamaha) calls 60 "C3", so a name copied from there is one octave off. These are also the names Strudel writes (`c4 e4 g4`, `#` for sharp, `b` for flat). Anything else, such as `H4` or `C-2`, stops with a one-line card saying how to write it. `notePressed(note, velocity)` still hands you the number: `noteName(note)` writes it back as a name. ## With OP Configurator [OP Configurator 3000](https://github.com/msawired/OPC) (OPC, built into OpenProcessing) names a variable and its range. With this line on the page the controller is wired to those variables in order, with no mapping step: ```js OPC.slider('radius', 60, 10, 300); // knob 1 OPC.slider('speed', 1, 0, 5, 0.1); // knob 2 OPC.toggle('trails', true); // pad 1 flips it ``` Knobs drive the sliders in the order they are declared, the faders continue after the knobs, and pads drive toggles and buttons. OPC's panel moves with the knob, and `parameterChanged()` fires as usual. OpenProcessing's OPC panel can also learn hardware by itself (its MIDI switch); once that mapping is seen working, hardware is left to it and this line keeps wiring the on-screen controller and the phone, which OpenProcessing cannot see. Avoid OPC names that are also p5 functions (`hue`, `fill`): p5 1.x clashes with them. **One knob, one job.** If the sketch also reads `knob1` itself, knob 1 is the sketch's: the first OPC slider no longer follows it (drag it on OPC's panel), the other knobs stay wired, and a hint says so once. The first line of the 🎹 panel is the wiring, for example *knob1 → knob1 (sketch; radius by its slider) · knob2 → speed (OPC)*. ## The microphone The same line reads the microphone, and the sketch reads it the same way: as variables. ```js function draw() { background(230, 40, 10); fill(200 + bass * 160, 70, 100); // bass: the low end, 0..1 circle(width / 2, height / 2, 50 + micLevel * 300); // micLevel: how loud, 0..1 } function beatDetected(level) { // like keyPressed(): a beat just landed // flash, spawn, jump to the next scene } ``` | read | value | |---|---| | `micLevel` | 0..1, how loud it sounds: the room's own noise is 0, talking is about 0.4–0.6, shouting is near 1 | | `bass`, `mid`, `treble` | 0..1, the energy in 20–250 Hz, 250–2000 Hz and 2000–8000 Hz | | `beat` | `true` for one frame per beat, like `mouseIsPressed` while a button is down | | `micIsOn` | whether the microphone is running | | `startMic()`, `stopMic()` | turn it on or off from code (they return a promise) | | `micSensitivity(2)` | twice as sensitive; `micSensitivity()` returns the current value | | write this function | when it is called | |---|---| | `beatDetected(level)` | a beat: a jump in loudness over what the room has been doing (level 0..1) | | `kickDetected(level)`, `snareDetected(level)`, `hatDetected(level)` | a hit in the low, middle or high band (20–120 Hz, 150–800 Hz, 6–14 kHz) | **It is off until the sketch asks.** Nothing is requested when the page loads. The first time the sketch reads `micLevel`, `bass`, `mid`, `treble` or `beat`, or calls `startMic()`, the browser asks for the microphone; until the answer arrives every value is `0`. A sketch that defines `beatDetected()` and reads nothing starts it on the first click or key. In the 🎹 panel the **🎤** button turns it on and off by hand and shows the level, and it says so when the browser blocked the microphone (click it again after allowing it from the address bar). - **Click first, if the browser wants one.** Browsers keep audio quiet until the page has been clicked. This file asks as soon as `draw()` reads a value and starts the microphone again on the first click or key if the browser held it back, so no `userStartAudio()` is needed. - **Next to p5.sound.** The analysis is [`@openav/audio`](packages.md) (an `AnalyserNode`), loaded from next to this file the first time it is needed, with its own `AudioContext`. `p5.sound` (`p5.AudioIn`, `p5.FFT`) can run on the same page; the two microphones do not share anything, and none of these names is a p5 or p5.sound name. It works in p5 1.x and 2.x global mode. - **How loud is loud.** `micLevel` is a loudness in decibels, measured from the room: its bottom follows the room's own noise (it settles in a second or two, and rises at most 1.5 dB a second, so a voice never becomes the new silence), and −10 dBFS, shouting into a laptop, is 1. So `50 + micLevel * 300` visibly grows when someone talks. A quiet or far microphone: `micSensitivity(3)` (+9.5 dB on `micLevel`, ×3 on the bands, still capped at 1). The raw numbers are `OAVp5.mic.rms` (0..1), `OAVp5.mic.db` and `OAVp5.mic.floor` (the room, in dBFS). `OAVp5.mic.on('beat', fn)` is the same as defining `beatDetected`, and returns a function that stops it. - **Your own names win.** A sketch that already has a `bass` or `mid` (a `let`, a function, a library) keeps it, and this file never starts the microphone for it. - **Off switch.** `?mic=click` (or `window.OAV_P5 = { mic: 'click' }`) means only 🎤 and `startMic()` turn it on; `mic=off` means nothing can start it (the names stay `0`). `https` or `localhost` is required, as for any microphone page. - A sketch with a microphone, ready to open: `p5/examples/mic/`. ## Sound: what is played is heard ```js function setup() { createCanvas(windowWidth, windowHeight); startSound('piano'); // every key is heard: a keyboard, the 🎹 on screen, a phone } ``` | call / read | value | |---|---| | `startSound(instrument)` | turn sound on (default `'piano'`), or switch instruments while it plays | | `stopSound()` | turn it off | | `soundIsOn` | whether notes are heard now | The instruments are those of [`@openav/sound`](packages.md): `piano` (the Salamander grand; its 2 MB of samples load the first time, and the electric piano plays meanwhile, or instead if they cannot load), `epiano`, `organ`, `strings`, `choir`, `harp`, `marimba`, `vibraphone`, `xylophone`, `glockenspiel`, `music-box` and the `synth/…` presets. In the 🎹 panel, **🔊** turns sound on and off and shows a picker with every instrument and the loading progress. - **Off until asked.** Nothing loads before `startSound()` or 🔊; then `@openav/sound` from next to this file and Tone.js 15 from jsDelivr (the option `tone=` loads it from elsewhere). - **The first click.** Browsers play audio only after the page has been clicked, and a key on a MIDI keyboard does not count. A sketch that calls `startSound()` in `setup()` is heard from the first click or key; until then the 🔊 on the 🎹 button is yellow. - **Pads are drums.** Notes on MIDI channel 10 (MiniLab 3 and LPD8 pads, the General MIDI drum channel) play the drum kit, not the piano. The sustain pedal holds notes. - **No code at all.** `?oav_sound=piano` on the page (any instrument id; on OpenProcessing `?sketch_oav_sound=piano`) makes any sketch heard from the first click; `sound=off` keeps it silent. - A sketch that plays and draws, ready to open: `p5/examples/keys/`. ## Drum machine ```js function setup() { startDrums('breakbeat', 110); // a 16-step grid on screen; click a cell to change the beat } function kickDetected(level) { // the microphone's own callback // swell, flash, spawn } function drumHit(lane, velocity) { // every hit: 'kick', 'snare', 'hat' or 'clap' } ``` | read | value | |---|---| | `drumsAreOn` | whether the machine is playing | | `drumStep` | 0..15, the step that just played | | `drumBar` | 1, 2, 3 …, counted from `startDrums()` | | `drumBpm` | the tempo (the grid's slider changes it too) | | `drumPhase` | 0..1, how far through the bar, smoothly, as heard: `rotate(drumPhase * TWO_PI)` turns once a bar | | `beatPhase` | 0..1, how far through the beat (a quarter note): `1 - beatPhase` swells on each beat and shrinks to the next | | `startDrums(pattern, bpm)`, `stopDrums()` | start (or change the pattern and tempo while it plays), stop | | write this function | when it is called | |---|---| | `drumHit(lane, velocity)` | every hit, every drum (a line of rhythm's crash, ride or toms too) | | `kickDetected(level)`, `snareDetected(level)`, `hatDetected(level)` | a kick; a snare, a clap or a rim; a hat (closed, open or pedal) | | `beatDetected(level)` | a kick or a snare, once per step (and `beat` is true for a frame) | | `rhythmChanged(src, how)` | the line changed: `startDrums()`, a click on a cell, a line typed, ↶ (0.7) | **A microphone sketch moves over unchanged.** Each hit calls the same functions the microphone does, so a sketch written for the microphone reacts to the machine as it is. Once a sketch has started the machine, defining `kickDetected()` no longer opens the microphone on the first click (a sketch that reads `micLevel` still asks, and then both call it). - **Patterns.** `'four on floor'`, `breakbeat`, `'half time'`, `latin`, `sparse`, your own `startDrums({ kick: 'x...x...x...x...', snare: '....x.......x...', hat: 'x.x.x.x.x.x.x.x.' })` (a lane left out is silent), or one line of rhythm (below). - **The card.** The lanes are named in the page's language (大鼓 小鼓 鈸 拍手 in a Chinese browser), the bar it is in shows beside the tempo, and the step that sounds is a yellow column (13:1 against the dark card, readable on a projector). - **Where the grid sits.** At the top centre: on OpenProcessing between the 🎹 and OPC's icon and clear of the console, and below the 🎹 panel while that is open. – folds it, × stops the machine, and **🥁** in the 🎹 panel opens and closes it with no code at all. 🥁 brings the grid up stopped: tick its play box to hear it (a sketch's own `startDrums()` plays at once). - **The kit.** [`@openav/drums`](packages.md)' synthesized kit (no samples), heard from the first click like sound; the callbacks run from the start either way. **On time, however heavy the sketch.** The hits are handed to the kit ahead of time, on its audio clock: a heartbeat in a Web Worker wakes every 25 ms and schedules what falls in the next 0.1 s at its exact time. Measured on the kit's own output with a `draw()` that keeps the page busy 30 ms a frame, the kick-to-kick interval varies by 0.000 ms (0.5, which played each hit in the frame it was due, varied by 15 ms). The callbacks still come in a frame: the one in which the hit is *heard* (its time plus the output latency the browser reports), right before `draw()`, in the same order as before. - **A tab in the background** keeps playing and counting: `drumBar` follows the clock on the wall and the cues fire, with one `barStarted` a bar. The hits and `stepStarted` are not told while nobody can see them, and nothing is told twice when the tab comes back, so a sketch that adds a particle per hit does not burst. - **A `draw()` longer than 0.1 s** makes the machine look 0.3 s ahead from then on (🩺 says so); a click on a cell is then heard up to 0.3 s later. - **A Bluetooth speaker** is late by 0.15–0.25 s; where the browser reports it the callbacks wait for it, and `?oav_drumlatency=200` (milliseconds) sets it by hand. - **The safety net**: `?oav_drums=frame` (on OpenProcessing `?sketch_oav_drums=frame`) brings back 0.5's machine, moved by the frames. ### One line of rhythm ```js function setup() { startDrums("bd*4, [~ sd]*2, hh*8", 110); // a kick on every beat, a snare on 2 and 4, eight hats } ``` Any text that is not a pattern name is a line of rhythm; names, grids and numbers work exactly as before. The line shows over the grid, and the word that sounds lights up and fades. | write | means | e.g. | |---|---|---| | a word | a drum: `bd` kick, `sd` snare, `hh` hat, `oh` open hat, `cp` clap, `rim`, `lt` `mt` `ht` toms, `cr` crash, `rd` ride, `cb` cowbell | `"bd sd bd sd"` | | `~` | a rest | `"bd ~ sd ~"` | | space | the words share the bar evenly | `"bd hh sd hh"` | | `[ ]` | a group shares one place | `"bd [sd sd]"` | | `*n` | n times in that place | `"hh*8"` | | `,` | another line on top | `"bd*4, hh*8"` | | `< >` | one per bar, in turn | `""` | | `(k,n)` | k hits spread over n steps | `"bd(3,8)"` | The whole list is in [`@openav/pattern`](packages.md), loaded from next to `oav-p5.js` with the drums. Drums the grid has no lane for (crash, ride, toms, open hat) get a row of their own; a hit off the sixteen steps (a triplet) is drawn where it falls inside its step. **A mistake** in `startDrums("…")` shows a card in the page's language with the character it is at, why, and how to fix it, and the machine plays `'four on floor'` meanwhile; `'x...x...'` on its own (one lane of the grid) says to put it inside `{ kick: … }`. **A line copied from strudel.cc** works too: `startDrums('s("bd*4, [~ sd]*2").fast(2)')` plays `[bd*4, [~ sd]*2]*2`. `s("…")`, `sound("…")`, with `"`, `'` or `` ` `` quotes, several of them (`$: s(…)`) as layers, `.fast(n)` and `.slow(n)`; `setcpm(110/4)` or `setcps(…)` sets the tempo when `startDrums()` is given none. `.bank("…")` is left out (OAV plays its own synthesized kit) and anything else in the chain (`.jux(rev)`, `.every(…)`, `.gain(…)`) gets a card that says OAV does not do it yet and points at **Open in Strudel**; the rest plays. The way of writing one line of rhythm comes from TidalCycles / Strudel ([strudel.cc](https://strudel.cc)); OAV implements it independently from their public documentation. ### The cells and the line edit each other Since 0.7 the line over the grid is always there (a preset shows the line its grid is: `'four on floor'` is `bd*4, [~ sd]*2, hh*8`) and both ways work: - **Click a cell**: the line is rewritten the way a person would write it, and the machine plays the new line at once. Steady hits become `bd*4` / `hh*8`, one hit per group `[~ sd]*2`, a Euclidean rhythm `bd(3,8)` (with its rotation: `bd(3,8,2)` starts two steps later), anything else beat by beat: `bd [~ ~ ~ bd] bd*2 ~`. Lines are joined with `, `. When the line holds more than sixteen steps can (triplets, `< >`, `/2`, `?`, `bd:2`), the first click shows a card saying what would go, and the next click rewrites it. - **Type in the line**: click it, change it; Enter, or a pause of 400 ms, plays it and the cells follow. A mistake marks the word in red, a card says what is wrong, and **the last line that worked keeps playing** (not `'four on floor'`). Escape drops what is not played yet. Typed keys never reach the sketch's `keyPressed()`. - **↶** on the card (or ⌘Z / Ctrl+Z after touching the card) goes back one line; `OAVp5.drums.history` lists them. - **Copy** puts `startDrums("…", 110)` on the clipboard, ready to paste into `setup()`. - **Open in Strudel** opens the same line at strudel.cc in a new tab, as `s("…")` with `setcpm(bpm/4)` (one OAV bar is one Strudel cycle of four beats; strudel.cc's docs: `setcpm(bpm/bpc)`). `OAVp5.drums.strudelUrl` is the link. ```js function rhythmChanged(src, how) { // how: 'code' · 'cell' · 'type' · 'undo' · 'menu' line = src; // e.g. draw the line on the canvas, so a recording shows it } ``` `OAVp5.drums.src` is the line that plays, `OAVp5.drums.edit("…")` plays a line as if typed, `OAVp5.drums.undo()` goes back. Keys that swap the rhythm, step back and open Strudel: `p5/examples/rhythm-lab/`. ### The rhythm on the canvas The card is part of the page, not the canvas, so a recording never shows it. `drawRhythm()` draws the rhythm on the canvas: ```js function draw() { background(0); drawRhythm(20, 20, 360, 90, 'grid'); // the sixteen steps, a playhead drawRhythm(20, 130, 300, 300, 'circle'); // a ring of 16 per drum, a hand turned by drumPhase drawRhythm(340, 130, 300, 300, 'spiral', { bars: 4, colors: { kick: '#ff7b54' } }); // the last 4 bars, wound outwards } ``` `drawRhythm(x, y, w, h, style, options)`: `style` is `'grid'`, `'circle'` or `'spiral'`; a hit lights up the moment it is heard (the clock `drumPhase` reads) and a triplet sits where it falls. Options: `colors` (`{ kick: '#f66', … }`, any p5 color), `color` (one for all), `off` (the empty steps), `playhead` (a color, or `false`), `bg`, `bars` (spiral). Drawn inside `push()` / `pop()` with p5's own shapes, in 2D and WEBGL, p5 1.x and 2.x; **in WEBGL (0, 0) is the middle of the canvas**: `drawRhythm(-width / 2, -height / 2, width, height, 'circle')`. With no box it draws a strip along the bottom (`'grid'`) or a square in the middle. A sketch with all three: `p5/examples/rhythm-visual/`. ## Timeline: bars and cues The drum machine is also the clock: sixteen steps a bar, counted from `startDrums()`. Declare what happens at which bar: ```js function setup() { startDrums('breakbeat', 110); cue(1, 'calm', 12); // bar 1, and again every 12 bars cue(5, 'bloom', 12); cue(9, 'storm', 12); } function cueFired(name, bar) { // or read cueName in draw() look = SCENES[name]; } ``` | call / read / define | | |---|---| | `cue(bar, name, every)` | at bar `bar` (1 is the first, `4.5` is halfway through bar 4); `every` bars again if given | | `cueName` | the last cue that fired (`''` before the first) | | `cueFired(name, bar)` | a cue's bar has come | | `barStarted(bar)`, `stepStarted(step, bar)` | the first step of every bar; every step | The same `cue()` twice adds nothing, so it may also sit in `draw()`. On one step the cues fire in the order they were declared, then `barStarted`, `stepStarted` and that step's hits. A silent clock: `startDrums({})`. The whole of it, ready to open: `p5/examples/drums/`; a line of rhythm with `drumPhase` and `beatPhase`: `p5/examples/rhythm/`; the rhythm drawn on the canvas: `p5/examples/rhythm-visual/`. ## Values over time: lag, env, speed, spring, trail A knob or `micLevel` jumps; what looks good is how it moves between the jumps. TouchDesigner does this with CHOPs; the same operators are here, one call each in `draw()`, nothing to declare, and seconds are seconds (the same curve at 30 and at 144 fps): ```js circle(width / 2, height / 2, 50 + lag(micLevel, 0.03, 0.4) * 300); // a VU meter: opens at once, closes slowly scale(1 + env('kick', 0.01, 0.3)); // the drum machine: every kick swells it, gone in 0.3 s rotate(spring(knob1, 100, 6) * TWO_PI); // flick a knob: it overshoots and settles ``` | TouchDesigner | oav-p5 | | |---|---|---| | Lag CHOP (Lag 1 / Lag 2) | `lag(value, up, down)` | follows `value`: `up` seconds while it rises, `down` while it falls (`down` = `up` if left out; 0.1) | | Trigger CHOP (attack, decay) | `env('kick', attack, decay)`, `trigger('boom')` | 0..1: up to 1 in `attack` seconds after each event, back to 0 in `decay` (0.01, 0.3). Events: `'kick'` `'snare'` `'hat'` `'beat'` (the microphone or the drum machine, the hits `kickDetected` hears), `'note'` (any key), `'bar'` `'step'` (the drums), a cue's name, `trigger('boom')`; or a switch: `env(mouseIsPressed, …)` | | Speed CHOP | `speed(rate)` | adds `rate` every second: `rotate(speed(TWO_PI))` turns once a second | | Spring CHOP | `spring(value, stiffness, damping)` | follows like a mass on a spring (100, 6: two or three bounces; damping 20: none) | | Trail CHOP | `trail(value, seconds)` | the last `seconds` (2) as an array, 60 values a second, oldest first, the newest last | | Slope CHOP | `slope(value)` | how fast it changes, per second | | Count CHOP | `count(beat)`, `count(beat, 4)`, `count('kick')` | how many times it turned on; with 4: 0 1 2 3 0 …; a name: how many of those events so far | | a CHOP's viewer | `scope(value, 'label')` | a small plot in the corner of the page (below) | | | `drawTrail(values, x, y, w, h)` | the same plot on the canvas, in the work and in a recording | **Seconds, the same as in TouchDesigner.** A lag's seconds mean what a Lag CHOP's do: about the time the value takes to follow 90% of a jump, so `lag(x, 0.1)` here moves like a Lag CHOP set to 0.1 ([Lag CHOP](https://docs.derivative.ca/Lag_CHOP)). Underneath it is a one-pole lag with a time constant of seconds ÷ 2.3 (ln 10); `@openav/mapping`'s `smooth` keeps its own 63% time constant. Attack and decay are lengths in both. **Nothing to declare.** The first `lag()` in `draw()` keeps the first lag's memory, the second the second, every frame; `spring()`, `speed()`, `trail()` … are counted each on their own. Put one inside an `if`, or in a loop whose length changes, and the order slips: a card says so, and from then on oav-p5 matches each call by the line it is written on. The sure way is a name, a string at the end: `lag(p.size, 0.1, 0.5, 'p' + p.id)`. A named one is the same wherever it is called, `draw()`, `setup()` or `mousePressed()`; an unnamed one outside `draw()` has nothing to remember by and returns its input (`speed` 0, `trail` `[]`), and a card says that too. `env()`, `trigger()` and `count('kick')` are about events and work anywhere. A sketch with its own `speed` or `count` keeps it; `OAVp5.chop.speed(…)` is the same function under another name. **Seeing the values.** `scope(value, 'label')` returns `value`, so it can wrap an expression: `circle(x, y, scope(lag(micLevel, 0.03, 0.4), 'lag') * 300)`. Each label is one line in a small oscilloscope across the page from the 🎹: the last 4 seconds and the value now; `scope(micLevel, 'mic')` beside it shows what the lag did. It lives on the page, not on the canvas: never in a recording or a 📷, never in the way of a click; `hints=off` or `scope=off` hides it. To draw it in the work: `stroke(255); drawTrail(trail(micLevel, 3), 0, height - 100, width, 100)` (or `drawTrail(micLevel, …)`, which keeps its own 2 seconds): 0 at the bottom of the box, 1 at the top (`lo`, `hi` after `h`), the sketch's `stroke()` (white under `noStroke()`). The math is [@openav/chop](packages.md#openav-chop), loaded from next to this file the first time one of these is called; until it arrives (a few frames) each returns its input as it is. What each parameter means follows Derivative's TouchDesigner documentation; the code is OAV's own. Raw against lagged, side by side, with the scope: `p5/examples/lag/`. `env`, `spring`, `speed`, `count` and `slope` driving a disc that a bass drum opens, with `startDrums()` as the only input: `p5/examples/chop/`. ## Recording: a video with its sound ```js function keyPressed() { if (key === 'r') recordingIsOn ? stopRecording() : startRecording(); // startRecording(10): ten seconds, then it stops } function recordingSaved(info) { // info.name, info.seconds, info.bytes, info.format ('mp4' or 'webm') } ``` | call / read / define | | |---|---| | `startRecording()`, `startRecording(seconds)`, `startRecording({ seconds, fps, mic, name, canvas, bitrate })` | record the canvas and the sketch's sound; with seconds it stops by itself; a take is 5 minutes at most. `canvas` is the canvas to record (a p5 canvas or `createGraphics()`; the default is the biggest p5 canvas on the page), `bitrate` is bits a second (the default follows the canvas size and frame rate: 4 to 16 Mb/s, up to 24 Mb/s above 30 frames a second) | | `stopRecording()` | stop; the file downloads | | `recordingIsOn` | whether a take is running | | `recordingSaved(info)` | the file is saved: `name`, `seconds`, `bytes`, `format`, `width`, `height`, `audio`, `sources`, `warnings`, `blob`, `url` | In the 🎹 panel, **⏺** starts and stops a take with no code; while it runs, the 🎹 button shows a red dot and the time (a click on the dot stops it). **📷** saves the canvas as a PNG. - **What is in the video.** The p5 canvas at its own pixel size (a retina screen doubles it), 30 frames a second, and every sound this file makes: `startSound()`, the drum machine and the pads, the microphone while it is on, and p5.sound 1.x, mixed into one track while the speakers play as before. The 🎹, the drum grid, the cards and OPC's panel are not on the canvas, so they are not in the video. - **The file.** MP4 (H.264 + AAC), which Instagram, YouTube and every editor take, wherever the browser can write it: Chrome 126 and later, Edge, Safari. Elsewhere (Firefox, an older Chrome) WebM, and a card says Instagram will not take it. It is named `oav-20261015-183005.mp4`; on a computer it lands in Downloads, on a phone a **Share** button on the card saves it to the photo library or sends it to Instagram. - **Checked after every take.** A card says what was recorded (format, length, size, with or without sound) and, when something is wrong, what and why: nothing reached the recording although a sound was on (no click on the page yet, nothing played), WebM, an almost empty file, a black video of a canvas that was not black (Safari recording a WebGL canvas, WebKit 229611), an image from another site on the canvas (no CORS), no recorder in this browser. - **The microphone** is in the take while it is on: wear headphones, or the speakers are recorded twice. `record=nomic` leaves it out. - **p5.sound.** p5.sound 1.x (the one for p5 1.x) is recorded. The p5.sound made for p5 2.x connects each sound straight to the speakers, with no output a recording can tap: it is not in the take, and a card says so. Switch the sketch to p5 1.11, or use `startSound()` and `startDrums()`. Sound the sketch makes with its own Tone.js or Web Audio code is not recorded either. - **Stills.** 📷 (or `OAVp5.record.shot()`) saves the canvas alone as `oav--