The short version. Several dancers each stand in front of their own camera — a phone on a tripod, or your laptop webcam. Each device watches its dancer and sends a stick-figure skeleton to one program on your Mac. That program compares the dancers to each other and sends the results out as OSC, the message format TouchDesigner, Max/MSP and Ableton listen for. You see a live skeleton on every phone screen and one dashboard on the Mac showing everything at once.
No video ever leaves the phone — only numbers, about 4 KB per frame.
For what this is and why it is built this way, read SPEC.md. This file is just how to run it.
You do not need phones, a studio, or another person to see the whole thing working.
1 — Install (one time):
uv venv --python 3.11
uv pip install -e ".[dev]"
bash scripts/make_cert.sh # so phones will turn their cameras on
python scripts/fetch_assets.py # so it works with no internet later2 — Start the server:
.venv/bin/python -m server.mainIt prints the addresses to open. Open the dashboard link in your browser.
3 — In another terminal, run three fake dancers who move in a canon, each trailing the one before by 0.4 seconds:
.venv/bin/python scripts/fake_dancers.py --dancers 3 --lag 0.4 \
--names stage-left stage-right upstageThe dashboard now shows three stick figures moving, live numbers underneath, and OSC is streaming to
127.0.0.1:8000. You should see canon_lag sitting at 0.40 and 0.80 — the system correctly
finding the offsets the fake dancers were given.
4 — Now use your real webcam. Open the camera?id=laptop link and press Start camera. You get
a live skeleton and hand overlay, and a fourth figure appears on the dashboard.
Useful variations:
# one dancer goes silent after 12s -- watch /cam/upstage/alive flip to 0
python scripts/fake_dancers.py --dancers 3 --drop upstage
# scatter them, to see group disagreement climb
python scripts/fake_dancers.py --dancers 4 --chaos 0.6Every phone needs https. Browsers refuse camera access over a plain connection, and the failure
is silent — the camera simply never turns on, with no error. scripts/make_cert.sh handles this;
each phone taps through one "not private" warning.
- Phone and Mac on the same WiFi.
- Open the address the server printed, with a name you choose:
https://192.168.1.50:8443/camera?id=stage-left - Tap through the certificate warning, then Start camera.
- Tap Keep awake so the screen does not lock.
Names, not numbers. The ?id= is remembered by that phone. Use a real name (stage-left,
upstage) rather than a number, so a phone that reconnects after a WiFi blip cannot come back as a
different identity and drive the wrong cue.
Phone hygiene for a real session: turn off auto-lock, silence notifications, and turn on Do Not Disturb. An incoming FaceTime call takes over the screen and kills the camera feed.
If you end up running more than two or three devices, Tailscale is worth the setup — it gives the Mac a genuinely trusted certificate, so there is no warning to tap on any device.
Guest, hotel, campus and conference WiFi block devices from reaching each other. That is what a guest network is for, so no server setting works around it: your phone simply cannot see your Mac. The server detects this at startup and prints a NETWORK WARNING.
The giveaway, if you want to check yourself: run arp -a -n. If several different addresses all
answer with the same hardware address, everything is being relayed through the gateway and devices
are isolated from each other.
Three ways out, fastest first:
| How | Good for | |
|---|---|---|
| Phone hotspot | Personal Hotspot on one phone; join the Mac and the other phones to it | Working in two minutes, costs nothing. Pose data stays local — cellular is only for internet. |
| Tailscale | Install on the Mac and each phone, same account, switch it on | Works anywhere, survives network changes. Best permanent answer. |
| Your own router | Any cheap travel router — it needs no internet at all | The right answer for a real show: predictable, isolated, nothing else on it. |
Your IP changes when you switch networks, so re-run bash scripts/make_cert.sh and restart. The
startup banner warns you when the certificate no longer covers your current address.
OSC to 127.0.0.1:8000 by default. Change it in setups/unison-study.yaml.
/group/cohesion 0.71 is the ensemble together
/group/disagreement [33 floats] where in the body they are apart
/group/energy 0.45 average movement
/group/outlier stage-right who is furthest from the group
/pair/stage-left/upstage/similarity 0.82 same shape?
/pair/stage-left/upstage/direction 0.88 moving the same way?
/pair/stage-left/upstage/canon_lag 0.40 same phrase, 0.4s apart
/pair/stage-left/upstage/canon_score 0.91 and it is a good match
/dancer/stage-left/effort 0.34 how much this dancer is moving
/dancer/stage-left/stillness 0.66
/dancer/stage-left/pose [132 floats] the normalized skeleton
/dancer/stage-left/hand/left/gesture Open_Palm
/cam/stage-left/alive 1 0 means this camera went quiet
Always watch /cam/<name>/alive. When a camera drops, the server stops sending that dancer's
skeleton and flips this to 0. Without checking it, a frozen skeleton looks exactly like a motionless
dancer, and you will hold a cue on dead data.
Two message shapes, set per setup with format::
| good for | note | |
|---|---|---|
packed (default) |
TouchDesigner, Unity, Notch | one message per dancer per frame |
joint |
Max/MSP, Ableton | one message per body part; ~4,000/sec with four cameras, which Max struggles with |
A different instrument, not a reduced version of the group one. together has no meaning
alone, so all four controls are self-referential.
python -m server.main
python sound/soundscape.py --preset solo --device headphones| OSC (port 9001) | Dancer does | You hear |
|---|---|---|
/solo/width |
wrists tucked in → reaching wide | the chord opens — from one tight mono note to a wide hard-panned stack |
/solo/height |
hands low → overhead | brightness, nearly 3 octaves |
/solo/active |
frozen → moving fast | loudness, 25 dB |
/solo/still |
just moved → held for seconds | a shimmer that grows the longer the shape is held |
Four rather than three, and it holds up: active and still are anti-correlated in
time, so they never compete for attention. width is spatial where height is spectral.
still is not 1 − active. It counts the seconds of the current hold and snaps back
to zero the moment you move, so holding a shape for four seconds is a different musical
event from moving slowly. Stillness costs a dancer nothing and reads enormously; this
gives it somewhere to go.
Hear it with no dancer: python sound/soundscape.py --preset solo --sweep.
Both presets can run at once — soundscape.yaml streams to 9000, solo.yaml to 9001,
unison-study.yaml to 8000. Point the synth at whichever you want to hear.
A worked example, and a good place to start if that's your use case.
python -m server.main # setups/two-dancers-sound.yaml loads automatically
uv pip install sounddevice # one time
python sound/soundscape.py # in another terminalThree signals, on purpose. More than three and you cannot hear which is which, so the extra channels stop being information:
| OSC | What the dancers do | What you hear |
|---|---|---|
/snd/together |
same shape or not | harmony — a clean triad, or the same notes gone sour |
/snd/energy |
how much they're moving | loudness and density |
/snd/height |
grounded, or reaching up | brightness — near-sine, or full of harmonics |
Sent to port 9000 (not 8000, so it can run alongside unison-study.yaml).
Hear it with no dancers: python sound/soundscape.py --sweep walks each control through its
range in turn and names which one is moving. That's how to develop the sound design at your desk.
Into a DJ setup rather than laptop speakers: --list-devices, then --device N.
Knobs worth touching first: full_speed in the yaml (lower it if energy pins at 1 too easily),
and --root / --floor on the sound script (--floor 0 makes stillness silent instead of a held
drone).
The whole sound design lives in one readable file, sound/soundscape.py — about 80 lines of numpy,
no framework. Two things in it were measured rather than guessed, and the comments explain why:
brightness is done by adding harmonics rather than filtering, and the height-to-brightness curve is
numerically inverted so equal arm movement gives equal steps by ear.
A setup is a file listing what to compute and where to send it. Copy
setups/unison-study.yaml, change a few lines, done. Every .yaml in setups/ loads and runs, so
several can stream at once — coarse and smoothed to the lighting desk on one port, twitchy and
per-joint to the sound desk on another.
Five measures ship built in: pose_distance, group_disagreement, effort, direction, canon.
To write your own, drop a file in setups/measures/:
from server.measures import Measure, register
@register
class ArmsOnly(Measure):
name = "arms_only"
def compute(self, t, dancers):
a, b = dancers["stage-left"], dancers["stage-right"]
gap, _ = self.weighted_distance(a.pose, b.pose,
a.confidence, b.confidence,
self.idx_for("arms"))
return {"/arms/gap": float(gap)}Then name it in a setup file. Save the file and it reloads live — no restart, cameras stay connected, calibration is kept. With dancers standing in the room waiting, that is the difference between exploring and settling for the first thing that works.
Each dancer handed to compute gives you .pose, .velocity, .confidence, .history(seconds),
.hands, .gesture("Left"), .calib and .alive. The five built-in measures are written through
this same interface, so read one of them as a worked example.
Everything works without it, and off is the default. Uncertainty is part of the material.
Turning it on means a dancer stands still for five seconds; the server records their body
proportions and facing, and comparisons get a little cleaner. Profiles save to calib/ and are
reused across sessions — you do not redo this every rehearsal.
Hit Calibrate on a camera card in the dashboard. The phone shows a countdown.
Turn it on for a whole setup with calibration: t_pose in the yaml.
Every session records the skeletons to recordings/. That lets you run a brand-new measure against
last Tuesday's rehearsal, at your desk, with nobody in the room:
python -m server.replay recordings/2026-07-29-165726.jsonl.gz --speed 4 --no-oscSizes, measured on three dancers at 30fps: about 170 MB an hour gzipped, 620 MB uncompressed.
Pass --no-compress-recordings to the server if you want plain text you can grep.
.venv/bin/python -m pytest tests/ -q32 tests. They focus on the failures that would be invisible in a room: a normalization that flips, a similarity number that is really measuring network lag, a dead camera that keeps reporting its last position.
The phone shows "This browser will not give us the camera." The page came over plain http. Run
scripts/make_cert.sh, restart the server, and load the https address.
Camera page loads but nothing happens after "Start camera". It is probably fetching the tracking
models from the internet and failing. Run python scripts/fetch_assets.py on the Mac and reload.
"This browser has no graphics support." Body tracking needs WebGL, the browser feature that lets a page use the graphics card — and MediaPipe needs it even when inference runs on the CPU, so there is no slower fallback. On a desktop, turn hardware acceleration back on in the browser settings. On a phone it means a very old device or a locked-down browser.
You edited camera.js but the phone still behaves the old way. It shouldn't — the server sends
no-cache headers for its own pages and scripts. If it happens anyway, force-reload on the phone
(pull down in Safari). The big vendored model files are cached deliberately; re-run
scripts/fetch_assets.py if you suspect one is corrupt.
Dashboard shows a camera but no stick figure. The dancer is out of frame, or too dark. The dot on the camera page goes amber when it has a connection but no body.
The camera page shows a skeleton, but the dashboard preview says "no signal" — and the recording
keeps growing. This means frames are arriving and being thrown away: the camera's clock is out of
step with the server, so every timestamp looks implausible. The camera card now says so directly
(N frames rejected), and the server logs it. Reload the camera page. If it persists, open the
camera page's console and check __poseStreaming.offsetMs — anything beyond a few hundred is wrong.
Left and right are swapped. The camera page shows a big L and R. Raise the arm it calls L and check the dashboard agrees. Toggle Mirror if not.
Similarity numbers look erratic. Raise delay_ms in the setup (try 350) — a busy network needs
more slack for frames to arrive before they are compared. Raise smooth to steady the output.
OSC is not arriving. Check the host and port in the setup file. The dashboard shows a running message count per setup, so you can tell whether the server is sending or the receiver is not listening.
- One dancer per camera. No multi-person tracking, by design.
- No room positions. You get each dancer's body shape, not where they stand relative to each other. That needs a per-camera room calibration — a good future project.
- Only eight built-in hand gestures (fist, open palm, pointing up, thumb up, thumb down, victory, "I love you", none). That is a hand-sign vocabulary, not a dance vocabulary. All 21 points per hand stream out, so you can build your own logic downstream.
canoncan alias on strictly periodic movement. A repeated bounce matches itself at every multiple of its period. Keepmax_lagshorter than the shortest loop.
MIT — see LICENSE. Use it, change it, ship it.
Two things in here are not covered by that, because they are not mine to license:
- The body-tracking models (
pose_landmarker_*.task,gesture_recognizer.task) are Google's, under Apache 2.0. They are not in this repository —scripts/fetch_assets.pydownloads them from Google's servers on first run, so you get them from the source under their own terms. - MediaPipe itself (the
@mediapipe/tasks-visionJavaScript the camera page loads) is also Google's, Apache 2.0, and likewise fetched rather than vendored here.