diff --git a/README.md b/README.md index 0e984a2..5c5b8db 100644 --- a/README.md +++ b/README.md @@ -203,11 +203,14 @@ spine of it (everything except the models): - ✗ **A model behind the service** — the v1 step; needs weights + likely a GPU. *Not built.* ``` -fixtures/ canonical motion JSON (hand-authored) + generator +fixtures/ canonical motion JSON (hand-authored) + generator +docs/abstract.md the accepted abstract — the canonical framing docs/motion-schema.md the exchange-format spec -service/ FastAPI /generate stub (uv) -frontend/app/ Vite + three.js Lab Bench (the live screen) -frontend/mockups/ the original static mockups (reference) +docs/usage.md how to use the tool — every view, control and shortcut +docs/v0-stub.md what v0 fakes — the complete honesty inventory +service/ FastAPI /generate stub (uv) +frontend/app/ Vite + three.js Lab Bench (the live screen) +frontend/mockups/ the original static mockups (reference) ``` **Stack:** three.js + TypeScript + Vite (frontend, pnpm); Python + FastAPI (service, uv); @@ -236,6 +239,9 @@ Open , type a phrase, click **Generate** — a 3D stick f animates; drag to orbit, use play/pause and the scrub bar. To re-author the motions, edit and re-run `python3 fixtures/_generate.py`. +📖 **[`docs/usage.md`](docs/usage.md) is the full guide** — every view, every control, every +keyboard shortcut, and how to read each of the four notation registers. + ### Reading it Hit **Read** (or press R) for the four **notation registers** — the same motion @@ -301,6 +307,13 @@ keeping its native way of authoring) and **performance mode** (the projectable s **They are not three models interpreting a theme.** Nothing in that view can be read as a finding about model behaviour. +> **There are five hand-authored motions in the entire system.** Every movement you have ever +> seen BodyPrompt produce is one of those five, wearing a little seeded jitter. + +⚠️ **[`docs/v0-stub.md`](docs/v0-stub.md) is the complete inventory of what v0 fakes** — every +stand-in, written down in one place, so that nothing in a screenshot can be mistaken for a +finding. Read it before citing anything this tool shows you. + Everything above is the research **instrument** — deliberately built first, so that v1 only has to swap the stub for a model and every one of these views becomes real at once. Repo: **Public**. diff --git a/docs/usage.md b/docs/usage.md new file mode 100644 index 0000000..2937c5a --- /dev/null +++ b/docs/usage.md @@ -0,0 +1,242 @@ +# Using BodyPrompt + +The complete guide to the running app: every view, every control, every keyboard shortcut. + +**Before you start, one thing.** The app does not read your prompt. It hashes it, and returns +one of five hand-authored motions. Everything below describes a working *instrument* pointed +at *fake data* — see [`v0-stub.md`](v0-stub.md) for exactly what is faked and where. Nothing +you see in this tool is yet a finding about a model. + +--- + +## Running it + +Two processes. Needs **Python 3.10+ with [uv](https://docs.astral.sh/uv/)** and **Node 18+ +with [pnpm](https://pnpm.io/)**. + +```bash +# 1) the service — serves motions on http://localhost:8000 +cd service +uv run uvicorn app.main:app --port 8000 + +# 2) the app — http://localhost:5173 (second terminal) +cd frontend/app +pnpm install +pnpm dev +``` + +Check the service is alive: `curl localhost:8000/health` → +`{"ok":true,"backend":"stub","ml":false,"ready":true}`. The `"ml":false` is the tell — no +model is running. + +If the app can't reach the service it says so on the stage rather than failing silently, and +prints the command to start it. + +--- + +## Keyboard shortcuts + +Every shortcut, in one table. **Shortcuts are ignored while your cursor is in the prompt box** +— otherwise typing the letter "p" would drop you into performance mode mid-sentence. The one +exception is esc, which always works, because you do not want to be hunting for a +mouse in front of an audience. + +| Key | Does | +|---|---| +| R | **Read** — open / close the four notation registers | +| C | **Compare** — open / close the multi-model triptych | +| P | **Perform** — enter / leave performance mode | +| space | play / pause | +| T | cycle tempo: 0.5× → 0.25× → 1× | +| G | ghost-cloud on / off | +| esc | leave the current mode; from the prompt box, unfocus it | +| enter | (in the prompt box) generate | + +**Mouse, on the 3D stage:** drag to orbit, scroll to zoom, right-drag to pan. Standard +three.js orbit controls. + +## Boot flags + +Append to the URL. Useful for projectors and for scripted screenshots — you don't want to be +clicking through chrome in front of a room. + +| URL | Opens | +|---|---| +| `localhost:5173/` | the Lab Bench | +| `localhost:5173/?registers=1` | straight into the four notation registers | +| `localhost:5173/?compare=1` | straight into the triptych | +| `localhost:5173/?perform=1` | straight into the projectable performance stage | + +--- + +## The Lab Bench (the default view) + +Type a phrase, press enter or click **Generate**. A 3D stick figure plays the +returned motion. + +**Deliberately not a realistic avatar.** A realistic body sells an illusion and invites you to +read a *character*; a stick figure exposes the computational body directly — joints, +trajectories, timing, weight. See the README on why. + +Around the stage: + +- **Telemetry** (top left) — model, prompt, seed, joint count, how many other seeds are in the + cloud, and an amber `stub · hand-authored fixture (no ML)` line. That last line is there so + that no screenshot of this app can be honestly mistaken for model output. +- **Lineage rail** (left) — the search so far. See below. +- **Notation rail** (right) — two of the four registers, small: the notation strip and the + floor path. R opens all four, large. +- **Transport** (bottom) — play/pause, a scrub bar, the ghost-cloud toggle, and a frame + counter. Scrubbing moves the figure, the ghosts, and every open register together. + +### The prompt lineage tree — the core contribution + +**Every prompt revision is kept.** Generating does not replace what came before; it adds a +child node. The search branches, and nothing is undone. + +- **Generate** from the current node → extends the line. +- **Click a past node**, then Generate → **branches** from there. The old line stays. +- Clicking a node **replays its stored motion** — no re-fetch, and it restores that node's own + ghost-cloud too. + +The tree is the artefact, not the leftovers. In performance it is the set: the audience +watches not just generated movement but the *evolution of thought* — how a phrase mutated, +where it branched, which possibility was followed and which was left open. + +### The ghost-cloud + +One prompt, many seeds. The solid amber figure is the primary motion; the translucent blue +figures around it are three siblings with different seeds. Toggle with G or the +checkbox. + +Where the lineage tree shows *the search across time*, the ghost-cloud shows *the possibilities +at one moment*. It is hidden in the triptych and the registers view, because there it would +only add noise — the ghost-cloud compares **seeds**, the triptych compares **models**. + +*(In v0 this is a seeded perturbation, not a model sampling. See [`v0-stub.md`](v0-stub.md).)* + +--- + +## The four notation registers — R + +The same motion, made legible four ways at once. A stick figure lets you *watch* movement; it +does not let you **read** it. + +**1 · Chronophotograph.** Marey's plate: seven exposures fading from past to present, so the +whole phrase is visible at once instead of streaming past. The lit pose is *now*. +The horizontal axis is **time, not distance** — as on Marey's *moving* plate. Each pose is +centred on its own pelvis, so the body's travel is dropped (otherwise a motion that slides a +metre sideways walks its last exposure clean off the plate). Seen from a quarter-turn, because +dead-on the hips are only 9 cm apart and the two legs collapse onto a single line. + +**2 · Notation strip.** A time-scored staff, one row per limb — L arm, R arm, spine, weight, +feet. Each glyph's **angle** is the direction that limb travelled, its **length** is how far, +and its **height in the row** is the level. Limbs are read *relative to their anchor* (a wrist +against its shoulder is the arm's gesture; the wrist's absolute position would just re-tell you +where the body is). "Weight" is the exception — the pelvis in absolute terms *is* where the +weight is. A bucket in which nothing moved still gets a dot, so the score never lies by +omission. + +Level and magnitude are normalised **per track**, so "high" means high *for that limb* — right +for reading one motion closely, but it means two strips are not directly comparable. + +**3 · Floor path.** The movement from above: the weight's trace, the feet faint behind it, a +marker where it began and a dot where it is now. Never zoomed in past 0.7 m — a body that +barely travels should *look* like a body that barely travels. + +**4 · Laban-inspired score.** A vertical staff read **bottom → top**. It is a **designed +reduction, not strict Labanotation** — designing that reduction is itself part of the research. + +| Element | Means | +|---|---| +| The central column pair | **support** — which foot is bearing weight. Read from the foot joint; if both feet leave the floor the column goes empty, and the *gap* is the notation. | +| Support shading | how deep the body is sitting (a crouch shades it solid) | +| Outer columns | the limb **gestures** — the body's own left and right, as the performer would read them, not as you watching would | +| Glyph **fill** | **level**: solid = low · hatched = middle · hollow = high | +| Glyph **lean** | which way it went, sideways | +| Glyph **width** | how far it went | + +Unlike the notation strip, levels here are read **anatomically, not statistically** — a hanging +wrist sits ~0.57 m below its shoulder (*low*), level with the shoulder is *middle*, above is +*high*; a standing ankle rests ~0.09 m up (*low*). That means "high" means the same thing in +every motion, so two Laban scores **can** be compared. This is a real difference in kind +between register 2 and register 4, and it is deliberate. + +### No register is complete — that is the point + +Each one throws information away, and **which** thing it throws away is the argument. The floor +path cannot show you a raised arm. The chronophotograph drops the body's travel. The Laban +score leaves forward/back to the floor path. Reading them *against* each other — and noticing +what falls *between* them — is the instrument. All four playheads walk the phrase together, so +that reading-against is possible at all. + +**Try this:** the only fixture in which an arm rises above the shoulder is `reach-and-return`, +so on most prompts the Laban score's level shading is honestly, monotonously "low". To see the +fill actually change (solid → hatched → hollow), find a prompt that lands on that fixture — with +model **snapmogen**, the phrase "slip away" does it. (Yes, really. That is fake #1 in +[`v0-stub.md`](v0-stub.md), and it is exactly as arbitrary as it sounds.) + +--- + +## The triptych — C + +One prompt, three models, side by side: SnapMoGen, Language of Motion, Kimodo. Each keeps its +**native way of authoring** — write / voice / sculpt — because the difference in *how you +author* is itself part of the research. (Only *write* is wired up; the others are labelled, not +built.) Each panel has its own accent colour, and all three play in step — comparing motions +that were out of step with each other would tell you nothing. + +> ⚠️ **The three models differ only because the stub hashes `(model, prompt)`.** They are not +> three models interpreting a theme. In fact two of the three panels usually show the *same* +> base fixture wearing different seeded jitter. There is a banner in the UI saying so, and it +> is not being coy. **Do not use a triptych screenshot as evidence of model comparison.** + +--- + +## Performance mode — P + +The projectable stage, for the lecture performance. Not a separate page: the **same session, +the same lineage.** The performer keeps working — typing, generating, branching — while the +room sees only the body, the phrase and the score. + +What changes: the instrument chrome falls away, the background darkens, a spotlight gathers the +eye onto the body, the phrase goes large beneath it, and **playback drops to half speed** — +because a human has to be able to *follow and re-embody it*. The score gets thicker strokes and +bigger labels: it has to be readable by a body, from across a room. The lineage keeps growing in +the corner — research log as set. + +T cycles the tempo, space plays and pauses, esc gets you out. +`?perform=1` boots straight into it, for plugging into a projector. + +--- + +## Changing the movement itself + +The five motions are hand-authored keyframes in +[`fixtures/_generate.py`](../fixtures/_generate.py). Edit the keyframe lists and re-compile: + +```bash +python3 fixtures/_generate.py +``` + +**One trap, and it has drawn blood.** The fixture authoring uses **translation-only forward +kinematics**: a joint's total offset is the sum of its own delta and all its ancestors'. So +**dropping the pelvis drags the legs down with it** — a crouch needs compensating `+y` knee and +ankle deltas or the feet sink through the floor. `gather` shipped with its feet 7 cm +underground before a foot-plant check caught it. If you author a crouch, check the feet. + +The service loads fixtures at startup, so restart it after re-compiling. + +## Swapping the backend + +The generator behind `POST /generate` is chosen by an environment variable: + +```bash +BODYPROMPT_BACKEND=stub uv run uvicorn app.main:app --port 8000 # the default; the only one that exists +``` + +A real backend implements `Generator` (`name`, `ml`, `ready()`, `generate(model, prompt, +variants)`), returns canonical motion, and registers itself in `_BACKENDS`. **Nothing else in +the system changes** — not the frontend, not the renderer, not one of the four registers. That +is the whole reason v0 was built in this order. See +[`motion-schema.md`](motion-schema.md) for the format a backend must emit. diff --git a/docs/v0-stub.md b/docs/v0-stub.md new file mode 100644 index 0000000..c37c7ca --- /dev/null +++ b/docs/v0-stub.md @@ -0,0 +1,159 @@ +# What v0 fakes + +**A complete inventory of every stand-in in the system, written down in one place so that +nothing in a screenshot can be mistaken for a finding.** + +v0 built the *research instruments* before the model, deliberately: the search loop, the +lineage tree, the ghost-cloud, the four notation registers, the triptych and performance +mode all run end-to-end — on hand-authored data. That was the right order to build in, but +it leaves the repo in a state where **the instrument looks like it is working when in fact +it is not reading your prompt at all.** This document says exactly where the seams are. + +The single sentence version: + +> **There are five hand-authored motions in the entire system. Every movement you have ever +> seen BodyPrompt produce is one of those five, wearing a little seeded jitter. The prompt +> is not read; it is hashed.** + +--- + +## The five fixtures + +Hand-authored keyframes in [`fixtures/_generate.py`](../fixtures/_generate.py), compiled to +canonical motion JSON. All five are 90 frames at 30 fps — exactly 3.0 seconds. + +| Fixture | Phrase it was authored *for* | Tagged model | Seed | What the body does | +|---|---|---|---|---| +| `reach-and-return` | "a body remembers a place it cannot return to" | snapmogen | 4021 | The right arm reaches up and forward, then almost — but not quite — returns. A residual 8 cm remains: the memory. **The only fixture in which an arm goes above the shoulder.** | +| `slip-away` | "slip away" | kimodo | 1177 | The whole body drifts 0.46 m sideways and turns, arms trailing behind. **The only fixture that really travels.** | +| `gather` | "coming home" | language-of-motion | 8802 | Both arms sweep inward to the chest; a slight crouch. | +| `fall-and-rise` | "the ground remembers" | snapmogen | 5310 | A fold down toward the floor (the pelvis drops 45 cm), then a rise, ending a little taller than rest — opened out. | +| `turn-away` | "look back, then go" | kimodo | 2287 | The torso twists and the head turns back over the shoulder, one arm sweeping behind; then the body unwinds and leaves. | + +The "phrase it was authored for" column is **not** a lookup table. Read on. + +--- + +## Fake #1 — the prompt is hashed, not read + +[`StubGenerator.generate`](../service/app/generators.py) picks a fixture like this: + +```python +key = f"{model}:{prompt}" +idx = (sum(ord(c) for c in key)) % len(self._fixtures) # 5 fixtures +``` + +The sum of the character codes, modulo five. That is the entire "understanding" of language +in v0. It is **stable** (the same prompt always returns the same motion, which is what makes +the lineage tree meaningful) and it is **meaningless** (the mapping carries no relation +whatever between what you typed and what the body does). + +Here is the proof, and it is sharper than I expected. These are the five phrases the fixtures +were *authored for*, run through the real hash: + +| You type… | snapmogen gives you | language-of-motion gives you | kimodo gives you | +|---|---|---|---| +| "a body remembers a place it cannot return to" | `slip-away` | `gather` | `slip-away` | +| "slip away" | `reach-and-return` | `fall-and-rise` | `reach-and-return` | +| "coming home" | `fall-and-rise` | `slip-away` | `fall-and-rise` | +| "the ground remembers" | `slip-away` | `gather` | `slip-away` | +| "look back, then go" | `turn-away` | `reach-and-return` | `turn-away` | + +**Not one phrase returns the motion it was written for.** Type "slip away" and the body +reaches for something. Type "a body remembers a place it cannot return to" — the project's +own signature phrase, the one in the abstract — and you get a body sliding sideways, which +is not what that fixture was authored to express at all. + +This is not a bug to be fixed. It is what "the system does not understand what you type" +*looks like* when you write it down honestly. It is fixed by a model, not by a better hash. + +## Fake #2 — the triptych's three models + +Two models collide. `sum(ord(c))` for `"snapmogen"` and for `"kimodo"` happen to be congruent +modulo 5 — so for **every** prompt, those two panels select the **same base fixture**. Look at +the table above: the snapmogen and kimodo columns are identical, row for row. + +What stops the triptych from showing literal twins is this, in `generate()`: + +```python +model_sig = sum(ord(c) for c in model) * 7919 if model else 0 +seed = int(base.get("seed", 0)) + model_sig +motion = vary(base, seed) if model_sig else dict(base) +``` + +Each model gets a stable signature, and the motion is jittered by it. So in a two-thirds of +the triptych, **the visible difference between "SnapMoGen" and "Kimodo" is nothing but seeded +noise applied to one identical fixture.** The remaining panel differs only because a modulo +landed elsewhere. + +The triptych carries an in-UI banner saying the differences are a stub artefact. That banner +is doing heavy lifting, and it is not overstating the case. **Nothing in that view can be read +as a finding about model behaviour.** Do not put a triptych screenshot in a paper as evidence +of models interpreting a theme. + +## Fake #3 — the ghost-cloud is a perturbation, not a sampling + +`vary(motion, seed)` gives every joint a smooth, low-frequency sinusoidal wander — amplitude, +frequency and phase drawn from a seeded RNG, amplitude scaled per joint by the `_WANDER` +table (pelvis ~1.2 cm, wrists ~7 cm; vertical wander damped to 0.6 so the figure doesn't bob +off the floor). + +That table is a *guess at the shape* of a model's variance — the root and spine barely move +while the extremities vary a lot, so the cloud reads as "the same intention, differently +expressed" rather than as noise. A real model's variance does behave roughly like that. But +this cloud is **not a model sampling different outputs.** It is one motion, wobbled five ways. + +The claim the ghost-cloud is currently entitled to make is only: *here is what a variance +display would look like, and here is the interface for reading one.* + +## Fake #4 — rotations are empty, and the FK is translation-only + +Every frame carries a `rotations` array, and in v0 it is **entirely identity quaternions** — +reserved space in the schema, filled with nothing. All motion lives in `positions`. + +The fixtures themselves are authored by summing per-joint translation offsets down the +kinematic chain (see `total_offset` in `_generate.py`) — not by rotating limbs. This has a +consequence that bit once already: **dropping the pelvis drags the legs down with it**, so +every crouch needs compensating `+y` knee and ankle deltas or the feet sink through the +floor. `gather` shipped with its feet 7 cm underground until a foot-plant check caught it. + +A real model will emit rotations. When it does, the renderer and the notation registers keep +working (they read positions), but anything that wants joint *orientation* — a facing +direction, a proper Labanotation front/back symbol — becomes possible for the first time. + +--- + +## What is **not** fake + +Everything the fakes are wearing. This is the part that survives v1 untouched, and it is why +building the instrument first was worth doing: + +- **The canonical schema** (`bodyprompt.motion/v0`) — the exchange format, and the whole + reason a model swap is a swap rather than a rewrite. → [`motion-schema.md`](motion-schema.md) +- **The `Generator` interface and the `POST /generate` contract.** A real backend implements + three methods and registers itself in `_BACKENDS`. Nothing else in the system changes. +- **The stick-figure renderer** — reads canonical motion, knows nothing of where it came from. +- **The prompt lineage tree** — the core contribution. Branching, retention, click-to-replay. + Real now, real later. +- **All four notation registers** — chronophotograph, notation strip, floor path, + Laban-inspired score. Every glyph is derived from joint trajectories. They are *real + readings of fake data*, and the moment the data is real, so are they. +- **The triptych as an instrument** (the comparison is real; the models are not). +- **Performance mode**, the transport, the tempo control, the ghost-cloud *display*. + +## Retiring the fakes + +Every one of them dies at v1, and they die together — that is the point of having built it +this way. One backend that returns joints instead of a fixture, and: + +| Fake | Retired by | +|---|---| +| #1 the prompt hash | the model reading the prompt | +| #2 the triptych's model signature | three actual models | +| #3 the ghost-cloud perturbation | asking the model for *n* samples | +| #4 identity rotations | a model that emits them | + +The one thing v1 must not do is return **rendered video**. Several hosted motion-generation +APIs do exactly that, and a video cannot be fed to the renderer, the ghost-cloud, or any of +the four registers — it would throw away the entire instrument this repo is. **v1 needs +joints.** diff --git a/frontend/app/src/main.ts b/frontend/app/src/main.ts index 025237a..227492a 100644 --- a/frontend/app/src/main.ts +++ b/frontend/app/src/main.ts @@ -187,8 +187,11 @@ function setReading(on: boolean): void { // score. Slowed down, because a human has to be able to follow and re-embody it. const TEMPOS = [0.5, 0.25, 1]; // performance opens at half speed; T cycles +const FULL_SPEED = TEMPOS.indexOf(1); let performing = false; -let tempoIdx = 0; +// Where T picks up from. The bench plays at full speed, so it starts here and the first +// press steps to 0.5x; performance mode re-seats it at 0.5x (index 0) when it opens. +let tempoIdx = FULL_SPEED; function setTempo(rate: number): void { renderer.setTempo(rate); @@ -205,11 +208,12 @@ function setPerforming(on: boolean): void { scoreTitleEl.textContent = on ? "the score · for the body" : "notation · the score"; if (on) { - tempoIdx = 0; + tempoIdx = 0; // performance opens at half speed — a body has to be able to follow it setTempo(TEMPOS[tempoIdx]); renderer.play(); } else { - setTempo(1); + tempoIdx = FULL_SPEED; + setTempo(TEMPOS[tempoIdx]); } }