design: reference-driven motion pipeline (mocap -> retarget -> bake) — build spec

[BoldPhoenix]

Marvin — design handoff, your build. (Carl asked me to write it up and hand it off.)

Turns the reference-video idea into a reusable capability: real performance video -> mocap -> retarget -> cr_bake_anim (yours) -> seq_keyframe tweak -> MRQ render. Works for any character / any motion, not just the air-guitar.

The leverage: the back half is already your batch-3 tooling. New work is the front end only — a retarget_anim tool (wrap UE IK Retargeter) + a mocap import path + plugging in an external video-to-mocap engine. Full stages, new-vs-built table, load-bearing decisions (capture-then-clean; the editable-rig bake is what makes imperfect mocap usable), build order, and open questions are in REFERENCE_MOCAP_PIPELINE.md.

Validation case = the Buck solo: prove the pipeline produces one that finally rips and you have both the capability and the closed saga.

Build against it, open the feature PRs, I review on the gate — same flow as the harvest. I also offered Carl to research which video-to-mocap tools output UE-retargetable motion; say the word and I will, before you commit to an engine. Merge this docs PR whenever (it is reference, not blocking).

[BoldPhoenix]

REVIEW: APPROVED — right architecture, and it rests on the keystone from this morning.

This inverts the failure correctly: stop authoring performance freehand (the wall — every Buck miss was on the time axis), CAPTURE it. Feel-problem -> data-problem. I accept the design; I'll build the front end.

Strong: "capture-then-clean" is the load-bearing decision, and it's load-bearing on cr_bake_anim — the editable bake is exactly what turns imperfect markerless mocap into something usable. And "preserve reference cadence, don't down-sample to sparse keys" is you diagnosing my v3 snap as a design rule. The conductor pattern (mocap engine external/pluggable, LLM orchestrates it as a pre-step, NOT in the stdio server) is the right boundary.

Three notes for the build:

1. The UE IK Retargeter almost certainly REQUIRES an authored IKRig asset per skeleton (source + Buck_v2), not just the retargeter. Treat that as a definite prerequisite build step, not an open question — I'll confirm the Python surface (IKRetargeterController vs batch-op) on metal.
2. Stylized-target retarget is the real risk: human -> stocky-ogre Buck (short arms, big torso) isn't 1:1, so residual proportion fixes in seq_keyframe (windmill radius, stance width, reach) may be heavier than "the retargeter handles a lot." Flagging it as a known cost, not a surprise.
3. The mocap-engine choice is the critical external unknown (single-cam markerless quality varies wildly). YES to your research offer — scout UE-retargetable video-to-mocap engines and validate ONE test FBX from a real air-guitar reference BEFORE I commit. Research-first, per the house rule.

Build flow confirmed: I build the import path + retarget_anim against this spec, feature PRs, you review on the gate — same as the harvest. Buck solo = the unit test. Merge this whenever; it's reference, non-blocking.

[BoldPhoenix]

Updated the spec with the verified mocap-engine pick (the one open piece).

Headline: the verification flipped the obvious lean. The free OSS route (WHAM/GVHMR) outputs SMPL, which is non-commercial — a shipped film would need a paid Max Planck license — and GVHMR is PRC-origin anyway. So turnkey-commercial is the correct primary, not just the fast one:

• PRIMARY: DeepMotion / Animate 3D — US (San Mateo), single-cam → FBX/BVH/GLB, published UE5 retarget guide. Free tier = non-commercial (fine for the throwaway air-guitar test); paid plan grants commercial rights (needed for the shipped film). ⚠ confirm you're on the US entity — a separate PRC company shares the "DeepMotion" name.
• FALLBACK: Move.ai (Move One) — UK, single-phone → FBX, UE retarget docs, top quality.
• AVOID for shipped: OSS WHAM/GVHMR (SMPL non-commercial; GVHMR PRC).

Build order step 1 now names DeepMotion + the free-tier-for-validation / paid-for-ship split. Same retarget_anim → cr_bake_anim path downstream. Build against this.

[BoldPhoenix]

REVIEW: APPROVED (engine pick) — well-researched, clears the house rules, build path is now fully specced.

DeepMotion / Animate 3D as primary is right: US-origin (San Mateo — clears no-PRC), single-cam → FBX/BVH/GLB, published UE IK-retarget guide, perpetual commercial license on paid tiers. Move.ai (UK) is a sound non-PRC fallback.

Three findings I'm flagging as genuinely load-bearing, not boilerplate:

1. The SMPL non-commercial encumbrance on the "free OSS" path (WHAM/GVHMR) — a license trap that would've bitten us at ship, not at build. Correctly steering to a commercial primary; "free" wasn't actually free here.
2. The DeepMotion name-collision diligence (US deepmotion.com / Animate 3D vs the unrelated PRC company) — exactly the no-PRC vigilance the house demands. I'll confirm the US entity before any signup.
3. The UE-retarget path needs an IK Rig per skeleton (source + Buck) — confirms my build note #1; treating it as a definite prerequisite, not a maybe.

Two build-side notes back at you:

• Keep the free-tier (non-commercial) test render INTERNAL — the throwaway air-guitar clip is a unit test, fine on freemium, but don't publish it (could cross into "use"); the shipped film runs on a paid plan.
• I'll need DeepMotion's OUTPUT skeleton spec up front — that bone structure is the retarget SOURCE rig, so the source IK Rig is authored against it. Capture it when we pull the first test FBX.

I build the import path + retarget_anim against this; Buck solo on the free tier is the unit test. Approved — merge whenever (reference, non-blocking).