Skip to content

Magenta RT 2 backend family (sidecar-hosted, append-only)#230

Draft
ryanontheinside wants to merge 2 commits into
mainfrom
ryanontheinside/feat/models/mrt2-on-backend
Draft

Magenta RT 2 backend family (sidecar-hosted, append-only)#230
ryanontheinside wants to merge 2 commits into
mainfrom
ryanontheinside/feat/models/mrt2-on-backend

Conversation

@ryanontheinside

Copy link
Copy Markdown
Collaborator

DRAFT: Magenta RT 2 as a backend family (sidecar-hosted, append-only)

Merge order

Draft, stacked: opened against ryanontheinside/feat/models/backend. Merge #224 -> #220 -> models/backend before this.
Stack: main <- #224 <- #220 <- models/backend <- this (draft). Sibling leaf: sa3 (draft, same parent).
Heads-up for whoever merges the second leaf: families.py will conflict (both leaves register their family in the same dicts); streaming/session.py resolves clean apart from one comment line.

First non-diffusion model family behind the GeneratorBackend seam, validating that the seam absorbs a generator with a completely different shape: Google's Magenta RealTime 2, running in a WSL sidecar process, append-only (no refinement, no positional source), rolling-window song instead of fixed-length.

What's here (two commits)

a9ff406 mrt2 backend.

  • acestep/streaming/mrt2/protocol.py: versioned, language-neutral length-prefixed frame protocol (JSON control + raw PCM audio frames) over TCP. Deliberately generic: this is the template for absorbing ANY future generator in any runtime.
  • scripts/mrt2_sidecar.py: the model host. Owns the Magenta RT 2 checkpoint inside its own venv/runtime (WSL), streams PCM frames on credit grants. Model variant is picked at sidecar launch (--model=X).
  • acestep/streaming/mrt2/backend.py: MRT2Backend behind the seam. Frontier-driven emission with crossfade splicing into a rolling window; credit accounting; prompt and style-blend forwarded to the sidecar's MusicCoCa conditioning (handle_set_prompt, handle_set_prompt_blend). Capability mask is all-off except generation: every ACE-shaped command (swap, timbre, structure, LoRA, depth) fails loudly via command_failed.
  • create_mrt2_session registered in families.SESSION_CREATORS: the family bypasses the ACE create path entirely (no TRT profiles, no model load, no demucs, no conditioning encode). Honest nullable metadata (no bpm/key) per the contract surface.
  • tests/unit/test_mrt2_backend.py: protocol framing, frontier/window math, credit flow, prompt forwarding, capability mask (13 tests, no sidecar needed).

dc2dc9c magenta pedal. /magenta web page: a minimal pedal-style UI (prompt, blend, temperature) built on the @demon/client SDK and gated by the served capability mask, demonstrating that a non-ACE family gets a working frontend without touching the shipped app.

Transactional create (carried from #222's semantics)

The sidecar TCP link is a real acquired resource, so a failed session create cannot strand it: MRT2Backend.__init__ closes a self-acquired client if construction fails after connect (injected test clients stay caller-owned), StreamingSession.__init__ closes the backend if init fails after make_backend, and the factory registers the audio engine on an ExitStack with pop_all() ownership transfer, mirroring the ACE create path. StreamingSession.close() closes backend-owned resources first.

Status / why draft

Live-verified end-to-end against the WSL sidecar (append-only frontier contract, prompt steering). Remaining before ready-for-review: sidecar deployment story for pods, and a golden-harness scenario for the family. scripts/mrt2_live_check.py is the manual smoke (needs the sidecar running).

Validation (2026-06-06, restacked tip)

160 unit tests passing at this tip (mrt2 suite + all inherited guards), web typecheck clean.

Base automatically changed from ryanontheinside/feat/models/backend to main June 8, 2026 13:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant