fix(server): extract shared runtime into jobs_registry.py (P0: double-imported server.py)

[chazmaniandinkle]

Field incident

PR #139 (job retention + adaptive pre-playback buffer) was merged to main (198910c) and deployed. Shortly after, a P0 surfaced in production and the deploy was rolled back to ce42990 (pinned, healthy). Observed in order:

1. Daemon restarted on the merged main (198910c).
2. A /v1/speak utterance began playing audibly.
3. The operator heard it cut off mid-utterance at the moment a GET /v1/jobs/{id} poll hit the daemon.
4. The job then sat frozen in status "speaking" / HUD "encoding" / progress 0.0 for minutes, never terminal.
5. /diagnostics showed jobs: {total: 0} while GET /v1/jobs/{id} did return the record.

Root causes — confirmed vs refuted

Confirmed: server.py's __main__/import duality double-executes the whole speech runtime. In production, python server.py --http makes server.py execute as __main__. http_api.py read shared state via six separate lazy from server import ... calls (_bus, _jobs, _speech_queue/pipeline_state, _start_speech/check_bargein_gate, _dashboard_chat_broadcast, _dashboard_chat_register/_dashboard_chat_unregister) — some going back further than PR #139. Because a module run as __main__ is never the same module object as one later imported by its own name, the very first such import silently re-executes server.py's entire top-level body a second time under the module name "server". I verified this empirically with a temporary instrumentation probe (module-level log line + thread id) launched via a real subprocess exactly matching production's invocation: two distinct executions, two distinct _bargein_watcher thread ids, same OS pid. Both instances (__main__ and the phantom "server") construct their own ModalityBus, job registry, SpeechQueue, pipeline_state, and barge-in watcher thread. install_mcp_route()'s mcp reference resolves through __main__'s own globals (it's defined in server.py itself), so MCP-tool calls (speak, diagnostics, ...) and HTTP calls (/v1/speak, /v1/jobs) could end up talking to different instances depending on which surface was used — exactly matching symptom (5): /diagnostics (MCP-tool-backed) read __main__'s empty registry while /v1/jobs/{id} (HTTP, using the lazily-imported "server") found the record.

Refuted (or: unproven and out of scope) — the double-import causing the audible mid-utterance cutoff specifically. Re-examined against the daemon log (~/workspaces/cog/.cog/run/services/mod3.log): both the mid-utterance cut on the new build (~13:47) and the identical cut on the rolled-back pre-merge build (13:54:11, job d7105d13) show the same sequence — Barge-in: cross-process interrupt (pid=10048) → Barge-in from mic_vad: paused local playback (36%/52 of 156 chars delivered) → Speaking lock no longer ours — stopping generation. pid 10048 is the daemon's own process (its own mic_vad listener; SuperWhisper also running on the host) — this is a real, in-room voice triggering barge-in, not an artifact of the double-import. It reproduces on the rolled-back build too, so it predates this PR and isn't attributable to the import bug. Barge-in is behaving as designed here.

Confirmed (new, found via the same log evidence): a barge-in-interrupted job never reaches a terminal state. pipeline_state.interrupt() calls player.flush() directly — silences audio immediately — but nothing told _run_speech_job's generation loop to stop calling the TTS engine for more chunks. Its only prior stop signal was the file-based speaking-lock check, which an in-process interrupt never touches. The loop kept synthesizing the rest of the (unheard) text and the job stayed "speaking"/HUD "encoding" until that finished on its own — this is what actually produced symptom (4), on both builds, independent of the import bug. (The /diagnostics: total:0 disagreement, symptom (5), also has a simpler standalone explanation I confirmed by reading the code directly: the HTTP GET /diagnostics endpoint only ever read http_api.py's own _jobs ledger, never the /v1/speak registry at all — now fixed as part of the same-class merge below.)

Fix

• New jobs_registry.py — the single shared runtime: job registry (_jobs, _prune_jobs), SpeechQueue/_start_speech/_run_speech_job, the ModalityBus (_bus), pipeline_state, the barge-in watcher thread + registry + speaking-lock file helpers, the dashboard-chat relay, check_bargein_gate, and the @mcp.tool()-decorated MCP surface (speak, stop, speech_status, diagnostics, list_voices, set_output_device, register_session, output, mod3_dashboard_post, ...). This module is never run as __main__, so it is only ever imported once, under one name, regardless of who imports it first.
• server.py is now a thin CLI entrypoint: argument parsing, _run_http, install_mcp_route, _start_inbound_pipeline_if_enabled, _prewarm_tts_if_enabled — importing everything else from jobs_registry.py.
• http_api.py now imports directly from jobs_registry.py everywhere (no more lazy from server import ... anywhere in the file — audited and converted all six call sites).
• GET /diagnostics now reports topology.server_reimported ("server" in sys.modules) — should always be False; a direct, permanent observability hook for this exact regression class, plus its job counts now include the /v1/speak registry (closing the standalone gap noted above).
• Barge-in job finalization: the generation loop now also checks the same InterruptInfo its own interrupt callback already receives (in addition to the pre-existing lock-loss check); either path finalizes the job as "interrupted" (not "done") with partial spoken_pct/delivered_text/reason plus the buffer/starvation fields from the adaptive-buffer feature. /v1/stop and MCP stop()'s active-job branch now route through pipeline_state.interrupt() too (falling back to a direct player.flush() only if pipeline_state didn't think anything was speaking), so a manual stop finalizes the job record the same way a barge-in now does.
• Env vars MOD3_BARGEIN_SIGNAL_PATH / MOD3_SPEAKING_LOCK_PATH (default unchanged) let tests (and any isolated invocation) point the barge-in file/lock at a tmp path instead of the real daemon's files.

New tests

• tests/test_server_topology.py — launches server.py --http as a real subprocess (the actual deployed topology; an in-process import never exhibits this bug because pytest always imports server.py under its own name). Starts a real speak job, polls GET /v1/jobs/{id} mid-flight (the exact operator action from the incident), and asserts: the poll doesn't disturb the job, the job reaches a clean "done" (not killed), and GET /diagnostics reports topology.server_reimported == False. Gated on Apple Silicon / mlx + mcp availability (same skip convention as the rest of the suite).
• tests/test_barge_in_finalization.py — an in-process interrupt mid-generation finalizes the job as "interrupted" with correct partial metrics and buffer fields (not stuck at "speaking"), the generation loop stops pulling chunks promptly rather than draining a 200-chunk fake utterance, the pre-existing lock-loss path finalizes identically, and stop()'s active-job branch routes through pipeline_state.interrupt() (with a direct-flush fallback verified for the edge case where nothing was speaking per pipeline_state).
• Updated 12 existing test files whose mocks/patches targeted server.X for names that moved to jobs_registry.X (SpeechQueue, _jobs, _run_speech_job, speak/stop/speech_status/diagnostics, the barge-in/speaking-lock helpers, _dashboard_chat_*, etc.) — audited every from server import / import server / patch("server....") / monkeypatch.setattr(server, ...) in the test suite; only the ones for names that genuinely stayed in server.py (install_mcp_route, _start_inbound_pipeline_if_enabled) were left as-is.

Test evidence

• ruff check ., ruff format --check . clean.
• pyright --pythonversion 3.12 http_api.py server.py jobs_registry.py clean (the one new pyright complaint the split introduced — mcp._mod3_route_installed = True looking like an unknown attribute once mcp arrives via cross-module import rather than local construction — fixed with setattr).
• Full suite: 1059 passed, 1 skipped, ~28s.
• Manual empirical verification of the fix and the original bug: with the topology probe instrumentation, the pristine pre-fix code reproducibly showed two server.py executions (distinct _bargein_watcher thread ids, same pid) on a bare server.py --http startup, before any request ever arrived; after this fix, "server" not in sys.modules throughout.

Deployment

Do NOT restart or redeploy anything — ~/worktrees/mod3-deploy stays pinned at ce42990 per the coordinator's instruction; deploy/redeploy stays with the seat. This PR is ready to merge to main; the fix takes effect on the next intentional daemon restart, same as the original PR.