Skip to content

fix(playback): deliver the default-audio correction across reconnects - #233

Merged
drondeseries merged 16 commits into
mainfrom
fix/cold-start-default-audio
Oct 6, 2026
Merged

drondeseries merged 16 commits into
mainfrom
fix/cold-start-default-audio

Conversation

@drondeseries

@drondeseries drondeseries commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Problem

A movie whose probe reorders the audio tracks starts in the wrong language. The server detects the stale default, settles one correction, and withdraws the plan — but if the client disconnects, fails the replan, or reconnects cold, the withdrawal was already marked delivered and never repeated. The correction was lost for the rest of the session and the wrong language played on until restart.

Two further defects surfaced during review. The v2 replan schema did not declare answers_plan_invalidation, and because that body forbids additional properties a capable client's replan was refused outright rather than losing the correlation — so the feature worked on v1 and was broken on v2. And both Postgres ledger reads cast audio_reconcile_ledger->>'revision' to bigint; a fresh attempt's ledger is {}, which yields SQL NULL, so the first read of any new attempt failed and the durable ledger could never bootstrap on the production store.

Related issue: N/A
Validation tasks: none

Approach

Delivery now stops on completion, not on send: the withdrawal repeats until the attempt's plan id moves off the withdrawn plan, bounded by a burst of 20 accepted deliveries, a 15-minute cooldown, and budget rearmed on a genuine reconnect (the reconnect rearms budget; subsequent evaluations deliver again). The budget is process-local, not cluster-global. Missing sockets do not consume budget. Capacity is reserved atomically before each send under one lock, and failed writes refund against an epoch so a stale failure cannot weaken a rearmed burst.

The correction is gated on a new client capability, default_audio_reconcile_response_v1. A client that advertises it answers a withdrawal by naming its reason on the replan; the server treats a missing marker from such a client as an ordinary re-pick and keeps the viewer's selection. The identity heuristic remains the fallback for clients that have not adopted the echo, and for them alone.

The v2 replan body declares the answer as an optional request property. The contract linter classifies it as a new optional request property — additive and non-breaking, as the v2 additive-only rule requires before it locks.

Validation

  • Full Go suite passes, including the API handler and planstore packages.
  • Build, formatting, vet, and changed-line lint are clean.
  • Contract, OpenAPI, web-type, playback-fixture, and settings-binding gates are current; the only wire change is the additive optional property.
  • The Postgres regression was run against a real pgvector database migrated with this branch's own code (563 migrations applied). Reverting the fix makes it fail with cannot scan NULL into *int64, so the test is proven non-vacuous.
  • Not run: a live client session. The reconnect-during-pending-correction path is the one to watch on a real deployment.

Risks

Adds one migration (two nullable jsonb columns defaulting to {}) and one client capability. The capability is opt-in; clients that do not advertise it keep the previous behavior, with the correction deferred to the next cold start. Server-owned provenance is still stripped at the ingress boundary, so clients cannot claim server-owned reconciliation provenance. Apple and Android clients do not yet send the answer, so they continue on the deferred path until they adopt the capability.

Checklist

  • I read and can explain the complete diff.
  • This pull request addresses one concern.

AI Disclosure

  • Harness: OpenCode
  • Tool(s): oracle (review subagent)
  • Model(s): workhorse (omniroute)

drondeseries and others added 16 commits October 5, 2026 23:04
…eorder

Persist the start-time audio selection intent (origin, preferred language,
series snapshot, committed signature) on the attempt and the session, then
replay it against the verified inventory when probe evidence lands. An
automatic track_change replan moves the executable selection only when the
language moved; identical selections stay a byte-equal no-op.
…orged provenance

Establish the reconcile budget once at the entry point and propagate it
instead of letting each interior caller discard cancellation and arm a
fresh timeout. Clear client-supplied Automatic on the inbound replan
boundary so only server-built reconciliation can claim that provenance.
…ption

Reconciliation no longer self-commits a replacement plan the client
never adopts. It persists the corrected decision atomically with the
canonical request, withdraws the active plan with plan_invalidated, and
lets the client's own replan commit the corrected recipe.
…keep the winner

The pending default-audio correction was layered onto the replan request
after the selection had already been copied onto the executable start,
so audio resolution never saw it. Apply the correction first. A later
divergence refusal is an audit record, not a verdict on the settled
decision, so it must not strand a correction already announced.
…ntics

The cold-start audio correction withdraws a plan whose route is healthy —
the committed recipe plays the wrong audio stream, not wrong bytes — but
the client treated every `plan_invalidated` as a `failure_recovery`, which
folds the current plan's attempt key into `attempted_plan_keys`. That
excluded the route that was playing from its own replacement plan, pushing
a working session onto a worse route or a terminal.

Replan off `default_audio_reconciliation` as `track_change` instead: it
carries the audio correction, and since nothing failed the route stays
eligible. Position and pause state are preserved as before, and
`fallback_reason` still reports the server's own reason, so telemetry is
unchanged and already distinguishes the two cases. Every other reason keeps
the recovery semantics §6.1 specifies.

The browser cannot import the Go constant, so each side pins the other's
string with a test. No new operation or feature: the path reuses
`track_change`, so the client capability list and the v3 operation enum are
unchanged.
An explicit audio identity on the answering replan supersedes the pending
automatic correction. A correction the server applies restores its own
Automatic provenance, which the inbound boundary strips from clients, so
a server decision is never persisted as a viewer preference.
…ponse

The web request builder echoes the plan's own audio on every replan, so
the answering request carries the withdrawn selection even when the viewer
chose nothing. Presence is not intent: only an identity the withdrawn plan
did not select is a viewer choice, and everything else still receives the
correction, marked automatic.
The server now gates the default-audio reconciliation withdrawal on a new
client capability, default_audio_reconcile_response_v1, and correlates the
client's answer with its pending decision via an answers_plan_invalidation
field on the replan. An older client that lacks the token gets no
withdrawal and plays the corrected audio on its next start, instead of
laundering the withdrawal through failure_recovery.
A client that would replay the withdrawal as failure_recovery excludes a
healthy route from its own replacement, so the withdrawal is gated on a
capability that means the client answers it correctly; older clients
deliberately receive none and get the correction on their next start.
The answering replan now correlates through answers_plan_invalidation
instead of an inferred request shape, with the identity heuristic kept as
the fallback.
…lity

The capability golden records are the published list of what the server
advertises, so the new feature token has to appear in them. The negative
native fixtures are derived from the valid one and must differ from it in
exactly their intended violation, so they carry the token too.
A capability-aware client is distinguished by its answers_plan_invalidation
marker. Absent that marker on such a client, the request keeps its own
selection rather than being treated as a reconciliation response, so a
deliberate same-identity pick can no longer be silently replaced.
A successful hub send proved only that the server wrote the command. A
client that disconnected before processing it, failed its replan, or
reconnected later lost the correction for the rest of the session while
the ledger claimed delivery. Delivery now stops when the attempt's current
plan moves off the withdrawn plan, which any replan commits, and retries
under a stable per-(session, generation) command id until then.
The web client sends answers_plan_invalidation when it answers a
default_audio_reconciliation withdrawal, but the v2 replan schema did not
declare the property. Because the v2 body forbids additional properties, a
capable client's replan was refused outright rather than merely losing the
correlation, so the correction could not land on the native surface at all.

Add the optional request property to PlaybackReplanBody, map it in domain(),
and regenerate the OpenAPI artifact, web types, and contract fixtures. The
contract linter classifies the change as a new optional request property:
additive and non-breaking, which is what the v2 additive-only rule requires
before it locks.

Also correct two protocol-doc errors found alongside it: the identity
heuristic is a fallback for clients WITHOUT the echo capability, not for
every mismatching echo, and the plan_invalidated example carried an HTML
comment inside its JSON block. Add the missing conformance scenarios for the
marked answer and the unmarked re-pick, so the correlation is pinned on the
wire instead of only in prose.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
A newly saved attempt carries an empty audio_reconcile_ledger '{}', and
'{}'::jsonb->>'revision' is SQL NULL. Both ledger reads cast that expression
to bigint and scan it into an int64, so the first read failed and the durable
reconciliation ledger could never bootstrap on the production Postgres store:
no correction could be recorded or announced for an ordinary new attempt.

Wrap both reads in COALESCE(..., 0). Changing only the JSON tag or the migration
default would not cover rows already written as '{}'.

Add a Postgres-backed regression that saves an ordinary attempt, reads revision
zero, records the first decision, and reads revision one. It is gated on
SILO_TEST_DATABASE_URL like the rest of the planstore suite, so it skips where
no migrated database is configured and runs in CI.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The repeated "mul" and "unknown" literals in audio_select.go tripped
goconst and turned both lint CI jobs red. Extract the sentinel strings into
named constants used at both sites, following the typed-const convention of
the surrounding playback package. No behavior change; the playback suite
still passes and lint-changed reports 0 issues.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@drondeseries
drondeseries merged commit 9303ab4 into main Oct 6, 2026
8 checks passed
@drondeseries

Copy link
Copy Markdown
Collaborator Author

Oracle production-readiness review of b59b799d — not ready to merge: two integration gaps still strand corrections.

  1. Retries never run on the v2 lifecycle. reconcilePendingAudioStartup is only called by legacy HandleUpdateProgress (playback.go:2251). V2 progress (playback_service.go:439–454) does not call it, and the reconnect hook (playback_reconcile_audio.go:1417–1426) only rearms counters. A missed initial send can remain undelivered across reconnects. Wire evaluation into v2 progress/socket attachment and test those entry points.

  2. The web client discards retries after failure. Withdrawals reuse a deterministic command ID (playback_reconcile_audio.go:1144–1149), but usePlaybackRealtime.ts:132–159 marks it seen before execution and retains it after rejection. A transient replan failure makes later deliveries no-ops until reconnect. Keep in-flight/success deduplication while allowing failed corrections to retry; cover this through the realtime hook.

Also, reconnect marks expire after 10 minutes while cooldown lasts 15, so an exhausted session reconnecting between those bounds does not rearm.

The migration and NULL-safe ledger reads look correct. Both columns are actually NOT NULL DEFAULT '{}', contrary to the summary. The v2 property is optional and additive. The Postgres regression uses real storage, but skips without its database environment; the inspected CI configuration does not explicitly run it against Postgres.

I reviewed source at b59b799d; I did not rerun tests or perform live-client validation.

NEEDS_FIX

@drondeseries

Copy link
Copy Markdown
Collaborator Author

@randrini — playback diagnosis since the last Vio start (build 932, rev 9303ab4e8, container up 2026-10-06 13:12:41 UTC). Times UTC.

One caveat: the provider log (provider candidates fetched) carries no request id, so per-play Stremio time cannot be joined exactly. Per-play resolve below is Vio's file_load_probe_ms. Aggregate Stremio numbers are separate.

Aggregates

  • Click-play to Vio response (protocol v3 start timing total_ms, n=22): avg 1,541 ms, min 6 ms, max 7,704 ms. Inside it: resolve/probe avg 1,055 ms, commit avg 517 ms, planner ~0 ms.
  • Stremio/addon fetch (provider candidates fetched duration_ms, n=23): avg 2,449 ms, min 514 ms, max 4,245 ms.
  • Virtual resolve (n=23): avg total 991 ms, cache hits 0/23.
  • Route events: plan_selected 40, stopped 20, first_frame 18, plan_failed 1, terminal 1.

Per play

Click-play is admin_playback_history.started_at. Vio start is protocol v3 start timing total_ms. Resolve is file_load_probe_ms. Commit is session_transport_commit_ms. Sel to ff is first_frame.received_at minus plan_selected.received_at for the same attempt.

Click play Vio start Resolve Commit Sel to ff Method/cand Watch Note
13:23:14 movie 967 ms 931 17 3 s direct/1 23 s/21 s ~1 s resolve
13:23:45 movie 1,761 ms 1,738 13 4 s direct/1 6 s/3 s ~1.7 s resolve
13:24:03 series ep 2,261 ms, no stage split — — none terminal virtual_source_unavailable no history row failed: virtual playback candidate failed on the same series content, empty diagnostics
13:24:23 series ep 791 ms 766 11 4 s direct/5 12 s/6 s ok
13:24:50 series ep 13 ms 2 6 2 s direct/1 24 s/23 s healthy, warm
13:25:14 series ep 6 ms, no stage split — — none no plan row no history row failed: virtual playback candidate failed on the same content
13:25:22 series ep 1,727 ms 1,709 9 8 s direct/5 13 s/3 s slow resolve plus 8 s to first frame
13:25:45 series ep 1,570 ms 1,553 8 none direct/5 46 s/0 s failed: background probe fails, then 4x transport startup failures with empty error
13:26:38 series ep 1,655 ms 1,633 10 4 s direct/5 10 s/4 s ok after retry
13:27:05 series ep 3,335 ms 3,325 5 none direct/0 cand 9 s failed: zero candidates, then client 3003 PARSING_CONTAINER_UNSUPPORTED on wifi
13:27:25 series ep 3,368 ms 3,358 5 4 s direct/1 completed slow resolve, then completed
13:27:46 series ep 2,123 ms 2,096 19 4 s direct/5 20 s/15 s ~2 s resolve
13:28:15 series ep 1,880 ms 1,856 14 4 s direct/5 9 s/4 s ~1.9 s resolve
13:28:38 movie 49 ms 30 8 4 s direct/5 10 s/4 s healthy
13:29:54 movie 37 ms 19 5 3 s direct/5 5 s/1 s healthy
13:30:04 movie 81 ms 67 8 4 s direct/5 8 s/2 s healthy
13:30:16 movie 10 ms 0 5 3 s direct/1 6 s/4 s healthy
13:30:27 movie 39 ms 28 5 7 s direct/5 13 s/2 s Vio fast, 7 s server to first frame is client side
13:30:47 movie 1,845 ms 1,831 5 4 s direct/5 6 s/1 s ~1.8 s resolve
13:30:58 movie remux 7,704 ms 92 7,598 2 s remux/5 hls_audio_adaptation 30 s/18 s remux commit dominates, resolve fast
13:31:33 movie remux 2,650 ms 53 2,588 1 s remux/5 hls_audio_adaptation 9 s/3 s same remux commit pattern
13:31:45 movie 26 ms 15 6 10 s direct/5 14 s/2 s Vio fast, 10 s to first frame is client/transport side

Issues

  1. 13:24:03 and 13:25:14 series plays fail before planning. virtual playback candidate failed twice for the same series content, one terminal virtual_source_unavailable with empty diagnostics. Neither leaves a history row.
  2. 13:25:45 series play never reaches first frame. 0 s watched, background probe fails for the candidate, then four virtual stream transport failed plus stream transport startup failed pairs with empty error:{}.
  3. 13:27:05 zero-candidate parse failure. Only candidate_count=0 in the window, then client 3003 PARSING_CONTAINER_UNSUPPORTED. A refusing to substitute a dead session-bound virtual candidate with empty identity tiers fires at 13:30:17.
  4. Remux commit stalls. Both remux plays spend 2.6–7.6 s in session_transport_commit against ~5–19 ms for direct. HLS startup cost, but it is the whole wait on those plays.
  5. Cold resolve 1–3.4 s, zero cache hits. Warm retries drop to 0–67 ms.

Planning holds at ~0 ms with validated_original_playback rank 0 on all direct plays.

Sources: admin_playback_history, operational_logs (protocol v3 start timing, playback plan decided, virtual resolve timing, provider candidates fetched), playback_route_events, all filtered to the build-932 window.

Harness: OpenCode. Model: workhorse (omniroute).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant