Skip to content

fix(playback): recover stale pins on identity-less rows with guarded re-pin - #234

Merged
drondeseries merged 4 commits into
mainfrom
wip/stale-pin-partial
Oct 6, 2026
Merged

drondeseries merged 4 commits into
mainfrom
wip/stale-pin-partial

Conversation

@randrini

@randrini randrini commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator

Problem

Closes #227

Cold starts on rows pinned to ?result= IDs that upstream renumbers within minutes fail terminally when the row carries no durable provider identity, so same-release rematch cannot apply — even while the live listing holds the release under a new ID.

Approach

Cold-start-only guarded recovery in the version-fallback walk: pin absent from a NON-EMPTY listing + identity-less row + same-content fingerprint match (size±5%, codec, duration±2%) → re-pin in-memory (never the DB pin; old candidate URL/headers/identity/evidence cleared) and retry once. Session-bound path explicitly refuses; empty listings and cross-content adoption preserve terminal. Kill-switch: SILO_DISABLE_VIRTUAL_STALE_PIN_RECOVERY=1 (default on).

Validation

  • New stale-pin tests: recovery on match, terminal on empty/no-match, no cross-content adoption, single-retry bound, session-bound refusal, old-state clearing (integrated walk test proven load-bearing)
  • Full handlers suite green incl. -race; changed-lines lint 0 issues; gofmt/vet clean
  • Full CI green on the final head required

Risks

Fingerprint re-pin can mismatch on volatile listings (guarded by strict tiers + non-empty-listing requirement + single retry). Verdict/delivery stamps untouched. No API, migration, or config impact.

Checklist

  • I read and can explain the complete diff.
  • This pull request addresses one concern.

AI Disclosure

  • Harness: OpenCode
  • Tool(s): OpenCode (oracle, fixer subagents)
  • Model(s): opencode-go/muse-spark-1.3-contributor
  • Involvement: AI-assisted

@drondeseries

Copy link
Copy Markdown
Collaborator

Oracle production-readiness review of ef379f24 — not ready to merge. Source review only; tests were not rerun. The reported eight successful CI checks do not cover these gaps:

  1. Enforce an exact-candidate, single attempt (playback_virtual_stale_pin.go:151). The general resolver can iterate siblings and invoke stale-source fallback. Rejecting a different returned ID afterward does not enforce the promised retry bound.

  2. Validate before side effects (playback_virtual_stale_pin.go:151–178). The resolver can persist evidence and update the sticky pin before recovery checks identity and duration. A rejected recovery must not leave a new sticky selection or publish evidence.

  3. Use measured candidate duration (playback_virtual_stale_pin.go:170,204–209). The copied row retains its old duration, and no-prober/probe-failure returns can carry that value into the comparison. Existing recovery tests have no prober. Add actual probe-result and failure-path coverage.

  4. Reserve time for alternatives (playback_v3.go:2638–2644). The 2-second parent caps the 5-second listing timeout, but synchronous recovery can exhaust the whole budget before healthy alternate rows run.

  5. Preserve session-bound intent (playback_v3.go:2629). Unconditionally setting binding to false defeats the claimed protection against future bound callers reusing this walk. Test refusal through the walk, not just the helper.

The kill switch, empty-list refusal, identity gate, same-path input filtering, and copied-state clearing are present. The integrated disabled-recovery control is useful, but stronger collaborator coverage is needed.

NEEDS_FIX

randrini added 3 commits October 6, 2026 16:34
…rows

Finish the #227 partial: the version-fallback walk now declares its
cold-start (unbound) intent, the recovery refuses a session-bound resolve
explicitly, and a re-pinned row no longer carries the previous pin's stored
URL, headers, durable identity, or probe evidence onto the matched candidate
(clearing lifecycle verdict/delivery untouched).

Rewrite the integrated walk test to be load-bearing: the primary resolve
genuinely terminals before the recovery is consulted, and the recovery-disabled
control proves the recovery is what recovers it. Add coverage for the
session-bound refusal and the re-pin field hygiene.
Address the PR #234 review remainders:

1. Enforce an exact-candidate single attempt. The recovery no longer calls
   the general resolver (which iterates siblings and invokes the stale-source
   fallback, paying extra listings and breaking the promised retry bound). It
   resolves the re-pinned candidate once through resolveStalePinCandidateOnceV3,
   which has no sibling iteration, no fallback invocation, and rejects a
   substituted result id.

2. Validate before side effects. Identity and duration are checked on the
   probed result before anything is persisted or pinned; a rejected recovery
   leaves no new sticky selection and publishes no evidence.

3. Use the measured candidate duration. The probe copies the row with its
   duration zeroed and the fingerprint's duration tier is compared against the
   probe's own measured runtime; a no-prober or failed probe is a no-match
   rather than a comparison against the copied stale value.

4. Reserve time for alternatives. The recovery runs inside its own
   virtualStalePinRecoveryBudget (well under the walk's decision budget) so a
   slow recovery yields to the healthy alternates the walk still has to try.

5. Preserve session-bound intent. The walk threads the caller's binding intent
   instead of forcing it false, and the fresh-start path declares unbound at the
   call site; a session-bound caller reusing the walk keeps refusing.

Tests cover the exact-candidate single attempt, substitution rejection,
no-prober and probe-failure no-matches, measured-duration accept/reject with
side-effect checks, a slow recovery yielding to a healthy alternate, and
session-bound refusal through the walk.

AI-assisted: implemented by AI coding agents on behalf of the repository owner.
@randrini

randrini commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Review addressed and pushed (9b5a95bb, rebased onto post-#233 main): recovery now resolves the exact re-pinned candidate once (no sibling iteration/fallback, substituted IDs rejected); identity+duration validated BEFORE any sticky persist or evidence write (measured probe duration only, failures are no-match); recovery runs under a 500ms sub-budget so alternates still run; walk threads the caller's binding intent (fresh declares unbound, bound refuses — proven through the walk). Full handlers suite + -race green, changed-lines lint clean. One deliberate non-scope: recovery re-list is not bound by the virtualRecoveryRelists damper — suggest tracking as follow-up if wanted.

@randrini
randrini force-pushed the wip/stale-pin-partial branch from ef379f2 to 9b5a95b Compare October 6, 2026 15:40
@drondeseries

Copy link
Copy Markdown
Collaborator

Production-readiness: NOT READY — do not merge (review of 9b5a95b)

Gate chain, single-retry bound, and rejection tests are solid. Four blockers remain, all in the new recovery file.

Must-fix

1. Recovery can rewrite the durable pin — playback_virtual_stale_pin.go:208
Calls persistVirtualProbeEvidence with the original row + replacement URI. For non-collection rows that helper takes the path-adoption branch (AdoptPath=resolvedPath, RequireAdopt=true, playback_virtual.go:4095-4097,4146-4151) — a path-adoption write, not evidence-only. Breaks the 'never the DB pin' contract.
Fix: keep recovery transient; do not persist replacement evidence onto the old row via the adoption-capable helper. Persist only against a verified existing owner without changing its path. Add a saver-argument assertion + DB-backed test proving the original pin is unchanged (existing test :170-173 asserts only the Go struct, not the DB).

2. Sticky generation allocated too late — playback_virtual_stale_pin.go:207
nextVirtualCacheGeneration() at publication time means an earlier-started but later-finishing recovery can overwrite a newer selection, contradicting the ordering guarantee (playback_virtual.go:6564-6575; general resolver allocates at :2008).
Fix: allocate the generation before listing/resolving, carry it through publication. Add an interleaving test where a newer selection finishes first.

3. Ambiguous fingerprints pick by rank — playback_virtual_stale_pin.go:332-344
First size/codec match wins; same-title releases routinely share codec, similar size/runtime, and duration does not separate language/edition/track inventory. The 'strongest rank' comment (:317-319) is unsupported.
Fix: refuse on multiple matching candidate identities instead of rank-picking; veto known resolution/track-language conflicts. Keep the single-resolve bound. Add two same-title releases in both listing orders asserting refusal. Describe a unique match as heuristic, not renumber proof.

4. 'Exact candidate' check incomplete — playback_virtual_stale_pin.go:259-266
Rejects only when both result IDs are nonempty and differ. A neutral URI or a different content path reusing the same result ID passes; listing filtering never validates the resolver's returned URI.
Fix: require a concrete matched candidate; validate returned content/provider scope + result identity; reject inconsistent CandidateID when supplied. Test neutral-URI and cross-content substitutions with zero publication.

Advisory (not blocking)

  • Retry bound is one handler listing + one detailed-resolver call + ≤1 probe — does not bound provider listings inside the detailed resolver.
  • Rewritten copy retains FailedAt, which vetoes the new URI via virtualCandidateVerdictError (:4335-4337): conservative refusal, keep it — add an active-failure test and do not clear verdicts casually.
  • 500ms covers lister+resolver+probe cooperatively; persistence uses parent ctx, so the whole-recovery bound is incomplete — fine as best-effort, document as such.
  • Kill-switch: ANY nonempty value disables (:55), not only 1 — document actual semantics.
  • Measured-duration test uses identical stored/probed values (:426-435); clearing test exercises the matcher despite walk-coverage claim (:638).

Recovery for an identity-less stale pin re-pinned in memory but then
handed the adoption-capable evidence writer the original row with the
replacement URI, which took the path-adoption branch and rewrote the
durable ?result= pin it promised never to touch. Persist replacement
evidence only against a row that already verifiably owns the matched
candidate's concrete path, and then as metadata only; with no existing
owner the recovery leaves no durable trace.

Allocate the write generation before the listing instead of at
publication, so an earlier-started, later-finishing recovery cannot
overwrite a newer selection that finished first.

Refuse when more than one live candidate identity carries the row's
fingerprint rather than rank-picking the first: same-title releases
share codec, size and runtime, so size+codec cannot separate language or
edition variants. Veto known resolution and audio-language conflicts so
those variants are excluded before the uniqueness count.

Require a concrete matched candidate from the resolver: reject a
provider-neutral URI, a different release under the same neutral key, a
candidate id inconsistent with the returned URI, and an owner outside
the matched release scope.

Add DB-backed tests that read the whole media_files row back and prove
the durable pin is byte-identical, plus unit tests for ambiguous
listings in both orders, interleaved generation, and the substitution
cases.
@randrini

randrini commented Oct 6, 2026

Copy link
Copy Markdown
Collaborator Author

Review addressed and pushed (130c36f3): recovery no longer touches the durable pin — publication resolves the existing path owner first and writes metadata-only, or nothing (DB-backed tests prove byte-identical pin + zero writes); sticky generation allocated before listing with an interleaving test; ambiguous fingerprints refuse (plus resolution/audio-language vetoes, heuristic wording); exact-candidate check requires concrete id + scope validation (neutral/cross-content rejected, zero publication). Full handlers suite green incl. -race; two failures (TestAdminSubtitleListPageDB, TestCatalogTransferPersistsJobsAndSignedLink) reproduce on the clean tree (missing Redis / pre-existing fixture), unrelated. Full CI required before re-review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(playback): recover stale pins on identity-less rows with guarded re-pin

2 participants