Epic (v2): Aftercare-driven iterative tuning — the real-world tuning lab

[coaxk]

The insight

Aftercare (#156) judges real completed jobs across the actual library at scale — a far richer signal than the synthetic-clip arena (Tuning Lab). It is, in effect, the real-world tuning lab. This epic turns that signal into action: requeue bad jobs with better settings, iterate, and converge toward the best achievable subtitle — capturing real (file, language, config) -> quality data the whole time (the firehose passive-tuning #95 / federated #124 want).

Grounding finding (2026-06-09)

Testing aftercare on a real foreign-language library showed CPS issues on nearly every job (271-644 per file). Investigated a flagged file (Hellground S01E07): median CPS 21.6 (borderline-normal for translated dialogue), but a tail of sub-second cues — e.g. "You lied to me all this time." (29 chars) shown for 0.26s, "No." for 0.06s. So:

• It is REAL, not a threshold artifact.
• Root cause = micro-duration cues from segmentation (subgen CUSTOM_REGROUP / word-timing), NOT text density. CPS is the symptom; cue duration is the cause (these also trip too_short).
• It is systemic (config-level), so it hits the whole library — which is precisely why a tuning loop, not per-file luck, is the answer.

Two leverage points

1. Aggregate config insight (fleet-level, highest leverage): aftercare aggregates → "X% of your jobs trip CPS via sub-second cues -> your regroup/segmentation default is producing micro-cues; here is a better one." One config change fixes the library. This is the real-world tuning lab at the fleet level.
2. Per-file iterative "Improve" (user-facing): a flagged file -> try a better config -> re-judge -> keep iterating.

North star: greedy hill-climb + lock (Judd)

Iterate a file (or a language profile): change a config lever -> re-transcribe -> judge with the aftercare judges -> if it improved, LOCK that change -> move to the next lever -> repeat -> converge toward "as good as it gets" (zero detected failures now; true accuracy once the QE judge #123 lands) or until the user stops. Coordinate-descent over the config space, scored by real output.

Design questions to work through (brainstorm -> spec before building)

• Where do candidate settings come from? failure-mode heuristics (CPS/too-short -> merge-short-cues / regroup; repeats -> repetition_penalty / no_repeat_ngram_size / lower temp) -> Tuning Lab learned per-language winners (Tuning Lab: global recipe leaderboard (overall recipe ranking) #146) -> a small real-file sweep judged by aftercare.
• Convergence target & stop: "no detected failures", a score target, no-improvement plateau, max attempts, or explicit user Stop. Per-file vs per-language-profile (lock at the language level so one convergence fixes many files).
• Clear status UX: per-file/per-profile attempt strip — Attempt 2/5 · CPS 644->120 · repeats 6%->1% · score 41->78, live progress, improving/plateaued state, Stop.
• Guardrails (hard): every attempt is a full GPU transcription (minutes). Cap attempts, surface GPU cost, never auto-loop without a ceiling. Suggest -> cap -> gate (existing rule).
• Data capture: every (file, language, config) -> aftercare score is a federated data point (Capture a structured media/subtitle data DB (local-rich, transmit-anonymous) #95). This is the operations firehose feeding Epic: federated tournament tuning loop (crowd-sourced per-language config refinement) #124/Tuning Lab: global recipe leaderboard (overall recipe ranking) #146.
• "Perfect" needs a trustworthy positive judge: structural failure-flags can reach "no problems detected"; true accuracy convergence depends on the QE judge (QE/adequacy summit — reference-free translation-quality judge (per-language accuracy ranking) #123). So: optimize to zero-structural-failures now, accuracy later.

Relationships

Builds on #156 (aftercare). Supersedes/absorbs #165 (open-in-Tuning-Lab from aftercare = the seed). Feeds #95 (passive capture), #124 (federated loop), #146 (leaderboard). Gated for "perfect" on #123 (QE).

Status: v2 epic — needs a proper brainstorm -> spec pass before implementation.

[coaxk]

Hard dependency surfaced during testing (2026-06-09): the Aftercare Requeue action is currently a no-op on any file that already has a subtitle — subgen has SKIP_IF_TARGET_SUBTITLES_EXIST=true, so it sees the existing .srt and skips ("matching .srt already on disk — subgen had nothing to do").

This blocks the entire "requeue with better settings / iterate" premise (and makes the basic Requeue button misleading). The loop needs an explicit force-overwrite path: an explicit user requeue is a deliberate "redo this", not an accidental re-scan, so it must bypass the skip. Options:

• Add a force/overwrite query param to subarr-subgen /batch that bypasses SKIP_IF_TARGET_SUBTITLES_EXIST for that call (mirrors the audio_language_override skip-bypass pattern — a subarr-subgen patch). Preferred.
• Or have subarr delete the sidecar .srt before requeue (file mutation; less clean).

Also applies to the plain #156 Requeue button independent of this epic.

[coaxk]

Regroup tuning result (2026-06-09) — pad solves micro-cues; density is the per-language residual:

config	sub-0.5s micro-cues	%>25 CPS
old (split-only)	many	~33%
v1 (sentence/gap split + merge)	13%	23%
v2 (+ pad sub-1s cues)	0%	10% (Dutch) / 30% (Spanish)

pad (extend only sub-1s cues into surrounding gaps, validated 0.1s->0.80s) eliminated the sub-second micro-cues entirely. CPS-issue counts roughly halved. BUT a dense file (Presumed Guilty, Spanish) still showed 30% >25 CPS with 0% micro-cues — i.e. the residual is pure text density (normal-duration cues, too much translated text), which segmentation cannot fix. Reinforces the per-language thesis: Dutch came out clean, Spanish dense → different CPS tolerance / handling per language. Locked v2 as the subgen-next default; rolling to live subgen + subarr-subgen shipped default is a separate decision.

[coaxk]

Regroup arena results (2026-06-09) — 4 languages x 6 configs, full files. Transcribe-once-per-file (raw, large-v3 translate) then apply each regroup config to the same result (the cheap transcribe-once/regroup-many pattern this epic is about).

% cues over 25 CPS (lower better):

config            fr   ru   es   ja
old (split-only)  53   34   39   34
v2 (sub-1s pad)   27   19   19   16
v2-strongpad      15   11    8    9   <- winner
da+pad            28   19   18   16
da (stable default)58  40   46   40

micro-cues (sub-0.5s): v2 / strongpad / strongmerge = 0% everywhere; da = 17-24% (over-splits, 643-781 cues vs ~500).

Findings:

1. v2-strongpad (pad sub-1.5s cues +.3/+.7 into gaps) wins on every language — ~halves %>25 vs current v2; median CPS down to 13-15; micro-cues 0. Adopted as subgen-next default.
2. The optimal regroup config is UNIVERSAL, not per-language — strongpad is best for fr/ru/es/ja alike. So per-language tuning is NOT about the regroup string.
3. The residual IS language-bound — French consistently hardest (53->15), Spanish/Japanese easiest (8-9) at every config. So the per-language lever is CPS tolerance / transcription density (a per-language CPS threshold, or transcription kwargs/prompt), NOT segmentation.
4. stable-ts default da is worse than the old config (over-splits) — hand-tuned beats the library default here.

Implication for the epic: the auto-iterating loop should treat regroup as a (largely) global lever, and put the per-language search on CPS-tolerance / transcription-side knobs. The arena method (transcribe-once, regroup-many) is a working, GPU-cheap prototype for the loop.