captions that follow the cut - #37
Merged
Merged
Conversation
A transcript is in source time; a text overlay is in timeline time. Between them sit every trim, reorder, speed change and removed silence — so captions_from_transcript only ever lined up on an untouched asset sitting at zero, which is not a cut. Its own doc comment conceded as much. Remove silence, the op this app leads with, desynced every caption it had just written. Timeline::captions projects each transcript segment through the clips that actually show its footage (Clip::source_span_to_timeline, honoring speed and reverse), so a caption lands on the words that survived and words that were cut out get none. It reads through for_render, so a muted track is as uncaptioned as it is unheard, and it dedupes the same words reaching two clips — extract_audio leaves picture and detached audio both referencing the asset. Chunking comes with it: a speech model emits whole sentences and a whole sentence does not fit a 9:16 frame. Lines are split to 4 words / 28 chars and timed by character share, which is the honest approximation when neither backend reports word timings. Lines too short to read merge back into a neighbour rather than flashing for two frames. TextOverlay.generated marks what the generator wrote, so regenerating after a trim replaces the set instead of stacking a second one on it — and leaves the title the editor typed alone.
generate_captions / clear_captions replace captions_from_transcript on the Tauri and MCP surfaces. The asset-scoped call had nothing left to offer: its one correct case — an untouched asset at zero — is what the timeline-scoped one does anyway. The MCP instructions now say to caption last and to re-run after any further edit. An agent that captions and then keeps trimming would otherwise leave every line on the wrong words, and it has no way to know that from the tool list.
Found by driving the harness: the sample cut carries the same asset twice at different trims, and both clips captioned the same words — so two different lines were drawn on top of each other at the same screen position. Captions are one lane of text; two at once is two unreadable ones. The same footage reaching the cut twice is not exotic — a callback shot, or a full source parked under the edit. First line in wins the slot, the next starts where it ends, and one with nothing readable left is dropped. That also subsumes the identical-line case the dedupe was catching.
The Text overlays section captions the whole timeline rather than one asset, and relabels to Recaption once it has: captions are placed in timeline time, so a later trim moves the words out from under them, and the button admitting that is cheaper than a user discovering it on export. Clear sits next to it and takes only the generated ones. A `Caption the cut` agent-queue chip analyzes whatever is in the cut but not yet transcribed, then captions — the same shape as the other presets. src/lib/captions.ts is the bun-tested mirror of the timing math, so the browser harness produces the captions the backend would rather than the sentence-per-segment stand-in it had. Same arrangement as platforms.ts and smart-crop.ts. Unlike those, the whole feature is arithmetic, so the harness is faithful here rather than approximate.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The problem
captions_from_transcriptwrote the transcript's source timestamps straightinto
TextOverlay.start/end, which are timeline seconds. Its own doc commentconceded the limit:
That is not a cut. The moment anything is trimmed, reordered, retimed or
silence-removed, source time and timeline time diverge and every caption is on
the wrong word.
remove_silence— the op this app leads with — desynced everycaption it had just written, so the two headline social-video features could not
be used together.
Second problem: a whisper segment is a whole sentence. One sentence at
size: 0.05does not fit a 9:16 frame.What this does
Timeline::captions(pure, unit-tested) projects each transcript segmentthrough the clips that actually show its footage —
Clip::source_span_to_timeline,honoring trim, speed and reverse. Captions land on the words that survived the
cut; words that were cut out get none. A sentence cut in half captions only the
half still in the cut, because chunking happens over the segment's whole
projected span and each line is then clipped to the clip.
It reads through
for_render, so a muted track is as uncaptioned as it isunheard.
Chunking comes with it: 4 words / 28 chars by default, timed by character
share — the honest approximation, since neither speech backend reports word
timings (
TranscriptSegmenthas only a start and an end). Lines too short toread merge back into a neighbour instead of flashing for two frames.
TextOverlay.generatedmarks what the generator wrote, so re-running after atrim replaces its own set instead of stacking a second one — and leaves the title
the editor typed alone.
clear_captionstakes only the generated ones.The bug the harness found
Captions are one lane of text at one screen position, so two at once is two
unreadable ones. The sample cut carries the same asset twice at different trims,
and both clips captioned the same words — two different lines drawn on top of
each other. Not exotic: a callback shot, or a full source parked under the edit.
First line in wins the slot, the next starts where it ends, and one with nothing
readable left is dropped.
Surfaces
generate_captions/clear_captions— Tauri commands and MCP tools,replacing
captions_from_transcript. The asset-scoped call had nothing left tooffer: its one correct case is what the timeline-scoped one does anyway.
instructionsnow say to caption last and re-run after furtheredits. An agent has no way to infer that from a tool list.
Recaption once it has, with Clear beside it.
Caption the cutagent-queue chip: analyzes whatever is in the cut but notyet transcribed, then captions.
src/lib/captions.ts— bun-tested mirror, same arrangement asplatforms.tsand
smart-crop.ts. Unlike those it is faithful rather than approximate,because captioning is arithmetic all the way down — which is how the harness
caught the collision above.
Verification
as-is, then a 3s head trim → recaption moves every line with it; regenerate
replaces without touching a typed title; clear leaves only the title; the
agent chip reports
Captioned the cut — 8 lines.