Skip to content

captions that follow the cut - #37

Merged
OrellBuehler merged 5 commits into
mainfrom
feat/caption-the-cut
Aug 27, 2026
Merged

captions that follow the cut#37
OrellBuehler merged 5 commits into
mainfrom
feat/caption-the-cut

Conversation

@OrellBuehler

Copy link
Copy Markdown
Owner

The problem

captions_from_transcript wrote the transcript's source timestamps straight
into TextOverlay.start/end, which are timeline seconds. Its own doc comment
conceded the limit:

The segments keep the transcript's own timestamps, so they line up when the
asset sits at the start of the timeline at normal speed.

That is not a cut. The moment anything is trimmed, reordered, retimed or
silence-removed, source time and timeline time diverge and every caption is on
the wrong word. remove_silence — the op this app leads with — desynced every
caption it had just written, so the two headline social-video features could not
be used together.

Second problem: a whisper segment is a whole sentence. One sentence at size: 0.05 does not fit a 9:16 frame.

What this does

Timeline::captions (pure, unit-tested) projects each transcript segment
through the clips that actually show its footage — Clip::source_span_to_timeline,
honoring trim, speed and reverse. Captions land on the words that survived the
cut; words that were cut out get none. A sentence cut in half captions only the
half still in the cut, because chunking happens over the segment's whole
projected span and each line is then clipped to the clip.

It reads through for_render, so a muted track is as uncaptioned as it is
unheard.

Chunking comes with it: 4 words / 28 chars by default, timed by character
share
— the honest approximation, since neither speech backend reports word
timings (TranscriptSegment has only a start and an end). Lines too short to
read merge back into a neighbour instead of flashing for two frames.

TextOverlay.generated marks what the generator wrote, so re-running after a
trim replaces its own set instead of stacking a second one — and leaves the title
the editor typed alone. clear_captions takes only the generated ones.

The bug the harness found

Captions are one lane of text at one screen position, so two at once is two
unreadable ones. The sample cut carries the same asset twice at different trims,
and both clips captioned the same words — two different lines drawn on top of
each other. Not exotic: a callback shot, or a full source parked under the edit.
First line in wins the slot, the next starts where it ends, and one with nothing
readable left is dropped.

Surfaces

  • generate_captions / clear_captions — Tauri commands and MCP tools,
    replacing captions_from_transcript. The asset-scoped call had nothing left to
    offer: its one correct case is what the timeline-scoped one does anyway.
  • The MCP instructions now say to caption last and re-run after further
    edits. An agent has no way to infer that from a tool list.
  • Inspector's Text overlays section captions the whole cut and relabels to
    Recaption once it has, with Clear beside it.
  • A Caption the cut agent-queue chip: analyzes whatever is in the cut but not
    yet transcribed, then captions.
  • src/lib/captions.ts — bun-tested mirror, same arrangement as platforms.ts
    and smart-crop.ts. Unlike those it is faithful rather than approximate,
    because captioning is arithmetic all the way down — which is how the harness
    caught the collision above.

Verification

  • 261 Rust tests (9 new), 52 bun tests (10 new), clippy and rustfmt clean.
  • Driven end-to-end through the Playwright browser harness: captions on the cut
    as-is, then a 3s head trim → recaption moves every line with it; regenerate
    replaces without touching a typed title; clear leaves only the title; the
    agent chip reports Captioned the cut — 8 lines.

A transcript is in source time; a text overlay is in timeline time. Between
them sit every trim, reorder, speed change and removed silence — so
captions_from_transcript only ever lined up on an untouched asset sitting at
zero, which is not a cut. Its own doc comment conceded as much. Remove
silence, the op this app leads with, desynced every caption it had just
written.

Timeline::captions projects each transcript segment through the clips that
actually show its footage (Clip::source_span_to_timeline, honoring speed and
reverse), so a caption lands on the words that survived and words that were
cut out get none. It reads through for_render, so a muted track is as
uncaptioned as it is unheard, and it dedupes the same words reaching two
clips — extract_audio leaves picture and detached audio both referencing the
asset.

Chunking comes with it: a speech model emits whole sentences and a whole
sentence does not fit a 9:16 frame. Lines are split to 4 words / 28 chars and
timed by character share, which is the honest approximation when neither
backend reports word timings. Lines too short to read merge back into a
neighbour rather than flashing for two frames.

TextOverlay.generated marks what the generator wrote, so regenerating after a
trim replaces the set instead of stacking a second one on it — and leaves the
title the editor typed alone.
generate_captions / clear_captions replace captions_from_transcript on the
Tauri and MCP surfaces. The asset-scoped call had nothing left to offer: its
one correct case — an untouched asset at zero — is what the timeline-scoped
one does anyway.

The MCP instructions now say to caption last and to re-run after any further
edit. An agent that captions and then keeps trimming would otherwise leave
every line on the wrong words, and it has no way to know that from the tool
list.
Found by driving the harness: the sample cut carries the same asset twice at
different trims, and both clips captioned the same words — so two different
lines were drawn on top of each other at the same screen position. Captions
are one lane of text; two at once is two unreadable ones.

The same footage reaching the cut twice is not exotic — a callback shot, or a
full source parked under the edit. First line in wins the slot, the next
starts where it ends, and one with nothing readable left is dropped. That also
subsumes the identical-line case the dedupe was catching.
The Text overlays section captions the whole timeline rather than one asset,
and relabels to Recaption once it has: captions are placed in timeline time,
so a later trim moves the words out from under them, and the button admitting
that is cheaper than a user discovering it on export. Clear sits next to it
and takes only the generated ones.

A `Caption the cut` agent-queue chip analyzes whatever is in the cut but not
yet transcribed, then captions — the same shape as the other presets.

src/lib/captions.ts is the bun-tested mirror of the timing math, so the
browser harness produces the captions the backend would rather than the
sentence-per-segment stand-in it had. Same arrangement as platforms.ts and
smart-crop.ts. Unlike those, the whole feature is arithmetic, so the harness
is faithful here rather than approximate.
@OrellBuehler
OrellBuehler merged commit ebee3c0 into main Aug 27, 2026
23 of 24 checks passed
@OrellBuehler
OrellBuehler deleted the feat/caption-the-cut branch August 27, 2026 19:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant