captions that follow the cut

CLAUDE.md

@@ -267,7 +267,22 @@ no editing logic in the adapter.
   **animation** — `Clip::transform_at` interpolates it, the engine renders the
   motion). Text titles / lower-thirds / captions live on the timeline itself as
   `Timeline.overlays: Vec<TextOverlay>` (each with its own `TextKeyframe`
-  animation); `transcript_to_srt` serializes a transcript to SubRip. A `Track`
+  animation); `transcript_to_srt` serializes a transcript to SubRip.
+  **Captions are timeline math, not a transcript dump**, and pure +
+  unit-tested: a transcript is in *source* time and an overlay is in *timeline*
+  time, so `Timeline::captions` projects each segment through the clips that
+  actually show its footage (`Clip::source_span_to_timeline`, honoring trim /
+  speed / reverse) — captions land on the words that survived the cut and words
+  that were cut out get none. It reads through `for_render`, so a muted track is
+  as uncaptioned as it is unheard; it chunks a sentence to `CaptionOptions`
+  (4 words / 28 chars by default — a speech model emits whole sentences and a
+  whole sentence does not fit a 9:16 frame), timing lines by *character share*
+  because neither speech backend reports word timings; lines too short to read
+  merge back into a neighbour instead of flashing; and no two lines are ever on
+  screen at once (captions are one lane of text, and the same footage reaching
+  the cut twice would otherwise collide with itself). `TextOverlay.generated`
+  marks what it wrote, so regenerating replaces its own set and leaves a typed
+  title alone. A `Track`
   carries a `duck` flag (sidechain-ducked under the rest of the mix on export).
   `Fit` and `Delivery` live here (the domain owns the delivery shape; `engine::cli`
   re-exports `Fit`), and `Timeline.format` is the frame the project is cut for.
@@ -415,7 +430,11 @@ proposal appears for review, not that the cut changes: the read tools
 `timeline_summary` carries `staged_changes` so it cannot mistake one for the other.
 `smart_crop` frames each shot for the delivery frame (the server `instructions`
 pair it with `set_delivery_format`, since reshaping to 9:16 otherwise keeps
-whatever was in the middle).
+whatever was in the middle). `generate_captions` / `clear_captions` caption the
+cut; the `instructions` say to caption **last** and to re-run after any further
+edit, because captions are placed in timeline time and a later trim moves the
+words out from under them — which an agent has no way to infer from the tool
+list.
 `platform_check` tells it whether the cut is publishable where it is going
 (and the server `instructions` tell it to run that before reporting a cut
 finished — an agent that assembles a four-minute Reel has done the work and lost
@@ -445,7 +464,8 @@ width/height to clear it), `remove_clip`, `set_volume`, `set_fade`,
 `set_reframe` / `clear_reframe` / `set_reframe_keyframes` / `add_reframe_keyframe`,
 `set_asset_projection` (asset-level 360 mark; returns the `Asset`),
 `add_overlay` / `update_overlay` / `remove_overlay` / `set_overlay_keyframes`,
-`captions_from_transcript`, `export_srt`, `remove_silence`, `snap_to_beats`,
+`generate_captions` / `clear_captions` (caption the whole cut, in timeline
+time), `export_srt`, `remove_silence`, `snap_to_beats`,
 `smart_crop` (frame each shot for the delivery frame),
 `extract_audio`, `concatenate` — each returns the
 refreshed `Timeline`), media (`get_frame` → base64 PNG data URL, `get_waveform`,
@@ -568,8 +588,11 @@ at the playhead like Transform — note its `lerpAngle` takes the shortest arc,
 plain `lerp` would read as a 340° swing across the seam; for a source Kerf did not
 detect as 360 it instead offers a projection picker that marks the whole asset via
 `set_asset_projection`), and an always-visible
-**Text overlays** section (add titles / lower-thirds, generate captions, edit
-text / timing / position / size / color / box / bold).
+**Text overlays** section (add titles / lower-thirds, caption the whole cut —
+the button relabels to `Recaption` once there are generated captions, since a
+later trim moves the words out from under them, with `Clear` beside it taking
+only the generated ones — and edit text / timing / position / size / color /
+box / bold).
 **Polish presets** (`src/lib/style-presets.ts`, pure data over the existing
 surfaces): the Color section leads with one-click **looks** —
 Punchy / Warm / Cool / Faded / B&W chips (the active one highlights; the sliders
@@ -579,7 +602,7 @@ omitted at 0 so old graphs stay byte-identical; plain saturation/gamma can't
 tint) — and the Text overlays section leads with **Title / Lower third /
 Caption** style chips that create a styled overlay at the playhead with
 fade-in/out opacity keyframes; the caption style matches what
-`captions_from_transcript` generates, so manual and generated captions look
+`generate_captions` generates, so manual and generated captions look
 alike.
 Everything is styled with the CSS-variable tokens directly (inline `style`), not Tailwind
 utilities. The **timeline is a bespoke NLE timeline** that renders **real `editor.timeline`
@@ -631,7 +654,11 @@ mirror used **only** by the browser harness, so the panel is drivable under
 `bun run dev`. `src/lib/smart-crop.ts` is the same arrangement for smart crop: only
 the *shape* arithmetic is mirrored (bun-tested), because the harness has no decoder
 to sample with and so lands on the centre window — which part of the shot survives
-is the half that only exists with media behind it. The **cover frame** is saved from the preview's context menu
+is the half that only exists with media behind it. `src/lib/captions.ts` is the
+same arrangement again, but *faithful* rather than approximate — captioning is
+arithmetic all the way down, so the harness produces exactly the captions the
+backend would (the mirror caught the two-captions-at-once collision the Rust
+tests had not). The **cover frame** is saved from the preview's context menu
 (`Save cover frame…` → `export_cover` at the playhead), and both a finished
 export and a saved cover offer **Show in folder** in their toast.
 `Preview` shows the composited frame under the playhead, and during
@@ -665,8 +692,9 @@ queue** (status · queue · history · add-task) — Kerf has no in-app chat; a
 LLM claims tasks over MCP. The queue is `agent` state (`src/lib/agent.svelte.ts`, a third
 runes singleton) backed by the `tasks` table over Tauri/MCP: the add-task box and preset chips
 `agent.add(...)` real tasks, and `ready` tasks show Apply/Dismiss (`resolve_task`/`remove_task`).
-Four preset chips (`Remove silences` / `Assemble rough cut` / `Frame for the delivery`
-/ `Cut to the beat` — which
+Five preset chips (`Remove silences` / `Assemble rough cut` / `Frame for the delivery`
+/ `Caption the cut` (analyzes whatever is in the cut but not yet transcribed,
+then captions it) / `Cut to the beat` — which
 analyzes whatever is on the audio tracks first, then calls `snap_to_beats`, and says
 "No cuts were near a beat" instead of claiming an alignment when the grid never reached
 them) also run the matching local op and

crates/kerf-app/src/lib.rs

@@ -20,8 +20,9 @@ use std::sync::{Arc, Mutex};
 
 use base64::Engine as _;
 use kerf_core::{
-    Asset, AssetAnalysis, AudioEffect, Delivery, EditSource, ExportOptions, Fit, Keyframe, Project, Projection, ReframeKeyframe,
-    Revision, StagedEdit, StreamKind, Task, TextKeyframe, Timeline, TimelineDiff, Transition, TransitionKind, VideoEffect,
+    Asset, AssetAnalysis, AudioEffect, CaptionOptions, Delivery, EditSource, ExportOptions, Fit, Keyframe, Project, Projection,
+    ReframeKeyframe, Revision, StagedEdit, StreamKind, Task, TextKeyframe, Timeline, TimelineDiff, Transition, TransitionKind,
+    VideoEffect,
 };
 use serde::Serialize;
 use tauri::{AppHandle, Emitter, Manager, State};
@@ -887,10 +888,18 @@ fn set_overlay_keyframes(state: State<'_, AppState>, overlay_id: String, keyfram
 }
 
 #[tauri::command(async)]
-fn captions_from_transcript(state: State<'_, AppState>, asset_id: String) -> CmdResult<Timeline> {
-    let id = id(&asset_id)?;
+fn generate_captions(state: State<'_, AppState>, options: Option<CaptionOptions>) -> CmdResult<Timeline> {
+    let project = state.project();
+    project
+        .generate_captions(options.unwrap_or_default())
+        .map_err(|e| e.to_string())?;
+    project.timeline().map_err(|e| e.to_string())
+}
+
+#[tauri::command(async)]
+fn clear_captions(state: State<'_, AppState>) -> CmdResult<Timeline> {
     let project = state.project();
-    project.captions_from_transcript(id).map_err(|e| e.to_string())?;
+    project.clear_captions().map_err(|e| e.to_string())?;
     project.timeline().map_err(|e| e.to_string())
 }
 
@@ -1604,7 +1613,8 @@ pub fn run() {
             update_overlay,
             remove_overlay,
             set_overlay_keyframes,
-            captions_from_transcript,
+            generate_captions,
+            clear_captions,
             export_srt,
             remove_silence,
             snap_to_beats,

crates/kerf-app/src/mcp.rs

@@ -15,8 +15,8 @@ use std::sync::{Arc, Mutex, MutexGuard};
 
 use base64::Engine as _;
 use kerf_core::{
-    AudioEffect, Delivery, EditSource, ExportOptions, Fit, Keyframe, Project, Projection, ReframeKeyframe, StreamKind,
-    TextKeyframe, Transition, TransitionKind, VideoEffect,
+    AudioEffect, CaptionOptions, Delivery, EditSource, ExportOptions, Fit, Keyframe, Project, Projection, ReframeKeyframe,
+    StreamKind, TextKeyframe, Transition, TransitionKind, VideoEffect,
 };
 use rmcp::handler::server::wrapper::Parameters;
 use rmcp::model::{CallToolResult, ContentBlock, Implementation, ServerCapabilities, ServerInfo};
@@ -67,6 +67,18 @@ struct AssetIdParams {
     asset_id: String,
 }
 
+#[derive(Debug, Default, serde::Deserialize, schemars::JsonSchema)]
+struct CaptionParams {
+    #[schemars(description = "Most words on one caption line (default 4)")]
+    max_words: Option<usize>,
+    #[schemars(description = "Most characters on one caption line (default 28); the tighter of the two limits wins")]
+    max_chars: Option<usize>,
+    #[schemars(description = "Vertical position as a fraction of frame height, 0 = top (default 0.88)")]
+    pos_y: Option<f64>,
+    #[schemars(description = "Font height as a fraction of frame height (default 0.05)")]
+    size: Option<f64>,
+}
+
 #[derive(Debug, serde::Deserialize, schemars::JsonSchema)]
 struct SpeechModelParams {
     #[schemars(
@@ -1257,16 +1269,38 @@ impl KerfMcp {
     }
 
     #[tool(
-        description = "Generate caption overlays from an asset's cached transcript (run analyze_asset first), one per segment, low-center with a translucent box. Captions use the transcript's timestamps, so they align when the asset sits at the start of the timeline at normal speed. Returns the overlays created."
+        description = "Caption the cut: project every clip's cached transcript (run analyze_asset first) through the current edit and write the result as text overlays, replacing any previously generated set. Captions are placed in TIMELINE time, so they follow trims, reorders, speed changes and removed silences, and words that were cut out get no caption. Long sentences are split into readable lines (defaults: 4 words / 28 characters). Hand-made titles and lower-thirds are left alone. Returns the overlays created."
     )]
-    fn captions_from_transcript(&self, Parameters(p): Parameters<AssetIdParams>) -> Result<String, McpError> {
-        let id = parse_id(&p.asset_id)?;
+    fn generate_captions(&self, Parameters(p): Parameters<CaptionParams>) -> Result<String, McpError> {
+        let mut opts = CaptionOptions::default();
+        if let Some(v) = p.max_words {
+            opts.max_words = v;
+        }
+        if let Some(v) = p.max_chars {
+            opts.max_chars = v;
+        }
+        if let Some(v) = p.pos_y {
+            opts.pos_y = v;
+        }
+        if let Some(v) = p.size {
+            opts.size = v;
+        }
         let project = self.lock();
-        let out = project.captions_from_transcript(id).map_err(core_err)?;
+        let out = project.generate_captions(opts).map_err(core_err)?;
         self.changed();
         json(&out)
     }
 
+    #[tool(
+        description = "Remove the captions generate_captions wrote, leaving hand-made titles and lower-thirds alone. Returns how many were removed."
+    )]
+    fn clear_captions(&self) -> Result<String, McpError> {
+        let project = self.lock();
+        let removed = project.clear_captions().map_err(core_err)?;
+        self.changed();
+        Ok(format!("removed {removed} generated caption(s)"))
+    }
+
     #[tool(description = "Write an asset's cached transcript to a SubRip (.srt) subtitle file (run analyze_asset first)")]
     fn export_srt(&self, Parameters(p): Parameters<ExportSrtParams>) -> Result<String, McpError> {
         let id = parse_id(&p.asset_id)?;
@@ -1777,8 +1811,12 @@ impl ServerHandler for KerfMcp {
              the shot is actually about. Add titles, lower-thirds \
              and captions with add_overlay / update_overlay / set_overlay_keyframes \
              (drawn over the cut; list_fonts lists installed system fonts to pass \
-             as update_overlay's font), or captions_from_transcript to caption an \
-             analyzed asset in one call; export_srt writes a subtitle file. \
+             as update_overlay's font), or generate_captions to caption the whole \
+             cut in one call. Caption LAST, after the cutting is done: captions \
+             are placed in timeline time, so a later trim or remove_silence moves \
+             the words out from under them — re-run generate_captions after any \
+             further edit and it replaces its own set, leaving typed titles \
+             alone. export_srt writes a subtitle file. \
              When the cut is going somewhere vertical, set_delivery_format sets \
              the frame it is being made for and smart_crop then frames each shot \
              for it — reshaping 16:9 footage to 9:16 throws away most of the \

crates/kerf-core/src/lib.rs

@@ -30,10 +30,10 @@ pub use engine::{
 pub use error::{Error, Result};
 pub use fonts::list_system_fonts;
 pub use model::{
-    Asset, AssetAnalysis, AudioEffect, Clip, Color, CropFrame, Delivery, DiffEntry, DiffKind, EditSource, Keyframe, Marker,
-    Projection, Reframe, ReframeKeyframe, ResolvedReframe, Revision, Rhythm, SalienceMap, StagedEdit, StreamInfo, StreamKind,
-    Task, TaskStatus, TextKeyframe, TextOverlay, TimeRange, Timeline, TimelineDiff, Track, TranscriptSegment, Transform,
-    Transition, TransitionKind, VideoEffect,
+    Asset, AssetAnalysis, AudioEffect, CaptionOptions, Clip, Color, CropFrame, Delivery, DiffEntry, DiffKind, EditSource,
+    Keyframe, Marker, Projection, Reframe, ReframeKeyframe, ResolvedReframe, Revision, Rhythm, SalienceMap, StagedEdit,
+    StreamInfo, StreamKind, Task, TaskStatus, TextKeyframe, TextOverlay, TimeRange, Timeline, TimelineDiff, Track,
+    TranscriptSegment, Transform, Transition, TransitionKind, VideoEffect,
 };
 pub use platform::{
     check_all as check_platforms, CutSummary, DeliveryCheck, DeliveryIssue, IssueKind, PlatformTarget, Severity,