Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 84 additions & 14 deletions CLAUDE.md
Original file line number Diff line number Diff line change
Expand Up @@ -218,6 +218,13 @@ transcript tab and an agent both read it to explain an empty transcript.
`analyze_asset_media_with_progress` streams a per-step `AnalysisProgress`
(`silence`/`scenes`/`loudness`/`rhythm`/`download_model`/`transcribe`/`done`), and
transcription runs **last** so the markers land before minutes of inference.
**A pass is abandonable** (`analyze_asset_media_cancellable` / `analyze_cancellable`,
a `CancelFn` alongside the `ProgressFn`): the check lands between steps *and* inside
transcription — the ffmpeg `whisper` run polls it about once a second off
`-stats_period 1` and kills the child, and the model download polls it per chunk,
keeping the `.part` file so the next attempt resumes rather than re-fetching 148 MB.
A cancelled pass returns `Error::Cancelled` and caches **nothing**: a half-analyzed
asset would read as analyzed, and its missing transcript as "no speech".

Two more optional features: `libav-render` (above) and `whisper` (in-process
`whisper-rs`; needs cmake, a C++ compiler and **libclang** at build time). Both are
Expand Down Expand Up @@ -446,8 +453,10 @@ no editing logic in the adapter.
re-decoding the whole file), `WhisperFilterTranscriber` (the ffmpeg `whisper`
filter, always compiled) and
`WhisperTranscriber` (in-process, `whisper` feature); `NullAnalyzer` is still the
fallback. `Transcriber::transcribe` takes a `ProgressFn` — alone among the
providers, because it can download a model and then run for minutes.
fallback. `Transcriber::transcribe` takes a `ProgressFn` *and* a `CancelFn` —
alone among the providers, because it can download a model and then run for
minutes, which is both the only step worth reporting on and the only one worth
being able to give up on.
`Project::analyze_asset` wires them and caches the `AssetAnalysis`.
- `error.rs` — `Error`/`Result`; the `Ffmpeg(#[from] ffmpeg_next::Error)` variant is
itself `#[cfg(feature = "ffmpeg")]`.
Expand All @@ -467,7 +476,13 @@ It is spawned from `lib.rs`'s Tauri `.setup` hook on
`tauri::async_runtime` and shares the **same** `Arc<Mutex<Project>>` the Tauri commands
hold, so the agent edits the project the user has open. Patterns that matter if you edit
it: `#[tool_router]` on the impl + `#[tool_handler]` on `impl ServerHandler` — **no
`tool_router` field on the struct** (the macro calls `Self::tool_router()`).
`tool_router` field on the struct** (the macro would call `Self::tool_router()`).
That default is also the reason for the `router()` `OnceLock`: the generated
`call_tool` / `list_tools` / `get_tool` each *evaluate* the router expression, so
`Self::tool_router()` rebuilds all ~85 routes — a schema lookup, a boxed handler
and a map insert apiece, ~250 µs of release-build work — on **every request**.
The routes are fixed at compile time, so it is built once and
`#[tool_handler(router = router())]` hands out a borrow.
`ServerInfo` is `#[non_exhaustive]`, so `get_info` builds it via `Default::default()`
then mutates fields — including `server_info` (`server_identity`), because that
default is filled from **rmcp's own** crate identity and left alone the server
Expand All @@ -479,13 +494,31 @@ parts) and `preview_timeline` (the composited cut at a timeline time) — return
`Content::text` plus a `Content::image(bare_base64, "image/jpeg")` block the LLM can
actually *see* (rmcp wants bare base64 + MIME, **not** a `data:` URL). The `lock()`
helper sets `EditSource::Agent` per-op under the shared lock (the GUI's `project()`
helper sets `User` the same way); every **mutating** tool calls `self.changed()`, which
emits a `project-changed` Tauri event so the webview re-fetches and the edit shows up
live in the GUI. Because agent edits **stage**, "live in the GUI" now means the
helper sets `User` the same way); every **mutating** tool goes through the `edit()`
helper, which runs the op under the lock, **releases it**, and only then emits a
`project-changed` Tauri event so the webview re-fetches and the edit shows up
live in the GUI — that order matters, because the re-fetch the event triggers
takes the same lock. `set_speech_model` emits `speech-model-changed` instead,
which the webview listens for to re-read the transcription status: it reads that
once at launch, and `project-changed` would re-fetch the timeline, history and
task queue, none of which moved. Because agent edits **stage**, "live in the GUI" now means the
proposal appears for review, not that the cut changes: the read tools
(`get_timeline_state`, `timeline_summary`, `preview_timeline`, `export`) go through
`working_timeline`, so the agent sees the cut it is building, and
`timeline_summary` carries `staged_changes` so it cannot mistake one for the other.
`timeline_summary` carries `staged_changes` so it cannot mistake one for the other,
and a per-track `gaps` list — a hole between clips (or before the first one) is
black picture, which is the kind of defect an agent has to be *told* about since
it never watches the cut. `core_err` splits the caller's mistakes (a stale id, an
out-of-range value, a stale staged edit) out as `invalid_params`: reported as
`internal_error`, a mistyped uuid reads to a model as a broken server rather than
as something it can fix and retry. Sizes an agent picks out of a schema
description — `get_waveform`/`get_energy` buckets, `get_frame`/`preview_timeline`
widths — are clamped rather than trusted, the way `skim_asset` already clamps its
grid. `set_speech_model` is the write side of `transcription_status`
(`download_speech_model` only fills the cache; transcription uses whichever model
is *selected*, so downloading without selecting was a silent no-op) — it makes
both writes the GUI picker makes, though the picker itself only re-reads at
launch, so a model an agent selects shows there on the next start.
`smart_crop` frames each shot for the delivery frame (the server `instructions`
pair it with `set_delivery_format`, since reshaping to 9:16 otherwise keeps
whatever was in the middle). `generate_captions` / `clear_captions` caption the
Expand All @@ -496,6 +529,17 @@ say to caption **last** and to re-run after any further
edit, because captions are placed in timeline time and a later trim moves the
words out from under them — which an agent has no way to infer from the tool
list.
`import_asset` is the one write that does **not** stage — a file on disk is not
an edit to the user's cut, so imported media (and its background proxy) lands for
them immediately, reporting on the same `import-progress` event a lens-pair
stitch drives for the GUI. `export` takes rmcp's `RequestContext` beside its
`Parameters`: a render runs for minutes, so it forwards ffmpeg's progress to the
client's `progressToken` and passes `context.ct` as the cancel callback, deleting
the half-written file on cancel the way the GUI's export does. Progress goes
through an unbounded channel to a spawned forwarder because the render itself is
on the blocking pool and `notify_progress` is async; the forwarder drains the
channel even with no token, so a client that asked for no progress doesn't leave
ticks piling up.
`platform_check` tells it whether the cut is publishable where it is going
(and the server `instructions` tell it to run that before reporting a cut
finished — an agent that assembles a four-minute Reel has done the work and lost
Expand Down Expand Up @@ -546,8 +590,16 @@ agent task queue (`list_tasks`, `add_task` → the new `Task`; `resolve_task` /
`remove_task` → the refreshed `Task[]`), the agent's staged proposal
(`get_staged_edit` → the `StagedEdit` *with its diff*, so the review card renders
from one round-trip; `get_staged_timeline` for previewing it; `apply_staged_edit` /
`discard_staged_edit`) and `revision_diff`, and `export_timeline` (emits
`export-progress` events) / `cancel_export`. **No command runs on the main thread** (a plain sync
`discard_staged_edit`) and `revision_diff`, `export_timeline` (emits
`export-progress` events) / `cancel_export`, `cancel_analysis` (the same shape,
for the analysis pass — importing ten clips must not be an unbreakable
commitment to ten transcriptions) and `agent_status` (the MCP endpoint plus how
many seconds ago an agent last spoke to it, or `null` if none ever has —
`mcp::LAST_AGENT_ACTIVITY`, stamped in `lock_agent` and in `get_info`, since
`initialize` is the one moment an agent is known to be there; a
streamable-HTTP client holds no connection between calls, so there is no socket
to report and the panel judges from the age instead of the green dot it used to
show unconditionally). **No command runs on the main thread** (a plain sync
command would freeze the window in Tauri v2): quick ops are
`#[tauri::command(async)]`, and every heavy one (ffmpeg decode / analysis /
export, disk-bound open/save) is an `async fn` that pushes its work onto the
Expand Down Expand Up @@ -638,7 +690,11 @@ The editor UI is implemented from the **Kerf design system** (claude.ai/design):
editor-grade workspace under `src/lib/components/editor/` — bespoke atoms (`Btn`,
`IconBtn`, `Badge`, `Icon`, `KerfMark`) plus `TitleBar`, `Toolbar`, `MediaBin`,
`Preview`, `Timeline`, `Inspector`, `AgentPanel`, `StatusBar`, composed by
`routes/+page.svelte`. The `Inspector` (right panel) edits the selected clip —
`routes/+page.svelte`. The `Inspector` (right panel) is **mounted whether or not a
clip is selected** and toggled from the toolbar (`ui.inspectorOpen`): its Text
overlays section belongs to the timeline rather than to any one clip, so gating the
whole panel on a selection made titles and captions unreachable until you clicked a
clip. It edits the selected clip —
trim, volume, fades, speed, transform, color, **transition** (a grouped picker
over `src/lib/transitions.ts` — fade / slide / push, then a direction, because
that is the order the choice is actually made and a flat list of eleven names
Expand Down Expand Up @@ -762,7 +818,11 @@ from the playhead rather than playing it out in slow motion against the sound.
playback moves under `bun run dev` and that failure mode is reproducible without a
desktop build. `ExportDialog` (⌘E) drives
the full `ExportOptions` surface — presets, containers/codecs, rate control, resolution,
loudness normalize, and a **Range: In → out** choice when marks are set. `MediaBin`'s
loudness normalize, and a **Range: In → out** choice when marks are set. It **opens on
the frame the project is cut for** (`initialExport`): the preset whose resolution is
that frame when one matches, else the default preset with its resolution cleared so
"Project frame" renders — otherwise a 9:16 project opened its export already
landscape and the readiness panel warned about the shape the user had just chosen. `MediaBin`'s
**Transcript tab is an editing surface**: lines resolve to the clip carrying them,
click seeks, the playhead line highlights, and `×` cuts the sentence from the timeline
(`cut_clip_range`); cut lines render struck through. When it is *empty* it says which
Expand Down Expand Up @@ -824,11 +884,21 @@ keeps its placeholder). This browser sample is a **dev harness only** — the de
uses the real backend and starts empty. State is two runes singletons: `src/lib/state.svelte.ts`
(`export const editor` — assets, timeline, analyses, selection, and the editing actions that
call the backend and apply the returned `Timeline`) and `src/lib/editor-ui.svelte.ts`
(`export const ui` — chrome state, playhead/zoom/playback, and `runAnalysis` which runs real
analysis and toggles the `analyzing` flag). There is **no scripted demo phase machine**: the
(`export const ui` — chrome state, playhead/zoom/playback, and `analyzeQueue` /
`runAnalysis` / `stopAnalysis`: a batch analyzes **one asset at a time** — each pass is
ffmpeg-bound, so running them together only makes each slower — and stopping drops
the whole rest of the queue). There is **no scripted demo phase machine**: the
editor chrome derives from real state — `MediaBin` shows a dropzone until `editor.assets` is
non-empty, `StatusBar` shows the selected asset's real fps/resolution/codec and timeline
duration, and `Preview` shows the decoded frame or a "No media loaded" placeholder.
duration (plus the analysis step, what is still queued behind it and a **Stop**), and
`Preview` shows the decoded frame or a "No media loaded" placeholder.
**Dropping files onto the window imports them** (`+page.svelte` listens for Tauri's
`onDragDropEvent`, filters by `isMediaPath` — the same extension list the picker
filters by, so a dropped folder of mixed files doesn't answer with one error per
README — and runs the same `editor.importPaths` the picker resolves to), which is
what the bin's "Drop media to start" had been promising. `editor.error` renders as a
dismissible banner under the toolbar: it was recorded and never shown, so a `.kerf`
that would not open opened as silence.

## Conventions

Expand Down
82 changes: 70 additions & 12 deletions crates/kerf-app/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@ struct AppState {
/// Set by `cancel_export` and polled by the in-flight export; lives outside
/// the project lock so a cancel lands even while a render holds it.
export_cancel: Arc<AtomicBool>,
/// Same, for the in-flight analysis pass. Analysis downloads a speech model
/// and then transcribes for minutes, so it has to be abandonable — and the
/// GUI runs imported assets through it one after another, which is a long
/// commitment to make on the user's behalf without an exit.
analysis_cancel: Arc<AtomicBool>,
}

#[derive(Serialize)]
Expand Down Expand Up @@ -192,13 +197,27 @@ async fn save_project_as(state: State<'_, AppState>, path: String) -> CmdResult<
/// Progress of a slow import (an Insta360 lens pair being stitched), tagged with
/// the file the user picked so the UI can label it while several import at once.
#[derive(Clone, serde::Serialize)]
struct ImportProgress {
pub(crate) struct ImportProgress {
path: String,
fraction: f64,
elapsed_secs: f64,
eta_secs: Option<f64>,
}

impl ImportProgress {
/// Tag a render-progress tick with the file it belongs to. Shared with the
/// MCP `import_asset` tool so an agent's import reports on the same event
/// and drives the same overlay the user's own import does.
pub(crate) fn new(path: &str, p: kerf_core::ExportProgress) -> Self {
Self {
path: path.to_string(),
fraction: p.fraction,
elapsed_secs: p.elapsed_secs,
eta_secs: p.eta_secs,
}
}
}

#[tauri::command]
async fn import_asset(app: AppHandle, state: State<'_, AppState>, path: String) -> CmdResult<Asset> {
let shared = state.project.clone();
Expand All @@ -207,15 +226,7 @@ async fn import_asset(app: AppHandle, state: State<'_, AppState>, path: String)
// parallel imports really run in parallel and a multi-minute stitch never
// freezes the GUI or the agent — then take it only for the quick insert.
let mut on_progress = |p: kerf_core::ExportProgress| {
let _ = app.emit(
"import-progress",
ImportProgress {
path: path.clone(),
fraction: p.fraction,
elapsed_secs: p.elapsed_secs,
eta_secs: p.eta_secs,
},
);
let _ = app.emit("import-progress", ImportProgress::new(&path, p));
};
let asset = Project::probe_import(std::path::Path::new(&path), &mut on_progress).map_err(|e| e.to_string())?;
// Importing the pair's other lens (or the same file twice) resolves to
Expand Down Expand Up @@ -308,6 +319,9 @@ struct AnalysisProgressEvent {
async fn analyze_asset(app: AppHandle, state: State<'_, AppState>, asset_id: String) -> CmdResult<AssetAnalysis> {
let id = id(&asset_id)?;
let shared = state.project.clone();
// Fresh cancel flag for this pass; `cancel_analysis` flips it from the UI.
let cancel = state.analysis_cancel.clone();
cancel.store(false, Ordering::SeqCst);
blocking(move || {
// Resolve the asset under the lock, run the multi-second ffmpeg analysis
// with the lock released, then re-acquire it only to cache the result —
Expand All @@ -327,7 +341,15 @@ async fn analyze_asset(app: AppHandle, state: State<'_, AppState>, asset_id: Str
},
);
};
let analysis = kerf_core::analyze_asset_media_with_progress(&asset, &mut on_progress).map_err(|e| e.to_string())?;
let analysis = kerf_core::analyze_asset_media_cancellable(&asset, &mut on_progress, &|| cancel.load(Ordering::SeqCst))
.map_err(|e| match e {
// A cancel is the user's own doing, not a failure — the
// caller keys off this string to stay quiet about it.
kerf_core::Error::Cancelled => ANALYSIS_CANCELLED.to_string(),
other => other.to_string(),
})?;
// Nothing is cached for a cancelled pass: a half-analyzed asset would
// read as analyzed, and the missing transcript as "no speech".
lock_user(&shared).set_analysis(&analysis).map_err(|e| e.to_string())?;
Ok(analysis)
})
Expand All @@ -337,7 +359,7 @@ async fn analyze_asset(app: AppHandle, state: State<'_, AppState>, asset_id: Str
// ---- speech-to-text -------------------------------------------------------

/// The project-meta key holding the user's speech-model choice.
const SPEECH_MODEL_KEY: &str = "speech_model";
pub(crate) const SPEECH_MODEL_KEY: &str = "speech_model";

/// Which transcription backend this build will use, and whether its model is
/// already downloaded. The transcript tab reads this to explain an empty
Expand Down Expand Up @@ -1432,6 +1454,18 @@ fn reveal_path(app: AppHandle, path: String) -> CmdResult<()> {
.map_err(|e| e.to_string())
}

/// The error an abandoned analysis pass returns. The webview matches on it to
/// tell "the user stopped this" apart from "this broke".
const ANALYSIS_CANCELLED: &str = "analysis cancelled";

/// Request cancellation of the in-flight analysis pass (if any). The running
/// [`analyze_asset`] observes the flag between steps — and, during
/// transcription, about once a second — then gives up and caches nothing.
#[tauri::command(async)]
fn cancel_analysis(state: State<'_, AppState>) {
state.analysis_cancel.store(true, Ordering::SeqCst);
}

/// Request cancellation of the in-flight export (if any). The running
/// [`export_timeline`] observes the flag on its next progress poll, stops
/// ffmpeg, and returns the `"export cancelled"` error.
Expand All @@ -1450,6 +1484,27 @@ fn mcp_endpoint() -> String {
mcp::endpoint_url()
}

/// Where the endpoint is, and how long ago an agent last used it.
///
/// A streamable-HTTP client holds no connection between calls, so there is no
/// "is it plugged in" to report — `last_seen_secs` is `None` until something
/// has spoken to the endpoint at all, and the panel decides from its age
/// whether to call that connected. Anything else would be the green dot the
/// panel used to show whether or not an agent existed.
#[derive(Serialize)]
struct AgentStatus {
endpoint: String,
last_seen_secs: Option<i64>,
}

#[tauri::command(async)]
fn agent_status() -> AgentStatus {
AgentStatus {
endpoint: mcp::endpoint_url(),
last_seen_secs: mcp::agent_last_seen_secs(),
}
}

// ---- diagnostics (logs) ----------------------------------------------------

#[tauri::command(async)]
Expand Down Expand Up @@ -1564,6 +1619,7 @@ pub fn run() {
.manage(AppState {
project: project.clone(),
export_cancel: Arc::new(AtomicBool::new(false)),
analysis_cancel: Arc::new(AtomicBool::new(false)),
})
.setup(move |app| {
// Logging needs the resolved platform log directory, so set it up here
Expand Down Expand Up @@ -1679,11 +1735,13 @@ pub fn run() {
hw_encoders,
export_timeline,
cancel_export,
cancel_analysis,
export_cover,
platform_targets,
platform_check,
reveal_path,
mcp_endpoint,
agent_status,
log_dir,
reveal_logs
])
Expand Down
Loading
Loading