smart crop: frame each shot for the delivery frame - #36
Merged
Conversation
Two halves of the last mile a social-video cut has to travel after the
render, both in kerf-core so the GUI and an agent get them together.
`export_still` writes the composited timeline at a given time to a real
file at full delivery resolution, through the same graph the export
renders — so a cover is literally a frame of the video it fronts, at the
shape the project is cut for, rather than a screenshot to crop back into
agreement. The still arg builder grew a sink (`StillOutput`) instead of a
second copy: the preview keeps its MJPEG pipe, a cover gets a JPEG or PNG
file, and every existing still test is untouched. An `--ignored` test
runs the binary and probes the result, because an arg builder that never
produced a file would look identical from here.
`platform.rs` answers whether the cut is ready to go somewhere. It keeps
two limits apart that are usually conflated: what a platform *rejects*,
and what it accepts and then stops distributing — a four-minute Reel
uploads fine and is shown only to existing followers, which is the worse
of the two because nothing tells you. Errors, warnings and tips, each
phrased with the real numbers ("0:20 over", "cutting 1:00 would keep it
in the feed"). Aspect is compared as a ratio, so 720x1280 reads as the
right shape and merely soft. Limits verified 2026-08-25 and advisory —
Kerf says what it thinks and exports what you ask for.
The GUI gets export_cover / platform_targets / platform_check, plus reveal_path so a finished render can be found — it opens the containing folder rather than the file, since "show me where it went" is not a request to launch a player. The heavy one follows the usual shape: resolve under the lock, decode with it released. The agent gets platform_check and export_cover, and the server instructions now tell it to check before reporting a cut finished — an agent that assembles a four-minute Reel has done the work and lost the audience, and nothing in the file would have told it.
The export dialog now leads with where this render can go: "Ready for Instagram Reels · YouTube Shorts · TikTok", then any length error or reach warning on its own line, then one collapsed line for shape. That last grouping is the point — a landscape cut earns a near-identical "will be letterboxed" from all four vertical feeds, and four lines say nothing four times, so issues carry an `IssueKind` and the panel folds them into one line naming the platforms. It judges the frame this render will actually produce, not the project's: a 9:16 project exported at 1920x1080 is a landscape file and is told so. The cover frame is saved from the preview's context menu at the playhead, and both a finished export and a saved cover offer "Show in folder" — a path in a toast is not much use on its own. kerf-core decides all of it. `platforms.ts` mirrors the check for the browser harness only, so the panel can be driven under `bun run dev`; verified there against both a 16:9 and a 9:16 render.
Reshaping a cut throws away most of one axis, and both fits pick that axis without looking: Cover takes the middle, Contain keeps everything and shrinks it into a letterboxed strip. Neither is right when the subject stands in the left third, which is where a subject usually stands. engine::salience_map decodes ~48 tiny gray frames of a source window in one ffmpeg pass and scores each cell by edge energy plus frame-to-frame motion, so a locked-off talking head scores on detail and a follow shot on both. Not face detection: no model to ship, and the answer only has to beat a centre crop. SalienceMap::crop_for then slides a window of the delivery aspect across that map and returns the crop that keeps the content, with a centre bias so a flat map resolves to the plain centre crop rather than to whichever edge won by rounding. Project::smart_crop applies it per clip as one revision, split for the lock-free pattern so the decodes don't hold the project lock. The result is an ordinary Transform crop, which the graph already applies before the fit scale — so the preview, the still and the export all follow, and the inspector's sliders still have the last word.
Both follow the shape every heavy op here uses: plan under the project lock, decode with it released, apply under it again — one short ffmpeg pass per clip must not freeze the window or stall the other MCP tools. The server instructions pair smart_crop with set_delivery_format, because an agent that reshapes a cut to 9:16 without it keeps whatever happened to be in the middle of every shot and has no way to know.
The inspector grows a Framing section above the crop sliders it writes: Smart crop for the selected shot, Reset crop, and — when the shot already matches the frame, or is 360 and framed by its virtual camera — a line saying why the button is off rather than a control that would refuse. The agent panel gains a "Frame for the delivery" chip for the whole cut, next to the other presets that run a local op. smart-crop.ts mirrors only the *shape* arithmetic for the browser harness, the way platforms.ts does: with no decoder to sample with it lands on the centre window, and which part of the shot survives is the half that only exists with media behind it.
The delivery last mile went in unformatted too, so this covers platform.rs and platform_check alongside the smart crop additions.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Kerf can already be told what frame a project is cut for (
Timeline.format), and the last release made it check whether the cut suits where it's going. What it could not do was decide which part of each shot survives the reshape — and that is most of the work when 16:9 footage becomes a 9:16 Reel.Both fits pick that axis without looking.
Covertakes the middle;Containkeeps everything and shrinks it into a letterboxed strip. Neither is right when the subject stands in the left third, which is where a subject usually stands.What
smart_cropsamples where each shot's content actually sits and writes the crop that keeps it — per clip, so a cut of six shots gets six framings rather than one compromise.engine::salience_map— one ffmpeg pass decodes ~48 tiny gray frames of a source window and scores each cell by edge energy plus frame-to-frame motion, so a locked-off talking head scores on detail and a follow shot on both. Deliberately not face detection: no model to ship, no licence to carry, and the answer only has to beat a centre crop — which is what the alternative actually is.SalienceMap::crop_for(pure, unit-tested) slides a window of the delivery aspect across that map and returns the crop that keeps the content, with aCENTER_BIASso a flat map resolves to the plain centre crop rather than to whichever edge won by rounding.Project::smart_cropapplies it as one undoableSmart croprevision, split three ways for the lock-free pattern so the decodes never hold the project lock. Clips already the delivery shape are skipped; so are 360-reframed clips, whose virtual camera is the framing decision. A pass that changes nothing writes no revision.The result is an ordinary
Transformcrop, which the graph already applies before the fit scale — so the preview, the scrubbed still and the export all follow from the one change, and the inspector's sliders still have the last word. No new filter code in the export graph; every pre-existing graph test is byte-identical.Surfaces
smart_crop(clip_id?)and MCPsmart_crop— both plan under the lock, decode with it released, apply under it again. The serverinstructionsnow pair it withset_delivery_format, since an agent that reshapes to 9:16 without it keeps whatever was in the middle of every shot and has no way to know.Smart crop,Reset crop, and a line explaining why the button is off when the shot already matches the frame or is 360.Frame for the deliverypreset chip for the whole cut.src/lib/smart-crop.tsmirrors only the shape arithmetic for the browser harness (the wayplatforms.tsdoes) — with no decoder it lands on the centre window, which keepsbun run devdrivable.Verification
--no-default-features), including 9 new ones over the pure crop math, the arg builder and the frame scoring, plus a graph test that the crop lands before the fit.#[ignore]d real-ffmpeg test: synthesizes a 16:9 shot with its only content in the left third, samples it, and asserts the 9:16 window keeps it — a centre crop would miss it entirely. Passes locally, along with the rest of the--ignoredsuite (downloads_a_real_modelstill fails in WSL2, as before and unrelated).cargo clippy --workspace --no-default-features --all-targetsclean;bun run checkclean; 42 bun tests green.🤖 Generated with Claude Code