transitions, track mixer, clip masks - #38
Merged
Merged
Conversation
A subtitle line is read; a punched word is watched. `CaptionStyle` picks between them, and it is one decision rather than four: the word count, the size, the position and the floors that stop a line flickering all move together, so asking for `word_punch` gets the whole look instead of one word left at subtitle size in the subtitle position. Held to `MIN_CAPTION` every short word would merge into a neighbour and the style would collapse back into lines, so word punch gets its own floors — words still merge, just far later. `CaptionOptions` is now style plus overrides: every number is optional and follows the style when omitted, resolved by `resolve()` into the `CaptionLayout` captioning actually works from. `CaptionOptions::default()` is unchanged, so every existing call captions exactly as before. Captions are also fitted to the frame now. `drawtext` neither wraps nor scales, and a 9:16 frame is barely half as wide as it is tall, so a long word — or a 28-character subtitle line, which was already true before this — was drawn off both edges. `fontsize` cannot be an expression over `text_w` (the width is what depends on the size), so `fit_size` estimates it from the character count against the project's delivery aspect. An unframed project assumes 16:9, which is wide enough that the fit never binds: nothing that never picked a frame captions differently than it did.
`generate_captions` takes a `style`, and the instructions say to prefer `word_punch` for a vertical cut: an agent asked for a Reel has no way to know from a tool list that the subtitle shape is not what social captions look like. The other params are documented as overrides on top of the style rather than as defaults, since that is what they now are.
The Inspector's Text overlays section leads with a Lines / Word punch choice, and the caption button generates in it. The selection is not derived from the overlays already on the timeline: a caption's style is not recoverable from the text it carries, and guessing it from the word count would flip the chip every time a sentence happened to be short. `captions.ts` mirrors the style table and the frame fit, so the browser harness still produces exactly what the backend would — which is how this was driven end to end: a 9:16 project captions `non-destructive` at 6% and `the` at the style's full 11%.
A piecewise keyframe expression contains commas, and the filtergraph parser reads an unquoted comma as the end of the filter — so `overlay=x=` for a clip with position keyframes, and the `drawtext` x/y of an animated text overlay, both made ffmpeg abort the whole render with `No such filter: '2)'`. Neither is exotic: the Text overlays style chips animate an overlay's opacity, which puts every overlay they create on the keyframed branch, so a project that used one could not be exported at all. Every unit test asserted on the graph string, and the graph string looked right — so the regression test renders one of each and asks ffmpeg to accept the graph.
Two options — a dissolve and a dip to black — is where a montage starts to look homemade, and a travel or brand cut is mostly shot-to-shot movement. Adds nine: a dip to white, and slide / push in four directions each. The families are what decide the render, so `TransitionKind` answers for them rather than the engine matching on every variant: a **dip** takes both sides through a solid colour and needs no handle, a **dissolve** mixes them, and a **motion** transition travels the incoming clip in over the outgoing one (slide) or carries the outgoing one out with it (push). No new machinery in the graph. A motion transition is an overlay offset, and an overlay offset is what `keyframe_expr` already writes — so a slide is two keyframes, reusing the tested expression builder rather than a second expression language. `ClipFx` grows the offsets, the existing crossfade rules for borrowing the outgoing clip's source handle apply unchanged (no handle still degrades to a hard cut), and the travel is *added* to whatever position the clip already has, so a picture-in-picture slides in to where it lives. The sound dissolves under a slide either way: the picture moving is no reason for the audio to cut hard. A slide and a push differ only in whether the outgoing clip moves, which no assertion on the graph string can see — so the real-ffmpeg test renders both and looks at where the stripe went.
`TransitionKind::parse` already accepted the new kinds, so what was missing was anyone knowing they exist: both surfaces named "crossfade or dip_to_black" in their errors and the MCP tool description, which is the whole of what an agent has to go on. The expected-kind list is now derived from the enum (`wire_names`), so an error message cannot drift from what parses. The tool description says what the directions mean and that a dissolve or motion transition borrows the outgoing clip's source handle — an agent hitting the hard-cut fallback otherwise has no way to know why nothing happened — and `preview_timeline` stops claiming it approximates a mid-transition blend, since the still path renders the plain cut.
The Inspector's Type dropdown listed the two kinds it was written against. It now lists all eleven, grouped — because the choice a user actually makes is between a fade, a slide and a push, and only then a direction, and a flat list of eleven names hides that. `src/lib/transitions.ts` holds the list with the labels and the grouping, the same arrangement as delivery-formats.ts; its test pins the ids against `TransitionKind::ALL`, so a kind added on one side of the boundary and not the other fails rather than silently going missing from the picker. Driven through the browser harness: all three groups render and each selection survives the round-trip back into the timeline.
Kerf had a per-clip gain, a duck flag and a loudnorm on the master, and nothing in between — no way to say "the music sits 6 dB under the voice", which is most of what makes a cut sound finished. There was no pan anywhere at all. `Track.volume` and `Track.pan` ride every clip on the track, applied in the clip's own chain *after* its gain and effects — a channel strip, so pulling a music bed down does not change what its compressor was reacting to. Both are omitted at their neutral values, so all 271 existing graph tests stay byte-identical and a project that never touched a fader renders exactly what it always did. The pan is a **balance**, not a constant-power pan: the side you turn towards stays at unity and the other is attenuated away. Constant power would boost the near side 3 dB at the extremes, which is right for placing a mono source and wrong for leaning a finished stereo track — nudging a music bed left should not make it louder. It is dropped entirely on a mono delivery, where there is nowhere to pan to.
`set_track_volume` / `set_track_pan` on Tauri and MCP. The tool descriptions carry the part an agent cannot infer from a signature: that a music bed belongs on its own track pulled to roughly 0.3 under speech, that a fader is the right tool when a level should simply sit lower where ducking is for dipping and recovering, and that a hard pan sounds broken on the phone speaker most of this footage is watched on.
Each track header grows a mixer strip: a level fader and a pan, on any track that can actually be heard — an audio track, or a video track whose clips carry sound. A silent track gets none; a mixer strip on a silent track is furniture, not a control. Double-click returns either to its neutral value, and the tooltip reads in dB and L/R, the units a level is actually judged in. Preview playback follows: the fader multiplies the clip gain exactly as the export does, and the pan is rendered as the *same balance*, not a StereoPannerNode — its constant-power law would quietly disagree with the file. `get_audio` hands back mono, so the two legs are the stereo pair. `src/lib/mixer.ts` is the faithful TS mirror of `Track::pan_gains`, with the same assertions as the Rust test on both sides of the boundary: centre is exactly unity, and no pan is ever a boost.
Adds `Clip.mask`: a rectangle or ellipse, positioned and sized in fractions of the rendered frame, feathered, optionally inverted. Outside the shape the clip goes transparent and whatever is on a lower track shows through. Deliberately one primitive rather than a masking mode per use, because the timeline already stacks tracks: blurring a face is a copy of the shot on the track above, blurred and masked to the face; a rounded-off picture-in-picture is a mask on the upper clip; a region grade is a masked copy with its own colour. One thing to learn, composing with what is there, instead of a second compositor. Which is also what keeps it inside the existing per-clip chain. The mask only rewrites the alpha plane, so it is one filter in the linear chain the graph builder already emits — no branch, no second pass over the export graph. One expression covers both shapes: each axis is scaled so the edge sits at distance 1, and `max` gives a rectangle where `hypot` gives an ellipse. Fields are clamped, since dragging a handle too far would otherwise blank the clip. `geq` is per-pixel and slow, the same cost keyframed opacity already pays. The real-ffmpeg test renders black over white through a hard ellipse and checks the middle is black and the corners white — the only way to know a lower track really shows through.
`set_mask` on Tauri and MCP. The tool description carries what a signature cannot: that the mask composes with the track stack rather than replacing it, and therefore *how* to blur a face — duplicate the shot onto the track above, blur the copy, mask the copy to an ellipse. It also says to look at the result with preview_timeline, since the position is a fraction of the frame and nothing but a picture tells you whether it covered the right thing.
A Mask section in the Inspector: None / Rectangle / Ellipse chips, then centre, size, feather and invert. Picking a shape starts from a sensible default rather than a collapsed one, so the first click shows a mask instead of nothing. The caption under it carries the recipe the shape alone does not suggest — that a lower track shows through, and that blurring a face is a duplicated, blurred copy above, masked. Driven through the browser harness: the chips create and clear, the sliders round-trip into the timeline, and the edit lands in history.
- pan the track after aformat: run before the upmix a mono source has no c1, so panning it right muted the clip instead of leaning it - let a dissolve, slide or push out of a still keep its transition: a still loops, so its source handle is unbounded, not zero - teach Timeline::diff the new surfaces (clip mask, track level and pan) so a staged agent proposal touching only them is no longer discarded as empty by apply_staged - fold a clip's mask and keyframed opacity into one geq pass - mcp set_mask: omitted fields keep the clip's current mask instead of resetting the shape's size and feather to the defaults - un-splice set_mask's doc from set_transition's; correct the Track.pan doc claiming a constant-power law - frontend: align DEFAULT_MASK with Mask::default(), unpin the fader slider for levels above 2, cache the audible-asset set behind hasSound
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three features plus a render-breaking bug fix, and the word-punch caption style from a parallel session (kept on this branch deliberately).
Filtergraph quoting fix (0eeee84)
A piecewise keyframe expression contains commas, and an unquoted comma is where ffmpeg's graph parser thinks the filter ended.
overlay=x=anddrawtextx/y were unquoted, so any clip with position keyframes — and any overlay with animated opacity, which the Title/Lower-third/Caption style chips always write — aborted the export withNo such filter. Invisible to the unit tests, which assert on the graph string. Fixed with a real-ffmpeg regression test.Transitions: 2 → 11
Dip to white, plus slide and push in four directions each. No new graph machinery — a motion transition is an overlay offset, which
keyframe_expralready renders, so a slide is two keyframes. Handle-borrowing rules unchanged (no handle left → hard cut). Grouped Inspector picker: fade / slide / push, then direction.Track mixer
Track.volume+Track.pan, riding each clip after its own gain and effects like a channel strip. Pan is a balance, not constant-power — leaning a finished stereo track shouldn't make it louder. Mixer strip on every audible track; preview playback renders the same balance the export does.Clip masks
Rect/ellipse, feathered, invertible — one primitive that composes with the track stack (face blur = duplicate the shot above, blur, mask), which keeps it a single
geqalpha filter in the linear per-clip chain.Word-punch captions
CaptionStylepicks the subtitle shape (lines) or the social one-word shape (word_punch), withfit_sizeshrinking captions that would draw off a 9:16 frame's edges.All neutral values are omitted from the graph, so every pre-existing project renders byte-identically. 278 unit tests + 9 real-ffmpeg tests + 66 frontend tests pass; clippy clean.
🤖 Generated with Claude Code