Document status: Draft v1.0
Date: 26 July 2026
Owner: Duncan Trevithick
Working repository: agent-effects
Primary CLI: agent-effects
Optional CLI alias: afx
Project file extension: .afx.yaml
Agent Effects is an agent-first motion-graphics engine that converts structured story, scene and animation descriptions into rendered video.
A compiled Rust application will interpret declarative project files, render typography and graphics natively, and stream raw frames directly to FFmpeg. AI coding agents will create and modify the project files without writing Rust, controlling a constrained library of story structures, semantic compositions, layouts, animation presets, transitions and visual effects.
The initial product will support both:
- short 30–60 second motion-graphics explainers
- long-form 5–15 minute editorial and YouTube videos composed from independently renderable scenes
The MVP will focus on:
- kinetic typography
- simple vector graphics
- statistics and comparisons
- lists and process diagrams
- photographs and editorial collage layouts
- website and app screenshots
- stylised browser and device frames
- screenshot callouts and annotations
- constrained maps of Spain
- textured backgrounds
- drop shadows and layered depth
- narration-aligned timing
- deterministic scene transitions
- bounded-memory long-form rendering
- automated visual feedback for AI agents
Agent Effects is not intended to reproduce After Effects.
Its purpose is to prove that an AI agent can autonomously create a coherent, attractive and efficiently rendered motion-graphics video using a small, opinionated visual language.
Current browser-based video systems such as HyperFrames and similar HTML-to-video pipelines incur substantial overhead when capturing each rendered frame.
In the current workflow:
- approximately 94% of render time is spent capturing frames
- each frame capture takes approximately 750 milliseconds
- Chromium must lay out, paint and rasterise the scene
- a screenshot must then be extracted from the browser
- the result may be compressed, transferred and decoded before video encoding
At 30 frames per second, an 11-minute video contains approximately 19,800 frames.
At 750 milliseconds per capture, frame extraction alone would take approximately:
19,800 × 0.75 seconds = 14,850 seconds
= 247.5 minutes
= 4 hours 7 minutes
This excludes other rendering, media processing and encoding work.
Rewriting only the browser controller in Rust would not remove the underlying bottleneck because Chromium would still perform the layout, rasterisation and screenshot extraction.
Replacing HTML with low-level Rust or GPU code would solve part of the performance problem, but it would create an authoring problem. An AI agent should not need to implement text shaping, geometry, shaders, compositing or frame generation for every video.
Agent Effects therefore needs to solve two problems simultaneously:
- Eliminate the browser screenshot boundary.
- Give AI agents a constrained and discoverable language for creating attractive videos.
Create an agent-first motion-graphics system with:
- an After Effects-like internal rendering model
- a Canva-like collection of semantic components
- a declarative language designed for AI coding agents
- a compiled native Rust renderer
- deterministic rendering
- reusable branded visual systems
- automated structural and perceptual feedback
- a lightweight human review and refinement interface
The long-term workflow should allow a user to provide:
- a script
- a narration track
- a transcript
- a visual theme
- photographs
- screenshots
- maps or supporting data
- a target duration
- an aspect ratio
An agent should then be able to:
- identify the central message
- construct a beat sheet
- organise beats into scenes
- choose semantic compositions
- create the project DSL
- validate it
- render representative frames
- inspect visual and motion feedback
- revise the project
- render the final video
Give an AI agent a script, narration, visual theme and assets. Receive a polished short- or long-form motion-graphics video without browser screenshot capture, per-video Rust compilation or manual After Effects work.
The Rust renderer is compiled when the engine changes.
Individual videos remain data:
agent-effects render project.afx.yaml --output video.mp4Motion should reinforce the meaning, hierarchy and rhythm of the story.
The system should discourage arbitrary animation used solely for visual variety.
A word such as “collapse” may compress or fall. A statistic such as “94%” may dominate the frame. A transition representing resolution may simplify or settle the composition.
The agent should identify what the audience needs to understand before choosing visual layouts.
The primary storytelling unit is a beat, not a scene.
Agents should usually choose concepts such as:
- headline
- statistic
- comparison
- evidence
- reveal
- emphasise
- connect
- resolve
They should not normally position every rectangle or manually define every keyframe.
A small collection of polished components should outperform a broad system with inconsistent visual results.
The agent should prefer:
entrance:
preset: words-rise
feel: confidentover:
position:
from: [0, 72]
to: [0, 0]
opacity:
from: 0
to: 1
duration: 0.57
easing:
cubic_bezier: [0.16, 1, 0.3, 1]Explicit control remains available as an escape hatch.
The same project, engine version, assets, font files and random seed should produce the same frames.
The CLI, UI and agents must all operate on the same declarative project files.
The UI must not create important hidden state that cannot be represented in the DSL.
Agents should inspect:
- validation results
- representative still frames
- contact sheets
- low-resolution previews
- motion telemetry
- optional video-model critique
before producing a full-quality final render.
Long videos must not require retaining all rendered frames, decoded media or scene surfaces in memory.
The authoring system should expose four levels:
- complete scene templates
- semantic compositions
- animation presets
- explicit layers and keyframes
A developer or technical creator using an AI coding agent to produce short or long-form explainer, educational, marketing, property or editorial videos.
- YouTube creators
- SaaS marketing teams
- automated content systems
- developer-tool companies
- educational-content producers
- agencies producing repeated branded videos
- humans reviewing agent-generated videos
Given a script, narration and visual theme, create an attractive motion-graphics video without manually animating every layer.
Turn a script or transcript into a clear visual argument made from beats, scenes and recurring motifs.
Discover the available visual language, produce a valid project, render previews, inspect feedback and improve the result without modifying the Rust renderer.
Quickly understand the story, pacing and visual problems, then approve or adjust the generated video.
Add reusable visual capabilities without breaking the appearance of existing projects.
A constrained system focused on kinetic typography, editorial graphics and reusable semantic compositions can cover most of the visual needs of:
- short product explainers
- technical explainers
- property and location videos
- narrated YouTube essays
- SaaS marketing clips
- educational videos
- social-video derivatives
The MVP succeeds if an AI agent can autonomously produce a coherent video containing:
- a clear hook
- an understandable narrative arc
- readable kinetic typography
- images or screenshots
- statistics, comparisons or simple processes
- deliberate transitions
- narration-aligned timing
- consistent visual styling
- no significant layout or rendering errors
The agent should accomplish this without:
- writing Rust
- compiling Rust per video
- controlling Chromium
- manually capturing screenshots
- creating arbitrary shaders
- using a traditional After Effects timeline
- retaining the entire video in memory
The primary MVP benchmark should be an approximately 11-minute editorial video containing:
- kinetic typography
- multiple sections and narrative beats
- maps of Spain
- highlighted provinces such as Almería and León
- map markers and simple zooms
- photographs of properties
- property-listing screenshots
- browser-window compositions
- Polaroid-style image cards
- tape and sticker overlays
- paper-like textured backgrounds
- drop shadows
- simple diagrams and statistics
- narration-synchronised transitions
- long settled still-image intervals
This benchmark is intended to test:
- long-form project organisation
- constrained map support
- editorial image treatments
- screenshot presentation
- memory-bounded rendering
- scene-level caching
- resumability
- real rendering performance
The MVP includes:
- story goal
- audience
- objective
- central takeaway
- tone
- narrative arc
- ordered beats
- beat purposes
- desired audience responses
- beat-to-scene grouping
- semantic relationships between scenes
- recurring visual motifs
- pacing metadata
The MVP includes:
- text
- numbers
- rectangles
- rounded rectangles
- circles
- ellipses
- lines
- arrows
- connectors
- SVG assets
- raster images
- groups
- clipping masks
- solid backgrounds
- gradient backgrounds
- raster-image backgrounds
- tiled texture backgrounds
- deterministic noise or grain overlays
The MVP includes:
- font family
- font weight
- font style
- font size
- line height
- tracking
- word spacing
- automatic wrapping
- automatic fitting
- minimum font-size constraints
- emphasis spans
- highlighted words
- foreground and accent colours
- left, centre and right alignment
- line splitting
- word splitting
- character splitting
- text masks
- safe-area validation
The MVP includes:
- image placement
- image cropping
- cover and contain fitting
- image scale and rotation
- rounded corners
- image borders
- drop shadows
- image tint
- Polaroid-style cards
- photo stacks
- taped photographs
- overlapping paper panels
- caption labels
- decorative PNG and SVG overlays
- collage layout presets
The MVP includes constrained semantic wrappers for:
- desktop browser
- generic laptop
- generic phone
Browser-frame support includes:
- stylised browser chrome
- configurable toolbar visibility
- URL-bar text
- page title
- macOS-style window controls
- screenshot viewport
- rounded corners
- border
- drop shadow
- viewport crop and pan
- optional cursor
- optional callout
- optional highlighted region
These are graphic compositions, not functional browsers.
The MVP includes:
- imported SVG maps with targetable element IDs
- built-in maps of Spain
- Spanish autonomous communities
- Spanish provinces
- region highlighting
- region dimming
- point markers
- pulsing markers
- text labels
- simple routes
- country-to-region zoom
- predefined map camera bounds
- pan and zoom transitions
- path fill, stroke and opacity animation
The MVP does not include a complete geographic-information system.
The MVP includes:
- position
- scale
- rotation
- opacity
- colour
- blur
- clipping and reveal progress
- image crop position
- image crop scale
- background position
- texture opacity
- texture scale
- shadow properties
- SVG fill and stroke
- map camera bounds
- browser viewport transforms
- named easing curves
- custom cubic Bézier curves
- deterministic springs
- stagger
- parent-child transforms
- basic motion paths
- constant-speed path traversal
- orient-to-path
- scene transitions
- basic motion blur where performance permits
The MVP includes:
- headline
- question
- statement
- statistic
- quote
- bullet list
- ranked list
- comparison
- before-and-after
- steps
- process
- callout
- screenshot
- screenshot with callouts
- browser frame
- device frame
- photo card
- photo stack
- SVG map
- location map
- simple route map
- end card
The MVP includes:
- narration audio
- external music track
- external sound-effects track
- word-level transcript timing
- phrase markers
- named timeline markers
- alignment to words and phrases
- scene-duration validation
- minimum settled-reading-time warnings
- audio muxing through FFmpeg
- scene timing independent of media decoding
The MVP includes:
- frame rendering at an arbitrary timestamp
- scene-specific rendering
- time-range rendering
- contact-sheet generation
- low-resolution preview generation
- full-resolution MP4 rendering
- chunked long-form rendering
- resumable rendering
- machine-readable diagnostics
- profiling output
- optional Kimi video-critique bridge
- local review UI
The MVP does not include:
- arbitrary HTML or CSS rendering
- live browser capture
- live webpages
- complete global mapping
- map tiles
- satellite imagery
- road routing
- arbitrary address geocoding
- terrain rendering
- advanced 3D
- character rigging
- particle systems
- arbitrary shader authoring
- freehand vector illustration
- complex path morphing
- full Lottie authoring
- full Rive authoring
- After Effects project compatibility
- third-party effects plug-ins
- full keyframe timeline editing
- Bézier graph editing
- collaborative cloud editing
- cloud rendering
- mobile editing
- generative image or video models embedded in the renderer
- a complete charting library
Simple bars, progress indicators and numeric comparisons may be represented through primitive or semantic components without introducing a full chart engine.
Agent Effects should distinguish storytelling from rendering.
Project
└── Story
└── Sections
└── Beats
└── Scenes
└── States
└── Compositions
└── Elements
└── Animations
The complete video, including:
- metadata
- theme
- assets
- story
- scenes
- output settings
- narration
- shared motifs
- renderer version
- random seed
The communication objective and narrative progression.
A long-form organisational unit such as:
- introduction
- context
- case study
- geographic comparison
- evidence
- conclusion
Sections group related scenes and permit long-form navigation.
One meaningful change in what the audience understands or feels.
A beat should be removable only at the cost of weakening the argument.
A visually coherent interval delivering one or more related beats.
A new scene is appropriate when there is a meaningful change in:
- layout
- subject
- dominant object
- argument stage
- background environment
- visual perspective
- narrative energy
A meaningful visual stage within a scene.
Example:
- display the whole rendering pipeline
- highlight frame capture
- dim the other stages
- enlarge the 94% statistic
These remain part of one scene because they form one continuous visual idea.
A reusable visual sentence structure such as:
- statistic
- comparison
- process
- map
- browser frame
A drawable object such as:
- text
- shape
- line
- image
- SVG
- marker
- connector
A time-dependent change to an element, group, composition or camera property.
Agent Effects should use three conceptual representations.
Describes what the video communicates.
story:
audience: "People considering cheap property in Spain"
objective: "Explain where cheap properties exist and the trade-offs"
takeaway: "Low prices often reflect location, condition or legal complexity"
tone: "curious, informative and practical"Describes how the story will be visualised.
beats:
- id: almeria-prices
type: evidence
message: "Some of Spain's lowest property prices are in Almería"
emphasis: "Almería"
desired_response: curiosity
scenes:
- id: almeria-map
beats: [almeria-prices]
composition:
type: location-map
map: spain-provinces
highlight: ["Almería"]Contains resolved:
- layers
- transforms
- bounds
- shaped text
- asset references
- timing
- keyframes
- effects
- clipping relationships
- scene transitions
This representation is generated by the compiler and consumed by the Rust renderer.
Agents should rarely edit it directly.
YAML will be the primary human- and agent-facing format.
Reasons:
- concise
- readable
- easy to diff
- supports comments
- suitable for AI generation
- maps cleanly to strongly typed Rust structures
- easier to inspect than raw animation JSON
The Rust type definitions are the source of truth.
Recommended libraries:
serdeserde_yamlschemars- tagged enums for composition types
- explicit project-version migration
Every project must declare:
version: 1Unknown fields should produce warnings or errors rather than being silently ignored.
Short projects may use one file:
project.afx.yaml
assets/
Long projects should support modular scene files:
project.afx.yaml
theme.afx.yaml
scenes/
├── 001-hook.afx.yaml
├── 002-context.afx.yaml
├── 003-spain-map.afx.yaml
├── 004-almeria.afx.yaml
├── 005-listing-example.afx.yaml
└── 006-conclusion.afx.yaml
assets/
├── narration.wav
├── narration.words.json
├── paper-grain.webp
├── spain-provinces.svg
├── listing-01.webp
└── property-01.webp
The root project may reference scene files:
version: 1
video:
width: 1920
height: 1080
fps: 30
scenes:
- include: "scenes/001-hook.afx.yaml"
- include: "scenes/002-context.afx.yaml"
- include: "scenes/003-spain-map.afx.yaml"Benefits:
- smaller agent-editable files
- clearer Git diffs
- independent validation
- scene-level caching
- scene-level rendering
- easier recovery
- parallel preparation
- selective invalidation
version: 1
video:
width: 1920
height: 1080
fps: 30
duration: 38
seed: 42817
theme:
background: "#F1EDE3"
foreground: "#1D2827"
muted: "#6F7975"
accent: "#EF694D"
secondary: "#55B7AD"
font_family: "Inter"
display_font_family: "Fraunces"
motion_style: editorial
layout_style: documentary
background:
colour: "@theme.background"
texture:
source: "assets/paper-grain.webp"
mode: tile
opacity: 0.16
scale: 0.8
story:
audience: "People looking for affordable property in Spain"
objective: "Show why some regions appear unusually cheap"
takeaway: "Cheap property is available, but location and paperwork matter"
tone: "curious, practical and slightly playful"
narration:
audio: "assets/narration.wav"
words: "assets/narration.words.json"
beats:
- id: hook
type: question
message: "Can you really buy a house in Spain for under €50,000?"
- id: map-context
type: context
message: "The cheapest listings cluster in specific inland regions"
- id: almeria-example
type: example
message: "Almería has low-priced properties close to the coast"
- id: tradeoff
type: contrast
message: "But the cheapest homes often need work or have paperwork risks"
scenes:
- id: opening
beats: [hook]
from: 0
to: 5
composition:
type: question
text: "Can you really buy a house\nin Spain for under €50,000?"
emphasis: "€50,000"
layout: editorial-left
entrance:
preset: lines-reveal
feel: curious
- id: national-map
beats: [map-context]
from: 5
to: 12
composition:
type: location-map
map: spain-provinces
highlight:
- province: "Almería"
colour: "@theme.accent"
- province: "León"
colour: "@theme.secondary"
markers:
- place: "Almería"
label: true
- place: "León"
label: true
transition_in:
relation: reveal
preset: camera-push
- id: listing-example
beats: [almeria-example]
from: 12
to: 23
composition:
type: browser-frame
style: minimal-macos
url: "idealista.com"
screenshot: "assets/listing-01.webp"
viewport:
fit: cover
position: [0.5, 0.28]
callouts:
- target: [0.72, 0.31]
text: "€43,000"
effects:
- type: drop-shadow
offset: [0, 24]
blur: 42
opacity: 0.22
- id: tradeoff-collage
beats: [tradeoff]
from: 23
to: 38
composition:
type: photo-stack
photos:
- source: "assets/property-01.webp"
rotation: -4
caption: "Low price"
- source: "assets/property-02.webp"
rotation: 3
caption: "Needs work"
- source: "assets/documents.webp"
rotation: -1
caption: "Check the paperwork"
transition_in:
relation: contrast
preset: split-wipeThe MVP vocabulary includes:
- hook
- question
- context
- claim
- evidence
- contrast
- cause
- mechanism
- example
- reveal
- consequence
- solution
- proof
- summary
- call to action
- curiosity
- surprise
- concern
- understanding
- confidence
- relief
- urgency
- excitement
- amusement
- continue
- contrast
- cause
- reveal
- resolve
- summarise
- escalate
- return
- introduce
- emphasise
- connect
- separate
- replace
- compare
- accumulate
- reduce
- reveal
- focus
- de-emphasise
- transform
- conclude
- settle
A motif is a recurring visual object or structure that creates continuity.
Examples:
- a map marker
- a horizontal pipeline
- a moving dot
- a paper card
- a price tag
- a highlighted word
- a route line
- a photograph frame
Example:
motifs:
- id: location-pin
type: map-marker
style: outlined
continuity: persistentA scene contains:
- ID
- section
- associated beats
- start and end timing
- composition
- optional states
- transition in
- transition out
- pacing metadata
- scene-local assets
- optional cached static surfaces
Example:
scene:
id: pipeline-breakdown
beats:
- capture-cause
states:
- id: full-pipeline
at: 0
- id: highlight-capture
at: 1.2
- id: reveal-statistic
at: 2.4A scene should not be created merely because one word changes.
Scenes are visual chapters. States are meaningful stages within those chapters.
Presents one dominant phrase with optional emphasis and supporting text.
Presents an opening or transitional question.
Presents a declarative idea without numeric dominance.
Presents a number, percentage, currency amount or duration as the primary visual.
Presents attributed speech or a source excerpt.
Presents a short unordered sequence.
Presents ordered items with numerical or visual ranking.
Presents two or more contrasting values or concepts.
Shows transformation from one state to another.
Presents ordered stages, flow or cause-and-effect.
Attaches a label to an element, image coordinate or map location.
Displays a screenshot with crop, pan, zoom and annotation.
Displays a screenshot inside stylised browser chrome.
Displays a screenshot inside a constrained laptop or phone frame.
Displays an image as a designed physical object.
Displays overlapping images with rotation, depth and captions.
Displays an imported targetable SVG map.
Displays a built-in map with named regions and markers.
Displays a simple origin-to-destination route.
Presents a summary, brand, call to action or final statement.
Agent Effects should encode the following defaults.
Animation should support the meaning of the phrase.
The system should not default to presenting every narrated word as animated subtitles.
Each scene should usually have:
- one primary phrase or object
- one supporting phrase or object
- optional accents
Important text should normally remain still and legible for longer than it animates.
Important visual changes should align to stressed words, phrase boundaries or deliberate pauses.
Constant motion weakens emphasis.
Linear motion should be reserved for intentionally mechanical movement.
Important words or objects should carry between scenes where this strengthens the visual argument.
Typography should respect:
- safe areas
- negative space
- alignment
- balance
- line length
- visual eye flow
The renderer should discourage:
- excessive rotation
- prolonged blur
- unreadably fast motion
- long periods at tiny scale
- overlapping moving text
A video should have a coherent motion grammar rather than unrelated effects in every scene.
Most polished typography can be produced with:
- translation
- scale
- opacity
- clipping
- stagger
- colour
- blur
- line drawing
The MVP should encode or warn around these guidelines:
- One dominant phrase per scene.
- No more than two primary movement directions at once.
- Typical entrance duration of 300–700 milliseconds.
- Important text remains settled longer than its entrance.
- Entrances default to ease-out.
- Exits default to ease-in.
- Word and line stagger are preferred over character stagger.
- Emphasis generally applies to 10–25% of words.
- Safe margins are enforced.
- Major ideas receive brief pauses.
- Important moving text resolves to a sharp still state.
- Simultaneous competing emphasis is discouraged.
These are defaults and diagnostics, not rigid creative restrictions.
- position
- scale
- rotation
- opacity
- fill colour
- stroke colour
- blur
- tracking
- line spacing
- clipping bounds
- mask progress
- line-drawing progress
- image crop
- image scale
- image rotation
- background position
- texture opacity
- texture scale
- shadow offset
- shadow blur
- shadow opacity
- SVG fill
- SVG stroke
- SVG opacity
- map camera bounds
- browser viewport position
- browser viewport scale
- effect parameters
- linear
- sine
- cubic
- quartic
- quintic
- exponential
- back
- elastic
- bounce
- custom cubic Bézier
- deterministic spring
- calm
- confident
- energetic
- urgent
- playful
- heavy
- elegant
- mechanical
- dramatic
- documentary
- understated
Semantic values map to tested combinations of:
- duration
- easing
- overshoot
- stagger
- travel distance
- blur
- motion intensity
- straight-line interpolation
- cubic Bézier path
- Catmull–Rom path
- points-through-path generation
- arc-length lookup
- constant-speed path traversal
- orient-to-path
- parent-relative transforms
Support stagger by:
- line
- word
- character
- list item
- process step
- child element
- map marker
- photo card
The MVP should launch with approximately 25 polished presets.
- words-rise
- lines-reveal
- masked-slide
- blur-resolve
- scale-reveal
- tracking-tighten
- split-reveal
- fade-slide
- scale-emphasis
- colour-emphasis
- highlight-sweep
- underline-draw
- pulse-once
- isolate-and-dim
- count-up
- photo-drop
- card-stack
- tape-in
- image-pan
- focus-crop
- browser-zoom
- callout-pop
- line-draw
- progress-fill
- ring-progress
- connector-grow
- marker-pulse
- region-highlight
- push-left
- push-up
- split-wipe
- directional-wipe
- shared-word
- shared-object
- simplify
- camera-push
- settle
- paper-slide
Each preset must have:
- valid targets
- default duration
- supported parameters
- semantic description
- visual preview
- canonical example
- stable version behaviour
The MVP includes:
- drop shadow
- blur
- border
- stroke
- corner radius
- clipping mask
- colour tint
- limited blend modes
- deterministic noise
- texture overlay
Drop-shadow example:
effects:
- type: drop-shadow
offset: [0, 18]
blur: 32
opacity: 0.22Supported drop-shadow animation properties:
- horizontal offset
- vertical offset
- blur radius
- opacity
- optional spread
Effects should use caching where possible.
The importer must retain or expose:
- element IDs
- element type
- document bounds
- element bounds
- transforms
- fill
- stroke
- opacity
The CLI must allow agents to inspect targetable elements:
agent-effects inspect-svg assets/spain.svg --list-idsExample result:
{
"elements": [
{
"id": "almeria",
"type": "path",
"bounds": [742.1, 611.4, 63.8, 54.2]
},
{
"id": "leon",
"type": "path",
"bounds": [389.2, 251.5, 78.7, 69.3]
}
]
}composition:
type: svg-map
source: "assets/spain-provinces.svg"
regions:
- selector: "#almeria"
fill: "@theme.accent"
- selector: "#leon"
fill: "@theme.secondary"Errors should suggest close matches:
{
"error": "svg_selector_not_found",
"selector": "#almeria",
"source": "assets/spain-provinces.svg",
"available_close_matches": [
"#almería",
"#province-almeria"
]
}The MVP should ship with a stable map package containing:
- Spain outline
- autonomous-community boundaries
- province boundaries
- canonical stable IDs
- named camera bounds
- major city coordinates where useful
- fade in
- draw outline
- highlight region
- dim other regions
- marker appearance
- marker pulse
- route drawing
- full-country to province zoom
- label attachment
- map pan
Project YAML
↓
Parser
↓
Schema validation
↓
Story and scene compiler
↓
Resolved render IR
↓
Timeline evaluator
↓
Layout and text shaping
↓
Native vector and image renderer
↓
Bounded pixel buffers
↓
FFmpeg stdin
↓
Video output
- Rust
serdeschemarsskia-safefor initial 2D renderingresvgor equivalent for SVG- FFmpeg subprocess receiving raw frames
- optional Rayon parallelism
- bounded producer-consumer queue
- CPU-backed renderer first
- GPU-backed Skia or
wgpuoptimisation after correctness
The Rust engine is compiled only when capabilities change.
A video is interpreted data:
agent-effects render project.afx.yamlFor each frame:
- identify the active scene and state
- evaluate keyframes and expressions
- resolve layout
- shape required text
- load or reuse required assets
- render the scene
- copy pixels into a bounded output buffer
- send the frame to FFmpeg
- reuse or release the buffer
The renderer must not retain all frames.
A 1920×1080 RGBA frame requires approximately 8 MB.
Retaining 19,800 frames would require roughly 158 GB before additional overhead.
Agent Effects must instead use a small bounded pool of reusable frame buffers.
Global resident information may include:
- output configuration
- theme
- font registry
- asset manifest
- transcript timing index
- scene manifest
Scene-local render information should be released after the scene finishes unless required by a transition or cache.
agent-effects render project.afx.yaml \
--from 240 \
--to 300 \
--output section.mp4agent-effects render project.afx.yaml \
--scene almeria-property-exampleLong renders should support safe chunks, initially aligned to scene boundaries.
agent-effects render project.afx.yaml \
--chunk-duration 20Automatic chunk boundaries must not cut through unsafe transitions.
The renderer should produce a manifest:
{
"engine_version": "0.1.0",
"project_hash": "abc123",
"completed_chunks": [
{
"from_frame": 0,
"to_frame": 899,
"output": ".agent-effects/chunks/0000.mkv"
}
]
}Rendering can resume when:
- the project hash is compatible
- relevant assets are unchanged
- the engine version is compatible
- completed chunks pass validation
Changing one scene should ideally invalidate:
- that scene
- dependent transition frames
- affected final assembly metadata
It should not require rerendering unrelated scenes.
Safe approaches include:
- chunking only at transition-safe scene boundaries
- rendering overlap frames
- treating transitions as separate bridge scenes
- rerendering a small boundary range
The first MVP implementation may restrict chunks to safe scene boundaries.
The renderer should not require fully decoded audio to remain in memory.
Permitted approaches include:
- passing original audio directly to FFmpeg
- reading transcript timings without decoding audio
- decoding only required windows
- assembling visual chunks before final audio muxing
- using external audio files during final encoding
The MVP should support:
- narration
- music
- sound effects
- volume envelopes
- fade in and fade out
- delayed starts
- final muxing
Advanced audio editing remains out of scope.
The engine should cache:
- shaped text runs
- loaded fonts
- decoded SVG structures
- raster images
- browser chrome
- static textures
- map geometry
- reusable shadow surfaces
- unchanged scene layers
- static composition surfaces
Caching must be bounded through:
- configurable limits
- least-recently-used eviction
- scene-aware release
- disk-backed cache where justified
- project and asset hashes
Caching must not compromise deterministic output.
Long-form editorial videos may contain intervals where most or all visual content is unchanged.
The DSL may declare:
hold:
from: 18.2
to: 21.8Optimisations may include:
- reusing the previous rendered frame
- reusing unchanged layer surfaces
- skipping layout work
- skipping repeated text shaping
- minimising repeated image processing
Output timing and frame cadence must remain correct.
The MVP should support a constrained expression or reference system.
Preferred semantic references:
anchor:
to: headline.right
offset: [40, 0]Timeline references:
start: "@word:Almería - 0.1"Property references:
position:
x: "= headline.right + 40"
y: "= headline.center_y"Arbitrary Rust or unrestricted JavaScript must not be embedded in project files.
A small safe expression evaluator may be used.
- invalid YAML
- unsupported project version
- missing required field
- unknown composition
- unknown preset
- invalid property type
- missing asset
- missing font
- circular reference
- invalid scene timing
- unresolved include
- text outside safe area
- text overflow
- fitted text below minimum size
- insufficient contrast
- overlapping text
- element outside canvas
- callout target outside image
- missing SVG selector
- map label outside viewport
- browser screenshot crop outside image
- shadow clipping
- important text settled too briefly
- excessive character stagger
- too many simultaneous animations
- abrupt position jump
- abrupt velocity discontinuity
- spring not settled before exit
- transition overlap
- motion path discontinuity
- unreadably fast moving text
- conflicting emphasis
- scene without associated beat
- beat not used by any scene
- repeated composition without variation
- no conclusion
- excessively dense scene
- too many unrelated ideas in one scene
- long interval without meaningful visual change
Errors should contain:
- exact DSL path
- received value
- expected values
- close matches
- documentation command
- suggested correction
Example:
{
"error": "unknown_preset",
"path": "scenes[1].entrance.preset",
"received": "smooth-pop",
"suggestions": [
"scale-reveal",
"fade-slide",
"photo-drop"
],
"docs_command": "agent-effects docs scale-reveal"
}The CLI is the primary interface for coding agents.
agent-effects init
agent-effects validate project.afx.yaml
agent-effects docs <component-or-preset>
agent-effects schema
agent-effects examples search "<query>"
agent-effects inspect project.afx.yaml
agent-effects inspect-svg assets/map.svg
agent-effects frame project.afx.yaml --time 3.2
agent-effects contact-sheet project.afx.yaml
agent-effects preview project.afx.yaml
agent-effects render project.afx.yaml
agent-effects critique project.afx.yamlagent-effects render project.afx.yaml --scene <id>
agent-effects render project.afx.yaml --from <seconds> --to <seconds>
agent-effects render project.afx.yaml --chunk-duration 20
agent-effects render project.afx.yaml --resumeReturns structured errors and warnings.
Displays concise machine- and human-readable documentation for one component, property or preset.
Outputs the complete generated JSON Schema.
Returns a small number of canonical examples relevant to a natural-language query.
Outputs resolved scene, layer, timing and asset metadata.
Outputs targetable SVG IDs, types and bounds.
Renders one frame at an arbitrary timestamp.
Renders representative frames with:
- timestamps
- scene IDs
- beat labels
- issue markers
Renders a lightweight video, for example:
- 854×480
- 15 fps
- H.264
- simplified effects where permitted
Produces the final output.
Combines:
- deterministic validation
- visual diagnostics
- contact-sheet output
- low-resolution preview
- optional video-model critique
- machine-readable recommendations
Agents should not be expected to memorise the DSL.
Every project should include concise instructions:
# Agent Effects project
Edit `project.afx.yaml` and files under `scenes/`.
Prefer semantic compositions and named presets.
Avoid explicit coordinates and keyframes unless required.
Preserve the visual theme and recurring motifs.
After editing:
1. Run `agent-effects validate project.afx.yaml`
2. Render and inspect the contact sheet
3. Render the affected scene or preview
4. Run `agent-effects critique project.afx.yaml`
5. Address high-severity issues
6. Render the final video only after the preview passesAgents must be able to:
- list compositions
- describe a composition
- list presets
- describe a preset
- inspect SVGs
- search examples
- validate projects
- render specific scenes
- inspect warnings
- inspect source paths
The repository should include approximately 25–30 excellent examples.
Suggested examples:
- headline word rise
- line reveal question
- dominant statistic
- price counter
- quote emphasis
- ranked list
- three-step process
- before-and-after comparison
- browser screenshot callout
- browser viewport zoom
- phone-frame animation
- photo card drop
- taped photo collage
- paper-stack transition
- Spain region highlight
- province zoom
- location markers
- route animation
- shared-word transition
- branded end card
- narration-aligned scene
- multi-file long-form project
- resumable chunked render
Examples should be curated, not generated in bulk.
The Rust types are the source of truth.
Documentation and JSON Schema should be generated from the same definitions to prevent drift.
The agent feedback system should have four layers.
Checks whether the project is syntactically and semantically valid.
Used to inspect:
- visual hierarchy
- composition
- scene progression
- text overflow
- settled states
- map framing
- image cropping
- narrative continuity
Used to inspect:
- pacing
- easing
- transitions
- motion continuity
- narration alignment
- visual rhythm
Used to detect:
- overlaps
- bounds violations
- movement discontinuities
- insufficient reading time
- unsettled springs
- timing conflicts
- static intervals
- expensive effects
- scene render cost
Because some coding-agent environments may not pass MP4 files directly to a multimodal model, Agent Effects should provide an optional bridge.
agent-effects critique may:
- render a lightweight MP4
- upload it directly through the Kimi API
- request timestamped structured criticism
- save the result to
.agent-effects/critique.json
Example:
{
"issues": [
{
"start": 16.2,
"end": 17.1,
"severity": "medium",
"scene": "national-map",
"problem": "The zoom reaches Almería before the label appears",
"suggestion": {
"label_start": 16.4,
"zoom_duration": 0.8
}
}
]
}The coding agent can then patch the affected scene.
Video-model feedback is advisory. Deterministic validation remains authoritative for measurable problems.
The MVP should include a local browser application.
The interface should function as an inspection and refinement tool, not an After Effects clone.
┌───────────────┬──────────────────────────────┬───────────────────┐
│ Sections │ │ Inspector │
│ and scenes │ Video preview │ │
│ │ │ Scene │
│ Introduction │ │ Composition │
│ Spain map │ │ Motion preset │
│ Almería │ │ Source DSL path │
├───────────────┴──────────────────────────────┴───────────────────┤
│ Narration | Scene 1 | Scene 2 | Scene 3 | Issues │
└─────────────────────────────────────────────────────────────────┘
- play
- pause
- timeline scrubbing
- frame stepping
- half-speed playback
- safe-area guides
- element bounds
- low-resolution preview
- full-resolution still
- jump to scene
- jump to issue
Display:
- section
- scene name
- associated beat
- start and end
- composition type
- warning count
- cache state
- render status
Display:
- narration duration
- waveform where available
- scene boundaries
- word or phrase markers
- issue markers
- chunk boundaries
- current playhead
Initially read-only:
- scene ID
- beat ID
- composition
- source file
- source DSL path
- resolved bounds
- active animation
- preset
- asset references
- render timing
Display:
- severity
- timestamp
- affected scene
- element
- DSL path
- deterministic or model source
- suggested fix
A later MVP stage may support:
- editing text
- changing a layout
- selecting a preset
- adjusting scene duration
- changing emphasis
- choosing theme tokens
All UI edits must patch the DSL.
The MVP UI does not include:
- freehand drawing
- full drag-and-drop authoring
- complex keyframe timelines
- easing graph editor
- arbitrary path editor
- general vector illustration
- advanced masking
- 3D editing
- hidden database-only projects
Visual dragging may be added later as an escape hatch.
Performance claims must be benchmarked rather than assumed.
- text
- masks
- simple shapes
- limited blur
- no large images
- several high-resolution images
- clipping
- rotation
- shadows
- textured background
- complex SVG
- region highlights
- labels
- pan and zoom
- browser chrome
- large screenshot
- crop and scale
- callouts
- shadow
- mostly unchanged composition
- long narration interval
On a suitable modern desktop:
- generate a representative contact sheet for a 30-second project in under 10 seconds
- render an arbitrary still without browser startup
- begin low-resolution preview generation promptly
- support random-access frame rendering
- avoid evaluating preceding frames for deterministic springs
- render only affected scenes during iteration
- eliminate the approximately 750 ms browser screenshot-extraction step
- avoid intermediate PNG sequences by default
- avoid base64 screenshot transfer
- stream raw frames directly to FFmpeg
- maintain bounded memory
- support interruption and resumption
- begin encoding before the entire project is evaluated
- substantially outperform the Chromium workflow on representative projects
An aspirational target is to render the representative 11-minute editorial video in approximately 5–10 minutes on suitable modern hardware.
This is not an acceptance requirement until validated by benchmarks.
Rendering performance will depend on:
- resolution
- frame rate
- image sizes
- shadows
- blur
- motion blur
- SVG complexity
- CPU
- GPU
- media codecs
- FFmpeg configuration
The renderer should report time spent in:
- project parsing
- scene compilation
- timeline evaluation
- layout
- text shaping
- SVG rendering
- image decoding
- image resizing
- effects
- compositing
- pixel transfer
- FFmpeg encoding
- no frame-ordering errors
- no silent asset failures
- no silent missing fonts
- no silent unsupported fields
- clear version incompatibility errors
- deterministic random effects
- bounded memory
- cancellable renders
- resumable long renders
- atomic chunk output
- corrupted chunks rejected
- reproducible output for the same project and engine version
- validation before final render
The user provides:
- script or narration
- word-level transcript where available
- visual theme
- fonts
- images
- screenshots
- map requirements
- target duration
- target aspect ratio
- Extract the central takeaway.
- Identify the audience.
- Create a beat sheet.
- Select the story arc.
- Group beats into sections and scenes.
- Select recurring motifs.
- Choose semantic compositions.
- Select transitions and motion intentions.
- Generate project and scene files.
- Validate the project.
- Render a contact sheet.
- Inspect and fix layout problems.
- Render affected scenes.
- Render a low-resolution preview.
- Run deterministic critique.
- Optionally run video-model critique.
- Patch the project.
- Render final chunks.
- Assemble and mux the final video.
- Open the local review UI.
- Review the beat and scene structure.
- Watch the preview.
- Inspect issue markers.
- Review map framing, screenshot crops and typography.
- Give natural-language feedback or edit project files.
- Approve the final render.
Percentage of agent-generated videos accepted after no more than two critique-and-revision cycles.
Initial targets:
- at least 70% structurally and visually usable
- at least 40% accepted with only minor human changes
- percentage of beats represented clearly
- number of scenes with layout warnings
- number of unreadable text intervals
- percentage of scenes using semantic compositions
- percentage of animations using named presets
- human rating of visual coherence
- human rating of story clarity
- human rating of pacing
- consistency across videos using the same theme
- first-draft validation-error count
- percentage of errors autonomously resolved
- number of agent retries caused by unclear documentation
- number of low-level escape-hatch uses
- number of project changes per accepted video
- time from script to first preview
- number of critique cycles
- median render time per frame
- render-to-video-duration ratio
- peak memory usage
- contact-sheet generation time
- preview generation time
- final assembly time
- cache hit rate
- number of scenes rerendered after a local change
- successful render-resume rate
The MVP is ready for internal use when all of the following are true.
- A compiled Rust executable can load a versioned
.afx.yamlproject. - Videos do not require recompiling Rust.
- Text, shapes, SVGs and images render correctly.
- Raw frames can be streamed directly to FFmpeg.
- Output frame order is correct.
- Rendering is deterministic.
- Projects can define beats, sections, scenes and states.
- Beats can be associated with scenes.
- Scene relationships can influence transition selection.
- Long projects can be split across scene files.
- Line-, word- and character-level kinetic typography works.
- At least 20 polished presets are available.
- Easing, stagger and deterministic springs work.
- Basic motion paths work.
- Parent-child transforms work.
- Important typography diagnostics are implemented.
- At least 15 semantic compositions are available.
- Browser-frame compositions can display and animate screenshots.
- Photo-card and photo-stack compositions work.
- Screenshot callouts work.
- Drop shadows and textured backgrounds work.
- SVG element IDs can be inspected.
- SVG elements can be targeted from the DSL.
- A built-in Spain map is available.
- At least two provinces can be highlighted independently.
- Markers and labels can be attached.
- The map can animate from country to province framing.
- Narration audio can be muxed.
- Word-level timings can be imported.
- Scene entrances can reference word or phrase markers.
- Reading-time warnings are generated.
- A 15-minute project renders with bounded memory.
- Scenes and time ranges can be rendered independently.
- Long renders can be chunked.
- Interrupted renders can resume.
- Frames are not retained for the full video.
- Static intervals can reuse cached results.
- A scene change invalidates only affected chunks where possible.
- The agent can inspect schema and documentation through the CLI.
- The agent can search canonical examples.
- Validation errors include corrective guidance.
- Contact sheets can be generated.
- Low-resolution previews can be generated.
- Machine-readable critique can be generated.
- The repository includes at least 25 canonical examples.
- The local UI can play a preview.
- It displays sections, scenes and issues.
- It can jump to timestamps.
- It displays source DSL paths.
- The DSL remains the source of truth.
- An AI agent can create a coherent 30-second video without changing Rust.
- An AI agent can create or modify a multi-scene long-form project.
- The representative 11-minute video can be recreated using supported features.
- The native pipeline substantially outperforms the existing Chromium capture workflow.
- Performance results are supported by profiler output rather than assumptions.
- Rust workspace
- versioned schema
- YAML parsing
- project includes
- asset manifest
- CLI skeleton
- structured errors
- Skia surface
- frame timeline
- text rendering
- rectangles and circles
- images
- raw FFmpeg pipeline
- single-frame rendering
- keyframes
- easing
- interpolation
- springs
- staggering
- groups
- parenting
- masks
- basic paths
- text shaping
- wrapping
- fitting
- emphasis spans
- line splitting
- word splitting
- character splitting
- typography presets
- text diagnostics
- cropping
- image transforms
- corner radius
- border
- drop shadow
- texture backgrounds
- deterministic grain
- collage layouts
- headline
- question
- statistic
- quote
- list
- comparison
- before-and-after
- process
- callout
- screenshot
- browser frame
- photo card
- photo stack
- end card
- SVG import
- stable element IDs
- bounds inspection
- path targeting
- Spain map package
- province highlights
- markers
- labels
- map camera
- zoom transitions
- story metadata
- beats
- sections
- scenes
- states
- relationships
- motifs
- semantic motion intentions
- render IR
- modular scene files
- scene-by-scene compilation
- bounded frame buffers
- scene rendering
- range rendering
- chunks
- manifests
- resumability
- selective invalidation
- generated schema
- CLI docs
- example search
- educational errors
- AGENTS.md
- canonical cookbook
- contact sheets
- preview rendering
- telemetry
- issue reports
- profiler
- optional Kimi bridge
- Axum local server
- scene list
- preview playback
- timeline
- issues panel
- read-only inspector
- source-path display
Use the target 11-minute editorial video and require an AI coding agent to:
- create the beat and scene structure
- construct project files
- use browser frames
- use maps
- create collage scenes
- validate
- inspect previews
- revise
- render chunks
- produce the final video
agent-effects/
├── Cargo.toml
├── AGENTS.md
├── README.md
├── crates/
│ ├── agent-effects-core/
│ ├── agent-effects-schema/
│ ├── agent-effects-story/
│ ├── agent-effects-layout/
│ ├── agent-effects-text/
│ ├── agent-effects-svg/
│ ├── agent-effects-maps/
│ ├── agent-effects-render/
│ ├── agent-effects-ffmpeg/
│ ├── agent-effects-diagnostics/
│ ├── agent-effects-cli/
│ └── agent-effects-server/
├── ui/
├── presets/
├── examples/
├── maps/
│ └── spain/
├── schemas/
├── tests/
├── benchmarks/
└── docs/
- automatic connectors
- flow diagrams
- browser-page scrolling
- cursor paths
- click indicators
- product-tour templates
- persistent shared elements
- richer screenshot annotations
- animated number
- progress bar
- progress ring
- bar chart
- line chart
- simple donut chart
- labels
- annotations
- restrained axes
- European countries
- world map
- GeoJSON ingestion
- more projections
- route arcs
- richer label placement
- additional country packages
- path morphing
- particles
- richer 2.5D camera
- advanced compositing
- custom GPU effects
- Rive playback
- Lottie playback
- drag with snapping
- resizing
- anchor controls
- basic motion-path editing
- basic keyframe editing
- before-and-after comparison
- semantic prompt-based scene revision
Risk: Agents frequently resort to coordinates and explicit keyframes.
Mitigation: Track escape-hatch usage and add semantic components for repeated patterns.
Risk: Attempting to reproduce all motion-graphics functionality delays the useful product.
Mitigation: Maintain the editorial explainer focus and explicit exclusions.
Risk: Videos feel templated.
Mitigation: Support controlled variation in layout, direction, duration, emphasis, composition and theme.
Risk: Schema compliance does not ensure visual quality.
Mitigation: Use strong defaults, curated examples, contact sheets, diagnostics and video critique.
Risk: Attractive scenes fail to form a coherent argument.
Mitigation: Require a takeaway and beat sheet before scene generation.
Risk: Missing or differing fonts change layout.
Mitigation: Require explicit approved fonts, verify availability and record font hashes in render manifests.
47.7 Maps become a hidden large project
Risk: Geographic support expands into a full GIS system.
Mitigation: Limit MVP support to built-in Spain maps and targetable SVGs.
Risk: Editorial effects undermine native-rendering gains.
Mitigation: Cache static effects, provide quality tiers and profile each effect independently.
Risk: Cross-scene transitions break during incremental rendering.
Mitigation: Restrict early chunks to safe boundaries and support overlap handles later.
Risk: Text shaping, image processing, shadows or FFmpeg remain expensive.
Mitigation: Benchmark representative scenes, optimise measured bottlenecks and avoid unverified performance claims.
Risk: Building a visual editor delays the core engine.
Mitigation: Begin with CLI, then read-only UI, then semantic editing.
The MVP will:
- be called Agent Effects
- use the repository name
agent-effects - use
.afx.yamlproject files - compile Rust once rather than per video
- use native rendering rather than Chromium
- begin with Skia rather than raw
wgpu - pipe raw frames directly to FFmpeg
- support short and long-form videos
- use semantic story beats and compositions
- focus on kinetic typography and editorial graphics
- include images, collage layouts and screenshot callouts
- include browser and constrained device frames
- include targetable SVG maps
- include built-in Spain province maps
- include textured backgrounds and drop shadows
- support modular scene files
- use bounded scene-by-scene rendering
- support resumable chunks
- include a CLI as the primary agent interface
- include a lightweight local review UI
- use contact sheets and previews for feedback
- optionally integrate Kimi as a video critic
- preserve project files as the source of truth
- expose low-level keyframes only as an escape hatch
- benchmark performance rather than promising assumed native speeds
Agent Effects is:
- a compiled Rust motion-graphics renderer
- an agent-facing declarative language
- a story and scene compiler
- a curated library of semantic visual compositions
- a preset motion system
- a memory-safe long-form rendering pipeline
- a validation and critique system
- a lightweight review interface
Its first repeatable outcome is:
An AI agent can reliably construct, inspect, revise and render a polished short or long-form editorial motion-graphics video using story beats, semantic scenes, targetable media and tested animation presets.
The central product principle is:
The agent decides what the story means, which ideas matter and how they relate. Agent Effects turns that intent into coherent, tasteful and efficiently rendered motion graphics.