Agent Cutroom

Agent Cutroom gives ChatGPT Codex and other coding agents a local toolbelt for video review and editing.

The agent makes the editorial decisions. The CLI handles deterministic media work: probing videos, generating/importing timestamped transcripts through transcribe-audio, detecting silence, extracting frames/contact sheets, writing review packs, recording agent observations, finding candidate moments, creating edit plans, rendering rough cuts, burning word-timed captions, verifying renders, packaging social outputs, exporting OTIO, and preparing HyperFrames briefs.

Why This Exists

Most video tools hide the timeline inside a UI. Agent Cutroom exposes the timeline as files an agent can inspect and edit:

raw video + transcript
  -> timeline.json
  -> transcript/raw txt+json
  -> frames/contact-sheets
  -> review/review-pack.md
  -> agent observations
  -> analysis/highlight-candidates.json
  -> analysis/story-candidates.json
  -> analysis/clip-slate.json
  -> review/content-inventory.md
  -> review/clip-candidate-evidence.md
  -> agent-authored review/clip-slate.md
  -> approved clip ids
  -> edit-plan.json
  -> plans/clips/<candidate-id>/edit-plan.json
  -> renders/rough-cut.mp4
  -> plans/short-form-pacing.json
  -> plans/color-grade.json
  -> captions/captions.ass
  -> renders/captioned.mp4
  -> renders/verify-report.json
  -> plans/platform-export.json
  -> renders/platform-<platform>.mp4
  -> plans/social-package.json
  -> exports/edit.otio
  -> HyperFrames polish pass

This is designed for workflows where Codex is the editor/producer and needs tools to see, reason about, and render video.

Requirements

• Bun 1.3+
• FFmpeg and ffprobe on PATH
• Optional for generated transcripts: transcribe-audio on PATH

Check your machine:

bun install
bun run build
bun dist/cli/index.js doctor

Quick Start

bun install
bun run build

bun dist/cli/index.js init ./input.mp4 --transcript ./transcript.json --out ./my-cut --title "My Cut"
bun dist/cli/index.js prepare ./my-cut
open ./my-cut/review/review-pack.md

Or generate the transcript locally with the transcription-to-vault workbench:

bun dist/cli/index.js init ./input.mp4 --out ./my-cut --title "My Cut"
bun dist/cli/index.js transcribe ./my-cut \
  --prompt "Names: product names, people, places"
bun dist/cli/index.js prepare ./my-cut

After inspecting the frames/contact sheet, record observations:

bun dist/cli/index.js observe ./my-cut \
  --window window-001 \
  --summary "Talking head over a dashboard; the screen supports the spoken point." \
  --visible-text "Dashboard, Export, Revenue" \
  --editing-use keep \
  --broll none \
  --note "Good opening section."

For source-backed talking-head packaging, build a content package from a recipe and profile:

bun dist/cli/index.js content-package ./my-cut \
  --recipe talking-head-story \
  --profile hanif \
  --target-seconds 75

This writes review/content-inventory.md, analysis/story-candidates.json, analysis/clip-slate.json, review/clip-candidate-evidence.md, and analysis/story-selection.md. The recipe is generic; the hanif profile contains transcript cleanup and heuristic scoring defaults for Hanif's videos. It does not write titles, hooks, points, recommendations, or post copy.

The running agent must read the evidence, author review/clip-slate.md, and show that slate before rendering:

open ./my-cut/review/clip-candidate-evidence.md
# agent writes ./my-cut/review/clip-slate.md
bun dist/cli/index.js content-package ./my-cut \
  --recipe talking-head-story \
  --profile hanif \
  --approve story-000055000-000095000,story-000120000-000170000

Approved clips get their own edit plans under plans/clips/<candidate-id>/edit-plan.json. Render a specific approved clip with --source-plan.

Create a short-form-paced rough cut:

bun dist/cli/index.js shortform-pacing ./my-cut \
  --source-plan plans/clips/story-000055000-000095000/edit-plan.json \
  --out-plan plans/clips/story-000055000-000095000/paced-edit-plan.json
bun dist/cli/index.js render ./my-cut \
  --source-plan plans/clips/story-000055000-000095000/paced-edit-plan.json \
  --out renders/story-000055000-000095000/rough-cut.mp4

Before final grade, captions, or release, use the cutroom-cut-review skill to audit the proposed cut boundaries and patch the approved clip edit plan when a deterministic cut clips a word, rushes a thought, or breaks a rhetorical chain.

Preview and apply a highlight-protected subject shadow lift when the speaker is underexposed:

bun dist/cli/index.js grade-preview ./my-cut --target renders/rough-cut.mp4
bun dist/cli/index.js grade-apply ./my-cut --target renders/rough-cut.mp4 --out renders/graded.mp4

Find moments, caption, verify, and package:

bun dist/cli/index.js find-moments ./my-cut \
  --objective "Find one complete Instagram-ready moment" \
  --target-seconds 30
bun dist/cli/index.js caption ./my-cut --target renders/graded.mp4 --out renders/captioned.mp4
bun dist/cli/index.js verify ./my-cut --target renders/captioned.mp4
bun dist/cli/index.js social-package ./my-cut --platform instagram
# or explicitly:
bun dist/cli/index.js platform-export ./my-cut --platform instagram --target renders/captioned.mp4
bun dist/cli/index.js export-otio ./my-cut

Generate a HyperFrames brief for the polish pass:

bun dist/cli/index.js hyperframes-brief ./my-cut

Commands

agent-cutroom doctor
agent-cutroom init <video> --out <dir> [--transcript <path>] [--title <title>] [--link-source]
agent-cutroom probe <project>
agent-cutroom transcribe <project> [--backend mlx-whisper] [--model large-v3] [--vault-note <path>]
agent-cutroom transcript <project>
agent-cutroom silence <project>
agent-cutroom frames <project>
agent-cutroom review-pack <project>
agent-cutroom prepare <project>
agent-cutroom observe <project> --window <id> --summary <text>
agent-cutroom plan <project>
agent-cutroom render <project>
agent-cutroom find-moments <project>
agent-cutroom content-package <project> [--recipe talking-head-story] [--profile hanif] [--approve <ids>]
agent-cutroom shortform-pacing <project>
agent-cutroom render <project> [--source-plan <path>]
agent-cutroom grade-preview <project>
agent-cutroom grade-apply <project>
agent-cutroom caption <project>
agent-cutroom verify <project>
agent-cutroom platform-export <project>
agent-cutroom social-package <project>
agent-cutroom export-otio <project>
agent-cutroom hyperframes-brief <project>

Agent Skill

This repo includes a starter Codex skill at skills/agent-cutroom/SKILL.md. Copy it into your skills directory and set CUTROOM_DIR to this checkout:

mkdir -p ~/.codex/skills/agent-cutroom
cp skills/agent-cutroom/SKILL.md ~/.codex/skills/agent-cutroom/SKILL.md
export CUTROOM_DIR="$PWD"

The skill keeps the workflow predictable: initialize a project, generate or import transcript metadata, prepare frames/contact sheets, record agent observations, plan and render rough cuts, then hand off to HyperFrames or caption rendering only after the rough cut exists.

Transcript Format

Agent Cutroom imports timestamped JSON transcripts. It supports Whisper-like segments:

{
  "segments": [
    { "start": 0.0, "end": 2.4, "text": "Here is the opening idea." }
  ],
  "text": "Here is the opening idea."
}

It also supports millisecond fields:

{
  "segments": [
    { "startMs": 0, "endMs": 2400, "text": "Here is the opening idea." }
  ]
}

Plain text transcripts are stored as untimed text with a warning. The CLI does not invent timestamps.

Active-word captions require real word timings:

{
  "segments": [
    {
      "start": 0.0,
      "end": 1.0,
      "text": "Hello world.",
      "words": [
        { "word": "Hello", "start": 0.0, "end": 0.4 },
        { "word": "world.", "start": 0.4, "end": 1.0 }
      ]
    }
  ]
}

agent-cutroom transcribe wraps the separate transcribe-audio workbench. It extracts project audio to audio/source.wav, runs local transcription, keeps raw txt and json artifacts in transcript/, imports timestamped segments into timeline.json, records transcript provenance, and can optionally write an Obsidian/vault note through transcribe-audio note.

How Vision Works

There is no separate OCR service or hidden vision model.

The frames command extracts JPEG frames and a contact sheet. The running agent inspects those images directly, then writes observations through agent-cutroom observe. Those observations become timeline metadata used by find-moments, plan, render, social-package, and hyperframes-brief.

HyperFrames Fit

Agent Cutroom owns the raw footage workflow: timeline metadata, transcript, frames, silence, agent observations, rough cuts.

HyperFrames fits after the rough cut, when the agent wants a polished composition with captions, title cards, lower thirds, callouts, pull quotes, B-roll layouts, or social-ready packaging. agent-cutroom hyperframes-brief exports the timeline context needed for that pass.

Artifact Contract

See docs/artifact-contract.md for the project file contract, including transcript words, highlight candidates, story candidates, caption plans, social packages, verification reports, and OTIO exports.

Content Packages

agent-cutroom content-package turns prepared transcript/windows/frame evidence into a content inventory, ranked source-backed candidates, and approval state. It does not create audience-facing titles, hooks, points, recommendations, or social copy.

agent-cutroom content-package <project> \
  --recipe talking-head-story \
  --profile hanif

The built-in talking-head-story recipe owns deterministic candidate generation, evidence writing, and edit-plan writing. The hanif content profile owns transcript corrections and heuristic scoring signals. See docs/content-package.md for profile shape, workflow, and quality gates.

The approval loop is explicit: the agent reads review/clip-candidate-evidence.md, authors review/clip-slate.md, shows it to the operator, gets approved candidate IDs, then renders each approved clip from its own plan in plans/clips/<candidate-id>/.

Short-Form Polish

agent-cutroom shortform-pacing tightens the approved edit plan using transcript word timings, writing plans/short-form-pacing.json and an output edit plan. It preserves short rhetorical question-chain pauses and skips cuts too small to justify a jump. Treat that as a proposed edit, not final editorial judgment: run cutroom-cut-review before final user-facing output. agent-cutroom grade-preview and agent-cutroom grade-apply use a feathered FFmpeg subject mask plus a luma shadow mask to lift the speaker while protecting bright wall pixels. See docs/short-form-polish.md.

MCP Server

Agent Cutroom includes a local stdio MCP server that wraps the CLI and exposes project artifacts as resources:

bun run build
agent-cutroom-mcp

See docs/mcp.md for tools, resources, prompts, and verification notes.

AI Video Workflow Research

See docs/ai-video-workflow-research.md for research on current AI video workflows, transcript-first editing, creator style guides, open-source building blocks, agent skills, MCP surfaces, and the concrete workflow additions worth borrowing for Agent Cutroom.

Synthetic Smoke Fixtures

The repo includes a reproducible sample-media script:

examples/synthetic-fixtures/scripts/make-sample-media.sh

Generated media and project outputs are ignored by git.

See docs/synthetic-fixture-report.md for the synthetic fixture smoke run.

Real Talking-Head Dogfood

Real dogfood uses public real-person talking videos:

examples/real-talk/scripts/download-real-talking-videos.sh

The downloaded source media and generated project outputs are ignored by git. See docs/dogfood-report.md for the real-video dogfood run and verification results.

This repo also includes a checked-in HyperFrames dogfood project at videos/sister-smollett-instagram that turns the Agent Cutroom rough cut into a 1080x1920 Instagram-ready package.

Development

bun install
bun run check
bun run build

Status

This is an early public prototype. The stable contract is the file-based workflow: cutroom.json, timeline.json, review/review-pack.md, review/content-inventory.md, review/clip-candidate-evidence.md, agent-authored review/clip-slate.md, analysis/highlight-candidates.json, analysis/story-candidates.json, analysis/clip-slate.json, analysis/story-selection.md, edit-plan.json, plans/clips/<candidate-id>/edit-plan.json, plans/caption-plan.json, plans/platform-export.json, plans/social-package.json, renders/verify-report.json, exports/edit.otio, and rendered outputs.