Skip to content

feat(kernel): VideoProcessor — ffmpeg probing, oriented dimensions, deterministic frame extraction (#17) - #89

Merged
JArmandoAnaya merged 1 commit into
mainfrom
feat/video-processor
Jul 27, 2026
Merged

feat(kernel): VideoProcessor — ffmpeg probing, oriented dimensions, deterministic frame extraction (#17)#89
JArmandoAnaya merged 1 commit into
mainfrom
feat/video-processor

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Closes #17.

Declares ports/video_processor.py beside #16's image_processor.py — one file per port,
each declared by the task that implements it — and ships FfmpegVideoProcessor behind it.
workspace.video_processor is the fifth port on the composition point.

info = workspace.video_processor.probe(clip)          # VideoMetadata(fps=29.97, ...)
for frame in workspace.video_processor.frames(clip, fps=1):
    ...                                               # frame.index, frame.timestamp, frame.content

Acceptance criteria

  • Synthetic clip → expected frame count at 1/5/10 fps. The generated 10 fps / 2 s clip
    gives exactly 2 / 10 / 20 frames. That parametrised test is also the canary for the whole
    extraction command — almost any change to it moves one of those numbers.
  • Frame origin metadata attached to each produced frame. VideoFrame(index, timestamp, content), not inferred downstream.
  • Determinism. Two extractions of one clip are byte-identical, and two identical clips
    extract to identical SHA-256s. Repeatability, never a hardcoded hash.
  • Missing ffmpeg detected up front, with an install hint.

Design decisions, and why

Path, not BinaryIO. The one place this port diverges from ImageProcessor, deliberately.
A video decoder is an out-of-process program that seeks: handed a pipe it cannot say how long a
clip is without decoding all of it, and cannot revisit a byte. Nothing is lost — a source is a
path and a blob in the default store is a path too.

codec is a str; there is no VideoFormat enum. An image is an asset, a video is a
source. Curating ImageFormat buys something because those exact bytes enter the dataset; a
video's never do — they leave as PNG frames — so a closed codec list would gate nothing while
going stale. Same split as DatasetChange.operation vs DatasetOperation.

Rotation is applied, not reported. The EXIF rule, on a different mechanism: a phone held
upright writes a landscape stream plus a display matrix, and ffmpeg applies it when decoding. So
VideoMetadata reports the swapped edges and the frames come out at exactly those dimensions.
Reporting the stored numbers would put every frame at odds with the metadata stamped beside it.

The two media errors are reused, not extended. A container ffmpeg never opens is
UnsupportedMedia — including a clip whose index went missing with its tail, which is why
probing before extracting matters; a clip that opens, yields frames and then runs out is
CorruptMedia, and it hands back what decoded before it raises. MediaToolUnavailable is the
one new error and is not a MediaError: no file is at fault, so an ingest catching the
family per item must not record "ffmpeg missing" against five thousand innocent files. It is
#19's fatal cause, next to the per-file report.

frames() is not a generator — it validates, checks for ffmpeg and returns an inner one — so
a missing binary or a negative fps is reported at the call rather than at the first iteration
inside whatever loop consumed it. What comes back owns a live decoder until exhausted or closed;
an abandoned iterator terminates it, and a test asserts the process is reaped.

Three things ffmpeg taught us, pinned in the adapter

  • round=up on the fps filter. Under the default near, extracting a 10 fps clip at 1 fps
    yields the pictures from 0.4 s and 1.4 s while labelling them 0.0 s and 1.0 s. up makes the
    frame at the grid point the winner, which is what makes timestamp = index / fps honest.
    Frame counts are identical either way.
  • -xerror is load-bearing. Without it a truncated clip exits zero with the frames it
    managed, so a damaged file would ingest as a merely short one and nothing would say so. Its
    opposite number -err_detect explode is deliberately not used: it rejects intact files.
  • There is no seek. Input seeking lands on a keyframe and is approximate; output seeking
    interacts with the filter. Extraction reads from the start every time.

One consequence #19 and #20 should know about

Frames are content-addressed, and the encoder is pinned, so extraction is deterministic — within
one installed ffmpeg, not across builds.
That propagates: video-derived asset identity is
reproducible within an ffmpeg build, not across one.
Re-ingesting a clip after an ffmpeg upgrade
yields new hashes and therefore new assets. Images do not have this property; the asset is
something we were given. Video does; the asset is something we computed.

Fixtures

tests/fixtures/media.py gains write_corrupt_video and write_rotated_video, and write_video
now asks for -movflags +faststart — that index at the front is exactly what lets a truncated
clip stay partially readable, which is the only way to reach the CorruptMedia branch.
write_rotated_video uses -display_rotation; the older -metadata:s:v rotate= spelling is
dropped silently by recent ffmpeg, which would have produced a fixture that tests nothing and
fails nowhere. Both are covered in tests/fixtures/test_media.py.

No migration: FORMAT_VERSION stays 6, VERSION stays 0.0.1.dev0, no Asset field, no
openapi.json change. Pillow is a dependency and ffmpeg is a binary, so pyproject.toml is
untouched and no import contract moves.

Checks

uv run pytest                      737 passed
VISIONSET_REQUIRE_FFMPEG=1 …       87 passed (the CI tripwire path)
uv run lint-imports                2 kept, 0 broken
uv run mypy src/visionset/kernel   no issues in 48 source files
uv run mypy src/visionset          no issues in 58 source files
uv run ruff check . / format       all checks passed, 107 files formatted
uv run python examples/…           released v1.0, republished byte-identical
uv run python scripts/export_openapi.py   no drift

…eterministic frame extraction (#17)

Declares `ports/video_processor.py` beside #16's `image_processor.py` — one file
per port, each declared by the task that implements it — and ships
`FfmpegVideoProcessor` behind it. `probe` reports as-displayed dimensions, source
rate, duration and codec; `frames` streams PNG frames carrying `(index,
timestamp)` off a running ffmpeg, one at a time.

The two media errors are reused rather than extended: a container ffmpeg never
opens is `UnsupportedMedia`, a clip that decodes for a while and then runs out is
`CorruptMedia`. A missing binary is `MediaToolUnavailable`, deliberately outside
that family — no file is at fault, so an ingest must not record it against five
thousand innocent ones.

`workspace.video_processor` is the fifth port, composed the EventBus way. No
migration: FORMAT_VERSION stays 6, VERSION stays 0.0.1.dev0, no Asset field.
@JArmandoAnaya
JArmandoAnaya merged commit 326812b into main Jul 27, 2026
3 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/video-processor branch July 27, 2026 10:21
JArmandoAnaya added a commit that referenced this pull request Aug 21, 2026
…eterministic frame extraction (#17) (#89)

Declares `ports/video_processor.py` beside #16's `image_processor.py` — one file
per port, each declared by the task that implements it — and ships
`FfmpegVideoProcessor` behind it. `probe` reports as-displayed dimensions, source
rate, duration and codec; `frames` streams PNG frames carrying `(index,
timestamp)` off a running ffmpeg, one at a time.

The two media errors are reused rather than extended: a container ffmpeg never
opens is `UnsupportedMedia`, a clip that decodes for a while and then runs out is
`CorruptMedia`. A missing binary is `MediaToolUnavailable`, deliberately outside
that family — no file is at fault, so an ingest must not record it against five
thousand innocent ones.

`workspace.video_processor` is the fifth port, composed the EventBus way. No
migration: FORMAT_VERSION stays 6, VERSION stays 0.0.1.dev0, no Asset field.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernel: MediaProcessor adapter — ffmpeg for video-to-frames decomposition at a target fps

1 participant