feat(examples): ingest end-to-end — a synthetic clip to an approved, partitioned batch (#23) - #94
Merged
Merged
Conversation
…partitioned batch (#23) M2's exit criterion made executable, the way examples/sdk_end_to_end.py served M1's. A ten-second testsrc clip and a folder of stills go in; fifty hash-deduplicated assets in an approved batch of two jobs come out, with a pollable progress row and a per-file report of what could not be read. Nothing is annotated and nothing is released — the SDK example already covers that half of the cycle, and this one spends its length on where assets come from. Four properties demonstrated rather than described: - A clip cannot state its total and a directory can. processed climbs to 50 while total stays NULL, because VideoMetadata carries no frame count by design; the image directory states 4 before reading its first file. - One file registered at two rates is two sources whose frames are one set. Decomposition parameters live on the source, so 5 fps and 1 fps over the same path are two origins — and the coarse run still creates nothing, because identity is content and round=up lands both grids on whole seconds. That alignment is a property of this extractor, not a promise the port makes. - A file that is not an image is reported, not skipped. notes.txt yields one IngestFailure and the run still ends completed. - Re-ingesting a source creates nothing: created=0, deduplicated=50, into a new batch because the first froze at approval. The clip is 160x120 rather than the fixtures' 64x48, and that is load-bearing. testsrc moves a little between frames; below roughly 96x72 that movement falls under what the scaler and encoder still resolve, consecutive frames come out byte-identical, and content addressing collapses them — a ten-second clip at 5 fps then yields forty assets, the feature working and reading as a shortfall. Two deliberate divergences from the SDK example, both stated in the module docstring: the clip is generated by shelling out to ffmpeg, because a video is a container wrapped around a codec and the only honest way to write one is the tool that reads it; and the stills are Pillow's work rather than a second copy of M1's hand-rolled PNG encoder, Pillow having been a dependency since #16. The generation command is duplicated from tests/fixtures/media.py rather than imported — that module is a test fixture, it imports pytest, and its answer to a missing binary is pytest.skip, which means nothing in a script. The example checks shutil.which("ffmpeg") before writing anything and exits with an install hint instead. CI runs it twice, as the SDK example is: the smoke test calls main() and asserts on the returned Summary, and a new "Ingest end-to-end example" step runs the file as a plain script, which is the only thing that proves it works from a clean checkout. The smoke test gates on tests/fixtures/media.require_ffmpeg() — a skip locally, an error under VISIONSET_REQUIRE_FFMPEG=1. No new service, domain model, error, event or migration. FORMAT_VERSION stays 10, VERSION stays 0.0.1.dev0, openapi.json unchanged. 885 tests, up from 877.
Closed
2 tasks
JArmandoAnaya
added a commit
that referenced
this pull request
Aug 21, 2026
…partitioned batch (#23) (#94) M2's exit criterion made executable, the way examples/sdk_end_to_end.py served M1's. A ten-second testsrc clip and a folder of stills go in; fifty hash-deduplicated assets in an approved batch of two jobs come out, with a pollable progress row and a per-file report of what could not be read. Nothing is annotated and nothing is released — the SDK example already covers that half of the cycle, and this one spends its length on where assets come from. Four properties demonstrated rather than described: - A clip cannot state its total and a directory can. processed climbs to 50 while total stays NULL, because VideoMetadata carries no frame count by design; the image directory states 4 before reading its first file. - One file registered at two rates is two sources whose frames are one set. Decomposition parameters live on the source, so 5 fps and 1 fps over the same path are two origins — and the coarse run still creates nothing, because identity is content and round=up lands both grids on whole seconds. That alignment is a property of this extractor, not a promise the port makes. - A file that is not an image is reported, not skipped. notes.txt yields one IngestFailure and the run still ends completed. - Re-ingesting a source creates nothing: created=0, deduplicated=50, into a new batch because the first froze at approval. The clip is 160x120 rather than the fixtures' 64x48, and that is load-bearing. testsrc moves a little between frames; below roughly 96x72 that movement falls under what the scaler and encoder still resolve, consecutive frames come out byte-identical, and content addressing collapses them — a ten-second clip at 5 fps then yields forty assets, the feature working and reading as a shortfall. Two deliberate divergences from the SDK example, both stated in the module docstring: the clip is generated by shelling out to ffmpeg, because a video is a container wrapped around a codec and the only honest way to write one is the tool that reads it; and the stills are Pillow's work rather than a second copy of M1's hand-rolled PNG encoder, Pillow having been a dependency since #16. The generation command is duplicated from tests/fixtures/media.py rather than imported — that module is a test fixture, it imports pytest, and its answer to a missing binary is pytest.skip, which means nothing in a script. The example checks shutil.which("ffmpeg") before writing anything and exits with an install hint instead. CI runs it twice, as the SDK example is: the smoke test calls main() and asserts on the returned Summary, and a new "Ingest end-to-end example" step runs the file as a plain script, which is the only thing that proves it works from a clean checkout. The smoke test gates on tests/fixtures/media.require_ffmpeg() — a skip locally, an error under VISIONSET_REQUIRE_FFMPEG=1. No new service, domain model, error, event or migration. FORMAT_VERSION stays 10, VERSION stays 0.0.1.dev0, openapi.json unchanged. 885 tests, up from 877.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #23.
M2's exit criterion made executable, the way
examples/sdk_end_to_end.pyserved M1's. Aten-second
testsrcclip and a folder of stills go in; fifty hash-deduplicated assets in anapproved batch of two jobs come out, with a pollable progress row and a per-file report of what
could not be read. Nothing is annotated and nothing is released — the SDK example already covers
that half of the cycle, and this one spends its length on where assets come from.
What it drives
register_video(..., extraction_fps=5.0)→ingestIngestService.get(job_id)state=completed,processed=50,total=Noneapprove(BySize(size=25))created=0,deduplicated=50, into a new batchregister_video(..., extraction_fps=1.0)created=0notes.txttotal=4,processed=4,created=3, oneIngestFailurethumbnail_hashFour properties demonstrated rather than described
A clip cannot state its total, and a directory can.
processedclimbs to 50 whiletotalstays NULL —
VideoMetadatacarries no frame count by design, because it would be a guess for aVFR clip and the number an ingest wants is what extraction produced. The directory can be
listed, so it states
4before reading its first file.One file registered at two rates is two sources whose frames are one set. Decomposition
parameters live on the source, so
5.0and1.0over the same path are two origins. And yetthe coarse run creates nothing: identity is content, and
round=uplands both grids on wholeseconds, so its ten frames are byte-for-byte frames the finer run already stored. Verified before
writing the assertion — all 10 of 10 coarse frames hash-match, at matching timestamps. The
example and the test both say this is a property of this extractor, not a promise the port
makes about every rate pair.
A file that is not an image is reported, not skipped. One
IngestFailure(
UNSUPPORTED, reason not repeating the name so a surface can group by kind) and the run stillends
completed.Re-ingesting a source creates nothing. Into a new batch, because the first froze at
approval — a batch is an ephemeral unit of work and two of them may name the same assets.
One finding worth recording
The clip is 160×120, not the fixtures' 64×48, and that is load-bearing.
testsrcmoves alittle between frames; below roughly 96×72 that movement falls under what the scaler and encoder
still resolve, and consecutive frames come out byte-identical. Content addressing then does
exactly what it promises and collapses them — the first working version of this example reported
40 assets from 50 extraction slots, the feature working and reading as a shortfall. Measured
across six sizes: 64×48 gives 40/50 distinct, everything from 96×72 up gives 50/50. The constant
carries the explanation.
Two deliberate divergences from the SDK example
Both stated in the module docstring:
codec, and the only honest way to write one is the tool that reads it. The command is
tests/fixtures/media.write_video's, duplicated rather than imported — that module is atest fixture, it imports pytest, and its answer to a missing binary is
pytest.skip, whichmeans nothing in a script. The example checks
shutil.which("ffmpeg")before writing anythingand exits with an install hint instead.
has been a dependency since kernel: MediaProcessor adapter — Pillow for image validation/dimensions/thumbnails #16. This also retires the now-false "M1 has no image library to
lean on (Pillow arrives with kernel: MediaProcessor adapter — Pillow for image validation/dimensions/thumbnails #16)" lines in
docs/examples.mdandexamples/README.md.CI
Runs twice, as the SDK example does: the smoke test calls
main()and asserts on the returnedSummary, and a newIngest end-to-end examplestep runs the file as a plain script — the onlything that proves it works from a clean checkout. The
pythonjob already installs ffmpeg andsets
VISIONSET_REQUIRE_FFMPEG=1, so the smoke test'srequire_ffmpeg()gate is a skip locallyand an error on the runner.
Scope
No new service, domain model, error, event or migration.
FORMAT_VERSIONstays 10,VERSIONstays
0.0.1.dev0,openapi.jsonunchanged (drift gate run locally, clean). Docs updated:docs/examples.md(restructured for two examples),docs/README.md,docs/ingest.md,examples/README.md, rootREADME.md.Checks