feat(examples): ingest end-to-end — a synthetic clip to an approved, partitioned batch (#23)

.github/workflows/ci.yml

@@ -50,6 +50,11 @@ jobs:
       - name: SDK end-to-end example
         run: uv run python examples/sdk_end_to_end.py
 
+      # The ingest half of the same argument, and the one that needs the ffmpeg
+      # installed above: this example generates its own clip.
+      - name: Ingest end-to-end example
+        run: uv run python examples/ingest_end_to_end.py
+
   frontend:
     runs-on: ubuntu-latest
     steps:

README.md

@@ -23,6 +23,11 @@ empty directory to a hash-verified release in one pass, generating its own image
 no CLI, nothing to download. Run it with `uv run python examples/sdk_end_to_end.py`; the
 walkthrough is in [docs/examples.md](docs/examples.md).
 
+For where the assets themselves come from,
+[`examples/ingest_end_to_end.py`](examples/ingest_end_to_end.py) turns a generated ten-second clip
+into 50 deduplicated assets in an approved batch, then shows a re-run creating nothing. It needs
+ffmpeg.
+
 ## Monorepo map
 
 ```

docs/README.md

@@ -19,4 +19,4 @@ contracts (kernel purity, headless annotator) are described there and enforced i
 | [releases.md](releases.md) | The immutable artifact: what a manifest is and is not, why two publishes agree byte for byte, hash verification, and the seeded split recipe |
 | [events.md](events.md) | Domain events: subscribing by type, why emission follows the commit, at-most-once delivery, and what an isolated subscriber failure does |
 | [persistence.md](persistence.md) | The metadata store: repositories, unit of work, table layout, migrations and `format_version` |
-| [examples.md](examples.md) | The runnable end-to-end example: the whole cycle in one pass, and what it is built to demonstrate |
+| [examples.md](examples.md) | The two runnable examples: the whole cycle in one pass, ingest on its own, and what each is built to demonstrate |

docs/examples.md

@@ -1,17 +1,31 @@
 # Examples
 
-Every other document here explains one thing the kernel does. This one is about the single
-runnable file that does all of them at once: [`examples/sdk_end_to_end.py`](../examples/sdk_end_to_end.py)
-takes an empty directory and leaves behind a release whose every byte can be re-hashed and
-checked — with nothing but `import visionset`.
+Every other document here explains one thing the kernel does. This one is about the runnable
+files that do several of them at once. There are two, one per milestone:
+
+| Example | What it drives | Milestone |
+| --- | --- | --- |
+| [`sdk_end_to_end.py`](../examples/sdk_end_to_end.py) | an empty directory to a release whose every byte can be re-hashed and checked | M1 |
+| [`ingest_end_to_end.py`](../examples/ingest_end_to_end.py) | a ten-second clip and a folder of stills to an approved, partitioned batch | M2 |
 
 ```bash
 uv run python examples/sdk_end_to_end.py
+uv run python examples/ingest_end_to_end.py     # needs ffmpeg
 ```
 
-It is the milestone's exit criterion made executable, and it runs in CI twice: once as a
-[pytest smoke test](../tests/examples/test_sdk_end_to_end.py) that asserts on outcomes, and once
-as a plain script, which is the only way to prove it still works from a clean checkout.
+Each is its milestone's exit criterion made executable, and each runs in CI **twice**: once as a
+pytest smoke test ([M1](../tests/examples/test_sdk_end_to_end.py),
+[M2](../tests/examples/test_ingest_end_to_end.py)) that asserts on outcomes, and once as a plain
+script, which is the only way to prove it still works from a clean checkout.
+
+The two overlap by design and neither subsumes the other. The SDK example walks the whole cycle
+and treats ingest as one stage of thirteen; the ingest example stops at an approved batch and
+spends its length on where assets come from — two sources over one file, dedup, progress and the
+per-file report.
+
+---
+
+# The SDK example
 
 ## The cycle
 
@@ -70,7 +84,8 @@ the ingest is what makes the batch now.
 The frames are still generated by the example's own six-line PNG encoder rather than by Pillow,
 which is a dependency these days. Their bytes are fixed, and that is the point: an asset's
 identity is the SHA-256 of its content and the split keys on content hash, so the same pictures
-land in the same folds on every machine.
+land in the same folds on every machine. (The [ingest example](#the-ingest-example) does reach
+for Pillow — it is not carrying a determinism argument about folds.)
 
 ## Why three classes for "two classes"
 
@@ -83,7 +98,79 @@ attribute is *required* (`occlusion` on `stop-sign`), which is what makes
 ## No committed media, ever
 
 The frames are built by a short PNG encoder using only `zlib` and `struct`: a signature, an IHDR
-chunk, zlib-compressed filter-0 scanlines in IDAT, and IEND. M1 has no image library to lean on
-— Pillow arrives with the media processor (#16) — and v1 of this product shipped 929 MB of
-images into git history, which is why `**/workspace-data/` is ignored and why an example that
-needs pictures makes its own.
+chunk, zlib-compressed filter-0 scanlines in IDAT, and IEND. It was written when M1 had no image
+library to lean on, and it stays for the reason in the section above rather than out of nostalgia.
+v1 of this product shipped 929 MB of images into git history, which is why `**/workspace-data/`
+is ignored and why an example that needs pictures makes its own.
+
+---
+
+# The ingest example
+
+Where [`sdk_end_to_end.py`](../examples/sdk_end_to_end.py) treats ingest as one stage,
+[`ingest_end_to_end.py`](../examples/ingest_end_to_end.py) is about nothing else. It generates a
+ten-second clip, registers it twice at two different rates, ingests a folder of stills with one
+file that is not an image, and stops at an approved batch cut into two jobs. Nothing is annotated
+and nothing is released.
+
+## What it does
+
+| Stage | What happens | Owned by |
+| --- | --- | --- |
+| Project | `ProjectService.create` + `SchemaService.create_version` — one class, because approval needs a version to pin | [projects.md](projects.md), [schemas.md](schemas.md) |
+| Clip | ten seconds of `testsrc` at 10 fps, written by ffmpeg | — |
+| Source | `register_video(..., extraction_fps=5.0)` — the rate is part of *what the source is* | [sources.md](sources.md) |
+| Assets | `IngestService.ingest` → **50 assets** in a draft batch, each with a frame index and timestamp | [ingest.md](ingest.md) |
+| Progress | `IngestService.get(job_id)` → `processed=50`, `total=None` | [ingest.md](ingest.md) |
+| Batch | `approve(BySize(size=25))` → 2 jobs of 25, schema pinned | [batches.md](batches.md) |
+| Re-run | the same source again → `created=0`, `deduplicated=50` | [ingest.md](ingest.md) |
+| Second rate | `register_video(..., extraction_fps=1.0)` → a *different* source, 10 frames, none new | [sources.md](sources.md) |
+| Stills | three PNGs and a `notes.txt` → `total=4`, `created=3`, one `IngestFailure` | [ingest.md](ingest.md) |
+| Previews | every asset carries a `thumbnail_hash` | [media.md](media.md) |
+
+## Four things it is built to demonstrate
+
+**A clip cannot state its total, and a directory can.** `IngestJob.processed` climbs to 50 while
+`total` stays NULL, because `VideoMetadata` deliberately carries no frame count — it would be a
+guess for a variable-rate clip, and the number an ingest actually wants is what extraction
+produced. The image directory *can* be listed, so it states `4` before reading the first file.
+Both numbers are written to the row as the run goes, which is what makes them pollable from
+another process rather than a return value dressed up as progress.
+
+**One file registered at two rates is two sources whose frames are one set.** Decomposition
+parameters live on the source, so `extraction_fps=5.0` and `extraction_fps=1.0` over the same
+path are two origins — "the same source yields the same assets" only means something if the
+parameters deciding those assets are part of what the source *is*. And yet the coarse run creates
+nothing: identity is content, and the ten frames it cuts are byte-for-byte frames the finer run
+already stored. Their recorded origin stays the first sighting's, because origin is provenance
+and provenance is never rewritten.
+
+That alignment is a property of *this* extractor and not a promise the port makes. The fps filter
+rounds **up** onto the grid, so both rates land on whole seconds; under the default rounding a
+1 fps pass would take the picture from 0.4 s and label it 0.0.
+
+**A file that is not an image is reported, not skipped.** `notes.txt` produces one
+`IngestFailure` — `name`, `kind`, `reason`, where the reason never repeats the name so a surface
+can group by kind instead of reading prose — and the run still ends `completed`. Guessing which
+files an operator meant to offer is a policy the kernel would be inventing. Failure splits by
+remedy, which is also why a missing ffmpeg would fail the whole job instead: one broken machine
+is not five thousand broken files.
+
+**The clip is 160×120, and that is load-bearing.** `testsrc` moves a little between frames; below
+roughly 96×72 that movement falls under what the scaler and encoder still resolve, and
+consecutive frames come out byte-identical. Content addressing then does exactly what it promises
+and collapses them — a ten-second clip at 5 fps yields *forty* assets, the feature working and
+reading as a shortfall. The example says so in a comment where the constant is declared.
+
+## Why this one needs ffmpeg
+
+The SDK example boasts of needing nothing. This one checks `shutil.which("ffmpeg")` before it
+writes anything and exits with an install hint if the binary is absent, because a video is a
+container wrapped around a codec and the only honest way to write one is the tool that reads it.
+CI installs ffmpeg for exactly this reason, and the smoke test gates on
+`tests/fixtures/media.require_ffmpeg()` — a skip locally, an error under `VISIONSET_REQUIRE_FFMPEG=1`.
+
+The generation command is `tests/fixtures/media.write_video`'s, duplicated rather than imported:
+that module is a test fixture, it imports pytest, and its answer to a missing binary is
+`pytest.skip`, which means nothing in a script. The stills, by contrast, are Pillow's work — a
+real dependency since #16, so a second hand-rolled PNG encoder beside it would be archaeology.

docs/ingest.md

@@ -5,6 +5,9 @@ the bytes once, records what the decoder made of them, and puts the result in a
 [batch](batches.md) somebody can approve. Nothing else in the kernel creates an `Asset` —
 `examples/sdk_end_to_end.py` used to, and no longer does.
 
+Everything below is executed by [`examples/ingest_end_to_end.py`](../examples/ingest_end_to_end.py),
+which is walked through in [examples.md](examples.md).
+
 ```python
 source = sources.register_images(project.id, Path("~/dashcam/monday").expanduser())
 result = ingest.ingest(source.id, batch_name="monday")

examples/README.md

@@ -6,17 +6,18 @@ and `**/workspace-data/` is git-ignored by design.
 | Example | What it shows |
 | --- | --- |
 | [`sdk_end_to_end.py`](sdk_end_to_end.py) | The whole cycle in one pass: workspace → project → schema → synthetic frames → batch → jobs → annotations → curated trunk → verified release |
+| [`ingest_end_to_end.py`](ingest_end_to_end.py) | Where assets come from: a generated 10 s clip → source at 5 fps → 50 deduplicated assets → approved batch of 2 jobs, plus a re-run that creates nothing, the same clip at a second rate, and a folder of stills with one unreadable file. **Needs ffmpeg.** |
 
-## Running the end-to-end example
+## Running the SDK end-to-end example
 
 ```bash
 uv run python examples/sdk_end_to_end.py            # into examples/workspace-data/sdk-e2e
 uv run python examples/sdk_end_to_end.py ./scratch  # or wherever you like
 ```
 
 It needs nothing but the package: no server, no CLI, no file the repository had to ship. The
-six 64×48 frames come from a short PNG encoder built out of `zlib` and `struct`, because M1 has
-no image library to lean on (Pillow arrives with the media processor in M2, #16).
+six 64×48 frames come from a short PNG encoder built out of `zlib` and `struct` — fixed bytes,
+so the release's split folds are the same on every machine.
 
 With no argument it writes into `examples/workspace-data/sdk-e2e` and clears a previous run
 first — but only after confirming that directory holds nothing except a workspace. A
@@ -34,4 +35,37 @@ with WorkspaceService.open("examples/workspace-data/sdk-e2e") as workspace:
         print(release.tag, release.asset_count, release.manifest_hash)
 ```
 
-[`docs/examples.md`](../docs/examples.md) walks through what each stage does and why.
+## Running the ingest example
+
+```bash
+uv run python examples/ingest_end_to_end.py            # into examples/workspace-data/ingest-e2e
+uv run python examples/ingest_end_to_end.py ./scratch  # or wherever you like
+```
+
+Same destination rules as above, with one extra requirement: **ffmpeg must be on `PATH`**, because
+this one generates its own ten-second clip. A video is a container wrapped around a codec, and the
+only honest way to write one is the tool that reads it — so the example checks for the binary
+before it writes anything and exits with an install hint if it is missing:
+
+```bash
+brew install ffmpeg              # macOS
+sudo apt-get install ffmpeg      # Debian/Ubuntu
+```
+
+It leaves a project holding 53 assets: 50 frames cut from the clip at 5 fps into an approved batch
+of two jobs, and three stills from `incoming/` — where a fourth file is deliberately not an image,
+so the run's per-file report has something in it.
+
+```python
+from visionset.kernel.services import IngestService, ProjectService, SourceService, WorkspaceService
+
+with WorkspaceService.open("examples/workspace-data/ingest-e2e") as workspace:
+    project = ProjectService(workspace).list()[0]
+    ingest = IngestService(workspace)
+    for source in SourceService(workspace).list(project.id):
+        for job in ingest.list(source.id):
+            print(source.kind.value, job.state.value, job.processed, job.total, job.failures)
+```
+
+[`docs/examples.md`](../docs/examples.md) walks through what each stage of both examples does and
+why.