feat(kernel): IngestJob lifecycle — transition table, persisted progress, per-file report, resume (#19)

[JArmandoAnaya]

Closes #19.

#20 shipped a working pipeline and deliberately left the job record thin: the row went in already at running, terminal states were written straight onto it, and the per-file report lived only in memory. Both domain/ingest.py and docs/ingest.md said so in as many words. This adds the lifecycle around that path rather than rewriting it.

Acceptance criteria

• Re-run test: second run creates zero new blobs/assets — test_re_ingesting_one_source_creates_nothing already asserted it for ingest; test_resuming_creates_no_new_blobs_for_what_was_already_stored now asserts the same for resume, which is what makes a redo affordable.
• Mid-run progress observable in a test — test_progress_is_visible_to_somebody_who_is_not_running_the_ingest watches the row through a second WorkspaceService opened on the same directory, not another call on the service doing the work. Only a separate connection can show the counters are committed as the run goes, which is the claim.
• Failed run reports which files failed and why — test_a_run_records_which_files_failed_and_why (per-file, run still completes) and test_a_fatal_cause_is_recorded_apart_from_the_per_file_report (the error column, with failures empty). test_a_failed_run_keeps_the_progress_it_had_made covers the position a failure stopped at.

Design

Resume re-enters the same row. INGEST_TRANSITIONS gets one backward edge, failed → running. The argument against reopening a batch does not carry over: a batch pins a schema version and its jobs are already cut against that pin, whereas nothing is pinned against an ingest run. A row per attempt would fork batch_id and turn list(source_id) into a list of retries. Refusing to resume a completed job then needs no new error — it is a move the table does not have, so it is an ordinary InvalidTransition. No new error class in this PR.

running → running is deliberately absent, so a job stuck at running cannot be resumed. That state is a process that died without reporting anything; the remedy already exists (ingest again, which creates nothing), and overwriting the row would erase the only evidence of the crash.

Counters are written once per item. An interval that suits five files and one that suits fifty thousand are different numbers and the service cannot know which it is looking at, so there is no cadence constant to guess. One small commit beside a decode-and-hash that costs an order of magnitude more. The progress writes are not a contradiction of the four-transaction rule: each opens and commits while nothing is being decoded — the hazard is a transaction held across the decode.

total is NULL for a clip, and that is honest. VideoMetadata carries no frame count by design (a guess for VFR; the number an ingest wants is what extraction produced), so a total there would be arithmetic presented as fact. A directory states its total before the first file, which is also what makes an empty one record 0 of 0.

The report is one JSON column, the way source.video holds a whole VideoProvenance: read whole, never queried by field. INGEST_JOBS becomes the eighth hand-written mapper pair.

A job is now created pending and moved to running by whoever picks it up. Today that is the same call and the state lasts microseconds — it is spelled out because it is the vocabulary a queue needs, and because adding it later would change what a stored row means. Every refusal still happens before the insert, so a fail-fast leaves no job row at all.

The hazard this closes

Repository.update is session.merge(to_row(entity)) — a whole-row replace. ingest() captured job in its first transaction and re-used that object after the decode. The moment counters are written in between, that stale copy would have silently undone all of them on completion. So nothing carries an IngestJob across the decode any more: only job_id travels, and every write re-reads the row inside its own transaction. That is what require_job(uow, job_id) was made public for in #20.

Migration 9 — FORMAT_VERSION is now 9

ingest_job gains batch_name, processed, total, failures. The plainest migration in the file, and the plainness is the point after 8: none of the four carries a foreign key, so ALTER can express all of them, and each has an honest value for a pre-#19 row (that run counted nothing and reported nothing). Nothing is refused and nothing is dropped.

batch_name earns its column rather than being convenience: a run that died during the decode reached no batch, so without it a resumed run would fall back to naming the batch after the source folder and quietly lose the name the caller asked for.

A coverage gap worth the entry. Migration 8 rebuilt this table, so its column-order exemption expired the moment this build started writing rows — but the fresh-versus-migrated test walks back to generation 1, from where migration 8 re-creates the table whole (from _tables, so including migration 9's columns) and migration 9 finds them present and does nothing. A migration whose only exercise is through an earlier rebuild is not exercised at all. test_migration_nine_alters_a_table_migration_eight_rebuilt sets a database to generation 8 and compares the real ALTER path against a fresh one. That baseline is produced by dropping the four columns rather than by retyping the CREATE TABLE, because these tests compare sqlite_master text and a hand-written one differs in whitespace — a false negative about this file rather than about the schema. That the drops are possible at all is migration 9's own argument restated.

Migration 9 needs no undo in _downgrade_to_version_one: its columns live on ingest_job, which migration 8's undo rebuilds from scratch. That is now commented there so the next migration on this table does not assume the same.

Also

• require_move promoted from job_service's module-private helper to domain/transitions.py, and BatchService's second spelling of the same sentence deleted — the standing "a gate two services need is promoted, not copied" rule. Message text is byte-identical, so every sweep still matches on "cannot become".
• tests/kernel/test_metadata_store.py seeds the ingest job with a populated failures tuple, per that file's convention that a nested shape left empty leaves the round trip untested.
• A function-local import of SchemaService/LabelClass in test_ingest_service.py folded into a Fixture.freeze() helper.
• docs/ingest.md gains a lifecycle section and a progress section; docs/persistence.md carries the ledger, FORMAT_VERSION 9, and the rebuild-exemption expiry.

Checks

uv run ruff check .                  All checks passed!
uv run ruff format --check .         114 files already formatted
uv run mypy src/visionset/kernel     Success: no issues found in 51 source files
uv run mypy src/visionset            Success: no issues found in 61 source files
uv run lint-imports                  Contracts: 2 kept, 0 broken.
uv run pytest                        854 passed  (was 828)
uv run python examples/sdk_end_to_end.py    green
uv run python scripts/export_openapi.py     no diff

VERSION stays 0.0.1.dev0; no new dependency; no new error class; ingest has no route yet, so openapi.json is unchanged.