feat(server): ingest endpoints — upload, launch, poll, and the pattern every long operation reuses (#28)

[JArmandoAnaya]

Closes #28. Sixth task of M3, branched from 6a1e750 (#27).

Ingest is the first API operation that outlives its request, so the shape it sets — launch returns an id, the client polls that id — is the one #29, #33 and #35 reuse. Getting it right once is the point of the task.

The seam the kernel was missing

IngestService.ingest() creates the job row and runs it, in one call. A 202 must carry a job id, so the API could not use it: submitting ingest() to a worker means the client gets a 202 with nothing to poll, and a fast refusal in transaction 1 leaves no job row at all.

The kernel had already reserved the vocabulary — IngestJob is created pending, INGEST_TRANSITIONS allows pending → running, and resume()'s docstring called pending "what a synchronous run never leaves behind but a queued one would". Only the enqueue half was never exposed.

• IngestService.enqueue(source_id, *, batch_id=None, batch_name=None) -> IngestJob — today's transaction 1, extracted. ingest() is now literally enqueue() + resume(), and all 95 existing ingest tests passed unchanged.
• IngestService.resumable(job_id) -> IngestJob — resume's own pre-check without the work, sharing one _resolve_for_run so there is a single spelling. Without it POST /ingest-jobs/{id}/resume would 202 a completed job and refuse only in the worker, leaving a client unable to tell a redo from a no-op.
• BatchService.assets(batch_id) -> list[Asset] — mirrors DatasetService.assets, with assets_of beside jobs_of for the same reason. This is what "→ assets listed" needed; no service read assets by batch.

No migration. FORMAT_VERSION stays 11, VERSION stays 0.0.1.dev0, no new error, event, or domain model. M3's agreed tripwire for logic leaking upward is a second migration; there is none.

Endpoints (nine)

POST /projects/{id}/sources/images	multipart, 201
POST /projects/{id}/sources/video	multipart + extraction_fps, 201
GET /projects/{id}/sources
GET /sources/{id}
POST /sources/{id}/ingest-jobs	202 + Location
GET /sources/{id}/ingest-jobs
GET /ingest-jobs/{id}	the polling contract
POST /ingest-jobs/{id}/resume	202 + Location
GET /batches/{id}/assets

A collection hangs off its owner; an individually addressable resource does not — nesting /projects/{p}/sources/{s}/ingest-jobs/{j} puts four segments in front of an id that already identifies one thing. A schema version has no id of its own, which is why it stays nested all the way down.

Four decisions

Uploads are staged content-addressed (server/uploads.py). The kernel registers a source by path; an HTTP client has bytes. Parts land under <workspace>/uploads/<digest>/, where the digest is sha-256 over the sorted name:sha256 lines — one rule for one clip and for fifty stills. The same files under the same names stage to the same path, so SourceService's (kind, path, extraction_fps) idempotency survives the trip to HTTP and a repeated upload returns the same source. Nothing is buffered whole: Starlette spools past 1 MiB and the module streams upload.file in chunks (upload.read() must never appear there). safe_name reduces every client filename to its last component — the path-traversal guard, with its own tests.

Upload only — no server-side-path registration. It would hand every token holder an arbitrary-directory read, and the CLI and MCP already hold real paths. The dividend: FileNotFoundError/NotADirectoryError become unreachable, and those are plain Python exceptions with no place in ERROR_RULES — so this PR adds nothing to the error table. gt=0 on the extraction_fps form field closes the last one, a bare ValueError that would have been a 500.

One background worker (server/runner.py), a ThreadPoolExecutor(max_workers=1) on app.state, drained in the lifespan before the workspace closes. One worker serializes writers against a single-writer store; what that buys the reader is #80's WAL, and test_polling_is_answered_while_another_writer_holds_the_workspace proves it by parking a second WorkspaceService mid-write and asserting the poll is still answered — with committed state, not the writer's.

Refusals split by when they can be known. Anything the request can refuse is refused synchronously (404 unknown source, 422 blank batch name, 409 non-resumable job); everything after the launch is reported on the job. batch_id targeting is deliberately not on the launch body — batches have no endpoints until #29, and leaving it out means the launch has no failure mode that produces no job row.

Two findings worth recording

IngestStart needs no _the_domain_accepts_it validator, and the contrast is the useful part. LabelClassBody needs one because LabelClass refuses with a pydantic ValidationError — neither a VisionSetError nor a RequestValidationError, so it reaches the catch-all as a 500. A blank batch name refuses with InvalidName, already in ERROR_RULES at 422 INVALID_NAME, so the kernel's own refusal arrives correctly and a validator would only restate it less precisely. The first draft had one; the test that expected VALIDATION_ERROR is what found it.

FastAPI emits Body_register_image_source / Body_register_video_source for multipart bodies, named off the operation id — which generate_unique_id_function=operation_id already pins to the handler name, so they are as stable as every other component.

Ledger

• openapi.json 34 KB → 73 KB. tests/architecture/test_tracked_file_sizes.py caps a tracked file at 200 KB; not close, and deliberately not allowlisted.
• python-multipart>=0.0.9 declared in [project].dependencies. It was already installed here — transitively, via mcp. An upload surface resting on somebody else's dependency tree breaks the day that tree changes.
• 1193 tests, up from 1116. New: tests/server/test_sources.py (23), test_ingest.py (14), test_batches.py (8), test_uploads.py (21), plus kernel tests for enqueue/resumable/BatchService.assets. Still no conftest.py anywhere; tests/server/_runner.py joins _api.py/_probe.py/_openapi.py as plain helpers.
• Nothing sleeps. Waiting is RecordingRunner.wait() joining the worker's future; sequencing is GatedRunner's events — the tests/kernel/test_concurrency.py doctrine. The threaded module ran 15 consecutive times, 0 failures before shipping.
• Fixture.job_in in tests/kernel/test_ingest_service.py now walks to pending via enqueue instead of planting it. running is the only planted state left, and it is the one no operation leaves behind.
• Docs: api.md (202 + the launch/poll block, multipart and size expectations, the paths block), ingest.md (the enqueue/resume split, a new "Over HTTP" section), sources.md (upload staging), workspaces.md (uploads/ in the layout), batches.md (assets()).

MCP parity (#24's standing rule)

Tools this capability implies, for #35 to implement: register_image_source, register_video_source, list_sources, get_source, start_ingest, get_ingest_job, list_ingest_jobs, resume_ingest, list_batch_assets.

Recorded now because it does not carry over cleanly: MCP takes a local path, not multipart. The staging area exists because HTTP has no paths; an agent on the same machine calls register_images directly, and visionset/server/uploads.py must not grow an MCP caller.

Checks

uv run pytest              1193 passed in 100.85s
uv run lint-imports        Contracts: 2 kept, 0 broken.
uv run mypy src/visionset/kernel   Success: no issues found in 54 source files
uv run mypy src/visionset          Success: no issues found in 78 source files
uv run ruff check .        All checks passed!
uv run ruff format --check .       156 files already formatted
uv run python scripts/export_openapi.py   committed
examples/ingest_end_to_end.py      53 assets, exit 0
examples/sdk_end_to_end.py         5 assets / 15 labels, exit 0