feat(server): ingest endpoints — upload, launch, poll, and the pattern every long operation reuses (#28)

docs/api.md

@@ -38,11 +38,26 @@ GET    /projects/{project_id}/schema                      the version in force
 POST   /projects/{project_id}/schema/versions
 GET    /projects/{project_id}/schema/versions
 GET    /projects/{project_id}/schema/versions/{version}
+POST   /projects/{project_id}/sources/images              multipart
+POST   /projects/{project_id}/sources/video               multipart
+GET    /projects/{project_id}/sources
+GET    /sources/{source_id}
+POST   /sources/{source_id}/ingest-jobs                   launch
+GET    /sources/{source_id}/ingest-jobs
+GET    /ingest-jobs/{job_id}                              poll
+POST   /ingest-jobs/{job_id}/resume
+GET    /batches/{batch_id}/assets
 ```
 
 The active schema is the collection's **parent**, not a member of it, because "in force" is a
 property of the schema rather than a version number a client could guess.
 
+A **collection** hangs off whatever owns it; an individually addressable **resource** does not.
+A source belongs to one project, so listing and creating happen under it — but a source has an
+id of its own, and nesting `/projects/{p}/sources/{s}/ingest-jobs/{j}` would put four segments
+in front of a job that one id already identifies. A schema version has no such id, which is why
+it stays nested all the way down.
+
 **Ids are UUIDs**, canonical hyphenated form, in the path. One deliberate exception: a **schema
 version is an integer 1..N**, because that is the handle the domain itself uses — an annotation
 records `schema_version`, and a batch pins one at approval. A malformed UUID never reaches a
@@ -68,7 +83,37 @@ pre-check either one: the flag goes to the SDK and the SDK's refusal is what car
 `CONFIRMATION_REQUIRED` or `DESTRUCTIVE_SCHEMA_CHANGE`.
 
 **Statuses.** 201 with the created resource in the body; 200 for a read or an update; 204 with an
-empty body for a delete.
+empty body for a delete. And **202 when the work has not happened yet** — see below.
+
+**A long operation is launched and then polled.** Ingest is the first one, and the shape it set
+is the one every later long operation uses:
+
+```
+POST /sources/{id}/ingest-jobs   →  202 Accepted
+                                    Location: /ingest-jobs/{job_id}
+                                    { "id": …, "state": "pending", "processed": 0, … }
+
+GET  /ingest-jobs/{job_id}       →  200 { "state": "running",   "processed": 12, … }
+GET  /ingest-jobs/{job_id}       →  200 { "state": "completed", "batch_id": …,   … }
+```
+
+**202, not 201**: the row exists, the work does not. The row is what makes the id worth handing
+back — it is written and committed before the response is sent, so the first poll always finds
+something. That is also why anything the request can refuse is refused *synchronously*: an
+unknown source is a 404 here rather than a 202 pointing at a job nobody wrote, and resuming a
+`completed` run is a 409 here rather than a background no-op a client could not distinguish from
+a redo. Everything that goes wrong *after* the launch is reported on the job — `error` for the
+one fatal cause, `failures` for the per-item report — because by then there is no request left
+to answer.
+
+**Uploads are multipart, and the only non-JSON request shape.** Registering a source means
+sending the bytes: one `files` part per image, or one `file` part plus an `extraction_fps` field
+for a clip. VisionSet sets **no size limit of its own** — parts are spooled to disk past 1 MiB
+and streamed from there, so memory does not grow with the file — which means the real ceilings
+are your reverse proxy's (`client_max_body_size` in nginx) and free disk. Uploaded bytes are
+staged under `<workspace>/uploads/<digest>/` and, like blobs, are **never deleted**: a workspace
+grows with what was offered to it, not only with what it kept. Re-uploading identical files
+under identical names is free — it stages to the same path and returns the same source.
 
 **Request bodies forbid unknown fields.** A misspelled key is a 422 `VALIDATION_ERROR`, never a
 silently ignored one — a typo that looked like it worked is worse than a refusal.

docs/batches.md

@@ -45,6 +45,13 @@ Membership is a **set** — adding an asset the batch already holds changes noth
 removing one it does not hold is a no-op. Order is the order assets were first added. Every
 asset must belong to the batch's project, else `AssetNotFound`.
 
+Reading it back is `batches.assets(batch_id)`, which answers with the `Asset` rows in that
+stored order — `DatasetService.assets` over the trunk, applied to a batch. It is how "what did
+that ingest actually gather" is answered, and a member whose asset is not stored is
+`WorkspaceCorrupt` rather than a silently shorter list: `batch_asset.asset_id` cascades from
+`asset`, so that cannot happen while foreign keys are on, and a batch quietly holding less than
+it says is worse than a refusal.
+
 Excluding an asset after approval is a different act: it is marked **`skipped`**, a per-asset
 progress decision the record keeps rather than a membership edit that erases it. Somebody
 decided not to label that asset, and that decision is worth more than a tidy list. The

docs/ingest.md

@@ -67,6 +67,29 @@ offer is a policy the kernel would be inventing.
 tests. Decoding each one again to re-confirm it would also route our own encoder's output into an
 operator's per-file report — a failure nobody could act on.
 
+## Asking for a run and doing it are two calls
+
+```python
+job = ingest.enqueue(source.id, batch_name="monday")  # refuses now, reads nothing
+result = ingest.resume(job.id)  # does the work
+```
+
+`ingest(...)` is those two composed, and that is all it is. The split exists because a caller
+that cannot wait — the [REST API](api.md), and one day a queue — needs the **row before the
+work**: the id it hands back has to name something the next request can find. Every refusal
+`enqueue` can make it makes before the insert, so a launch that fails leaves no job at all and
+a launch that succeeds leaves one that is already pollable.
+
+`resume` is what picks a `pending` job up, which is why `pending → running` was in the
+transition table from the start. `resumable(job_id)` is the same friendly pre-check without the
+work, for a caller that runs the second half elsewhere and needs the refusal on its own thread —
+a launch that answered "accepted" and only discovered in a worker that the job was already
+`completed` would give nobody a way to tell a redo from a no-op.
+
+Nothing here decides *when* the second half runs. That is deliberately not the kernel's
+business: the API supplies a single background worker (`server/runner.py`), the CLI just calls
+`ingest`, and neither arrangement is visible in this module.
+
 ## The run has a lifecycle, and it is a table
 
 `INGEST_TRANSITIONS` in `domain/ingest.py` is the whole of what is legal. `IngestService`
@@ -78,9 +101,11 @@ pending ──▶ running ──▶ completed
    └────────▶ failed ──▶ running        (resume)
 ```
 
-A job is created `pending` and moved to `running` by whoever picks the work up. Today that is
-the same call, and the state is over in microseconds — it is spelled out anyway because it is
-the vocabulary a queue needs, and adding it later would mean changing what a stored row means.
+A job is created `pending` and moved to `running` by whoever picks the work up. Through
+`ingest(...)` those are the same call and the state is over in microseconds; through
+`enqueue` + `resume` they are not, and a `pending` row is a run somebody asked for that has not
+started. That is why the state was spelled out before anything left one behind — adding it later
+would have meant changing what a stored row means.
 
 **`failed → running` is the only backward edge in this kernel**, and the argument against
 reopening a [batch](batches.md) does not carry over. A batch pins a schema version at approval
@@ -98,8 +123,8 @@ stuck row as the only evidence the crash left.
 
 `processed` and `total` are written to the row **as the run goes**, so
 `IngestService.get(job_id)` answers "where is it now" rather than "where did it end". That is
-the contract the HTTP API and the UI will reuse; nothing about it is specific to being in the
-same process.
+what `GET /ingest-jobs/{id}` returns and what the UI will poll; nothing about it is specific to
+being in the same process.
 
 | | what it means |
 | --- | --- |
@@ -255,10 +280,38 @@ Membership is everything the run ingested, deduplicated assets included: a dupli
 data, but it is part of what the run was asked to gather. Order is ingest order, which is filename
 order for a directory and frame order for a clip.
 
+## Over HTTP
+
+The [API](api.md) is `enqueue` and `resume` with a worker between them.
+
+```
+POST /projects/{id}/sources/video   multipart: the clip + extraction_fps   → 201 SourceOut
+POST /sources/{id}/ingest-jobs                                             → 202 IngestJobOut
+GET  /ingest-jobs/{id}                                                     → 200 IngestJobOut
+GET  /batches/{id}/assets                                                  → 200 the assets
+```
+
+The launch calls `enqueue` on the request thread and hands the `pending` job to a **single
+background worker** — one, so that runs serialize against a single-writer store rather than
+racing each other. What that buys the *reader* is what [#80's concurrency posture](workspaces.md)
+was for: a client polling while the worker holds a write transaction reads through WAL instead
+of waiting on it.
+
+**Where a refusal appears depends on when it can be known.** An unknown source or a blank batch
+name is refused synchronously, with a 404 or a 422 — the launch never returns 202 pointing at a
+job row nobody wrote. Everything after that is on the job: `state` becomes `failed` and `error`
+carries the cause, while individual unreadable items sit in `failures` and do not fail the run
+at all. That split is the same one this service already makes; HTTP just changes where you read
+it.
+
+Registration over HTTP is **upload-only**, and the bytes are staged content-addressed — see
+[sources.md](sources.md). Targeting an existing draft batch is not on the wire yet: batches have
+no endpoints until the batch and job API lands, so there is nothing for a client to name.
+
 ## What is deliberately not here yet
 
-- **No background execution.** A run is synchronous and in-process. The service API is shaped so
-  that moving it behind a queue changes the caller's waiting, not its vocabulary — which is why
-  a job is created `pending` and why progress is read off the row rather than off a callback.
+- **No scheduler.** `enqueue` and `resume` are two calls; nothing in the kernel decides when the
+  second one runs. The API supplies one background worker, the CLI supplies the calling thread,
+  and a queue would be a third arrangement neither of them would notice.
 - **No cross-attempt history.** The report and the counters describe the current attempt. A
   resumed run overwrites them, and a log of every attempt would be its own table.

docs/sources.md

@@ -117,6 +117,27 @@ where a directory was wanted is a `NotADirectoryError`. Both are about the machi
 the workspace, so both stay outside the `VisionSetError` tree — the same line
 `MediaToolUnavailable` sits on.
 
+## Over HTTP, a path is an upload
+
+`SourceService` registers by path, and an HTTP client has bytes rather than a path. So the
+[REST API](api.md) takes multipart — one `files` part per image, or one `file` part plus an
+`extraction_fps` field for a clip — writes the parts under `<workspace>/uploads/`, and registers
+what it wrote. There is **no route that accepts a server-side path**: it would hand every token
+holder an arbitrary-directory read, and the two surfaces that legitimately hold real paths, the
+CLI and MCP, call the SDK in-process and never go through HTTP.
+
+The staging directory is named by a **digest of the whole part set** — sha-256 over the sorted
+`name:sha256` lines — which is what makes the idempotency above survive the trip. The same files
+under the same names stage to the same path, so a repeated upload returns the *same* `Source`
+instead of a second one over a second copy on disk. Different bytes, or the same bytes under a
+different filename, are a different offer and stage apart.
+
+That upload-only choice has a quiet dividend: because the server just wrote the file, the
+`FileNotFoundError` and `NotADirectoryError` below are unreachable from HTTP. Neither is a
+`VisionSetError`, so neither has an entry in the API's error table — and neither needs one.
+
+A client never sees `path`. `SourceOut` publishes the filename and nothing about the machine.
+
 ## Registration is not a validation pass
 
 `register_video` probes; it does not decode. A clip whose tail has been truncated still has a

docs/workspaces.md

@@ -10,9 +10,17 @@ happens in the context of exactly one, so `WorkspaceService` is both the way in
   visionset.db-wal  SQLite's write-ahead log — present only while the workspace is open
   visionset.db-shm  its shared-memory index — likewise
   blobs/            FilesystemBlobStore root, sharded <hh>/<hh>/<hash>
+  uploads/          written only by the REST server — see below
 ```
 
-Nothing else is written. The store runs in **WAL mode**, which is why the two sidecars are
+Nothing the kernel writes is outside those four entries. `uploads/` is the exception and it
+belongs to somebody else: the [REST API](api.md) stages multipart uploads there, named by a
+digest of the part set, so that `SourceService` — which registers a source by *path* — has a
+path to be given. The kernel neither writes nor reads it, `open` simply tolerates it the way it
+tolerates anything else beside the database and `blobs/`, and the CLI and MCP surfaces never
+create one because they already hold real paths. Like blobs, staged uploads are never deleted.
+
+The store runs in **WAL mode**, which is why the two sidecars are
 part of the format: `close()` checkpoints them into `visionset.db` and removes them, so a
 workspace at rest is still just the database and the blobs — but a workspace that is *open*,
 or one whose process was killed, is all four entries.