feat(kernel): SourceService — source registration with provenance (#18)

docs/README.md

@@ -8,6 +8,7 @@ contracts (kernel purity, headless annotator) are described there and enforced i
 | --- | --- |
 | [workspaces.md](workspaces.md) | The workspace on disk: layout, `init`/`open`, project-name uniqueness, and how services are composed |
 | [projects.md](projects.md) | The project lifecycle: the 1:1 dataset, renaming, and what deletion does and does not destroy |
+| [sources.md](sources.md) | Where raw data comes from: the two registration methods, what a video source records from the probe, why decomposition parameters live on the source, and the idempotency rule and its named uniqueness gap |
 | [schemas.md](schemas.md) | The annotation schema: immutable monotonic versions, additive vs destructive change, and the two gates on narrowing |
 | [batches.md](batches.md) | The unit of annotation work: the state machine, membership frozen at approval, the schema pin, and the exact partition into jobs |
 | [jobs.md](jobs.md) | Annotation jobs: the job and per-asset progress machines, what counts as settled, ordered `next_pending`, and derived progress |

docs/examples.md

@@ -51,9 +51,10 @@ comparing folds by content hash, never by id.
 
 ## The one place it reaches below a service
 
-Creating an `Asset` has no door yet. Ingest is M2 — `SourceService` (#18), `IngestJob` (#19), and
-the pipeline that hashes, deduplicates, extracts dimensions and materializes assets into a batch
-(#20) — so `_add_assets` writes, by hand, the row that ingest will write:
+Creating an `Asset` has no door yet. `SourceService` (#18) has landed, but it registers *where*
+data comes from, not the assets themselves; the pipeline that hashes, deduplicates, extracts
+dimensions and materializes assets into a batch is #20, with `IngestJob` (#19) around it. Until
+#20 lands, `_add_assets` writes, by hand, the row that ingest will write:
 
 ```python
 hashes = [workspace.blob_store.put(BytesIO(frame_bytes(i))) for i in range(count)]

docs/media.md

@@ -338,5 +338,7 @@ kernel — FastAPI, Typer, MCP, uvicorn — not third-party libraries, and ffmpe
 - **No blob write.** `thumbnail()` and `frames()` hand back bytes. Storing them
   content-addressed, recording a `thumbnail_hash` and writing a frame's `index`/`timestamp` onto
   an asset are the ingest and thumbnail-cache tasks.
-- **No `Source`.** Nothing yet records that a clip was registered, at what original rate, or with
-  what decomposition parameters. `VideoMetadata.fps` is what that record will be built from.
+
+`Source` used to be on that list and no longer is: registering a clip records its original rate
+and the decomposition parameters chosen for it, built on `VideoMetadata` exactly as anticipated.
+See [sources.md](sources.md).

docs/persistence.md

@@ -95,8 +95,9 @@ MIGRATIONS: list[Migration] = [
     Migration(version=4, name="annotation_job_asset_position", upgrade=...),
     Migration(version=5, name="annotation_attributes", upgrade=...),
     Migration(version=6, name="release_manifest_pointer", upgrade=...),
+    Migration(version=7, name="source_provenance", upgrade=...),
 ]
-FORMAT_VERSION: int = MIGRATIONS[-1].version  # 6
+FORMAT_VERSION: int = MIGRATIONS[-1].version  # 7
 ```
 
 `initialize()` reads the version stamped in `_visionset_meta` and runs whatever is
@@ -138,24 +139,35 @@ one, where `batch_asset.position`, which was there from migration 001, does not.
 
 And a column arriving by `ALTER` must be declared **last** on its row class, because SQLite
 appends it: `batch.schema_version`, `annotation_job_asset.position` and `annotation.attributes`
-all sit at the end of their tables for that reason alone. Declared anywhere else, the
+all sit at the end of their tables for that reason alone. `source` is exempt — migration 007
+rebuilds it from `_tables` rather than altering it, so both paths run the same `CREATE TABLE` and
+the rule has nothing to bite on. Declared anywhere else, the
 `create_all` path and the `ALTER` path emit different `CREATE TABLE` text and the
 fresh-versus-migrated test fails — which is exactly what it is for.
 
-**Migration 006 is the one that drops a table**, and the bar it had to clear is worth writing
-down. `release` gained three `NOT NULL` columns with no honest default — an `ALTER` would have
-baked `manifest_hash DEFAULT ''` into every fresh database forever — and, decisively, a pre-#12
-release row carries its manifest as a JSON column with *no blob behind it*, so there is no value
-`manifest_hash` could be given that `verify` would ever accept. Adding the columns would have
-manufactured rows that are broken by construction. It is idempotent the same way 003–005 are (an
-inspector check for `manifest_hash`), and the emptiness it relies on is **checked** rather than
-argued: a workspace that somehow holds a release row raises `WorkspaceCorrupt` instead of being
-quietly emptied. A migration that would lose real data does not clear this bar.
+**Migrations 006 and 007 drop a table**, and the bar they had to clear is worth writing down.
+`release` gained three `NOT NULL` columns with no honest default — an `ALTER` would have baked
+`manifest_hash DEFAULT ''` into every fresh database forever — and, decisively, a pre-#12 release
+row carries its manifest as a JSON column with *no blob behind it*, so there is no value
+`manifest_hash` could be given that `verify` would ever accept. `source` is the same shape of
+argument twice over: `registered_at` is `NOT NULL` with no honest default, and a pre-#18 row's
+`kind` reads `'local_folder'`, which is not a value `SourceKind` has — those rows would come back
+as validation errors rather than as sources. Adding the columns would have manufactured rows that
+are broken by construction. Both are idempotent the same way 003–005 are (an inspector check for
+the new column), and the emptiness each relies on is **checked** rather than argued: a workspace
+that somehow holds such a row raises `WorkspaceCorrupt` instead of being quietly emptied. A
+migration that would lose real data does not clear this bar.
+
+Migration 007 carries one extra obligation that 006 did not. `ingest_job.source_id` is
+`ON DELETE CASCADE`, and this store sets `PRAGMA foreign_keys = ON` for every connection — so
+`DROP TABLE source` runs an implicit `DELETE FROM source` that takes the ingest jobs with it,
+**silently, without raising**. A rebuild of a table with children has to count the children too.
+`release` had none, which is why the precedent alone was not enough.
 
 The fresh-versus-migrated test is only as strong as how far back
 `_downgrade_to_version_one` walks, so every migration added there needs its undo added too.
-Migration 006 is the one place that undo cannot borrow its DDL from `_tables`, because `_tables`
-no longer describes the shape it is restoring.
+Migrations 006 and 007 are the two places that undo cannot borrow its DDL from `_tables`,
+because `_tables` no longer describes the shape it is restoring.
 
 `format_version` here is the *database* generation. Validating the on-disk workspace layout
 around it — directories, the blob-store root, what makes a directory a workspace at all —

docs/sources.md

@@ -0,0 +1,126 @@
+# Sources
+
+A **source** is the record that raw data was offered to a project: a directory of stills, or a
+video file. It is not annotatable and holds no pixels. Assets are what an ingest *materializes*
+from a source; the source is the receipt that says where they came from.
+
+`SourceService` is the one door to a `Source`. Nothing else writes `uow.sources`.
+
+## Two registration methods, not one
+
+```python
+sources = SourceService(workspace)
+
+stills = sources.register_images(project.id, Path("~/captures/2026-07").expanduser())
+clip = sources.register_video(project.id, Path("~/captures/drive.mp4"), extraction_fps=5.0)
+```
+
+There is no `register(kind=...)`, because the arguments genuinely differ. A clip needs a
+decomposition rate and gets probed; a directory needs neither and is not walked. That is the same
+argument that made `ImageProcessor` and `VideoProcessor` two protocols instead of one — see
+[media.md](media.md). A single entry point would have to accept parameters that are meaningless
+for half its callers.
+
+`register_images` checks that the directory exists and is a directory, and stops there. What is
+*in* it is read at ingest, because a count taken at registration would be stale by the time
+anything used it.
+
+`register_video` probes the file through the workspace's `VideoProcessor` and stores the answer.
+The probe runs **before** the transaction opens: it is an out-of-process decoder, and holding a
+write transaction open across a subprocess is how a single-writer SQLite store ends up reporting
+"database is locked".
+
+## What a source records
+
+| Field | Meaning |
+| --- | --- |
+| `kind` | `image_directory` or `video` — a `SourceKind` |
+| `path` | the canonical absolute path of the origin |
+| `registered_at` | timezone-aware UTC, the **first** registration |
+| `capture_params` | opaque operator-supplied provenance; nothing branches on it |
+| `video` | a `VideoProvenance`, present exactly when `kind` is `video` |
+
+`VideoProvenance` is the port's own `VideoMetadata` — original fps, duration, displayed
+dimensions, codec — plus the `extraction_fps` a decomposition will run at. The probe result is
+kept whole rather than re-spelled field by field, because `metadata.fps` is the rate the file was
+*shot* at and `extraction_fps` is the rate we chose to *cut* it at, and re-declaring the first
+beside the second is how the two come to be confused.
+
+The `video`/`kind` pairing is an invariant, enforced on construction **and** on assignment —
+`Source` is the only model in the domain with `validate_assignment` on, because a
+`model_validator` does not re-run when you assign to a field. Reading it goes through
+`source.require_video()`, which raises `WorkspaceCorrupt` rather than handing back a `None` that
+every caller would have to assert away.
+
+## Decomposition parameters live on the source, not on the job
+
+A source can be ingested more than once, and the promise is that the same source yields the same
+assets. That promise only means something if the parameters are part of what "the same source"
+*is* — put them on the ingest job and two runs of one source could legitimately disagree, leaving
+idempotency with nothing to be measured against.
+
+The consequence is deliberate: **one clip registered at 1 fps and again at 5 fps is two sources
+over one file**, not one source with a history.
+
+## Registration is idempotent
+
+The match key is `(kind, path, extraction_fps)`. Registering the same origin twice returns the
+same `Source` rather than a second one, so that once ingest gives `asset.source_id` a target,
+"which source did this asset come from?" has one answer.
+
+Two things the key deliberately leaves out:
+
+- **`capture_params`.** Fragmenting one directory into two sources because an operator typed a
+  different lens note would defeat the point. Differing params are written onto the matched
+  source instead.
+- **The probed `VideoMetadata`.** A clip replaced at a known path is still that path's source, so
+  its recorded provenance is *refreshed in place* rather than left describing a file that is
+  gone. `registered_at` is never rewritten — it is the first registration, not the last.
+
+That second rule has a corollary worth knowing: re-registering an already-known clip still needs
+ffmpeg, because the fresh probe is what keeps the record honest.
+
+### The gap this leaves, and when to close it
+
+`docs/persistence.md` says a rule with no backstop is a wish, and every other uniqueness rule in
+this store has a unique index behind it. This one does not. Two concurrent registrations of one
+folder can both pass the pre-check and both insert.
+
+That is tolerated today because no row references a source, so a duplicate is inert. It stops
+being tolerable when ingest gives `asset.source_id` a target and the winner of a race starts
+deciding an asset's recorded origin — **that is when this needs an index under it**. It sits
+alongside the store's other known concurrency gap, the untranslated `OperationalError`.
+
+## Paths are canonicalized once
+
+`canonical_path` is `str(Path.resolve(strict=True))`: absolute, symlinks followed, so `./data`,
+`../project/data` and `/abs/data` are one source. Two things it does not do:
+
+- **It does not normalize case.** On a case-insensitive filesystem — macOS by default, Windows
+  always — `/Data` and `/data` are one directory and would register as two sources. Lower-casing
+  would be wrong on Linux, where they are genuinely two.
+- **It does not look at the content.** Two hard links to one inode read as two origins. What the
+  bytes *are* is asked at ingest, where the answer is a content hash.
+
+`strict=True` means an origin that is not on disk is a `FileNotFoundError`, and a file offered
+where a directory was wanted is a `NotADirectoryError`. Both are about the machine rather than
+the workspace, so both stay outside the `VisionSetError` tree — the same line
+`MediaToolUnavailable` sits on.
+
+## Registration is not a validation pass
+
+`register_video` probes; it does not decode. A clip whose tail has been truncated still has a
+readable header, so it registers successfully and records the duration the intact file would have
+had. The damage surfaces when frames are actually extracted. Anything downstream that treats a
+successful registration as proof the file will decode is wrong.
+
+## What is deliberately not here yet
+
+- **No delete.** A source disappears with its project's cascade and no sooner. Nothing yet
+  references one, so there is no orphan to reason about; when ingest gives `asset.source_id` a
+  target, deletion becomes a real question with a real answer.
+- **No event.** Registering a source announces nothing. `IngestCompleted` is the event this area
+  will emit, and the ingest pipeline owns it.
+- **No remote kinds.** `SourceKind` has two members and grows by a deliberate kernel change with
+  a service method behind it — see the enum's own docstring for why it is an enum where
+  `DatasetChange.operation` is a plain `str`.

examples/sdk_end_to_end.py

@@ -14,10 +14,11 @@
 library to lean on (Pillow arrives with the media processor in M2, #16).
 
 **One call in here reaches below a service, on purpose.** Creating an ``Asset``
-has no door yet — ingest is M2 (#18 SourceService, #19 IngestJob, #20 the
-pipeline that hashes, deduplicates and materializes into a batch). Until #20
-lands, ``_add_assets`` writes the row that ingest would write, through the same
-public port a service uses. Every other step below goes through the service that
+has no door yet. #18's ``SourceService`` registers *where* data comes from, not
+the assets themselves; the pipeline that hashes, deduplicates and materializes
+into a batch is #20, with #19's ``IngestJob`` around it. Until #20 lands,
+``_add_assets`` writes the row that ingest would write, through the same public
+port a service uses. Every other step below goes through the service that
 owns it, which is how the rest of the SDK is meant to be used.
 """
 

src/visionset/kernel/__init__.py

@@ -45,6 +45,7 @@
     SchemaChangeWouldOrphan,
     SchemaNotFound,
     SchemaVersionConflict,
+    SourceNotFound,
     UnknownAttribute,
     UnserializableManifest,
     UnsupportedGeometry,
@@ -95,6 +96,7 @@
     "SchemaChangeWouldOrphan",
     "SchemaNotFound",
     "SchemaVersionConflict",
+    "SourceNotFound",
     "UnknownAttribute",
     "UnserializableManifest",
     "UnsupportedGeometry",

src/visionset/kernel/adapters/_mappers.py

@@ -7,14 +7,15 @@
 fourteen times against fourteen tables.
 
 Most entities are flat — every field is a column — and share
-``_flat_mapping``. The six that are not say so explicitly:
+``_flat_mapping``. The seven that are not say so explicitly:
 
 - ``AnnotationSchema`` and ``Annotation`` hold immutable nested values, encoded
   as JSON.
 - ``Batch`` and ``AnnotationJob`` own child tables, so their mappings carry a
   ``sync_children`` hook and rebuild their collections on read.
-- ``DatasetChange`` and ``Release`` encode a timezone-aware timestamp, which a
-  ``String`` column must be handed as text rather than as a ``datetime``.
+- ``DatasetChange``, ``Release`` and ``Source`` encode a timezone-aware
+  timestamp, which a ``String`` column must be handed as text rather than as a
+  ``datetime``. ``Source`` also carries a nested ``VideoProvenance`` as JSON.
 """
 
 from __future__ import annotations
@@ -46,8 +47,10 @@
     Project,
     Release,
     Source,
+    SourceKind,
     SplitRecipe,
     TaskGroup,
+    VideoProvenance,
     Workspace,
 )
 
@@ -172,6 +175,34 @@ def _change_to_domain(_: Session, row: Any) -> DatasetChange:
     )
 
 
+def _source_to_row(entity: Source) -> t.Base:
+    return t.SourceRow(
+        id=entity.id,
+        project_id=entity.project_id,
+        kind=entity.kind,
+        path=entity.path,
+        # Spelled out for the reason ``_release_to_row`` is: ``_flat_mapping``
+        # dumps in python mode and would hand a ``datetime`` to a ``String``
+        # column, which sqlite3 accepts through a deprecated adapter and writes
+        # in a second timestamp format.
+        registered_at=entity.registered_at.isoformat(),
+        capture_params=dict(entity.capture_params),
+        video=None if entity.video is None else entity.video.model_dump(mode="json"),
+    )
+
+
+def _source_to_domain(_: Session, row: Any) -> Source:
+    return Source(
+        id=row.id,
+        project_id=row.project_id,
+        kind=SourceKind(row.kind),
+        path=row.path,
+        registered_at=datetime.fromisoformat(row.registered_at),
+        capture_params=row.capture_params,
+        video=None if row.video is None else VideoProvenance.model_validate(row.video),
+    )
+
+
 def _release_to_row(entity: Release) -> t.Base:
     return t.ReleaseRow(
         id=entity.id,
@@ -278,7 +309,6 @@ def _job_sync_children(session: Session, entity: AnnotationJob) -> None:
 
 WORKSPACES = _flat_mapping(Workspace, t.WorkspaceRow, None)
 PROJECTS = _flat_mapping(Project, t.ProjectRow, "workspace_id")
-SOURCES = _flat_mapping(Source, t.SourceRow, "project_id")
 INGEST_JOBS = _flat_mapping(IngestJob, t.IngestJobRow, "source_id")
 ASSETS = _flat_mapping(Asset, t.AssetRow, "project_id")
 TASK_GROUPS = _flat_mapping(TaskGroup, t.TaskGroupRow, "batch_id")
@@ -297,6 +327,12 @@ def _job_sync_children(session: Session, entity: AnnotationJob) -> None:
     to_row=_annotation_to_row,
     to_domain=_annotation_to_domain,
 )
+SOURCES: EntityMapping[Source] = EntityMapping(
+    row=t.SourceRow,
+    parent_column="project_id",
+    to_row=_source_to_row,
+    to_domain=_source_to_domain,
+)
 RELEASES: EntityMapping[Release] = EntityMapping(
     row=t.ReleaseRow,
     parent_column="dataset_id",

src/visionset/kernel/adapters/_tables.py

@@ -98,14 +98,29 @@ class AnnotationSchemaRow(Base):
 
 
 class SourceRow(Base):
+    """Registered origins. Rebuilt by migration 7, so column order is free here.
+
+    Every other table with a migrated column declares it last, because SQLite's
+    ``ALTER TABLE ... ADD COLUMN`` appends and the two creation paths would
+    otherwise emit different DDL. Migration 7 drops and re-creates this table
+    from this class instead, so both paths run the same ``CREATE TABLE`` and the
+    rule has nothing to bite on.
+    """
+
     __tablename__ = "source"
 
     id: Mapped[UUID] = mapped_column(SaUuid, primary_key=True)
     project_id: Mapped[UUID] = mapped_column(
         SaUuid, ForeignKey("project.id", ondelete="CASCADE"), index=True, nullable=False
     )
     kind: Mapped[str] = mapped_column(String, nullable=False)
-    uri: Mapped[str] = mapped_column(String, nullable=False)
+    #: The canonical absolute path of the origin — see ``domain.canonical_path``.
+    path: Mapped[str] = mapped_column(String, nullable=False)
+    #: ISO-8601 with offset, never SQLite ``DATETIME``. See the module docstring.
+    registered_at: Mapped[str] = mapped_column(String, nullable=False)
+    capture_params: Mapped[dict[str, Any]] = mapped_column(JSON, nullable=False)
+    #: A ``VideoProvenance``, or NULL for anything that is not a clip.
+    video: Mapped[dict[str, Any] | None] = mapped_column(JSON, nullable=True)
 
 
 class IngestJobRow(Base):