Skip to content

feat(kernel): thumbnails cached at ingest, with a backfill for what predates them (#21) - #93

Merged
JArmandoAnaya merged 1 commit into
mainfrom
feat/21-thumbnail-cache
Jul 27, 2026
Merged

feat(kernel): thumbnails cached at ingest, with a backfill for what predates them (#21)#93
JArmandoAnaya merged 1 commit into
mainfrom
feat/21-thumbnail-cache

Conversation

@JArmandoAnaya

Copy link
Copy Markdown
Contributor

Closes #21. M2 goes to 7/8; only #23 (close-out and the v0.0.1-alpha.2 tag) is left.

Every ingested item now gets a preview, stored content-addressed beside its content and named by Asset.thumbnail_hash. The M5 gallery is the reason: decoding full-resolution images per tile at request time is the wrong shape, and the cost amortizes at ingest, where the bytes are already open and already decoded once.

The constraint everything falls out of

A thumbnail hash is a cache key, not an identity.

Set by #16, restated in docs/media.md. It never enters a release manifest, ReleaseService.verify never recomputes it, and two machines may legitimately hold different preview bytes for one image. Losing every thumbnail blob loses only the time to render them again.

Decisions, and why

  1. A preview that will not render is not an IngestFailure. That error means "this file did not become an asset, so fix the file"; here the asset exists and nothing was lost. The try is nested inside the caller's — letting a refusal reach _read_directory's except MediaError would report a perfectly good file as unreadable and leave the orphan blob that probing first exists to prevent. Nothing is logged: the NULL is the record, and it is exactly what the backfill queries for.
  2. Frames get previews too. _read_video's "frames are never re-probed" rule is about metadata an operator reads back as fact. A cache artifact is reported to nobody, and a gallery with tiles for stills and blanks for frames would be the worse outcome.
  3. One edge, one cache. max_edge is not a parameter on ingest or on the backfill — a per-call edge would fork the cache into variants nothing can tell apart from a hash.
  4. _store fills a NULL and never replaces a value. Origin fields record the first sighting and are never rewritten; a preview is not provenance, so filling an empty one is not a rewrite. That is what makes re-ingesting a source enough to catch up assets written before the cache existed.
  5. The backfill reads the blob, not asset.uri — that path may be gone, renamed, or on another machine.
  6. It reports rather than raises, on ReleaseService.verify's terms. missing (the content blob is gone — damage a preview pass cannot repair) stays apart from unreadable (the bytes are there and will not decode). Only the second reuses IngestFailure: IngestFailureKind answers "what is wrong with this file", and a blob that is not there is not a file. Raising would abandon the repair of every healthy asset over one bad row.
  7. Three phases, rendering in none of them — the module's own rule applied to a different job. Only ids and hashes cross out of the first transaction, and the last re-reads each asset before writing so a concurrent ingest is not clobbered.

Migration 10 (asset_thumbnail) — FORMAT_VERSION 9 → 10

One nullable column. No foreign key, so migration 8's "a column carrying a key cannot arrive by ALTER at all" limit does not bite. No data pre-check: the column is a cache, so NULL is the ordinary state rather than a legacy value to tolerate.

Unlike migration 9 it needs its own undo in _downgrade_to_version_one. Migration 9's columns rode back on migration 8's rebuild of ingest_job; asset is only ever altered — four cascading children, legitimate pre-pipeline rows — so nothing later removes thumbnail_hash on the way to generation 1. The flip side is that the walk back does run migration 10's real ALTER, which is why it needs no generation twin of test_migration_nine_alters_a_table_migration_eight_rebuilt. Both halves of that reasoning are written down where a future reader will look.

Checks

uv run ruff format .                    114 files left unchanged
uv run ruff check .                     All checks passed!
uv run mypy src/visionset/kernel        Success: no issues found in 51 source files
uv run lint-imports                     Contracts: 2 kept, 0 broken.
uv run pytest                           877 passed (was 854)
uv run python scripts/export_openapi.py no diff
uv run python examples/sdk_end_to_end.py  ok — verified: 5 blobs, ok=True
pnpm version:check                      All frontend packages are at 0.0.1-dev.0.

VERSION stays 0.0.1.dev0. No new dependency, no new error class, no new event, no frontend change.

Tests

  • # --- a preview per asset --- — retrievable by hash on both paths, JPEG whatever the source was, bounded by the port constant (never a copied number), never enlarged, repeatable across projects (never a hardcoded hash), a refusal still leaves no blob of either kind, and a thumbnail that raises leaves a NULL with an empty report.
  • # --- the backfill --- — fills what is missing, idempotent on a second pass, reads the blob after the source file is deleted, reports a deleted blob as missing while still filling its neighbour, reports undecodable bytes as unreadable with the right kind, and refuses an unknown or foreign project.
  • Migration 10 gets a column test and a data-preservation test stamped at generation 9.
  • Fixture.content_blob_count() is new: every dedup claim in that file is about content, and a bare blob_count would make "one image, one blob" read as two. blob_count keeps its meaning and is asserted directly where the preview blobs are the subject.

…redates them (#21)

Every ingested item now gets a preview, stored content-addressed beside its
content and named by `Asset.thumbnail_hash`. The M5 gallery is the reason:
decoding full-resolution images per tile at request time is the wrong shape,
and the cost amortizes here, where the bytes are already open.

A thumbnail hash is a cache key, not an identity — absent from every release
manifest, never recomputed by `ReleaseService.verify`, and legitimately
different across Pillow builds. Everything else follows from that:

- A preview that will not render is NOT an `IngestFailure`. The asset exists
  and nothing was lost, so the hash stays NULL and the run carries on with an
  empty report. The `try` is nested inside the caller's, because reaching the
  outer `except MediaError` would report a good file as unreadable and leave
  the orphan blob probe-first exists to prevent. The NULL is the record.
- Frames get previews too. The no-re-probe rule is about metadata an operator
  reads back as fact; a cache is reported to nobody.
- One edge, one cache: `max_edge` is not a parameter on either path.
- `_store` fills a NULL on a deduplicated asset and never replaces a value, so
  re-ingesting a source catches up assets written before the cache existed.
  Origin still records the first sighting and is never rewritten.

`IngestService.backfill_thumbnails(project_id) -> ThumbnailBackfill` is the
remedy for the three things a NULL can mean. It reads the blob rather than
`asset.uri`, runs in three phases with the rendering in no transaction, and
re-reads each asset before writing. It reports rather than raises, on
`ReleaseService.verify`'s terms: `missing` (the blob is gone) stays apart from
`unreadable` (the bytes will not decode), and only the second reuses
`IngestFailure` — `IngestFailureKind` answers "what is wrong with this file",
and a blob that is not there is not a file.

Migration 10 (`asset_thumbnail`), so `FORMAT_VERSION` is now 10. One nullable
column, no foreign key, no data pre-check. Unlike migration 9 it needs its own
undo in `_downgrade_to_version_one`: `asset` is only ever altered, so nothing
later rebuilds it. The flip side is that the walk back to generation 1 runs its
real ALTER, which is why it needs no generation twin of
`test_migration_nine_alters_a_table_migration_eight_rebuilt`.

877 tests, up from 854. `VERSION` unchanged, no `openapi.json` drift, no new
dependency, no new error class, no new event.
@JArmandoAnaya
JArmandoAnaya merged commit a7847c1 into main Jul 27, 2026
3 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/21-thumbnail-cache branch July 27, 2026 14:13
JArmandoAnaya added a commit that referenced this pull request Aug 21, 2026
…redates them (#21) (#93)

Every ingested item now gets a preview, stored content-addressed beside its
content and named by `Asset.thumbnail_hash`. The M5 gallery is the reason:
decoding full-resolution images per tile at request time is the wrong shape,
and the cost amortizes here, where the bytes are already open.

A thumbnail hash is a cache key, not an identity — absent from every release
manifest, never recomputed by `ReleaseService.verify`, and legitimately
different across Pillow builds. Everything else follows from that:

- A preview that will not render is NOT an `IngestFailure`. The asset exists
  and nothing was lost, so the hash stays NULL and the run carries on with an
  empty report. The `try` is nested inside the caller's, because reaching the
  outer `except MediaError` would report a good file as unreadable and leave
  the orphan blob probe-first exists to prevent. The NULL is the record.
- Frames get previews too. The no-re-probe rule is about metadata an operator
  reads back as fact; a cache is reported to nobody.
- One edge, one cache: `max_edge` is not a parameter on either path.
- `_store` fills a NULL on a deduplicated asset and never replaces a value, so
  re-ingesting a source catches up assets written before the cache existed.
  Origin still records the first sighting and is never rewritten.

`IngestService.backfill_thumbnails(project_id) -> ThumbnailBackfill` is the
remedy for the three things a NULL can mean. It reads the blob rather than
`asset.uri`, runs in three phases with the rendering in no transaction, and
re-reads each asset before writing. It reports rather than raises, on
`ReleaseService.verify`'s terms: `missing` (the blob is gone) stays apart from
`unreadable` (the bytes will not decode), and only the second reuses
`IngestFailure` — `IngestFailureKind` answers "what is wrong with this file",
and a blob that is not there is not a file.

Migration 10 (`asset_thumbnail`), so `FORMAT_VERSION` is now 10. One nullable
column, no foreign key, no data pre-check. Unlike migration 9 it needs its own
undo in `_downgrade_to_version_one`: `asset` is only ever altered, so nothing
later rebuilds it. The flip side is that the walk back to generation 1 runs its
real ALTER, which is why it needs no generation twin of
`test_migration_nine_alters_a_table_migration_eight_rebuilt`.

877 tests, up from 854. `VERSION` unchanged, no `openapi.json` drift, no new
dependency, no new error class, no new event.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernel: thumbnails cached in the blob store (needed by the M5 gallery)

1 participant