Skip to content

feat(ingest): accept what Pillow and ffmpeg decode, normalized into JPEG and PNG - #813

Merged
JArmandoAnaya merged 8 commits into
mainfrom
feat/ingest-media-formats
Aug 28, 2026
Merged

feat(ingest): accept what Pillow and ffmpeg decode, normalized into JPEG and PNG#813
JArmandoAnaya merged 8 commits into
mainfrom
feat/ingest-media-formats

Conversation

@JArmandoAnaya

@JArmandoAnaya JArmandoAnaya commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

What this changes

An image ingest accepted exactly two encodings, so a directory holding WebP files, GIFs, or the HEIC photos an iPhone writes reported every one of them as unsupported. This change widens acceptance to anything Pillow decodes while freezing the dataset vocabulary at JPEG and PNG: everything outside the two is normalized into them at ingest, so a dataset consumer never needs a third decoder.

  • A native JPEG or PNG passes through untouched — the exact bytes remain the asset, so nothing changes for existing workspaces.
  • Any other image Pillow decodes (WebP, HEIC/HEIF, BMP, TIFF, and the rest) is re-encoded to JPEG with a pinned encoder, orientation applied and alpha composited onto white. pillow-heif becomes a core dependency, registered as a Pillow plugin inside the adapter, so phone photos work in the base install.
  • An animated GIF or animated WebP decomposes into one PNG asset per frame with anim.gif#frame=3 URIs — the fragment convention video frames already use. The decode completes before anything is stored, so a truncated animation is one failure and zero assets, preserving the refused-file-leaves-no-blob invariant.
  • The video side needed no code: a video source has deliberately no format gate, so MP4, MOV, AVI, WebM and MKV already ingested. New container fixtures pin that breadth so an ffmpeg upgrade that drops one goes red in the suite instead of in a user's ingest report.

Out of scope, recorded in the docs: RAW camera formats (CR2, NEF, DNG), AVIF, JPEG XL, and Live Photo pairing.

How

One new domain value (DecodedStill) and one new port method (ImageProcessor.stills) implemented in the Pillow adapter; probe and thumbnail are untouched, and probe keeps its narrow meaning for its other callers. The directory reader consumes stills(); processed now counts files dealt with (assets would overcount once one file yields forty), and the stable-order key groups a file with its own frames by base uri so a directory holding stills and a decomposed animation lists them interleaved by name. No migration and no wire change beyond the regenerated ImageFormat schema description: assets already carry format, frame_index, frame_timestamp, and uri keeps the source path in storage, so a transcode's provenance stays legible as photo.heic plus format=jpeg.

Transcoding is repeatable within one Pillow build, not across builds — the caveat extracted video frames already carry, documented the same way.

Every surface goes through the same door

An earlier revision of this description flagged the MCP surface as possibly still validating uploads through the narrow probe. Verified since: no such surface exists. There is no MCP upload tool — MCP registers sources by path and ingests through the kernel, REST uploads stage bytes to a staged directory and ingest the same way, and the one probe call in the MCP package measures the thumbnail get_asset_image serves, which is always JPEG. A new end-to-end test uploads an HEIC and an animated GIF through POST /projects/{id}/sources/images and asserts they come out as a JPEG asset and ordered PNG frames, so the claim is pinned rather than asserted.

Checks run

tests/kernel/test_image_processor.py, tests/kernel/test_ingest_service.py, tests/kernel/test_video_processor.py and tests/server/test_ingest.py, mypy src/visionset/kernel, lint-imports (4 contracts kept), ruff check, the architecture suite, and the docs link checker — all green locally. CI's matrix is the exhaustive gate; its first run caught the stale openapi.json after the ImageFormat description change, regenerated together with the typed client in a follow-up commit.

@JArmandoAnaya
JArmandoAnaya merged commit 400bf8d into main Aug 28, 2026
15 checks passed
@JArmandoAnaya
JArmandoAnaya deleted the feat/ingest-media-formats branch August 28, 2026 07:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant