feat(ingest): accept what Pillow and ffmpeg decode, normalized into JPEG and PNG - #813
Merged
Conversation
…eFormat frozen by design
…malize stills to JPEG
…ecomposes animations
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this changes
An image ingest accepted exactly two encodings, so a directory holding WebP files, GIFs, or the HEIC photos an iPhone writes reported every one of them as unsupported. This change widens acceptance to anything Pillow decodes while freezing the dataset vocabulary at JPEG and PNG: everything outside the two is normalized into them at ingest, so a dataset consumer never needs a third decoder.
pillow-heifbecomes a core dependency, registered as a Pillow plugin inside the adapter, so phone photos work in the base install.anim.gif#frame=3URIs — the fragment convention video frames already use. The decode completes before anything is stored, so a truncated animation is one failure and zero assets, preserving the refused-file-leaves-no-blob invariant.Out of scope, recorded in the docs: RAW camera formats (CR2, NEF, DNG), AVIF, JPEG XL, and Live Photo pairing.
How
One new domain value (
DecodedStill) and one new port method (ImageProcessor.stills) implemented in the Pillow adapter;probeandthumbnailare untouched, andprobekeeps its narrow meaning for its other callers. The directory reader consumesstills();processednow counts files dealt with (assets would overcount once one file yields forty), and the stable-order key groups a file with its own frames by base uri so a directory holding stills and a decomposed animation lists them interleaved by name. No migration and no wire change beyond the regeneratedImageFormatschema description: assets already carryformat,frame_index,frame_timestamp, andurikeeps the source path in storage, so a transcode's provenance stays legible asphoto.heicplusformat=jpeg.Transcoding is repeatable within one Pillow build, not across builds — the caveat extracted video frames already carry, documented the same way.
Every surface goes through the same door
An earlier revision of this description flagged the MCP surface as possibly still validating uploads through the narrow
probe. Verified since: no such surface exists. There is no MCP upload tool — MCP registers sources by path and ingests through the kernel, REST uploads stage bytes to a staged directory and ingest the same way, and the oneprobecall in the MCP package measures the thumbnailget_asset_imageserves, which is always JPEG. A new end-to-end test uploads an HEIC and an animated GIF throughPOST /projects/{id}/sources/imagesand asserts they come out as a JPEG asset and ordered PNG frames, so the claim is pinned rather than asserted.Checks run
tests/kernel/test_image_processor.py,tests/kernel/test_ingest_service.py,tests/kernel/test_video_processor.pyandtests/server/test_ingest.py,mypy src/visionset/kernel,lint-imports(4 contracts kept),ruff check, the architecture suite, and the docs link checker — all green locally. CI's matrix is the exhaustive gate; its first run caught the staleopenapi.jsonafter theImageFormatdescription change, regenerated together with the typed client in a follow-up commit.