feat(ingest): accept what Pillow and ffmpeg decode, normalized into JPEG and PNG

docs/content/ingest.md

@@ -52,10 +52,10 @@ reports every item as deduplicated, and is how a folder that grew by three files
 | | image directory | video |
 | --- | --- | --- |
 | what is read | every file at the **top level**, in filename order | one frame per extraction slot, at the rate the source records |
-| decoded by | `ImageProcessor.probe` - dimensions and format from the bytes | `VideoProcessor.frames` - ffmpeg, deterministic within one build |
-| `uri` | the file's absolute path | `/path/clip.mp4#frame=7` |
-| frame position | none | `frame_index` and `frame_timestamp` |
-| damage | one report line per file; the run carries on | the frames ffmpeg managed are kept, plus one report line |
+| decoded by | `ImageProcessor.stills` - anything Pillow reads, normalized to JPEG/PNG | `VideoProcessor.frames` - ffmpeg, deterministic within one build |
+| `uri` | the file's absolute path; `/path/anim.gif#frame=3` for a decomposed animation frame | `/path/clip.mp4#frame=7` |
+| frame position | none for a still; `frame_index` and `frame_timestamp` for an animation frame | `frame_index` and `frame_timestamp` |
+| damage | one report line per file, the whole file refused; the run carries on | the frames ffmpeg managed are kept, plus one report line |
 
 Subdirectories are stepped over and recorded nowhere. Recursion is not a per-run option but a
 question about what *the source is* - "the same source yields the same assets" - so it belongs to

docs/content/media.md

@@ -45,19 +45,32 @@ disk, and a blob in the default blob store is a path too.
 | Produces | `thumbnail` → JPEG bytes | `frames` → an iterator of `VideoFrame` |
 | Needs | Pillow, a dependency | ffmpeg, a binary on `PATH` |
 
-## The accepted list is two formats, and widening it is a decision
-
-`ImageFormat` names everything VisionSet decodes: **JPEG and PNG**. Extending it is exactly two
-edits - a member on the enum, and the decoder's own spelling of it in the adapter's
-`_FORMAT_BY_PILLOW_NAME` - and a test asserts the two cover the same set, so a half-done
-extension fails on the first run rather than at the first file.
-
-That cost is the point. Accepting a format is a promise that VisionSet will decode it, hash it,
-thumbnail it and export it for as long as the workspaces written today are readable. A
-`try: decode` that admitted whatever the installed Pillow happened to support would make that
-promise depend on a wheel, and it would quietly let in the long tail where image-decoder CVEs
-live. WEBP is the obvious next member; it is not here yet because a format with no generated
-fixture is a format nobody is testing.
+## The dataset holds two formats; ingest reads far more
+
+`ImageFormat` names everything a dataset may contain: **JPEG and PNG**, frozen. Acceptance is
+deliberately wider. `ImageProcessor.stills` reads anything Pillow decodes - WebP, HEIC and HEIF
+(what an iPhone writes), BMP, TIFF, GIF and the rest - and normalizes everything outside the two
+into them at ingest, so a dataset consumer never needs a third decoder.
+
+| You give VisionSet | The dataset holds |
+| --- | --- |
+| JPG / JPEG, PNG | your exact bytes, untouched |
+| WebP, HEIC / HEIF, BMP, TIFF, anything else Pillow decodes | a JPEG, re-encoded at ingest |
+| animated GIF, animated WebP | one PNG per frame, `anim.gif#frame=3` |
+| MP4, MOV, AVI, WebM, MKV - anything ffmpeg reads | one PNG per extracted frame |
+
+A converted or decomposed asset keeps the original path in its `uri`, so provenance stays
+legible: an asset at `photo.heic` whose `format` reads `jpeg` was transcoded on the way in, and
+the original file on disk is untouched. Transcoding is repeatable within one Pillow build, not
+across builds - the same caveat extracted video frames carry.
+
+Out of scope for this distribution: RAW camera formats (CR2, NEF, DNG), AVIF, and JPEG XL. A
+Live Photo is two files; the photo ingests as a still, and its clip may be registered as an
+ordinary video source.
+
+The frozen enum is the promise that matters: VisionSet will decode, hash, thumbnail and export
+JPEG and PNG for as long as the workspaces written today are readable. What can be *read* may
+drift with the installed Pillow; what a dataset *holds* may not.
 
 `MPO` is a special case worth knowing about and is **not** a third format. It is a
 multi-picture JPEG container - what phones write in portrait and burst modes - so it is an

frontend/ui-core/src/generated/api.ts

@@ -4349,11 +4349,12 @@ export interface components {
         };
         /**
          * ImageFormat
-         * @description Every still-image encoding VisionSet accepts. See the module docstring.
+         * @description Every still-image encoding a dataset may contain. See the module docstring.
          *
          *     A ``StrEnum`` rather than a ``Literal``, unlike ``Asset.modality``: that one
-         *     has a single member, where an enum would be ceremony, and this one is a
-         *     closed set whose whole purpose is to grow deliberately. It costs the
+         *     has a single member, where an enum would be ceremony. This set is **frozen
+         *     at JPEG and PNG**: acceptance is wider — anything Pillow decodes is read —
+         *     but everything else is normalized into these two at ingest. It costs the
          *     persistence layer nothing — a ``StrEnum`` member *is* a ``str``, and the
          *     tables already store every other enum as ``String``.
          * @enum {string}

openapi.json

@@ -3088,7 +3088,7 @@
         "type": "object"
       },
       "ImageFormat": {
-        "description": "Every still-image encoding VisionSet accepts. See the module docstring.\n\nA ``StrEnum`` rather than a ``Literal``, unlike ``Asset.modality``: that one\nhas a single member, where an enum would be ceremony, and this one is a\nclosed set whose whole purpose is to grow deliberately. It costs the\npersistence layer nothing \u2014 a ``StrEnum`` member *is* a ``str``, and the\ntables already store every other enum as ``String``.",
+        "description": "Every still-image encoding a dataset may contain. See the module docstring.\n\nA ``StrEnum`` rather than a ``Literal``, unlike ``Asset.modality``: that one\nhas a single member, where an enum would be ceremony. This set is **frozen\nat JPEG and PNG**: acceptance is wider \u2014 anything Pillow decodes is read \u2014\nbut everything else is normalized into these two at ingest. It costs the\npersistence layer nothing \u2014 a ``StrEnum`` member *is* a ``str``, and the\ntables already store every other enum as ``String``.",
         "enum": [
           "jpeg",
           "png"

pyproject.toml

@@ -30,6 +30,11 @@ dependencies = [
     # so it is allowed inside kernel/adapters/ and needs no import-linter change. The 11.0
     # floor is the first release with Python 3.13 wheels, which the classifiers promise.
     "pillow>=11.0",
+    # HEIC/HEIF decoding, registered as a Pillow plugin inside kernel/adapters/ —
+    # the same third-party-library allowance pillow has. Core rather than an
+    # extra: phone photos are a first-contact input, and an ingest that refuses
+    # them until a second install is a broken first run.
+    "pillow-heif>=1.5.0",
     # Settings for the embedded job executor (#328). The first pydantic-settings
     # use in this repository; `server/settings.py` argues why the executor is what
     # finally earned one, and why the four existing bare `os.environ` reads stay
@@ -208,6 +213,12 @@ python_version = "3.12"
 warn_unused_configs = true
 warn_redundant_casts = true
 
+[[tool.mypy.overrides]]
+# pillow-heif publishes no py.typed marker, so the one symbol imported from it
+# (register_heif_opener) arrives untyped.
+module = "pillow_heif.*"
+ignore_missing_imports = true
+
 [[tool.mypy.overrides]]
 module = "visionset.kernel.*"
 disallow_untyped_defs = true

src/visionset/kernel/adapters/pillow_image_processor.py

@@ -3,21 +3,27 @@
 from __future__ import annotations
 
 import io
+from collections.abc import Iterator
 from typing import BinaryIO, Final
 
 from PIL import Image, ImageOps, UnidentifiedImageError
+from pillow_heif import register_heif_opener
 
-from visionset.kernel.domain import ImageFormat, ImageMetadata
+from visionset.kernel.domain import DecodedStill, ImageFormat, ImageMetadata
 from visionset.kernel.errors import CorruptMedia, UnsupportedMedia
 from visionset.kernel.ports.image_processor import DEFAULT_THUMBNAIL_MAX_EDGE
 
-#: Pillow's spelling of every format we accept. Pillow's vocabulary stops here,
-#: the way SQLAlchemy's stops at ``_tables.py``: a second image adapter would
-#: bring its own table rather than teach the domain a third set of names.
+register_heif_opener()
+
+#: Pillow's spelling of every format that passes through untouched. Pillow's
+#: vocabulary stops here, the way SQLAlchemy's stops at ``_tables.py``: a second
+#: image adapter would bring its own table rather than teach the domain a third
+#: set of names.
 #:
-#: Extending the accepted list is this dict plus a member on ``ImageFormat``, and
-#: ``test_image_processor.py`` asserts the two cover the same set, so a half-done
-#: extension fails on the first run instead of at the first file.
+#: This is the pass-through table, not the accepted list — ``stills`` reads
+#: anything Pillow decodes and transcodes what is not in here. It covers exactly
+#: ``ImageFormat``, which ``test_image_processor.py`` asserts, because the bytes
+#: that skip the transcode are precisely the bytes a dataset may hold.
 #:
 #: ``MPO`` is not a fourth format: it is a multi-picture JPEG container — what
 #: phones write in portrait and burst modes — and VisionSet reads its primary
@@ -52,6 +58,15 @@
     "progressive": False,
 }
 
+#: The dataset-still encoder, pinned for the reason ``_THUMBNAIL_ENCODER`` is.
+#: ``quality=95`` because these bytes are training input, not a preview.
+_STILL_ENCODER: Final[dict[str, object]] = {
+    "quality": 95,
+    "subsampling": 0,
+    "optimize": False,
+    "progressive": False,
+}
+
 #: Pinned by name rather than by its integer value, which says nothing.
 _RESAMPLING: Final = Image.Resampling.LANCZOS
 
@@ -135,14 +150,24 @@ def _fit(image: Image.Image, max_edge: int) -> Image.Image:
     """
     working = image.convert("RGBA") if _has_alpha(image) else image.convert("RGB")
     working.thumbnail((max_edge, max_edge), _RESAMPLING, reducing_gap=None)
+    return _opaque_rgb(working)
 
+
+def _opaque_rgb(image: Image.Image) -> Image.Image:
+    """The compositing tail of :func:`_fit`, alone: full-size, no resampling.
+
+    What a convertible still goes through before the dataset encoder sees it —
+    the same fresh-canvas and known-background guarantees, at the image's own
+    size.
+    """
+    working = image.convert("RGBA") if _has_alpha(image) else image.convert("RGB")
     canvas = Image.new("RGB", working.size, _BACKGROUND)
     canvas.paste(working, mask=working.getchannel("A") if working.mode == "RGBA" else None)
     return canvas
 
 
 class PillowImageProcessor:
-    """Decodes JPEG and PNG, reports oriented dimensions, encodes fixed thumbnails.
+    """Decodes what Pillow reads, passes JPEG and PNG through, transcodes the rest.
 
     Holds no state at all — no cache, no handle, no configuration — which is why
     ``WorkspaceService`` builds one per workspace from a zero-argument factory and
@@ -215,6 +240,81 @@ def thumbnail(
         canvas.save(buffer, format=_THUMBNAIL_PILLOW_NAME, **_THUMBNAIL_ENCODER)
         return buffer.getvalue()
 
+    def stills(self, content: BinaryIO, *, name: str | None = None) -> Iterator[DecodedStill]:
+        """Every dataset-ready still in this file. See the port docstring.
+
+        The native gate runs before the frame count on purpose: MPO decodes
+        with ``n_frames > 1``, and checking frames first would decompose every
+        burst photo instead of passing its primary frame through.
+        """
+        source = _stream_name(content, name)
+        image = self._open(io.BytesIO(_read_all(content)), source)
+
+        native = _FORMAT_BY_PILLOW_NAME.get(image.format or "")
+        if native is not None:
+            self._load(image, source)
+            with image:
+                width, height = image.size
+            return iter(
+                [DecodedStill(metadata=ImageMetadata(width=width, height=height, format=native))]
+            )
+
+        if getattr(image, "n_frames", 1) > 1:
+            with image:
+                return iter(self._decomposed(image, source))
+
+        self._load(image, source)
+        with image:
+            flat = _opaque_rgb(image)
+        buffer = io.BytesIO()
+        flat.save(buffer, format="JPEG", **_STILL_ENCODER)
+        return iter(
+            [
+                DecodedStill(
+                    metadata=ImageMetadata(
+                        width=flat.width, height=flat.height, format=ImageFormat.JPEG
+                    ),
+                    payload=buffer.getvalue(),
+                )
+            ]
+        )
+
+    def _decomposed(self, image: Image.Image, source: str | None) -> list[DecodedStill]:
+        """One PNG still per frame — a list, not a generator, on purpose.
+
+        Every frame is decoded and encoded before the caller sees the first
+        one, so damage at frame four of six raises with nothing yielded and a
+        caller that stores as it consumes leaves no partial file behind. The
+        cost is the whole animation's encoded frames in memory at once.
+        """
+        stills: list[DecodedStill] = []
+        elapsed_ms = 0.0
+        for index in range(int(getattr(image, "n_frames", 1))):
+            try:
+                image.seek(index)
+                frame = _opaque_rgb(image)
+            except Image.DecompressionBombError as exc:
+                raise UnsupportedMedia(str(exc), name=source) from exc
+            except (EOFError, OSError, SyntaxError, ValueError) as exc:
+                raise CorruptMedia(
+                    f"the animation is damaged or truncated at frame {index} ({exc})",
+                    name=source,
+                ) from exc
+            buffer = io.BytesIO()
+            frame.save(buffer, format="PNG")
+            stills.append(
+                DecodedStill(
+                    metadata=ImageMetadata(
+                        width=frame.width, height=frame.height, format=ImageFormat.PNG
+                    ),
+                    payload=buffer.getvalue(),
+                    frame_index=index,
+                    frame_timestamp=elapsed_ms / 1000.0,
+                )
+            )
+            elapsed_ms += float(image.info.get("duration", 0) or 0)
+        return stills
+
     def _decode(self, content: BinaryIO, source: str | None) -> tuple[Image.Image, ImageFormat]:
         """Open, identify, refuse, decode, orient — in that order.
 
@@ -223,10 +323,25 @@ def _decode(self, content: BinaryIO, source: str | None) -> tuple[Image.Image, I
         bytes; the decode comes before the dimensions, so a truncated file is
         refused rather than measured.
         """
-        buffer = io.BytesIO(_read_all(content))
+        image = self._open(io.BytesIO(_read_all(content)), source)
+
+        # Read off the ImageFile before anything transforms it: convert() and the
+        # copying form of exif_transpose() both hand back a format of None.
+        image_format = _FORMAT_BY_PILLOW_NAME.get(image.format or "")
+        if image_format is None:
+            found = image.format or "an unrecognized encoding"
+            accepted = ", ".join(sorted(member.value for member in ImageFormat))
+            image.close()
+            raise UnsupportedMedia(
+                f"{found} is not accepted; VisionSet reads {accepted}", name=source
+            )
+
+        self._load(image, source)
+        return image, image_format
 
+    def _open(self, buffer: io.BytesIO, source: str | None) -> Image.Image:
         try:
-            image = Image.open(buffer)
+            return Image.open(buffer)
         except UnidentifiedImageError as exc:
             # Before the OSError clause, which this subclasses: the other way
             # round, every file that is not an image reads as a corrupt one.
@@ -241,17 +356,7 @@ def _decode(self, content: BinaryIO, source: str | None) -> tuple[Image.Image, I
                 f"the image header is damaged or truncated ({exc})", name=source
             ) from exc
 
-        # Read off the ImageFile before anything transforms it: convert() and the
-        # copying form of exif_transpose() both hand back a format of None.
-        image_format = _FORMAT_BY_PILLOW_NAME.get(image.format or "")
-        if image_format is None:
-            found = image.format or "an unrecognized encoding"
-            accepted = ", ".join(sorted(member.value for member in ImageFormat))
-            image.close()
-            raise UnsupportedMedia(
-                f"{found} is not accepted; VisionSet reads {accepted}", name=source
-            )
-
+    def _load(self, image: Image.Image, source: str | None) -> None:
         try:
             image.load()
             ImageOps.exif_transpose(image, in_place=True)
@@ -263,5 +368,3 @@ def _decode(self, content: BinaryIO, source: str | None) -> tuple[Image.Image, I
             raise CorruptMedia(
                 f"the image data is damaged or truncated ({exc})", name=source
             ) from exc
-
-        return image, image_format

src/visionset/kernel/domain/__init__.py

@@ -151,6 +151,7 @@
 from visionset.kernel.domain.media import (
     MEDIA_TYPES,
     OCTET_STREAM,
+    DecodedStill,
     ImageFormat,
     ImageMetadata,
     VideoFrame,
@@ -407,6 +408,7 @@
     "DatasetMember",
     "DatasetOperation",
     "DatasetStats",
+    "DecodedStill",
     "DomainEvent",
     "DownloadSize",
     "DraftAttribute",