Skip to content

Add intra-only 4:4:4 (ChromaArrayType 3) decoding — byte-exact - #1

Open
JensWalter wants to merge 1 commit into
roticv:trunkfrom
JensWalter:chroma444-support
Open

Add intra-only 4:4:4 (ChromaArrayType 3) decoding — byte-exact#1
JensWalter wants to merge 1 commit into
roticv:trunkfrom
JensWalter:chroma444-support

Conversation

@JensWalter

Copy link
Copy Markdown

I am currently working on a image gallery server (https://codeberg.org/JensWalter/everlock) and found that some iPhone generated pictures can be decoded, and some cannot. The distinction here was the selected Chromatype (Apple use one for Photos, and one for edited Photos).

So after digging around I could add some ChromaType 4:4:4 support, although my knowledge about this is rather superficial. So I tried to make the changes as least invasive as possible, because of this. I also added some tests, specifically with sample pictures.

If you are open to merge that would be great. If my PR does not meet your expectation, I am willing to refine whatever you see as not working or needs refinement.

Also here is some description from Claude about this change:

Summary

The decoder was 4:2:0-only. This adds intra 4:4:4 support (8-bit and
10-bit, ChromaArrayType == 3), which is the format HEIF/HEIC still images use
— in particular Apple "edited"/HDR photos, which are commonly 4:4:4 10-bit and
previously failed to decode with Unsupported("only 4:2:0 … supported").

Output is byte-exact against FFmpeg. The 4:2:0 path is untouched
(byte-identical), and every change is gated on ChromaArrayType, so existing
behaviour is preserved. 4:4:4 inter (P/B) slices are explicitly rejected
(see Limitations) rather than reconstructed with the wrong chroma.

Motivation

HEVC-in-HEIF still images from recent iPhones are frequently coded as 4:4:4
10-bit (e.g. after on-device editing or HDR capture). Decoding one previously
returned an Unsupported error at SPS parse. This change makes those images
decodable while keeping the byte-exact 4:2:0 guarantees intact.

Testing

New byte-exact tests and fixtures (same style as the existing suite —
decode_and_hash against a committed .h265, with the generating command
documented above each test):

  • test_decode_444_intra_byte_exact — 96×96 8-bit 4:4:4 intra. Exercises
    full-resolution chroma TBs, chroma reference smoothing (PLANAR/angular/DC),
    per-PU intra_chroma_pred_mode for PART_NxN (62 such CUs in the fixture),
    chroma SAO/deblock, and the conformance-window crop (96 is not CTB-aligned at
    CTU 64).
  • test_decode_444_intra_10bit_byte_exact — 96×96 10-bit 4:4:4 intra
    (the HEIC photo format).
  • test_decode_444_inter_rejected — asserts a 4:4:4 P/B slice returns
    Unsupported.

Verification performed:

  • All existing 4:2:0 tests pass unchanged (169 total, was 166); the 4:2:0
    reconstruction is byte-identical.
  • Byte-exact vs FFmpeg on synthetic 4:4:4 streams across presets (8-bit and
    10-bit; SAO, deblock, sign-hiding, strong-intra-smoothing, PART_NxN).
  • Byte-exact vs FFmpeg on a real iPhone 4:4:4 10-bit HEIC (1685×2247):
    0 / 11,358,585 samples differ, max|Δ| = 0 on Y, U, and V.
  • cargo clippy clean.

Limitations / follow-ups

  • 4:4:4 inter (video) is not supported and is gated with Unsupported.
    Enabling it would require parameterizing the chroma motion-compensation call
    sites in inter_pred.rs (chroma block size, stride, coordinates, and the
    mv << (1 - shift) scaling) and enlarging the internal chroma scratch
    buffers (currently sized for 4:2:0). Luma inter already works for 4:4:4.
  • 4:2:2 is not supported (rejected at SPS parse).
  • Unrelated observation (not addressed here): the inter path assumes
    CTB-aligned picture dimensions for the reference luma stride; non-aligned
    resolutions are decoded correctly for intra but not for inter. The existing
    test fixtures all use CTB-aligned resolutions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant