Added support for decmpfs raw compression (9 and 10)

[ecc521]

Summary

Types 9 and 10 are the "stored" (uncompressed) decmpfs variants. The library currently hits the default: error branch for both. This PR adds support for reading files compressed with these types.

Format

Both types use a single byte 0xCC as a sentinel marking stored (uncompressed) data: for type 9 it sits immediately after the 16-byte decmpfs header; for type 10 it prefixes each resource-fork chunk. All remaining bytes are raw plaintext.

• Type 9: data is inline in com.apple.decmpfs xattr (max ~3785 bytes of plaintext)
• Type 10: data is in com.apple.ResourceFork using the same chunk-table format as types 8/12/14, with each chunk starting with 0xCC followed by raw data

The 0xCC sentinel presumably serves the same role as 0xFF in DEFLATE stored chunks and 0x06 in LZVN stored chunks to cause a compressor-skip.

decmpfs types	algorithm	stored-chunk sentinel
3 / 4	DEFLATE	0xFF
7 / 8	LZVN	0x06
9 / 10	STORED	0xCC
11 / 12	LZFSE	0xFF

Type 9 files exist within the OS, however I was unable to locate any type 10 files. The type 10 decompression behavior does exist in the kernel, however, and the behavior matched up cleanly with expectations in testing.

Implementation

1. libfsapfs_definitions.h.in: Add LIBFSAPFS_COMPRESSION_METHOD_STORED = 4
2. libfsapfs_file_entry.c: Dispatch cases 9 and 10 to STORED; add type 10 to the resource-fork routing condition (even-numbered types use rsrc fork)
3. libfsapfs_compression.c: Add STORED branch that validates the 0xCC sentinel and memory_copys the remaining bytes. Structurally identical to the existing LZVN stored-chunk path
4. libfsapfs/libfsapfs_compressed_data_handle.c: Add STORED to supported formats in method checks.

dtrace + kernel disassembly (macOS 26.4, ARM VM)

It's possible I made mistakes here - that said, my findings matched perfectly with observable behavior, so there is no reason to believe they are incorrect.

Type 9 stored data is copied verbatim - no decompressor in the path. The read path computes src = xattr_buf + 17 (past the 16-byte decmpfs header + 1 sentinel byte) and len = uncompressed_size, then calls uiomove64. No branch to any decompressor or compression kext appears in the path.

0xCC is an open-time validation sentinel, not a stream opcode. It is checked once in decmpfs_validate_compressed_file (loads byte 16 of the xattr, bit-tests it), not by any decompressor.

non-0xCC With a wrong sentinel, validation fails, the decmpfs cnode is never populated, and reads return 0 bytes. Since there is no decompressor in the path, the empty read cannot be "a decompressor rejecting bad input". dtrace apfs_vnop_getxattr shows only com.apple.decmpfs is ever read for a type-9 file.

Type 10 uses the same chunk table as types 8/12/14 (table_size = (num_chunks + 1) * 4, chunk_ends[i] = absolute end offset from rsrc start); on-disk type-10 files were created and read back correctly.

[joachimmetz]

Can you provide corresponding test data as well?

[ecc521]

@joachimmetz Apologies for the confusion - meant to open this as a draft initially.

Added test cases for type 9 - still need to go through and clean up type 10. Will mark this for review when I'm done.

[joachimmetz]

thanks for the proposed changes and the explanation. Would be good to have test files with the corresponding format cases as well.

[ecc521]

Alright - tests have been added in. I believe everything matches with the existing harness - quite a bit of template-y code for a rather simple format here.

In testing, I found 0 real-world examples of type 10 files, and zero instances of corrupt non 0xCC type 9 files. This was checked on both Intel (Sequoia) and Apple Silicon (Tahoe).

-> The 0xCC behavior is clear, but non-0xCC returns empty on MacOS rather than actually providing an error. This is almost certainly an error state based on the behavior, so I set things up to throw an error. I imagine this is the desired behavior.

I do have test code I used locally to create an APFS volume and confirm MacOS + libfsapfs return identical results. That can be added to the CI/CD pipeline if you're okay with something that only works on the MacOS runners, but is omitted from this PR as-is.

[joachimmetz]

In testing, I found 0 real-world examples of type 10 files, and zero instances of corrupt non 0xCC type 9 files. This was checked on both Intel (Sequoia) and Apple Silicon (Tahoe).

This is why it was never added. Previously I was able to create other cases with afstools see https://github.com/dfirlabs/apfs-specimens

[ecc521]

Makes sense.

As far as I can tell Apple is using type 9 to minimize file reads during boot. Still find it a bit weird they have an explicit uncompressed type instead of just using sentinels with another type, but it is cleaner.

Honestly if I'd known type 9 to be decompressed I wouldn't have implemented anything here. I was sort of hoping type 9/10 would lzma/zstd, and didn't quite clue in to the concept of an actually fully uncompressed type until it became obvious there literally was no other code path.

Any interest in LZFSE / the bitmaps? I've already done about 80% of the work for those as well, so it's just a matter of packaging it up.

[joachimmetz]

Honestly if I'd expected type 9 to be decompressed I wouldn't have implemented anything here. I was sort of hoping type 9/10 would be some stronger format like lzma. Should've done more homework there.

Still useful research, maybe add additional notes to the format documentation?

Any interest in LZFSE / the bitmaps? I've already done about 80% of the work for those as well, so it's just a matter of packaging it up.

Sure. Would recommend doing this in a separate PR though.

[ecc521]

Separate PR for sure for LZFSE / bitmaps. No guarantees I get around to that.

Want me to add type 9/10 doc updates to this PR? Nothing groundbreaking but having the specific format would be a mild improvement.

[codecov]

Codecov Report

❌ Patch coverage is 77.77778% with 6 lines in your changes missing coverage. Please review.
✅ Project coverage is 28.50%. Comparing base (af6712f) to head (6ddbefa).

Files with missing lines	Patch %	Lines
libfsapfs/libfsapfs_file_entry.c	0.00%	5 Missing ⚠️
libfsapfs/libfsapfs_compression.c	94.73%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main      #88      +/-   ##
==========================================
+ Coverage   28.05%   28.50%   +0.44%     
==========================================
  Files          73       73              
  Lines       15918    15885      -33     
  Branches     3659     3664       +5     
==========================================
+ Hits         4466     4528      +62     
+ Misses      10281    10170     -111     
- Partials     1171     1187      +16     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[joachimmetz]

A couple of updates, I'll make some tweaks to this PR to merge it:

• looks like "Stored" is referred to as "Raw", looks like type 9 is used, so indeed good to add.
• I was able to create LZFSE test data, and I'll make the corresponding changes. I'll include the documentation change into that.

[joachimmetz]

BTW with bitmaps are you referring to LZBITMAP (#73) ? if so libfmos will need to be first updated with support for this compression method first.

[ecc521]

BTW with bitmaps are you referring to LZBITMAP (#73) ? if so libfmos will need to be first updated with support for this compression method first.

Yes

I currently have a compressor + decompressor that is compatible with Apple's here. Planning to release it within the next 2 months. Will be a GPL compatible license, undecided what yet.

Couldn't win on speed (LZBITMAP is stupidly fast), but got about a quarter of the way there in both directions. I did manage to win on ratio by ~5% if you don't care about compression speed.

There's also https://github.com/eafer/libzbitmap. It's a great starting point if you'd like to build your own. There's a few things that it seems the format should support that Apple's decompressor refuses, which probably is the reason behind whatever trick they're using to get 6GB/s decompression on a single thread.

[joachimmetz]

It's a great starting point if you'd like to build your own

like is a strong word in this context. Libfsapfs would need something that could be used cross platform for decompression only, but understanding the compression method is beneficial for troubleshooting data format edge cases.

[joachimmetz]

LGTM