examples/codec_mode_histogram: measure shipped hpc::codec mode split + rate-distortion curve#229
Conversation
…curve Drives the shipped hpc::codec rdo_select over synthetic fields across a coherence sweep (raster scan, causal N/W neighbours), reporting the actual %skip/merge/delta/escape histogram, bytes/cell, and mean distortion; plus a λ rate-distortion sweep on the weather-like field. Measured (256×256, 16-basin codebook): - coherent (exact basin) → 100% skip → 2.00 B/cell → 4.0× lossless - weather-like (smooth) → 85% delta / 13% merge → 2.98 B/cell → 2.68× lossless - incoherent in-range → 98% delta → 2.98 B/cell → 2.68× lossless - incoherent out-of-range→ 73% escape → 11.05 B/cell → 0.72× (regression) - λ sweep: 2.68× lossless (λ≥4) → 4.0× at meanDist 16.5 (λ=0 quantizes to basin) Findings: (1) continuous fields are Delta-floored at ~2.68×, not the 3.3× skip-dominant target, because smooth data lands NEAR basins not ON them; (2) Merge=3B=Delta=3B, so neighbour-coherence buys zero bytes — only Skip (2B) saves; (3) "no regression vs dense" fails when the codebook doesn't span the data. Read-before-write example; std/codec feature only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MLBnPuScZy6w9di2QEjsXM
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 98d8515c4a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| use ndarray::hpc::codec::rdo::{rdo_select, RdoConfig, RdoContext}; | ||
| use ndarray::hpc::codec::{packed_byte_len, CellMode, LeafCu}; |
There was a problem hiding this comment.
This new auto-discovered example imports ndarray::hpc::codec, but codec is disabled by the crate's default features and no [[example]] required-features = ["codec"] entry was added. Since cargo test --help states that cargo test will “build examples”, the repo's default cargo test path will try to compile this example without --features codec and fail resolving hpc::codec; add a Cargo.toml example target with the codec requirement so default CI skips it unless the feature is enabled.
Useful? React with 👍 / 👎.
…odec"] Addresses codex P1 (#229): the example imports hpc::codec, which is not a default feature. Without a [[example]] required-features entry, default cargo test (which builds examples) would try to compile it without --features codec and fail resolving hpc::codec. Now default builds skip the target; --features codec builds it. Matches the sibling-example convention. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MLBnPuScZy6w9di2QEjsXM
Fixes the rustfmt --check CI failure on #229 — rustfmt 1.95.0 collapses the multi-line float expression to a single line. Cosmetic only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MLBnPuScZy6w9di2QEjsXM
|
Codex Review: Didn't find any major issues. Chef's kiss. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
|
@cursor review |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
✅ Bugbot reviewed your changes and found no new issues!
Comment @cursor review or bugbot run to trigger another review on this PR
Reviewed by Cursor Bugbot for commit 5bdd864. Configure here.
2 participants
Idea
Characterize what the shipped
hpc::codecRDO selector actually delivers — the compression is entirely a function of the data's coherence, so this measures where different data lands instead of assuming the design's 3.3× / 10–50× targets. Drivesrdo_selectover a coherence sweep as a raster scan with causal N/W neighbours (exactly how a real encoder feeds Merge), reporting the actual %skip/merge/delta/escape histogram + bytes/cell + mean distortion, plus a λ rate-distortion sweep.Measured (256×256, 16-basin codebook)
λ sweep (weather-like): 2.68× lossless (λ≥4) → 4.0× at meanDist 16.5 (λ=0 quantizes to basin).
Findings
Read-before-write example,
std+codecfeature only; no new deps. Confirms the codec math is proven and lossless-correct at 2.68–4×, and precisely locates the codec/codebook/λ work needed to reach the headline.Generated by Claude Code