M1: motion compensation via cognitive-shader primitives (E-7 / H-7)#230
Conversation
Proof-of-concept for the codec↔cognitive-shader unification (E-7 / H-7 in .claude/knowledge/pr-x12-codec-cognitive-substrate-mapping.md): E-7 — block-matching motion estimation IS i8gemm. HEVC ME does SAD; reformulated as SSD ‖A‖² − 2·A·B + ‖B‖², the middle term A·B is a GEMM. This PoC runs the whole per-block search through the SHIPPED hpc::quantized::int8_gemm_i32 (the u8×i8→i32 VNNI kernel the cognitive shader uses for attention/distance) — not a bespoke SAD loop — and cross-checks every block's argmin against brute-force direct SSD. H-7 — the codec IS the substrate. Each block's residual magnitude classifies into the shipped hpc::codec::CellMode (Skip/Merge/Delta/ Escape); the codec's mode taxonomy IS the per-block MC decision. Measured (128×128, 8×8 blocks = 256 blocks, 7-bit luma, ±4 search): - E-7: i8gemm SSD argmin == brute-force direct SSD : 256/256 (100%) - MV recovery (ME == ground-truth motion) : 256/256 (100%) - MC reconstruction max abs pixel error : 0 (bit-exact) - CellMode histogram: skip=160 merge=6 delta=90 escape=0 - 256 int8_gemm_i32 calls in 1.66 ms → ~154k blocks/s Two hard asserts falsify the concept if broken: gemm_eq_brute == nblk (E-7) and max_err == 0 (bit-exact MC). Scope: this proves ME runs through the shader's i8 GEMM bit-identically to direct SSD and MC is exact gather+add — NOT a real .265 front-end (CABAC entropy decode + inverse transform to extract MVs/residuals), which is M2. Frames are synthetic 7-bit luma with integer-translation MVs so the i8 GEMM is exact (7-bit values fit both u8 and i8 operands). Feature-gated: [[example]] required-features = ["codec"]. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MLBnPuScZy6w9di2QEjsXM
📝 WalkthroughWalkthroughThis PR adds a new Cargo example, Changesmc_via_shader example
Estimated code review effort: 2 (Simple) | ~15 minutes Sequence Diagram(s)sequenceDiagram
participant Main
participant GemmKernel as int8_gemm_i32
participant BruteForce
Main->>Main: generate reference and current frames
Main->>GemmKernel: compute A·B for candidate MVs per block
GemmKernel-->>Main: dot product results
Main->>Main: derive SSD argmin from GEMM output
Main->>BruteForce: compute brute-force SSD argmin
BruteForce-->>Main: brute-force argmin
Main->>Main: cross-check argmin, gather+add residual for reconstruction
Main->>Main: classify CellMode, print metrics, assert correctness
Related Issues: None found in provided context. Related PRs: None found in provided context. Suggested labels: example, codec Suggested reviewers: None found in provided context. Poem
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit 6d4f691. Configure here.
| let mode = if maxres == 0 { | ||
| CellMode::Skip | ||
| } else if maxres <= 127 && recovered[by * BX + bx] == recovered[by.saturating_sub(1) * BX + bx] { | ||
| CellMode::Merge // MV inherited from N neighbour + in-range residual |
There was a problem hiding this comment.
Row zero false Merge classification
Medium Severity
CellMode::Merge is chosen when the recovered MV equals the block above, but by.saturating_sub(1) is zero on the top row, so each top-row block is compared to itself. Any top-row block with a non-zero residual is labeled Merge instead of Delta, which skews the H-7 mode histogram (likely all six reported merges).
Reviewed by Cursor Bugbot for commit 6d4f691. Configure here.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6d4f691110
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| for i in 0..B { | ||
| let k = j * B + i; | ||
| let cur = current[(py as usize + j) * W + (px as usize + i)] as i32; | ||
| let res = cur - pred[k] as i32; // stored residual = current − prediction |
There was a problem hiding this comment.
The reconstruction check is tautological because res is recomputed from the same current pixel being verified: for any in-bounds recovered MV, pred + (cur - pred) equals cur, so max_err stays 0 even if motion estimation/compensation is wrong. In this PoC's synthetic stream the residuals are known at frame construction time, so store/reuse those residuals or also assert mv_exact == nblk before claiming bit-exact MC.
Useful? React with 👍 / 👎.
| // per-block decision = codec mode: residual magnitude → CellMode | ||
| let mode = if maxres == 0 { | ||
| CellMode::Skip | ||
| } else if maxres <= 127 && recovered[by * BX + bx] == recovered[by.saturating_sub(1) * BX + bx] { |
There was a problem hiding this comment.
When by == 0, by.saturating_sub(1) is also 0, so this compares the top-row block's MV with itself. Any top-row block with a nonzero residual and maxres <= 127 is therefore labeled Merge despite having no north neighbor to inherit from; in the generated frame the reported Merge count comes from this top-row self-comparison rather than real neighbor inheritance.
Useful? React with 👍 / 👎.
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
examples/mc_via_shader.rs (1)
194-215: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low valueReconstruction clamp bound inconsistent with stated 7-bit luma domain.
reconis clamped to[0, 255]at line 211, but every other pixel path in this file (ref_pixel,residualclamp at line 108) treats the domain as 7-bit[0, 127]. Sinceres = cur - pred[k]is an exact inverse,pred + resalways reproducescurexactly, so this is functionally inert here — but it's misleading given the file's stated 7-bit-luma invariant and could mask a real out-of-range value if the residual formula changes later.♻️ Suggested consistency fix
- let r = (pred[k] as i32 + res).clamp(0, 255); // reconstruction = pred + residual + let r = (pred[k] as i32 + res).clamp(0, 127); // reconstruction = pred + residual🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@examples/mc_via_shader.rs` around lines 194 - 215, The reconstruction path in the MC loop uses an inconsistent clamp range with the file’s 7-bit luma invariant. Update the reconstruction logic in the block that computes `pred`, `res`, and writes into `recon` so it clamps to the same `[0, 127]` domain used by `ref_pixel` and the residual handling, keeping the `mc_via_shader` reconstruction and validation paths consistent.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@examples/mc_via_shader.rs`:
- Around line 216-226: The north-neighbour check in the mode selection logic is
incorrectly using by.saturating_sub(1), which makes the first row compare
against itself and misclassify top-row blocks. Update the decision in the
mc_via_shader block-mode calculation so the Merge path is only considered when
by > 0, and fall back to Delta for the first row; use the existing recovered
indexing logic around the mode assignment to locate the fix.
---
Nitpick comments:
In `@examples/mc_via_shader.rs`:
- Around line 194-215: The reconstruction path in the MC loop uses an
inconsistent clamp range with the file’s 7-bit luma invariant. Update the
reconstruction logic in the block that computes `pred`, `res`, and writes into
`recon` so it clamps to the same `[0, 127]` domain used by `ref_pixel` and the
residual handling, keeping the `mc_via_shader` reconstruction and validation
paths consistent.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 671d9ca2-39d9-43f0-a271-4913f1456f8f
📒 Files selected for processing (2)
Cargo.tomlexamples/mc_via_shader.rs
| // per-block decision = codec mode: residual magnitude → CellMode | ||
| let mode = if maxres == 0 { | ||
| CellMode::Skip | ||
| } else if maxres <= 127 && recovered[by * BX + bx] == recovered[by.saturating_sub(1) * BX + bx] { | ||
| CellMode::Merge // MV inherited from N neighbour + in-range residual | ||
| } else if maxres <= 127 { | ||
| CellMode::Delta | ||
| } else { | ||
| CellMode::Escape | ||
| }; | ||
| modes[mode as usize] += 1; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Show the relevant section with line numbers
sed -n '190,245p' examples/mc_via_shader.rs | cat -n
# Find where `recovered` is written and how `by` / `bx` are used elsewhere
rg -n "recovered\\[|saturating_sub\\(1\\)|CellMode::(Skip|Merge|Delta|Escape)|maxres|by\\b|bx\\b" examples/mc_via_shader.rsRepository: AdaWorldAPI/ndarray
Length of output: 4960
Guard the north-neighbour check on the first row examples/mc_via_shader.rs:219
by.saturating_sub(1) makes this compare a block against itself when by == 0, so every top-row block with 1 <= maxres <= 127 is tagged Merge instead of Delta.
Fix
- } else if maxres <= 127 && recovered[by * BX + bx] == recovered[by.saturating_sub(1) * BX + bx] {
+ } else if maxres <= 127
+ && by > 0
+ && recovered[by * BX + bx] == recovered[(by - 1) * BX + bx]
+ {
CellMode::Merge // MV inherited from N neighbour + in-range residual📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| // per-block decision = codec mode: residual magnitude → CellMode | |
| let mode = if maxres == 0 { | |
| CellMode::Skip | |
| } else if maxres <= 127 && recovered[by * BX + bx] == recovered[by.saturating_sub(1) * BX + bx] { | |
| CellMode::Merge // MV inherited from N neighbour + in-range residual | |
| } else if maxres <= 127 { | |
| CellMode::Delta | |
| } else { | |
| CellMode::Escape | |
| }; | |
| modes[mode as usize] += 1; | |
| // per-block decision = codec mode: residual magnitude → CellMode | |
| let mode = if maxres == 0 { | |
| CellMode::Skip | |
| } else if maxres <= 127 | |
| && by > 0 | |
| && recovered[by * BX + bx] == recovered[(by - 1) * BX + bx] | |
| { | |
| CellMode::Merge // MV inherited from N neighbour + in-range residual | |
| } else if maxres <= 127 { | |
| CellMode::Delta | |
| } else { | |
| CellMode::Escape | |
| }; | |
| modes[mode as usize] += 1; |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@examples/mc_via_shader.rs` around lines 216 - 226, The north-neighbour check
in the mode selection logic is incorrectly using by.saturating_sub(1), which
makes the first row compare against itself and misclassify top-row blocks.
Update the decision in the mc_via_shader block-mode calculation so the Merge
path is only considered when by > 0, and fall back to Delta for the first row;
use the existing recovered indexing logic around the mode assignment to locate
the fix.
…ubstrate The durable ledger for the operator's thesis: use the BF16 16×16 AMX GEMM / blasgraph neighborhood / Morton-tile pyramid / perturbation-shader cascade to amortize H.265/HEVC — the per-block cost collapsing to a gather (bit shift) + one TDPBF16PS tile op over a deterministic pyramid built once. Maps every HEVC DSP stage to the substrate primitive, each row tagged MEASURED (#PR) / MECHANISM-unmeasured (probe named) / DOESN'T-FIT: - Motion estimation/compensation → i8/bf16 GEMM + gather [MEASURED #230] - Inverse transform → separable tile GEMM; WHT no-op [MEASURED #232] - Intra prediction / covariance → field-coupling tile op [MEASURED shape #233] - Sub-pel interpolation (8-tap) → separable tile GEMM [MECHANISM — probe] - CTU quad-tree → Morton/HHTL cascade [MECHANISM — shipped] - Deblock/SAO → masked tile op + LUT (the "AMX masking") [PARTIAL — masking] - CABAC entropy → serial arithmetic coding [DOESN'T FIT] Two honest boundaries recorded: (1) CABAC is not a GEMM — the entropy layer is the M2 serial front-end the tile substrate does not amortize; (2) the golden-spiral codec is for directional/sparse data, not dense scalar residuals (#231, measured negative). Cross-repo tie grounded: OGAR's perturbation pyramid IS the Walsh-Hadamard transform of the address tree = the same WHT #232 measured as the codec transform. Includes a probe queue (subpel_tap_tile, intra_angular_tile, deblock_masked_tile, CTU routing parity, M2 CABAC). Anchors: PRs #230–#233; hpc::bf16_tile_gemm, splat3d::spd3, cascade, codec; blasgraph HHTL; OGAR perturbation pyramid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MLBnPuScZy6w9di2QEjsXM
…ubstrate The durable ledger for the operator's thesis: use the BF16 16×16 AMX GEMM / blasgraph neighborhood / Morton-tile pyramid / perturbation-shader cascade to amortize H.265/HEVC — the per-block cost collapsing to a gather (bit shift) + one TDPBF16PS tile op over a deterministic pyramid built once. Maps every HEVC DSP stage to the substrate primitive, each row tagged MEASURED (#PR) / MECHANISM-unmeasured (probe named) / DOESN'T-FIT: - Motion estimation/compensation → i8/bf16 GEMM + gather [MEASURED #230] - Inverse transform → separable tile GEMM; WHT no-op [MEASURED #232] - Intra prediction / covariance → field-coupling tile op [MEASURED shape #233] - Sub-pel interpolation (8-tap) → separable tile GEMM [MECHANISM — probe] - CTU quad-tree → Morton/HHTL cascade [MECHANISM — shipped] - Deblock/SAO → masked tile op + LUT (the "AMX masking") [PARTIAL — masking] - CABAC entropy → serial arithmetic coding [DOESN'T FIT] Two honest boundaries recorded: (1) CABAC is not a GEMM — the entropy layer is the M2 serial front-end the tile substrate does not amortize; (2) the golden-spiral codec is for directional/sparse data, not dense scalar residuals (#231, measured negative). Cross-repo tie grounded: OGAR's perturbation pyramid IS the Walsh-Hadamard transform of the address tree = the same WHT #232 measured as the codec transform. Includes a probe queue (subpel_tap_tile, intra_angular_tile, deblock_masked_tile, CTU routing parity, M2 CABAC). Anchors: PRs #230–#233; hpc::bf16_tile_gemm, splat3d::spd3, cascade, codec; blasgraph HHTL; OGAR perturbation pyramid. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MLBnPuScZy6w9di2QEjsXM
2 participants
What
A runnable proof-of-concept (
examples/mc_via_shader.rs) for the codec↔cognitive-shader unification documented in.claude/knowledge/pr-x12-codec-cognitive-substrate-mapping.md. It proves two claims from that plan against the shipped primitives, with hard falsifiers, not prose:‖A‖² − 2·A·B + ‖B‖², the middle termA·Bis a GEMM. The whole per-block search runs through the shippedhpc::quantized::int8_gemm_i32— the sameu8×i8→i32VNNI kernel the cognitive shader uses for attention/distance — and every block's argmin is cross-checked against a brute-force direct SSD.hpc::codec::CellMode(Skip/Merge/Delta/Escape). The codec's mode taxonomy is the per-block MC decision.Measured
128×128,8×8blocks = 256 blocks, 7-bit luma, ±4 search:int8_gemm_i32calls in 1.66 ms → ~154k blocks/sTwo hard asserts are the falsifiers:
gemm_eq_brute == nblk(E-7) andmax_err == 0(bit-exact MC). If either breaks, the run panics.Why this is only zero new substrate
The pipeline reuses shipped kernels end to end: reference texture → shifted-ref + small residual (known MVs) → ME via
int8_gemm_i32(A=[1×K]current block u8, B=[K×N]candidates i8, C=[1×N]dot products; SSD argmin over‖B_n‖² − 2·C[n]) → MC = gather ref block at recovered MV + residual → classify toCellMode→ assert bit-exact. The 7-bit luma trick ([0,127]) makes each value fit both theu8andi8operands of the VNNI kernel exactly, so no centering games.Scope / what this is NOT
This proves ME runs through the shader's i8 GEMM bit-identically to direct SSD, and MC is exact gather+add. It does not parse a real
.265bitstream — that means CABAC entropy decode + inverse transform to extract the MVs/residuals, plus sub-pel interpolation and deblock/SAO. That's M2 (the multi-month decoder). Frames here are synthetic 7-bit luma with integer-translation MVs so the i8 GEMM is exact.Notes
[[example]] required-features = ["codec"]— defaultcargo testwon't try to build it against thecodec-onlyhpc::codectypes.cargo fmt --checkclean,cargo clippyclean on 1.95.0.cargo run --release --example mc_via_shader --features codec🤖 Generated with Claude Code
Generated by Claude Code
Summary by CodeRabbit