refactor(tensor,arithmetic)!: O(chunk) conversion/accumulate scratch (#296 Stage 2)

[LeoBuron]

Stacked on #334 (Stage 1) — merge that first; this PR then retargets to develop. Note the known stacked-PR caveat: CI is vacuous until retarget (ci.yml triggers on main/develop targets only); local verification below stands in until then, and the workflow run must be verified after retarget.

Stage 2 of the bounded-scratch design (#296; spec docs/superpowers/specs/2026-07-09-bounded-scratch-design.md). Executed via SDD: five task reviews + final whole-branch review (verdict: merge-ready after one fix wave, applied).

What changes

Every element-local conversion/accumulate path streams in fixed 256-element chunks (ODT_CONVERSION_CHUNK_ELEMS, multiple of 8 so packed chunk starts stay byte-aligned for any qBits):

• Chunk infrastructure: packChunkGuarded (fit-guard → abort-on-overflow, same message/exit; the old check-all-before-write existed only for an unrecoverable exit(1)), unpack{Sign,Zero}ExtendChunk, public dequantChunkToFloat, quantizeFloatToAsym split into grid-derivation + chunked emit.
• All ~30 TensorConversion VLA sites rewritten: pack family, four two-pass derived-grid cross-cells, nine unpack/dequant cells. TensorConversion.c compiles clean under -Werror=vla.
• Streamed accumulate engines (FixedGrid / Rescale / AsymRescale; float* signatures unchanged) + tensor-typed epilogue entry points; accumulateSymInt32IntoSymInt32Rescale reproduces Strategy A bit-for-bit — pinned against the live Add.c implementation in-process (testAccDynamicSymIntoSymBitEqualsAddSymInplace). executeOp's Phase-4 staging VLAs and incrementAsFloatView are gone.
• Documented residual (spec §5): the float→SYM_INT32 dynamic-rescale arm keeps whole-tensor staging byte-identically — its two sequential rounding blocks cannot be chunked without changing SR streams; it is reached only by SYM_INT32-stored grads (optimizer: gradients should not be stored as SYM_INT32 (compute-format ≠ storage-format) #261-discouraged, unused by every example).

Bit-identity

The acceptance bar for every rewrite: same per-element expressions, grid scans never round, exactly one roundByMode per element in element order (SR-RNG streams unchanged). Sanctioned deltas only: n==0 no-ops (old was VLA-UB), abort-on-overflow timing, the two watermark fixture values. Whole-branch review verified the epilogue routing table combination-by-combination (no arm lost or silently changed) and chunk-size contracts at every internal call site.

Measured effect (HAR, EPOCHS=1, macOS arm64)

config	pre-#296	after Stage 1 (#334)	after Stage 2
float stack	99,560 B	27,328 B	27,768 B (+440 B fixed chunk-buffer frames)
sym8 stack	149,008 B	100,080 B	51,968 B (repack chain eliminated)

Total provisioned training RAM (heap categories + stack): sym8 ≈ 201.6 KiB now beats float ≈ 207.9 KiB — the #322 inversion is resolved at HAR scale; sym8's total dropped 32 % vs pre-#296. Remaining stack floor = one unpacked operand row + rawData at the largest tensor (Option-4/chunked-kernels territory, documented residual).

Verification

• ctest --preset unit_test_debug 72/72; uv run pytest 29/29; examples rot-guard builds all targets.
• Strict TDD: pin-then-refactor per task, every task with a recorded mutation kill (incl. the fix-wave's chunk-crossing tensor-increment pin: hardcoded-offset mutant → scale off by 2000×, RED).
• Watermark budgets recalibrated with full provenance history; check script passes both fresh logs.

Closes #296 (manual close after develop merge; closing comment will record the residual table: funnel operand row + rawData, float→SYM_INT32 arm, kernel/layer VLAs, VLA-aware alloc-locality check → Option-4 follow-ups).

🤖 Generated with Claude Code