M3 Ultra (512GB, Mac15,14) benchmark results — 512ch hard constraint

[pudepiedj]

Hardware: Apple M3 Ultra, 512 GB unified memory, Mac Studio (Mac15,14)
macOS: 15.7.4 (24G517)
Commit: efcf193
Clang: Apple clang 17.0.0

system_info.txt

Key finding: 512ch is the only valid channel count

Every channel configuration except exactly 512 fails with -4 (invalid config)
or -3 (compile failure). This is not "minimum 512" — it is exactly 512.

inmem_bench

inmem_bench.log

=== In-Memory ANE Benchmark ===
Config W (MB) ms/eval TFLOPS
256ch x64sp 0.1 FAIL(-4)
512ch x64sp 0.5 0.189 ms 0.18
1024ch x64sp 2.0 FAIL(-4)
2048ch x64sp 8.0 FAIL(-4)
3072ch x64sp 18.0 FAIL(-4)
4096ch x64sp 32.0 FAIL(-4)

inmem_peak

inmem_peak.log

=== Programmatic MIL → In-Memory ANE Peak ===
Config W(MB) GFLOP ms/eval TFLOPS
32x conv 512ch sp64 16.0 1.07 0.281 ms 3.82
48x conv 512ch sp64 24.0 1.61 0.315 ms 5.11
64x conv 512ch sp64 32.0 2.15 0.313 ms 6.86
96x conv 512ch sp64 48.0 3.22 0.380 ms 8.47
128x conv 512ch sp64 64.0 4.29 0.490 ms 8.77
64x conv 256ch sp64 8.0 0.54 FAIL(-3)
128x conv 256ch sp64 16.0 1.07 FAIL(-3)
256x conv 256ch sp64 32.0 2.15 FAIL(-3)
64x conv 384ch sp64 18.0 1.21 FAIL(-3)
128x conv 384ch sp64 36.0 2.42 FAIL(-3)

Peak sustained: 8.77 TFLOPS at 128x conv depth.

sram_probe

sram_probe.log

=== ANE SRAM Fine Probe (weights only vary, spatial=64) ===
Channels W (MB) ms/eval TFLOPS GFLOPS/MB
512 ch 0.5 0.173 ms 0.19 388.8
1024 ch 2.0 -4.000 ms 0.00
(all others: FAIL -4)

SRAM cliff is unmeasurable — the channel sweep is blocked by the 512ch
constraint. The benchmark's "spilling?" annotations are incorrect; these are
hard rejections, not DRAM spills.

sram_bench

Same pattern — only 512ch succeeds. No SRAM cliff data obtainable with
current benchmark design on this hardware.

sram_bench.log

Open questions

1. Whether both ANE dies (UltraFusion) are being addressed — 8.77 TFLOPS is
   consistent with a single M3 Max ANE die. If only one die is active, true
   dual-die peak may be ~17+ TFLOPS but currently unreachable(?) via _ANEClient.
2. Why exactly 512ch and no other value — possibly a UltraFusion tensor
   partitioning requirement.
3. SRAM size remains unknown for M3 Ultra ANE — would require a benchmark
   variant that sweeps spatial dimension or weight size while holding channels
   fixed at 512.

The %peak column in inmem_peak seems nonsensical (shows 46000%+) — denominator
appears hardcoded to M4 TFLOPS.

Happy to run additional experiments if the benchmark suite is extended to
handle the 512ch constraint.

[pudepiedj]

Pre-M4 Hardware Findings: M2 Max [& M3 Ultra - partial]

Hardware tested:

• Apple M2 Max (MacBook Pro), macOS 15.x — benchmarks + training
• Apple M3 Ultra 512GB (Mac Studio, Mac15,14), macOS 15.7.4 — benchmarks only

Repo commit: efcf1930752c8e3d62dbe339976885f6551b2934

---

1. Channel Constraint: Exactly 512ch Required (Both Machines)

On both M2 Max and M3 Ultra, the ANE rejects all channel configurations except exactly 512 via the direct _ANEClient API path. This was verified across all four benchmark tools on both machines.

inmem_bench (M3 Ultra)

 256ch x64sp     0.1  FAIL(-4)
 512ch x64sp     0.5     0.189 ms    0.18
1024ch x64sp     2.0  FAIL(-4)
2048ch x64sp     8.0  FAIL(-4)
3072ch x64sp    18.0  FAIL(-4)
4096ch x64sp    32.0  FAIL(-4)

inmem_bench (M2 Max)

 256ch x64sp     0.1  FAIL(-4)
 512ch x64sp     0.5     0.179 ms    0.19
1024ch x64sp     2.0  FAIL(-4)

The M2 Max result rules out UltraFusion as the cause. The 512ch constraint is a generational requirement of M2/M3 ANE architecture. M4 is the first generation to accept 256ch.

inmem_peak (M3 Ultra)

128x conv 512ch sp64    64.0    4.29    0.490 ms   8.77  [peak]
64x conv 256ch sp64      8.0    0.54  FAIL(-3)
128x conv 256ch sp64    16.0    1.07  FAIL(-3)
64x conv 384ch sp64     18.0    1.21  FAIL(-3)

Peak sustained: 8.77 TFLOPS at 128x conv depth on M3 Ultra. Consistent with single-die access — the second ANE die (UltraFusion) appears unreachable via the direct _ANEClient path.

SRAM cliff

Unmeasurable on both machines. The benchmark sweeps channel counts to find the cliff, but only 512ch succeeds — the sweep is blocked. SRAM size for M2/M3 ANE remains unknown via this approach. A variant sweeping spatial dimension or weight size at fixed 512ch would be needed.

---

2. Training Fixes: M2 Max Only

The following training fixes were developed and validated on M2 Max. They have not been tested on M3 Ultra, though the shared 512ch constraint suggests similar behaviour is likely.

2a. SEQ=512 Stabilises Compilation

The training code (train_large.m) uses DIM=768 and HIDDEN=2048 as its internal channel dimensions. With the default SEQ=256, compilation behaviour is unstable on M2 Max — the run fails early with InvalidMILProgram.

Fix: Set SEQ=512 in stories_config.h:

#define SEQ 512  // was 256

With SEQ=512, compile count stabilises at exactly 72 per batch (60 weight-bearing + 12 static sdpaBwd2) and remains constant throughout. The exec() restart fires cleanly every 10 steps.

2b. Gradient Explosion: Missing Gradient Clipping

With SEQ=512, the larger attention matrices (SEQ² scaling) produce occasional catastrophically large gradients. Without clipping, training explodes at around step 130 — loss jumps from ~3.9 to 10.37 and locks permanently, followed by InvalidMILProgram as the ANE rejects programs built from corrupted weights.

This was not observed on M4 with SEQ=256, which is why clipping was absent from the original code.

Fix: Add gradient clipping before the Adam update in train_large.m, after the gsc scaling loops (around line 636):

// Gradient clipping (max norm = 1.0)
float grad_norm_sq = 0.0f;
for (int L=0; L<NLAYERS; L++) {
    LayerGrads *g = &grads[L];
    for(size_t i=0;i<WQ_SZ;i++) grad_norm_sq += g->Wq[i]*g->Wq[i] + g->Wk[i]*g->Wk[i] + g->Wv[i]*g->Wv[i] + g->Wo[i]*g->Wo[i];
    for(size_t i=0;i<W1_SZ;i++) grad_norm_sq += g->W1[i]*g->W1[i];
    for(size_t i=0;i<W2_SZ;i++) grad_norm_sq += g->W2[i]*g->W2[i];
    for(size_t i=0;i<W3_SZ;i++) grad_norm_sq += g->W3[i]*g->W3[i];
    for(int i=0;i<DIM;i++) grad_norm_sq += g->rms_att[i]*g->rms_att[i] + g->rms_ffn[i]*g->rms_ffn[i];
}
for(int i=0;i<DIM;i++) grad_norm_sq += grms_final[i]*grms_final[i];
for(size_t i=0;i<(size_t)VOCAB*DIM;i++) grad_norm_sq += gembed[i]*gembed[i];
float grad_norm = sqrtf(grad_norm_sq);
float max_norm = 1.0f;
if (grad_norm > max_norm) {
    float clip_scale = max_norm / grad_norm;
    for (int L=0; L<NLAYERS; L++) {
        LayerGrads *g = &grads[L];
        for(size_t i=0;i<WQ_SZ;i++){g->Wq[i]*=clip_scale;g->Wk[i]*=clip_scale;g->Wv[i]*=clip_scale;g->Wo[i]*=clip_scale;}
        for(size_t i=0;i<W1_SZ;i++) g->W1[i]*=clip_scale;
        for(size_t i=0;i<W2_SZ;i++) g->W2[i]*=clip_scale;
        for(size_t i=0;i<W3_SZ;i++) g->W3[i]*=clip_scale;
        for(int i=0;i<DIM;i++){g->rms_att[i]*=clip_scale; g->rms_ffn[i]*=clip_scale;}
    }
    for(int i=0;i<DIM;i++) grms_final[i]*=clip_scale;
    for(size_t i=0;i<(size_t)VOCAB*DIM;i++) gembed[i]*=clip_scale;
    fprintf(stderr, "{\"type\":\"clip\",\"step\":%d,\"grad_norm\":%.4f}\n", total_steps_done, grad_norm);
}

2c. Learning Rate: Reduce to 5e-5 for SEQ=512

The default LR of 3e-4 produces gradient norms in the hundreds to thousands even with clipping applied, indicating a fundamentally unstable training regime at this sequence length. Clipping alone is insufficient.

Fix: Reduce learning rate to 5e-5. Training then proceeds stably with gradient norms remaining bounded throughout.

---

3. Confirmed Training Result: 500 Steps on M2 Max

With all three fixes applied (SEQ=512, gradient clipping, LR=5e-5):

=== Efficiency Report ===
Total steps:     500
Wall time:       433173 ms (433.2 s)
Compile time:    259243 ms (59.8%)
Train time:      119116 ms (27.5%)
Avg train:       238.2 ms/step
ANE TFLOPS:      0.83 sustained
Total TFLOPS:    1.51 (ANE+CPU)
ANE utilization: 5.3% of 15.8 TFLOPS *

* ANE utilisation uses M4 denominator. Against M2 Max ANE (~11 TFLOPS) the real figure is approximately 7.5%.

Loss trajectory: 4.40 → 3.19 over 500 steps, minimum 2.91.

Implied potential without compile overhead: eliminating the 59.8% compile time would bring effective throughput to approximately 65ms/step — the hardware's real capability at this configuration.

---

4. Summary of Fixes for Pre-M4 Hardware

Issue	Scope	Cause	Fix
Benchmark FAIL(-4)	M2 Max + M3 Ultra	M2/M3 ANE requires exactly 512ch	Pass 512 as channel arg to all benchmarks
Unstable compilation in training	M2 Max (M3 Ultra untested)	SEQ=256 produces inconsistent tensor shapes	Set SEQ=512 in stories_config.h
Gradient explosion ~step 130	M2 Max (M3 Ultra untested)	SEQ=512 doubles attention matrix size, producing large gradient outliers	Add gradient clipping (max_norm=1.0) before Adam update
Explosion persists despite clipping	M2 Max (M3 Ultra untested)	LR=3e-4 too high for SEQ=512	Reduce LR to 5e-5

---

Findings from pudepiedj, March 2026.
Hardware: M2 Max MacBook Pro + M3 Ultra Mac Studio (512GB, Mac15,14).