Cluster native Swift: serialized evaluation and quantized Metal repairs

[jonathan308]

Summary

Make the cluster's native Swift path safe when Swift evaluation and a raw C collective completion share Metal streams. The Execution SPI exposes a synchronous scope using the existing recursive evaluation lock. Refresh core to 9e22e1998a3b6f7f6f4ad58dc2038ecf9f4d0cb9 and regenerate all four quantized JIT resources for partial-tile bounds and distinct offline Metal loader symbols. The cached device discovery and sealed all-sum surface remain included. Add GLMCalibration SPI for install-once Metal backing caps and snapshots, with the matching C bridge pinned at c4154afc4ef5d22cea324391fb7052d1fc68ce5f. The cap reserves pending backing before allocation and refunds only after actual Metal deallocation; it does not cap host or whole-process memory.

Before

flowchart LR
  S["Swift evaluation"] --> M["Shared Metal command buffer"]
  C["Raw C collective completion"] --> M
  M --> R["Commit / completion-handler race"]
  K["Core kernel fixes"] --> G["Stale embedded JIT text"]
  A["Offline library"] --> N["Colliding loader symbols / NaNs"]
  B["Bounded calibration"] --> U["No allocation-enforced backing cap"]
  classDef default fill:#fff1f2,stroke:#be123c,color:#881337

Loading

After

flowchart LR
  S["Swift evaluation"] --> L["Existing recursive eval lock"]
  C["Execution SPI: withSerializedEvaluation"] --> L
  L --> M["Serialized native eval and stream fences"]
  K["Pinned core"] --> G["Four regenerated JIT resources"] --> V["Exact source-consistency gate"]
  A["Distinct quantized loader symbols"] --> O["Correct offline Metal outputs"]
  B["GLMCalibration SPI"] --> P["Install once before buffer attempts"] --> CAP["Core backing cap and coherent snapshots"]
  classDef default fill:#ecfdf5,stroke:#047857,color:#064e3b

Loading

The SPI scope is synchronous; callers must not suspend or wait for work submitted on another thread while holding it. It does not make arbitrary concurrent graph construction safe. Sealed all-sum remains stacked on #54; mlx-c moves to c4154afc4ef5d22cea324391fb7052d1fc68ce5f, adding the cap API atop the sealed all-sum branch. Its target base is included; the separate fork-main delta remains documentation/CI only. Current Swift main 9ad4718 and target base 64070c4 are integrated.

Evidence

• Current combined composition, M3: 12 standalone Metal cap cases, 1 fresh-process Swift cap test, and the full native suite 931 passed, 2 expected skips, 0 failed in 74.75 s. Full-suite skips are the separately executed fresh-process cap test and an opt-in performance test. All 192 actual Cmlx objects are linked; source/binary/AOT identities and all four embedded kernel strings were verified.
• M5 cap production passed 12 cases and its prior full suite 930 passed, 3 expected skips, 0 failed before the CPU-only core delta. This is affected-scope evidence, not an exact current M5 combined-binary qualification.
• Core CPU regression qualification: 292 C++ passed; 815 Python passed, 62 declared skips, 0 failed. Exact-value controls reproduce and repair low-precision partial-sum rounding without changing products, packing, GPU kernels or tolerances.
• sh tools/check-process-allocation-cap.sh passes the real VM/Blocks ledger and C bridge checks plus backend/header/Swift syntax checks. Pinned formatting passes. Current fork-diff coverage and upstream merge-base gate: 245/245. Independent allocator and CPU reviews found no outstanding scoped blocker.

Earlier kernel/execution evidence below is retained separately.

• Original mixed C/Swift execution reproduced a Metal addCompletedHandler-after-commit abort. The shared-lock integration passed the seven pipeline cases and five fresh-process greedy controls; the dedicated native regression exercises recursive access and 128 operations per competing queue.
• Exact quantized regression: 48/144 cases failed before the tail fix; 272 final geometry cases pass across affine packing widths/groups, MXFP4/MXFP8/NVFP4, partial tiles and adjacent NaNs. The matching full AOT library also passes 39 Python quantization tests and its 128-case regression; the old library fails 32 subcases.
• Final M3 native suite: 931 passed, 1 skipped, 0 failed; the published predecessor reproduced 26 NVFP4 partial-K assertion failures. Final M5 native suite: 930 passed, 2 skipped, 0 failed (932 reported; 90.599 s including both test bundles), with expert kernels enabled and resource limit 50,000. Skips are the explicit performance opt-in and the retract test requiring a non-NAX device. Prior OS-default allocation stress also passed, but is separate earlier-composition evidence.
• All four complete embedded kernel strings are present byte-for-byte in the tested native executable. Generated-source gate: 4 copied headers + 4 JIT resources pass; all 11 mutation tests pass. CI now checks them before building. The optional whole-tree audit still reports two pre-existing absolute-toolchain dependencies in unrelated NAX resources; this scoped gate does not hide or certify them.
• Coverage reporting now accepts both SwiftBuild per-target and native SwiftPM combined test bundles, and refuses a missing binary/profile. The old path failed against the actual SwiftBuild output; the repaired complete report step passes (91.72% instrumented Swift line coverage). This does not claim the remote CI failure was this later step.
• Independent reviews found no remaining scoped correctness issue or useful behavior-preserving refactor. Fork-diff: 241/241 changed files described. Swift-format reports only three existing vector_array naming warnings, with no formatting errors.
• Real GLM RDMA requests, installed endpoints and final provider lifecycle remain separate qualification gates. This is not a production-readiness claim.

Merge Danger

Door: two-way. Restore the consumer pin and matching native resources together.

Blast Radius: native. Shared Metal execution ordering and quantized kernels affect SDK consumers. The cap is optional and process-local, cannot be raised/reset, and includes fixed heap and pending allocations. It is not a host/driver/whole-process limit and grants no model qualification. Ordinary residency-registration failures now clean up their buffer ownership. Protected merges, release signing and deployment are separate actions.