feat(dispatch): seal collective-aware MoE step execution

[fivetide]

Status — incomplete historical draft; changes requested

Head 26fe251394499721679ae6a05a221d8f4997e3ab is an incomplete G5 attempt against superseded base 0f1628241f75. Current master is 344e17a2edd893d6e323bd69189844e95bd73a9e; the G1/G2/G4 partials included by this stack were merged and then backed out by a0fca0d6d, and #675 remains an open stale-stack draft.

This head is superseded by the G5 consistent-deliverable contract in #666. The replacement must follow accepted G1, G2, and G3, close every listed blocker, and route production registry qwen3.6:35b-a3b through the sealed executor with per-position numerical/state evidence, lifecycle coverage, and maintained physical validation for every multi-device claim. Substrate-only completion is not accepted.

This head is dirty/non-rebaseable and is not mergeable or review-ready. Do not push or rebase it. Keep this PR open and draft with CHANGES_REQUESTED. Do not close until a replacement G5 PR is published.

Summary

Relative to the historical G3 baseline, this PR proposes the G5 collective-aware MoE Step changeset tracked by #666; it does not complete G5 or establish current-tree acceptance. It turns the G3 expert manifest and G1 named DeviceMesh into a sealed, manifest-derived expert ownership plan, then executes only an explicitly typed route-to-combine program. The plan owns placement, rank views, source metadata, and the one collective row; the dispatch layer owns the generic route, indexed/grouped projection, activation, combine, and mesh execution protocol.

What changed

• Added hipfire_runtime::moe_plan::ExpertPlan as the sole resolver for a declared expert group. Construction validates the manifest source entries, fused projection layout, logical shapes and dtypes, resource requirements, parallelism, assignment, named mesh group, and the ordered G3 collective schedule. It deterministically derives (global expert, owner rank, local slot) placements, rank-local owner views, mesh epoch, and the manifest-authorized post-combine collective.
• Kept expert storage authority inside the plan. Rank pointer tables are committed once after ABI/capacity checks; owner-scoped load transactions support reserve/commit/rollback and idempotent unload. Families receive only an opaque, borrowed MoeExpertRef produced by the validated plan binding—no allocator, source path, WeightStore representation, or family-side teardown authority crosses that boundary.
• Added typed routing and execution vocabulary in hipfire-dispatch: RouterPlan makes softmax-top-k versus precomputed routing explicit, while MoeRoute, IndexedMoeGemv, MoeScatter, GroupedMoeGemm, MoeGateUpUnscatter, MoeActivation, and MoeCombine carry the operands and geometry needed by each operation. The only accepted executable grammars are:
  • indexed decode: route → gate/up → activation → down (expanded) → combine
  • grouped prefill: route → scatter → gate/up → unscatter → activation → down (expanded) → combine
• Made the grammar exhaustive and fail-closed. Every routed operand must be identity-linked to the one route and one owner view; gate/up and down shapes, route width, batch geometry, inverse permutation, buffer aliasing, logical capacity, physical byte capacity, dtype, and projection protocol are checked before launch. Every down projection is expanded, and MoeCombine is the sole weighted reduction step; there is no fallback grammar that can bypass ownership or collective checks. Unsealed MoE steps are rejected by the ordinary pipeline entry point.
• Added router, tensor, and rank-signature preflight. RouterPlan::validate_against checks normalized route metadata and score/indices/weights storage; schedule validation checks all step tensors and raw grouped metadata; MoeExecutionSignature compares protocol, execution identity, router identity/selection, top-k, dtype, dimensions, batch/hidden geometry, and the canonical ownership partition across ranks. Mesh preflight additionally checks rank ordering, owner rank, named group membership, collective axis/dimension, device uniqueness, and mesh epoch entirely on the host before any device method is called.
• Added exactly-once named-group collective execution. Single-device plans emit the explicit identity (None); parallel plans require one manifest-owned all-reduce attached to MoeCombine, with dimension batch_size × hidden. execute_sealed_steps_mesh runs each rank schedule in the manifest group order and performs that one reduction after all rank-local schedules complete.
• Closed the immediate generic executor path in MoeFamily: the typed steps dispatch through the existing route, indexed GEMV, scatter, grouped GEMM, unscatter, activation, and combine kernels for the sealed single-device and mesh protocols. The new planner-to-executor boundary is exposed through seal_steps, execute_sealed, and execute_sealed_mesh; family code no longer reconstructs ownership or collective membership.

Goals

Short term

• Finish G5's generic collective-aware MoE execution contract on top of G1 topology and G3 manifest/store work.
• Refuse malformed routes, schedules, ownership bindings, tensor metadata, and rank/mesh identities before GPU work instead of allowing a hand-built family schedule to select a second policy, hidden reduction, or incorrect expert table.
• Provide one immediate executor closure for indexed decode and grouped prefill, with one explicit post-combine reduction for parallel ownership and an identity path for single-device execution.

Long term

• Establish a reusable planner-to-executor substrate: manifest and topology resolution produces a sealed expert plan; model-family adapters build one typed schedule from that plan; host preflight proves rank agreement; the generic executor launches the approved program.
• Keep expert placement, storage lifetime, source identity, collective membership, and mesh generation in shared authorities so future model-family integrations extend explicit typed contracts rather than reimplementing family-local storage or collective policy.

Benefits

• Invalid source layouts, projection shapes/dtypes, route metadata, tensor capacities, aliases, schedule order, protocol identity, ownership partitions, and mesh generations fail closed before GPU work, reducing the chance of silent wrong-basis or wrong-expert results.
• Parallel reduction membership comes from the named manifest schedule, is compared across ranks, and occurs exactly once at combine over the complete batched output; duplicate, missing, partial-dimension, or off-group reductions are refused.
• The dispatch family is no longer a storage or collective authority. It receives borrowed plan-produced views and invokes kernels, while placement, transactional residency, teardown, and collective identity remain centralized and deterministic.
• The same host-only mesh preflight is used by the executor and its focused seam tests, so rank disagreement and stale/mismatched mesh identity are checked by the same logic that guards execution.

Dependency and landing order

• The current head depends on the stale feat(runtime): add device-mesh weight manifest and store pilot #675/feat(hardware): establish named device mesh ownership #673 stack; those types are absent from current master after the rollback.
• Replacement G5 work follows accepted G1, G2, and G3 and uses current ownership APIs; this historical head is not implementation ancestry.
• Use the existing hipfire_runtime::multi_gpu::Gpus; do not add the reverted hipfire-hardware crate.
• Corrected G2 admission is a semantic prerequisite before complete production multi-device enablement. feat(loader): centralize source-aware parallel admission #674 was reverted and its classifier, VMM commit, and topology-policy defects remain unresolved on this path.
• G4 is independent. Keep this PR draft; no ready or merge action is appropriate for the current head.

Remaining blockers before ready

• Bind EP ownership to rank-local expert computation rather than metadata-only ownership, and use shard-local TP dimensions.
• Prove grouped-scatter capacity before allocation or use.
• Seal activation-basis selection to dtype and execution-signature identity.
• Define combine initialization/local contribution and prove the one all-reduce/residual contract.
• Bind logical mesh ranks to physical GPU identities and complete alias validation.
• Make unload/rollback own every published storage table and clear stale tables.
• Route the legacy execute_pipeline entry point through sealing, or remove the bypass; no execution path may bypass sealing.
• Re-run current-master checks and produce the required hardware evidence.

Verification

Historical head-only evidence; none of these results establishes current-master acceptance:

• Changed Rust files passed scoped rustfmt on head 26fe251394499721679ae6a05a221d8f4997e3ab.
• Isolated cargo build passed on the displaced G1 → G3 → G5 stack.
• Scoped clippy and isolated cargo test passed on that historical stack (3,509 passed, 12 ignored).
• Six GitHub check-runs passed on the historical head.
• Required gates (ratchets, layering, registers) check failed on the historical head.
• Fresh current-master branch build, tests, formatting, lint, and repository gates.
• Real hardware registry runs: at minimum hipfire run qwen3.6:27b and one A3B, with decoded text inspected and recorded.
• Physical multi-device collective execution.
• python3 -m tools.change_gate run --base beta was not run; fork tracker output is not upstream acceptance evidence.
• Architecture-trait change? No Architecture trait surface change on the historical head; re-confirm on the re-land.

Nonclaims / follow-up

• This PR does not claim physical multi-device collective execution; that evidence remains outstanding.
• This PR does not enable a complete production multi-device route ahead of feat(loader): centralize source-aware parallel admission #674 (G2 admission), and it does not claim model-family/axis integration beyond the generic G5 planner and executor seam.
• The change-gate command remains unrun and is not represented as passing by this body.

Wall time: 0.70 seconds
No production multi-device enablement, physical collective, model-family/axis integration, or G5 completion claim is made by this historical head.

[Kaden-Schutt]

Blocking architectural issues remain in the G5 execution path:

• EP ownership is metadata-only: every rank still computes every expert.
• TP shards retain full dimensions rather than shard-local shapes.
• Grouped scatter capacity is not proven and can overrun its allocation.
• Activation-basis selection is not sealed to dtype/signature identity.
• Combine uses additive output without a proven zero/local contribution, and the all-reduce/residual contract is wrong.
• Mesh logical ranks are not bound to physical GPU identity.
• Alias validation is incomplete.
• Unload/rollback does not own all published storage and can leave stale tables.
• The legacy execute_pipeline entry point bypasses sealing.

These are correctness and ownership blockers, not polish. Keep this draft until the execution/storage model is made fail-closed and validated end to end.