Skip to content

feat(hardware): establish named device mesh ownership - #673

Merged
Kaden-Schutt merged 7 commits into
warpfront:masterfrom
fivetide:feat/device-mesh-g1-topology
Sep 1, 2026
Merged

feat(hardware): establish named device mesh ownership#673
Kaden-Schutt merged 7 commits into
warpfront:masterfrom
fivetide:feat/device-mesh-g1-topology

Conversation

@fivetide

@fivetide fivetide commented Sep 1, 2026

Copy link
Copy Markdown
Collaborator

Summary

Establish G1's topology authority for multi-device execution. The sole multi-device owner moves from hipfire-runtime::multi_gpu into the new hipfire-hardware leaf, while a pure, typed DeviceMesh defines named PP/TP/EP placement, mesh-generation identity, stage boundaries, and collective groups.

This is the root changeset for the topology work tracked by #666. It supplies the owner and topology contracts that later admission, manifest/store, and MoE work can consume without introducing another ownership or rank-policy implementation.

What changed

  • Added the hipfire-hardware workspace crate and moved crates/hipfire-runtime/src/multi_gpu.rs to crates/hipfire-hardware/src/lib.rs (an 89% rename). Gpus remains the single owner of resolved device IDs, VRAM/architecture preflight, peer access, boundary copies, RCCL/host/peer reductions, and peer-reduction scratch leases. The runtime module and compatibility re-export were removed; the ownership integration test asserts that the legacy owner path is absent.
  • Added the pure DeviceMesh module. DimKind::{Pp, Tp, Ep} names the parallelism axes; rectangular shapes use row-major coordinates with checked cardinality, group_along derives axis subgroups, stage_for_layer and stage_devices describe PP bands, and band_xfer_after emits the PP boundary hint. CollectiveHint distinguishes axis all-reduces from pipeline band transfers.
  • Added MeshEpoch as identity for an admitted mesh generation. New single/rect meshes receive fresh epochs; clone and squeezed preserve the epoch, so shape-equivalent meshes are not silently treated as the same generation.
  • Kept topology independent of GPU handles, loading, allocation, and carrier policy. RuntimeConfig::device_resolve_opts lowers the already-resolved logical visibility IDs and hardware knobs into the leaf; hardware consumes those values instead of rereading process environment. init_uniform, init_layers, init_tp, and single now use that explicit construction input.
  • Made collective membership explicit at the EP seam. run_layer_program_ep and its reduction helpers receive an ordered group; RCCL all_reduce_sum_f32 validates the full ordered owner group, while all_reduce_sum_f32_peer validates unique global device IDs and is the genuine subgroup-capable path for composed TP×EP layouts. Current architecture callers pass the ordered full-device group and all affected DeepSeek4, MiniMax, Qwen35, generator, loader, runtime, and example call sites now use hipfire_hardware::Gpus.
  • Added mesh unit coverage and hardware-leaf ownership coverage for composed coordinates, PP boundary hints, epoch preservation, overflow rejection, owner exposure, and removal of the runtime owner path.

Goals

Short term

Make hipfire-hardware the one topology/ownership authority now: resolve the visible logical devices once, expose named PP/TP/EP placement primitives, and require collective callers to state their participating group explicitly. Preserve the existing PP, TP, and EP routes while removing the duplicate runtime owner and its compatibility surface.

Long term

Provide a composable device-mesh substrate for the next admission and execution layers. #674 admission, #675 manifest/store, and #677 MoE execution should be able to consume typed coordinates, axis subgroups, stage transfers, and mesh epochs rather than reimplementing device ownership, rank arithmetic, or environment policy in each feature.

Benefits

  • There is one multi-device Gpus owner and one resolved hardware-policy boundary, so runtime, loader, architecture, and generated callers cannot drift onto parallel ownership implementations.
  • PP×TP×EP placement is represented as named axes and validated coordinates/groups instead of scattered positional rank math. The same substrate can derive a TP or EP subgroup from a composed mesh and retain identity across shape-preserving operations.
  • Collective routing is explicit at the API boundary: full-owner RCCL reductions are distinct from direct peer reductions that can serve real subgroups, making the current limitation and the future ncclCommSplit seam visible rather than implicit.
  • Later admission, manifest, and MoE work can be typed against topology data and mesh generations, reducing policy duplication and making ownership/placement errors fail at the hardware API boundary.

Dependency and landing order

Verification

  • Changed Rust files pass scoped rustfmt.
  • cargo build passes on the final rebased head.
  • cargo clippy -p hipfire-hardware --all-targets --no-deps -- -D warnings passes for the changed package; dependency warnings remain pre-existing.
  • cargo test passes: 3,454 passed, 12 ignored.
  • Physical multi-device route validation was not run in this changeset.
  • python3 -m tools.change_gate run --base beta was not run.
  • No Architecture trait surface change.

Nonclaims / follow-up

@Kaden-Schutt
Kaden-Schutt merged commit fc7dec9 into warpfront:master Sep 1, 2026
7 checks passed
@Kaden-Schutt

Copy link
Copy Markdown
Collaborator

Note for anyone landing here: this PR shows MERGED, but its content was backed out of master the same evening in a0fca0d6d after the G2 admission layer refused every Qwen3.5-family registry model on hardware. The merged state is preserved at tag archive/device-mesh-merged-20260901 (541b33c33). Full account, both reproductions, and the re-land path: #666 (comment)

Kaden-Schutt added a commit that referenced this pull request Sep 2, 2026
G1 DeviceMesh as a type inside hipfire_runtime::multi_gpu (fixed re-land of #673, no crate move).

hw-gate dry run (gate at feat/hw-gate 8729d6f, seats under fake gh):
Sol authorized hardware; both lanes (hiptrx gfx1201, hipx gfx1100) passed
qwen3.6:27b, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b, qwen3.8:27b-mq4-xt
battery; Sol needs-human (PP/TP not exercised); Fable merge-staging
overriding Sol: Gpus.mesh has zero production readers, single-device load
is the only regressable surface and passed on both architectures.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants