feat(hardware): establish named device mesh ownership - #673
Merged
Kaden-Schutt merged 7 commits intoSep 1, 2026
Conversation
added 5 commits
September 1, 2026 13:37
This was referenced Sep 1, 2026
Kaden-Schutt
approved these changes
Sep 1, 2026
Collaborator
|
Note for anyone landing here: this PR shows MERGED, but its content was backed out of |
This was referenced Sep 2, 2026
Kaden-Schutt
added a commit
that referenced
this pull request
Sep 2, 2026
G1 DeviceMesh as a type inside hipfire_runtime::multi_gpu (fixed re-land of #673, no crate move). hw-gate dry run (gate at feat/hw-gate 8729d6f, seats under fake gh): Sol authorized hardware; both lanes (hiptrx gfx1201, hipx gfx1100) passed qwen3.6:27b, ornith-1.5:35b-a3b-mq4r, lfm2.5:1.2b, qwen3.8:27b-mq4-xt battery; Sol needs-human (PP/TP not exercised); Fable merge-staging overriding Sol: Gpus.mesh has zero production readers, single-device load is the only regressable surface and passed on both architectures.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Establish G1's topology authority for multi-device execution. The sole multi-device owner moves from
hipfire-runtime::multi_gpuinto the newhipfire-hardwareleaf, while a pure, typedDeviceMeshdefines named PP/TP/EP placement, mesh-generation identity, stage boundaries, and collective groups.This is the root changeset for the topology work tracked by #666. It supplies the owner and topology contracts that later admission, manifest/store, and MoE work can consume without introducing another ownership or rank-policy implementation.
What changed
hipfire-hardwareworkspace crate and movedcrates/hipfire-runtime/src/multi_gpu.rstocrates/hipfire-hardware/src/lib.rs(an 89% rename).Gpusremains the single owner of resolved device IDs, VRAM/architecture preflight, peer access, boundary copies, RCCL/host/peer reductions, and peer-reduction scratch leases. The runtime module and compatibility re-export were removed; the ownership integration test asserts that the legacy owner path is absent.DeviceMeshmodule.DimKind::{Pp, Tp, Ep}names the parallelism axes; rectangular shapes use row-major coordinates with checked cardinality,group_alongderives axis subgroups,stage_for_layerandstage_devicesdescribe PP bands, andband_xfer_afteremits the PP boundary hint.CollectiveHintdistinguishes axis all-reduces from pipeline band transfers.MeshEpochas identity for an admitted mesh generation. Newsingle/rectmeshes receive fresh epochs; clone andsqueezedpreserve the epoch, so shape-equivalent meshes are not silently treated as the same generation.RuntimeConfig::device_resolve_optslowers the already-resolved logical visibility IDs and hardware knobs into the leaf; hardware consumes those values instead of rereading process environment.init_uniform,init_layers,init_tp, andsinglenow use that explicit construction input.run_layer_program_epand its reduction helpers receive an orderedgroup; RCCLall_reduce_sum_f32validates the full ordered owner group, whileall_reduce_sum_f32_peervalidates unique global device IDs and is the genuine subgroup-capable path for composed TP×EP layouts. Current architecture callers pass the ordered full-device group and all affected DeepSeek4, MiniMax, Qwen35, generator, loader, runtime, and example call sites now usehipfire_hardware::Gpus.Goals
Short term
Make
hipfire-hardwarethe one topology/ownership authority now: resolve the visible logical devices once, expose named PP/TP/EP placement primitives, and require collective callers to state their participating group explicitly. Preserve the existing PP, TP, and EP routes while removing the duplicate runtime owner and its compatibility surface.Long term
Provide a composable device-mesh substrate for the next admission and execution layers. #674 admission, #675 manifest/store, and #677 MoE execution should be able to consume typed coordinates, axis subgroups, stage transfers, and mesh epochs rather than reimplementing device ownership, rank arithmetic, or environment policy in each feature.
Benefits
Gpusowner and one resolved hardware-policy boundary, so runtime, loader, architecture, and generated callers cannot drift onto parallel ownership implementations.ncclCommSplitseam visible rather than implicit.Dependency and landing order
Verification
cargo buildpasses on the final rebased head.cargo clippy -p hipfire-hardware --all-targets --no-deps -- -D warningspasses for the changed package; dependency warnings remain pre-existing.cargo testpasses: 3,454 passed, 12 ignored.python3 -m tools.change_gate run --base betawas not run.Architecturetrait surface change.Nonclaims / follow-up
all_reduce_sum_f32remains full-group only); use the explicit peer subgroup API until the later communicator-split work lands.