feat(gpu): report a pgpu card's device profile, and build a node's cards concurrently - #661
Merged
github-actions[bot] merged 2 commits intoSep 14, 2026
Merged
Conversation
This was referenced Sep 14, 2026
raven-pan
approved these changes
Sep 14, 2026
A pgpu card's Cyborg device profile is the string a flavor's accel:device_profile has to name, and it was reachable only through a root shell. sriovVgpu and migBackedVgpu cards already carry their scheduling identifier in the response -- the PCI alias, per profile -- so this closes the gap for the one resource type that had none (#818). Fetched once per listing, not per card: hex_sdk answers for the whole node in a single Openstack round trip, which costs over a second on a real node. A node with no pgpu card makes no call at all, which is the common case. Three decisions worth stating: - A card whose profile has not been created reports null, not the name it would be given. A name that does not resolve sends the operator to NoValidHost with nothing to go on; null at least says "not yet". - A failed lookup also reports null and does not mark the card degraded. Degraded means the card's capacity numbers cannot be trusted, and a missing convenience string is not that. - hex_sdk fails rather than reporting an empty map when Cyborg is unreachable, so "no profiles" and "could not ask" stay distinguishable here (gpu_vgpu_profile_list conflated them and that was #1247). restoreGpuSeams now stubs the new seam by default, so tests that list a node holding a pgpu card do not shell out to a real hex_sdk. Refs #818 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Seki Xu <seki.xu@bigstack.co>
Listing a four-card node took 10.5s, and the time was not in the network, the proxy or Go: each card costs one `hex_sdk gpu_vgpu_profile_list`, and that cost is hex_sdk's own startup -- fork, then source every module -- rather than the query. Four cards ran four startups back to back. Building the cards concurrently measured 12.58s -> 4.11s on cn13 (3.06x), with the two responses byte-for-byte identical. Safe because nothing is shared for writing: Servers, NvidiaSmiDevices and VgpuInstances are prefetched before the loop and read-only after it (and VgpuInstances is already sliced per card), the enrichment state is created inside buildLocalGpuCard per card, buildGpuCardLinks is string assembly, the Mongo read is goroutine-safe, and the hex_sdk call writes only its own mktemp file. Two behaviours the concurrency must not change, both now pinned by tests: - Order. Results are written back by index into a fixed-length slice, never appended. The response order is the sorted PCI address order, and the UI renders cards in the order it receives them -- appending as goroutines finish would make the list jump around between requests. - The failure contract. One card failing to build still fails the whole listing rather than returning a short list. A card that merely lacks enrichment is not a failure; it comes back with Degraded, which stays per card and is never collapsed node-wide. Refs #818 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Seki Xu <seki.xu@bigstack.co>
SekiXu
force-pushed
the
feat/818-pgpu-device-profile-api-wip
branch
from
September 14, 2026 14:29
57cd208 to
1e0712f
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What type of PR is this?
Feature + perf. Two independent commits, reviewable separately.
Which issue(s) this PR fixes?
Refs bigstack-oss/cubecos#818
Deliberately
Refs, notFixes: #818 is a User Story whose UI (#876) and API(#886) halves are still open, so merging this must not close it.
Merge order — bigstack-oss/cube-cos-openapi#113 is already merged, and the third
commit here moves
api/cube-cos-openapionto that SHA (9495246, ondevelop). CIregenerates
api/docs.jsonfrom the submodule, so the field only reaches the served docsonce that pointer moves. The hex side
that actually produces the value is bigstack-oss/cubecos#1487, which can land in any
order relative to this one (the field simply reads
nulluntil it does).What this PR does?
1.
feat(gpu): report a pgpu card's device profile on the cardGET /nodes/{nodeName}/gpuCardsnow reportsdeviceProfileon every card. Fora pgpu card that string is what a flavor's
accel:device_profilehas to name;without it the operator needed a root shell, which the parent epic rules out.
sriovVgpu/migBackedVgpualready carry their PCI alias, so they staynull.Fetched once per listing, not per card — hex_sdk answers for the whole node
in one Openstack round trip (~1.2s on a real node), and a node with no pgpu card
makes no call at all, which is the common case.
Three decisions worth the reviewer's attention:
null, not the name it wouldbe given. A name that does not resolve sends the operator to
NoValidHostwith nothing to go on.
nulland does not mark the card degraded.degradedmeans the capacity numbers cannot be trusted; a missingconvenience string is not that.
so "no profiles" and "could not ask" stay distinguishable here. Conflating
them is what #1247 was.
2.
perf(gpu): build a node's gpu cards concurrentlyCarried in this PR on purpose — it rewrites the same loop, needs the same
build, and was verified on the same node in the same session. Splitting it
would mean two builds and two rounds of hardware verification of one function.
Listing a four-card node cost one
hex_sdk gpu_vgpu_profile_listper card, andthat cost is hex_sdk's own startup (fork, then source every module), not the
query. Four cards ran four startups back to back.
Nothing is shared for writing:
Servers,NvidiaSmiDevicesandVgpuInstancesare prefetched before the loop and read-only after it,enrichment state is created per card inside
buildLocalGpuCard, the Mongo readis goroutine-safe, and the hex_sdk call writes only its own
mktempfile.Two behaviours the concurrency must not change, both now pinned by tests:
appended. Response order is sorted PCI address order and the UI renders in
receipt order — appending as goroutines finish would make the list jump
between requests.
listing rather than returning a short list. A card that merely lacks
enrichment is not a failure: it comes back
degraded, per card, nevercollapsed node-wide.
Test results
1). api docs —
deviceProfileis added in bigstack-oss/cube-cos-openapi#113(
string | null, inrequired); the submodule here points at it.2). the api works properly — end-to-end on cn13 (4 cards, 5 runs each,
GET /api/v1/datacenters/cn13/nodes/cn13/gpuCards):−2.77s (−29%) while doing one more Openstack call than the old binary. Net
of that ~1.2s lookup, concurrency itself is worth ~4.0s. The third row swaps the
new binary back in to rule out ordering and warm-up effects. At the hex_sdk
subprocess layer the same change measured 12.58s → 4.11s, with the two
responses byte-for-byte identical.
go test -race ./...green (87 packages),go vetclean,gofmtclean onevery file this PR touches.
Verified on cn13: a pgpu card reports
rtx_pro_6000_blackwell_server_edition_1, the other three reportnull.Known limitations the reviewer should not be surprised by
reporting PGPU (#1262's fix only applies on Caracal; cn13 runs a March Yoga
ISO), so a pgpu request can be scheduled onto a vGPU VF and fail in libvirt.
Environment defect, not this change.
cyborg renamed its GPU traits and
sdk_os.shstill writes the old ones. ThisPR makes the name obtainable; #1478 makes it usable. They ship together.