Skip to content

feat(gpu): report a pgpu card's device profile, and build a node's cards concurrently - #661

Merged
github-actions[bot] merged 2 commits into
developfrom
feat/818-pgpu-device-profile-api-wip
Sep 14, 2026
Merged

github-actions[bot] merged 2 commits into
developfrom
feat/818-pgpu-device-profile-api-wip

Conversation

@SekiXu

@SekiXu SekiXu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

Feature + perf. Two independent commits, reviewable separately.


Which issue(s) this PR fixes?

Refs bigstack-oss/cubecos#818

Deliberately Refs, not Fixes: #818 is a User Story whose UI (#876) and API
(#886) halves are still open, so merging this must not close it.

Merge orderbigstack-oss/cube-cos-openapi#113 is already merged, and the third
commit here moves api/cube-cos-openapi onto that SHA (9495246, on develop). CI
regenerates api/docs.json from the submodule, so the field only reaches the served docs
once that pointer moves. The hex side
that actually produces the value is bigstack-oss/cubecos#1487, which can land in any
order relative to this one (the field simply reads null until it does).


What this PR does?

1. feat(gpu): report a pgpu card's device profile on the card

GET /nodes/{nodeName}/gpuCards now reports deviceProfile on every card. For
a pgpu card that string is what a flavor's accel:device_profile has to name;
without it the operator needed a root shell, which the parent epic rules out.
sriovVgpu / migBackedVgpu already carry their PCI alias, so they stay
null.

Fetched once per listing, not per card — hex_sdk answers for the whole node
in one Openstack round trip (~1.2s on a real node), and a node with no pgpu card
makes no call at all, which is the common case.

Three decisions worth the reviewer's attention:

  • A profile that has not been created reports null, not the name it would
    be given. A name that does not resolve sends the operator to NoValidHost
    with nothing to go on.
  • A failed lookup reports null and does not mark the card degraded.
    degraded means the capacity numbers cannot be trusted; a missing
    convenience string is not that.
  • hex_sdk fails rather than returning an empty map when Cyborg is unreachable,
    so "no profiles" and "could not ask" stay distinguishable here. Conflating
    them is what #1247 was.

2. perf(gpu): build a node's gpu cards concurrently

Carried in this PR on purpose — it rewrites the same loop, needs the same
build, and was verified on the same node in the same session. Splitting it
would mean two builds and two rounds of hardware verification of one function.

Listing a four-card node cost one hex_sdk gpu_vgpu_profile_list per card, and
that cost is hex_sdk's own startup (fork, then source every module), not the
query. Four cards ran four startups back to back.

Nothing is shared for writing: Servers, NvidiaSmiDevices and
VgpuInstances are prefetched before the loop and read-only after it,
enrichment state is created per card inside buildLocalGpuCard, the Mongo read
is goroutine-safe, and the hex_sdk call writes only its own mktemp file.

Two behaviours the concurrency must not change, both now pinned by tests:

  • Order. Results are written back by index into a fixed-length slice, never
    appended. Response order is sorted PCI address order and the UI renders in
    receipt order — appending as goroutines finish would make the list jump
    between requests.
  • The failure contract. One card failing to build still fails the whole
    listing rather than returning a short list. A card that merely lacks
    enrichment is not a failure: it comes back degraded, per card, never
    collapsed node-wide.

Test results

1). api docsdeviceProfile is added in bigstack-oss/cube-cos-openapi#113
(string | null, in required); the submodule here points at it.

2). the api works properly — end-to-end on cn13 (4 cards, 5 runs each,
GET /api/v1/datacenters/cn13/nodes/cn13/gpuCards):

binary median
new (concurrent + deviceProfile) 6.70s
old (serial, without the new lookup) 9.47s
new again, swapped back 6.69s

−2.77s (−29%) while doing one more Openstack call than the old binary. Net
of that ~1.2s lookup, concurrency itself is worth ~4.0s. The third row swaps the
new binary back in to rule out ordering and warm-up effects. At the hex_sdk
subprocess layer the same change measured 12.58s → 4.11s, with the two
responses byte-for-byte identical.

go test -race ./... green (87 packages), go vet clean, gofmt clean on
every file this PR touches.

Verified on cn13: a pgpu card reports
rtx_pro_6000_blackwell_server_edition_1, the other three report null.


Known limitations the reviewer should not be surprised by

  • cn13 cannot prove a pgpu VM boots. That node has 148 resource providers
    reporting PGPU (#1262's fix only applies on Caracal; cn13 runs a March Yoga
    ISO), so a pgpu request can be scheduled onto a vGPU VF and fail in libvirt.
    Environment defect, not this change.
  • On Caracal the profile name cannot yet be scheduled against (#1478):
    cyborg renamed its GPU traits and sdk_os.sh still writes the old ones. This
    PR makes the name obtainable; #1478 makes it usable. They ship together.

@SekiXu SekiXu self-assigned this Sep 14, 2026
@SekiXu SekiXu added done Merge the pull request and removed done Merge the pull request labels Sep 14, 2026
SekiXu and others added 2 commits September 14, 2026 22:28
A pgpu card's Cyborg device profile is the string a flavor's
accel:device_profile has to name, and it was reachable only through a root
shell. sriovVgpu and migBackedVgpu cards already carry their scheduling
identifier in the response -- the PCI alias, per profile -- so this closes
the gap for the one resource type that had none (#818).

Fetched once per listing, not per card: hex_sdk answers for the whole node
in a single Openstack round trip, which costs over a second on a real node.
A node with no pgpu card makes no call at all, which is the common case.

Three decisions worth stating:

- A card whose profile has not been created reports null, not the name it
  would be given. A name that does not resolve sends the operator to
  NoValidHost with nothing to go on; null at least says "not yet".
- A failed lookup also reports null and does not mark the card degraded.
  Degraded means the card's capacity numbers cannot be trusted, and a
  missing convenience string is not that.
- hex_sdk fails rather than reporting an empty map when Cyborg is
  unreachable, so "no profiles" and "could not ask" stay distinguishable
  here (gpu_vgpu_profile_list conflated them and that was #1247).

restoreGpuSeams now stubs the new seam by default, so tests that list a
node holding a pgpu card do not shell out to a real hex_sdk.

Refs #818

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Seki Xu <seki.xu@bigstack.co>
Listing a four-card node took 10.5s, and the time was not in the network,
the proxy or Go: each card costs one `hex_sdk gpu_vgpu_profile_list`, and
that cost is hex_sdk's own startup -- fork, then source every module --
rather than the query. Four cards ran four startups back to back.

Building the cards concurrently measured 12.58s -> 4.11s on cn13 (3.06x),
with the two responses byte-for-byte identical.

Safe because nothing is shared for writing: Servers, NvidiaSmiDevices and
VgpuInstances are prefetched before the loop and read-only after it (and
VgpuInstances is already sliced per card), the enrichment state is created
inside buildLocalGpuCard per card, buildGpuCardLinks is string assembly,
the Mongo read is goroutine-safe, and the hex_sdk call writes only its own
mktemp file.

Two behaviours the concurrency must not change, both now pinned by tests:

- Order. Results are written back by index into a fixed-length slice, never
  appended. The response order is the sorted PCI address order, and the UI
  renders cards in the order it receives them -- appending as goroutines
  finish would make the list jump around between requests.
- The failure contract. One card failing to build still fails the whole
  listing rather than returning a short list. A card that merely lacks
  enrichment is not a failure; it comes back with Degraded, which stays per
  card and is never collapsed node-wide.

Refs #818

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Seki Xu <seki.xu@bigstack.co>
@SekiXu
SekiXu force-pushed the feat/818-pgpu-device-profile-api-wip branch from 57cd208 to 1e0712f Compare September 14, 2026 14:29
@SekiXu SekiXu added the done Merge the pull request label Sep 14, 2026
@github-actions
github-actions Bot merged commit 1e0712f into develop Sep 14, 2026
9 checks passed
@github-actions
github-actions Bot deleted the feat/818-pgpu-device-profile-api-wip branch September 14, 2026 14:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

done Merge the pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants