Skip to content

feat(gpu): obtain and create a pgpu card's cyborg device profile - #1487

Merged
github-actions[bot] merged 5 commits into
developfrom
feat/818-pgpu-device-profile-api-wip
Sep 14, 2026
Merged

github-actions[bot] merged 5 commits into
developfrom
feat/818-pgpu-device-profile-api-wip

Conversation

@SekiXu

@SekiXu SekiXu commented Sep 14, 2026

Copy link
Copy Markdown
Contributor

What type of PR is this?

/kind feature

What this PR does / why we need it

A pgpu card is unusable without its Cyborg device profile name — that is the
string a flavor's accel:device_profile has to be set to — and the only ways to
get it were openstack accelerator device profile list or the interactive
hex_cli gpu menu. A Web UI user could set a card to pgpu and then get no
further, which is exactly what the parent epic's "no root shell access needed"
goal rules out. sriovVgpu and migBackedVgpu need nothing of the sort: their
PCI alias is minted by gpu_resource_set and already travels with the card.

Five commits, each self-contained:

commit what
refactor(gpu): derive a device profile name through one shared function one naming function, used by both the create and the report path
feat(gpu): report the device profile of every pgpu card on a node gpu_device_profile_map / _get — one Openstack round trip for the whole node
feat(gpu): create a pgpu card's device profile without asking cyborg gpu_device_profile_ensure — idempotent, no dependency on Cyborg's inventory
feat(gpu): give a pgpu card its device profile when it is carved config_gpu.cpp calls ensure on a successful carve
feat(gpu): add a per-card device_profile_ensure to the gpu CLI cli_gpu.cpp, for a card that predates this change

Ensure does not ask Cyborg for the card's model, because at the instant a carve
succeeds Cyborg does not know about the card yet: its nvidia driver learns which
cards are passthrough by reading hex's config.json, but only on the agent
period (default 60s). Asking would mean either blocking the carve for up to a
minute or creating nothing. Both values are available locally — the model name
in config.json, the product id in sysfs, which stays readable after the card
is bound to vfio-pci.

Which issue(s) this PR fixes

Refs #818

Deliberately Refs, not Fixes#818 is a User Story whose UI (#876) and API
(#886) halves are still open, so merging this must not close it.

Special notes for your reviewer

Two runtime defects that only a node could show are folded into the commits
that introduced them, rather than left as a follow-up fix commit. Both have
regression tests, and both are worth knowing about beyond this PR:

  1. hex_sdk does not load modules across files. It takes the command's
    first underscore-separated segment as the module name and sources only
    sdk_<MOD>*.sh, so a gpu_* command never has sdk_os.sh loaded and every
    call into it died with "command not found" — at runtime, not at parse time,
    and not in a unit test that stubs the callee. _gpu_load_os_module pulls it
    in on demand, the way sdk_health.sh does for sdk_ovn.sh, except lazily:
    sdk_os.sh is 3300 lines and drags in os_cinder.sh, and sourcing it at the
    top of the file would tax gpu_device_list, which the API runs on every
    listing. The tests now exercise the real load path against a fake SDK_DIR.

  2. gpu_sysfs_pci_addr returns a sentence, not an address, for exactly the
    cards this feature is about.
    nvidia-smi prints "No devices were found" on
    stdout (rc 6, empty stderr) for a vfio-bound card, so that helper's
    [ -z "$addr" ] fallback to config.json never fires and it hands the
    sentence back as an address. gpu_device_profile_ensure already reads the
    card's record, so it takes the address from there. The helper's own defect
    is left alone and gets its own ticket ([Bug] gpu_sysfs_pci_addr returns nvidia-smi's error sentence as a PCI address for vfio-bound cards #1486)
    — its entire fallback branch is dead
    code for the case it exists to handle.

One more thing that matters more than it looks: the groups document is built
in one place, os_device_profile_groups_for. Those trait names are wrong on
Caracal — cyborg emits a single CUSTOM_NVIDIA_<PID> there (#1478) — and
os_nova_pgpu_host_list_by_instance_id parses the groups back out by field
ordinal
, so a change in how many traits a group carries breaks live pgpu
migration silently. Keeping the document in one function keeps that fix to one
place.

Verification

  • 45 offline tests green (10 naming / 19 map / 16 ensure), and green at every
    commit in this PR, not just at the tip.
  • cn13, end to end: query → create → GET /gpuCards. ensure creates
    rtx_pro_6000_blackwell_server_edition_1 with the expected traits; the API
    reports it on the pgpu card and null on the other three.
  • cn13: automatic ensure on carve, including the restore path — the card used
    for that test is back to IDENTICAL config.json, numvfs=48, driver=nvidia.

Two limitations QA needs to know, neither caused by this PR

Related PRs: bigstack-oss/cube-cos-openapi#113 (field contract, merges first),
and the cube-cos-api handler PR, which opens once #113 is merged and its submodule
pointer can be re-pointed at a SHA on develop. This one can land in any order relative to the
API side — the field simply reads null until hex can answer.

Additional documentation


SekiXu and others added 5 commits September 14, 2026 14:34
os_device_profile_create built the Cyborg device profile name inline from
cyborg's `model` string. #818 needs the same name from a second caller that
cannot use cyborg's string at all: a card just carved to pgpu takes up to one
cyborg agent period (periodic_interval, default 60s) to appear in the
accelerator inventory, so right after the carve there is nothing to read.

Extract the derivation into os_device_profile_name_for, accepting either
source and normalising both onto the same slug:

  cyborg's model  "NVIDIA Corporation GA106 [RTX A2000]"  -> rtx_a2000_1
  hex's name      "NVIDIA RTX A2000"                      -> rtx_a2000_1

Two derivations would be worse than one shared one that takes two inputs: if
they ever disagreed the UI would show a profile name that does not exist.

The cyborg path is byte-for-byte what it was, so existing profile names are
unchanged; the test pins them as golden values for that reason. It also now
refuses an empty name rather than building "_1", which every unnamed card
would otherwise share.

Refs #818

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Seki Xu <seki.xu@bigstack.co>
A pgpu card is unusable without its Cyborg device profile name -- that is
what a flavor's accel:device_profile has to be set to -- and the only way to
obtain it was `openstack accelerator device profile list` or the interactive
`hex_cli gpu` menu. A Web UI user could set a card to pgpu and then get no
further (#818), which is exactly what the parent epic's "no root shell
access needed" goal rules out.

sriovVgpu and migBackedVgpu cards need nothing of the sort: their PCI alias
is minted by gpu_resource_set and already travels with the card. Only pgpu
is missing a way to get its identifier out.

gpu_device_profile_map answers for the whole node in one Openstack round
trip -- that call costs ~1.2s on a real node, and a per-card loop would put
that on every GET of a four-card node. A node with no pgpu card makes no
call at all. gpu_device_profile_get is the single-card accessor on top.

Two things are deliberate:

- A card whose profile does not exist yet reports null, not the name it
  would be given. A name is only worth showing if it resolves; handing the
  operator one that was never created sends them to NoValidHost with no clue.
- os_device_profile_names fails rather than reporting an empty list when
  Cyborg cannot be reached, so "no profiles" and "could not ask" stay
  distinguishable. gpu_vgpu_profile_list made the opposite choice and that
  was #1247.

hex_sdk sources only the module matching a command's first underscore-
separated segment, so a gpu_* command never has sdk_os.sh loaded and a call
into it dies at runtime with "command not found". _gpu_load_os_module pulls
it in on demand -- what sdk_health.sh does for sdk_ovn.sh, except lazily:
sdk_os.sh is 3300 lines and drags in os_cinder.sh, and sourcing it at the top
of this file would tax gpu_device_list, which the API runs on every listing.
The test exercises that load path against a fake SDK_DIR instead of stubbing
the callee -- stubbing is why the gap reached a real node once.

Refs #818

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Seki Xu <seki.xu@bigstack.co>
gpu_resource_set has to be able to create the profile the instant a carve
succeeds, and at that instant Cyborg does not know about the card: its nvidia
driver decides which cards are passthrough by reading hex's config.json, but
only on the agent period (periodic_interval, default 60s). Asking Cyborg for
the card's model -- which is what os_device_profile_create does -- would mean
either blocking the carve for up to a minute or creating nothing.

Both values are available locally. The model name is in config.json, and the
product id is in sysfs, which stays readable after the card is bound to
vfio-pci (the same reason gpu_sysfs_pci_addr falls back to it).

gpu_device_profile_ensure is idempotent and reports the name; a non-pgpu card
is a no-op rather than an error, so the carve path can call it unconditionally.

The groups document is now built in one place, os_device_profile_groups_for,
shared with os_device_profile_create. Those trait names are wrong on Caracal --
cyborg emits CUSTOM_NVIDIA_<PID> as a single trait there (#1478) -- and this
keeps that fix to one function. It matters more than it looks:
os_nova_pgpu_host_list_by_instance_id parses the groups back out by field
ordinal, so a change in how many traits a group carries breaks live pgpu
migration with no error at all.

The test deliberately stubs no accelerator-device call, so a future edit that
reintroduces a dependency on the inventory fails it.

The PCI address comes straight out of config.json rather than through
gpu_sysfs_pci_addr. That helper asks nvidia-smi first and falls back to
config.json only when the answer is empty -- but nvidia-smi prints "No devices
were found" on *stdout* (rc 6, empty stderr) for a vfio-bound card, so the
answer is never empty, the fallback never fires, and it hands that sentence
back as if it were an address. Every card this function cares about is a pgpu,
i.e. exactly the case nvidia-smi cannot see. The helper's own defect is left
for its own ticket; the regression test stubs it with the poisoned value so
this path cannot drift back onto it.

Refs #818

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Seki Xu <seki.xu@bigstack.co>
Carving a card to pgpu left the operator halfway: the card was passed
through correctly, but the Cyborg device profile that a flavor's
accel:device_profile has to name did not exist, and nothing in the product
created one. The only way to get it was the interactive `hex_cli gpu` menu
or raw `openstack accelerator` commands -- root shell either way, which is
what the parent epic set out to remove (#818).

gpu_resource_set now creates it on a successful pgpu carve, and
`hex_config gpu_device_profile_ensure <gpu_id>` covers the two cases that
path cannot: a carve whose profile creation failed, and a card that was
already pgpu before any of this existed.

A failure to create the profile does not fail the carve. The card really is
a pgpu at that point and config.json, the Nova drop-in and the hardware all
agree; what is missing is a name that can be recreated on demand. Rolling
the carve back for a label would be the worse trade, and the rollback can
itself fail -- a far worse state to end in than "no profile yet".

Refs #818

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Seki Xu <seki.xu@bigstack.co>
device_profile_create is node-wide and model-keyed: it walks every NVIDIA
device Cyborg can see and makes one profile per model. That is the wrong
shape for the thing an operator actually holds -- a card -- and it cannot
help a card Cyborg has not rescanned yet, because it reads the model out of
the accelerator inventory.

device_profile_ensure takes one card and goes through hex_config, which
derives the name from config.json and the product id from sysfs, so it works
immediately after a carve. It is idempotent and a no-op on a non-pgpu card.

The existing two commands are untouched: their names and behaviour are what
the public docs describe, and customer sites may already have profiles made
by them.

Refs #818

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Seki Xu <seki.xu@bigstack.co>
@Eandalf-Bigstack
Eandalf-Bigstack force-pushed the feat/818-pgpu-device-profile-api-wip branch from 2351ba5 to a6fc31a Compare September 14, 2026 06:35
@Eandalf-Bigstack Eandalf-Bigstack added the done Merge the pull request label Sep 14, 2026
@github-actions
github-actions Bot merged commit a6fc31a into develop Sep 14, 2026
9 checks passed
@github-actions
github-actions Bot deleted the feat/818-pgpu-device-profile-api-wip branch September 14, 2026 06:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

done Merge the pull request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants