Skip to content

feat(health): Add NVOS Streaming telemetry#1975

Open
mkoci wants to merge 46 commits into
NVIDIA:mainfrom
mkoci:feature-nvos-health
Open

feat(health): Add NVOS Streaming telemetry#1975
mkoci wants to merge 46 commits into
NVIDIA:mainfrom
mkoci:feature-nvos-health

Conversation

@mkoci
Copy link
Copy Markdown
Contributor

@mkoci mkoci commented May 28, 2026

Description

gNMI collector ([collectors.nvue.gnmi], disabled by default) subscribes to NVUE gNMI ON_CHANGE /system-events and SAMPLE paths for:

/components/component
/interfaces/interface
/platform-general/leak-sensors

It uses long-lived gRPC streams with reconnection (exp. back-off + jitter).

Builds on #711 (SSE streaming + OtlpSink for logs). Protos vendored for reproducible offline builds, same as #711.

Currently supports PrometheusSink and OtlpSink

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Related Issues

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Additional Notes

Future

It would be useful to port this to support HealthAlerts especially for ON_CHANGE from /system-events - there is extra information available here which is not available via the Switch BMC.

mkoci and others added 30 commits May 14, 2026 14:20
@mkoci
feat(health): add machine placement metadata to events
0e1d965 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
feat(health): forward machine metadata from API
0053968 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
feat(health): expose machine metadata as Prometheus labels
037dd50 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
feat(health): emit machine metadata as OTLP attributes
40c3d49 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
docs(health): document hardware health metadata surfaces
f22528b 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
feat(health): expose switch placement metadata
83149ec 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
Merge branch 'main' into feature/hw-health-machine-metadata
c52200e
@mkoci
refactor(health): rename SseConnectionGuard to StreamingConnectionGuard
6331ef8 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
feat(health): vendor openconfig/gnmi protos for reproducible builds
89acbe5 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
feat(health): add NVUE gNMI streaming collector
57f9b0b 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
feat(health): add OTLP metrics export via MetricsService
8c0f234 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
fix(health): adapt nvos streaming to metadata stack
c8e3c83 
Signed-off-by: mkoci <mkoci@nvidia.com>
@mkoci
Merge remote-tracking branch 'mk-origin/main' into feature/nvos_strea…
7f1445a 
…ming

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
fix(health): add support for positional metadata in telemetry
3aafde5 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
Merge branch 'NVIDIA:main' into feature/nvos_streaming
cc59345
@mkoci
Merge remote-tracking branch 'NVIDIA/main' into feature-nvos-health
3885ae9 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
feat(health): restore nvue gnmi on-change events
37371fb 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
fix(health): typo
543be1a
@mkoci
Merge remote-tracking branch 'NVIDIA/main' into feature/nvos_streaming
11efebf 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
Merge branch 'feature/nvos_streaming' into feature-nvos-health
57221f2 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
lint(health): tidy up
0f3fd25 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
bug(health): gate switch hosts and bmcs in spawn to avoid redfish cal…
438019d 
…ls on switch hosts

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
feat(health): add SwitchEndpointRole to distinguish switch BMC from Host
db6dca4 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
feat(health): add static config shape for switch bmc/host
3fbc376 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
feat(health): gate switch collection by endpoint role (host/bmc)
817c2f3 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
feat(api): expose switch host endpoints and nvos credentials for host…
83f05fe 
… monitoring

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
feat(health): wire switch host endpoint to health discovery
039a020 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
feat(health): discover switch bmc and host endpoints, respectively
58e8c24 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
fix(health): remove unnecessary helpers
4f8e492 
Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
Merge branch 'NVIDIA:main' into bug_1744_no_redfish_for_switch_hosts
7277eef
mkoci added 15 commits May 20, 2026 16:30
@mkoci
fix(api): use proper errors when bmc/switch host credentials are not …
eb11bb4 
…found

Signed-off-by: mkoci <26286151+mkoci@users.noreply.github.com>
@mkoci
fix(health): lint-police
4099f62
@mkoci
fix(api, health): reshape Switch API surface with switch bmc and nvos…
804fd0f 
… info in one endpoint. Remove dead code
@mkoci
fix(api): broken test
4ebcf2e
@mkoci
Merge branch 'main' into bug_1744_no_redfish_for_switch_hosts
135d494
@mkoci
Merge branch 'NVIDIA:main' into feature-nvos-health
f93a8ee
@mkoci
Merge remote-tracking branch 'upstream/main' into bug_1744_no_redfish…
230c2b9 
…_for_switch_hosts

# Conflicts:
#	crates/api/src/auth/internal_rbac_rules.rs
@mkoci
Merge branch 'main' into bug_1744_no_redfish_for_switch_hosts
eadec73
@mkoci
Merge branch 'NVIDIA:main' into feature-nvos-health
877af27
@mkoci
Merge branch 'bug_1744_no_redfish_for_switch_hosts' into feature-nvos…
9f7ec17 
…-health

# Conflicts:
#	crates/health/src/api_client.rs
@mkoci
fix(health): adds SwitchNvosInfo, explicit `spawn_collectors_for_endp…
6670948 
…oint` for BMCs.
@mkoci
Merge branch 'main' into bug_1744_no_redfish_for_switch_hosts
bb5ec30
@mkoci
Merge branch 'bug_1744_no_redfish_for_switch_hosts' into feature-nvos…
04a4d9f 
…-health
@mkoci
fix(health): parse nvl partition num-gpus properly (it returns a stri…
b048f03 
…ng :/)
@mkoci
fix(health): ensure collector removal like NVIDIA#1302
4a38257
@mkoci mkoci requested review from a team and Coco-Ben as code owners May 28, 2026 02:10
@copy-pr-bot
Copy link
Copy Markdown

copy-pr-bot Bot commented May 28, 2026

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@mkoci
Merge remote-tracking branch 'origin/main' into feature-nvos-health
a1f996c 
# Conflicts:
#	crates/health/src/api_client.rs
#	crates/health/src/discovery/spawn.rs
@mkoci mkoci force-pushed the feature-nvos-health branch from 35586c5 to ab24084 Compare May 28, 2026 02:26
@mkoci mkoci changed the title Feature nvos health feat(health): Add NVOS Streaming telemetry May 28, 2026
@mkoci mkoci force-pushed the feature-nvos-health branch 2 times, most recently from a4430a9 to a1f996c Compare May 28, 2026 02:52
@mkoci mkoci requested review from tmcroberts97 and yoks May 28, 2026 02:52
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Coco-Ben Coco-Ben Awaiting requested review from Coco-Ben Coco-Ben is a code owner

@yoks yoks Awaiting requested review from yoks

@tmcroberts97 tmcroberts97 Awaiting requested review from tmcroberts97

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@mkoci