test: validate Hermes routed provider OpenInference spans#235
Conversation
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
github-actions
Bot
added
size:L
|
Linter diff in the way? Review this PR in Change Stack to focus on meaningful changes and expand context only when needed. WalkthroughAdds workspace/dev OpenTelemetry deps and test tracing utilities: stable routed-test session marker, in-memory exporter/subscriber helpers, a reusable Hermes routed session driver (three routed LLM calls), and a test asserting emitted OpenInference span attributes. ChangesOpenInference Testing Infrastructure
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
willkill07
left a comment
willkill07
left a comment
There was a problem hiding this comment.
Everything looks good except for the potential package version drift. I included a comment on how to resolve.
Signed-off-by: mnajafian-nv <mnajafian@nvidia.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@Cargo.toml`:
- Around line 29-30: The Cargo.toml currently pins opentelemetry and
opentelemetry_sdk to 0.31; either add a comment/PR description justifying why
0.31 is required for compatibility/security, or update both dependencies to the
aligned 0.32.x releases (e.g., opentelemetry = "0.32.0" and opentelemetry_sdk =
"0.32.1") ensuring versions match the registry, then run cargo update and cargo
build/test to verify no breakages; reference the dependency names opentelemetry
and opentelemetry_sdk when making the change and in the commit/PR message.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 7288b804-5b83-4a79-b880-958e49aaaa80
📒 Files selected for processing (3)
Cargo.tomlcrates/cli/Cargo.tomlcrates/core/Cargo.toml
📜 Review details
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: Check / Run
- GitHub Check: Preview docs
🧰 Additional context used
📓 Path-based instructions (12)
**/{Cargo.toml,**/*.rs}
📄 CodeRabbit inference engine (.agents/skills/maintain-packaging/SKILL.md)
Maintain consistency between Rust package names in
Cargo.tomland their actual usage across the codebase
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
**/*.{rs,toml}
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update Rust crate names and module prefixes during coordinated rename operations
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
**/*.{py,txt,toml,cfg,yaml,yml}
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update Python package names and top-level module imports during coordinated rename operations
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
**/Cargo.toml
📄 CodeRabbit inference engine (.agents/skills/rename-surfaces/SKILL.md)
Update WebAssembly crate names and generated package names during coordinated rename operations
Confirm or infer the target release version from
upstream/main:Cargo.toml. Derive the release branch asrelease/<major>.<minor>.
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
{docs/**,README.md,**/Cargo.toml,**/package.json,**/*.md}
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
Ensure renamed public surfaces are reflected consistently in manifests and docs for large or public-facing changes
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
**/*.{md,mdx,py,sh,yaml,yml,toml,json}
📄 CodeRabbit inference engine (.agents/skills/contribute-docs/SKILL.md)
Keep package names, repo references, and build commands current
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
**/*.toml
📄 CodeRabbit inference engine (CONTRIBUTING.md)
Include SPDX license header in TOML configuration files using hash comment syntax
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
**/*.{rs,py,js,ts,tsx,jsx,go,sh,toml,yaml,yml,md}
📄 CodeRabbit inference engine (AGENTS.md)
Keep SPDX headers on source, docs, scripts, and configuration files. The project is Apache-2.0.
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
**
⚙️ CodeRabbit configuration file
**:AGENTS.md
This file provides guidance to agents, including Claude Code and OpenAI Codex, when working in this repository.
Project Overview
NeMo Relay is a multi-language agent runtime framework for execution scopes, lifecycle events, middleware, plugins, and observability around tool and LLM calls. The core runtime is Rust. Primary supported bindings are Rust, Python, and Node.js. Go, WebAssembly, and the raw C FFI are experimental and source-first.
The shared runtime model is:
- Scope stacks decide where work belongs and which scope-local behavior is visible.
- Middleware registries decide what guardrails and intercepts run around managed calls.
- Plugins install reusable runtime behavior from configuration.
- Events record runtime behavior in ATOF form.
- Subscribers and exporters consume events in-process or export them to ATIF, OpenTelemetry, OpenInference, or other backends.
Repository Structure
The repository layout separates the Rust runtime, language bindings, documentation,
integration patches, and agent-facing skills.crates/ core/ # Rust core runtime crate, published as nemo-relay adaptive/ # Adaptive runtime primitives and plugin components python/ # PyO3 native extension for the Python package ffi/ # Raw C ABI layer used by downstream bindings such as Go node/ # NAPI Node.js binding and JavaScript/TypeScript entry points wasm/ # wasm-bindgen WebAssembly binding and JS wrappers python/ nemo_relay/ # Python wrapper package: scopes, tools, LLM, middleware, typed helpers, plugins, adaptive helpers tests/ # Python tests go/ nemo_relay/ # Experimental Go CGo binding and tests fern/ # Fern documentation site scripts/ # Stable wrappers and helper scripts; build/test/docs entry points live in justfile third_party/ # P...
Files:
crates/cli/Cargo.tomlcrates/core/Cargo.tomlCargo.toml
{crates/core,crates/adaptive}/**/*
📄 CodeRabbit inference engine (.agents/skills/prepare-pr/SKILL.md)
Changes to
crates/coreorcrates/adaptivemust run the full language matrix
Files:
crates/core/Cargo.toml
crates/{core,adaptive}/**
📄 CodeRabbit inference engine (.agents/skills/validate-change/SKILL.md)
If
crates/coreorcrates/adaptivechanged, run the full matrix across Rust, Python, Go, Node.js, and WebAssembly
Files:
crates/core/Cargo.toml
Cargo.toml
📄 CodeRabbit inference engine (.agents/skills/update-project-version/SKILL.md)
Cargo.toml: MaintainCargo.toml[workspace.package].versionas the source of truth for Rust workspace and Python build versioning
KeepCargo.toml[workspace.dependencies]self-references aligned when the workspace version changes
Files:
Cargo.toml
🔇 Additional comments (2)
crates/cli/Cargo.toml (1)
53-54: LGTM!crates/core/Cargo.toml (1)
89-90: LGTM!Also applies to: 103-103
|
/merge |
3 participants
Overview
This PR validates that Hermes routed provider flows emit the expected OpenInference spans by adding explicit session-path coverage for the routed Anthropic
/v1/messages, OpenAI/v1/responses, and OpenAI/v1/chat/completionssurfaces.Details
SessionManager::start_llmandSessionManager::end_llm.Validated with:
cargo test -p nemo-relay-cli hermes_routed_provider_payloads_write_exact_atif_trajectory -- --nocapturecargo test -p nemo-relay-cli hermes_routed_provider_payloads_emit_openinference_text_usage_and_cost -- --nocapturecargo test -p nemo-relay-cli hermes -- --nocapturecargo test -p nemo-relay openinference -- --nocapturecargo fmt --alluv run pre-commit run --all-filesWhere should the reviewer start?
Start in
crates/cli/tests/coverage/session_tests.rswithhermes_routed_provider_payloads_emit_openinference_text_usage_and_cost, then review the shared routed-provider session helper used by both the new OpenInference regression and the existing routed ATIF regression.Related Issues: (use one of the action keywords Closes / Fixes / Resolves / Relates to)
Summary by CodeRabbit
Tests
Chores