feat(client): add dynamo_chat transport + routed_experts to renderer generate

[biswapanda]

Description

Adds a dynamo_chat transport to the renderer-based generate() client so it can run against NVIDIA Dynamo, which serves no /inference/v1/generate route. Selected per-call via transport=; defaults to the existing vLLM path, so behavior is unchanged unless opted in.

Two transports:

• vllm_generate (default): unchanged — messages → render_ids() → POST /inference/v1/generate → parse_response() (vLLM TITO surface).
• dynamo_chat: messages → render_ids() → POST /v1/chat/completions with nvext.token_data (pre-tokenized prompt) + nvext.extra_fields=["engine_data"]. Completion token IDs and logprobs are read back from nvext.engine_data.

Dynamo wire shape (_post_dynamo_chat)

Mirrors the verifiers token client so the payload is identical whether a rollout goes through the token client or the renderer client. nvext.token_data (Dynamo skips tokenization when present); cache_salt → nvext.cache_salt, priority → nvext.agent_hints.priority; a single placeholder user message; sampling remap (max_tokens → max_completion_tokens, logprobs=N → logprobs=true + top_logprobs=N); passthrough fields ride the Dynamo allowlist. Tools are baked into token_data by the renderer (not sent on the wire).

routed_experts (MoE expert replay) — now surfaced on dynamo_chat

(Supersedes the earlier "routed_experts intentionally NOT surfaced" note — it now is.) parse reads routed_experts from nvext.routed_experts (or nvext.engine_data.routed_experts) and maps it to the downstream RoutedExpertsPayload {data, shape, start, dtype}. The Dynamo worker returns full-sequence routing with start=0; the renderer row-trims the leading prompt rows only when the caller explicitly sets routed_experts_prompt_start — a first-turn request with no caller start stays full-sequence with start=0 (no phantom prefix). Completion logprobs prefer nvext.engine_data.completion_logprobs (the same authoritative source as the engine token IDs) over the chat echo; a present-but-empty engine list is authoritative and does not fall back to chat.

Other

• Public RendererTransport = Literal["vllm_generate", "dynamo_chat"] alias. A present-but-empty completion_token_ids is a valid zero-token completion; only a fully absent field raises. Multimodal renderers raise NotImplementedError on dynamo_chat (vLLM path / token-client TITO remain available for VLMs).

Type of Change

• New feature (non-breaking change which adds functionality)

Review

Codex adversarial review: SIGN-OFF (F1/F2/F3 + the N1 logprob-presence finding resolved; head 5f2a914). All review threads resolved.

Testing

tests/test_client.py covers the Dynamo request body shape (priority/detokenize/sampling remap), routed_experts parse + row-trim (explicit prompt_start vs first-turn full-sequence), engine-logprob preference incl. present-but-empty, and missing/empty completion IDs.

---

Note

Medium Risk
New inference path affects rollout token IDs, logprobs, and MoE expert replay for Dynamo users; default transport is unchanged but parsing/validation errors are now explicit on misconfigured Dynamo responses.

Overview
Adds a dynamo_chat backend to renderer generate(), selected per call via transport= (default vllm_generate leaves existing vLLM /inference/v1/generate behavior intact).

generate() now delegates HTTP to a _Transport strategy: vLLM posts token IDs to the cached absolute generate URL; Dynamo posts to /chat/completions with nvext.token_data, merged caller nvext, sampling remaps (max_completion_tokens, logprobs/top_logprobs), and denylisted vLLM-only keys routed into nvext (cache_salt, priority, routed_experts_prompt_start). Responses are normalized through _WireResult; Dynamo prefers nvext.engine_data for completion IDs and logprobs, validates routed_experts to {data, shape, start, dtype}, and can client-trim MoE routing when the worker returns full-sequence data with start=0. Multimodal on dynamo_chat raises NotImplementedError.

Tests cover Dynamo wire shape, nvext merging, missing/misaligned completions and logprobs, routed-experts trimming, and shared POST error propagation for both transports.

^{Reviewed by Cursor Bugbot for commit f5c480d. Bugbot is set up for automated code reviews on this repo. Configure here.}

Note

Add dynamo_chat transport and routed_experts support to generate()

• Introduces a transport parameter to generate() in renderers/client.py (default "vllm_generate"), routing requests to either /inference/v1/generate (vLLM) or /chat/completions (Dynamo); unknown values raise ValueError.
• Adds _DynamoChatTransport which builds an OpenAI-style chat body with nvext fields (token IDs, engine data, cache salt, priority, routed_experts_prompt_start), forwards sampling params while dropping vLLM-internal keys, and parses nvext.engine_data for completion token IDs and logprobs.
• Adds client-side trimming of routed_experts prompt rows via _trim_dynamo_routed_experts when the worker has not already applied the offset.
• Extracts the existing vLLM logic into _VllmGenerateTransport with no behavioral change to that path.
• Risk: Dynamo transport raises NotImplementedError for multimodal inputs and RuntimeError when completion_token_ids is absent or logprob length mismatches — these are new hard failures with no fallback.

^{Macroscope summarized f5c480d.}

[cursor]

Cursor Bugbot has reviewed your changes and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 30c01b6. Configure here.}