Conversation
resolve_worker_info took the MDC-derived model name ahead of the operator
supplied config.model_name. Discovery normalizes that name to lowercase, but
engine metrics carry the model's real casing, so every engine-metric query
silently returned empty:
model=deepseek-ai/DeepSeek-V4-Flash-0731-roce-disagg -> accept_length 3.67
model=deepseek-ai/deepseek-v4-flash-0731-roce-disagg -> None
With accept_length missing, _clamp_accept_length falls back to 1.0 and the ITL
estimator divides forward-pass time by 1 instead of the real ~3.9, inflating the
estimate ~4x. On DeepSeek-V4-Flash that read 265ms against a 39ms actual, so the
ITL arm breached its SLA forever and the planner recommended SCALE_UP every cycle
regardless of load.
After the fix, accept_length resolves to ~3.96 and the ITL estimate lands at
37.9ms against a measured 38.9ms.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_avg_kv_hit_rate sanitized the namespace to underscores unconditionally. The
router emits it that way, but some scrape paths relabel it back to the hyphenated
k8s form (nvidia.com/dynamo-namespace), so the query silently returned no data:
dynamo_namespace=deepseek_ai__DeepSeek_V4_Flash_0731_roce_disagg -> EMPTY
dynamo_namespace=deepseek-ai--DeepSeek-V4-Flash-0731-roce-disagg -> 5 series
DeepSeek-V3.2 carries both forms, so neither spelling is universally right. Try
the sanitized form first, then the raw one, and name both in the miss log.
With this and the model-name casing fix, the planner's Observed line goes from
"kv_hit_rate: n/a accept_length: n/a" to real values (0.36 / 3.08) and every
"No prometheus data" warning disappears.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Pernekhan
temporarily deployed
to
external_collaborator
August 31, 2026 20:24 — with
GitHub Actions
Inactive
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two fixes that together make the planner's Prometheus metrics resolve. Both were silent — no error, just
n/aon every tick.1. Model-name casing (
worker_info.py)resolve_worker_infopreferred the MDC-derived model name over the operator-suppliedconfig.model_name. Discovery normalizes that name to lowercase, but engine metrics carry the model's real casing:One character-case difference emptied every engine-metric query.
_clamp_accept_lengththen falls back to 1.0, so the ITL estimator divides forward-pass time by 1 instead of the real ~3.9 — inflating the estimate ~4x (265ms against a measured 39ms), breaching the ITL SLA every cycle and pinning the planner atSCALE_UP.Fix: config
model_namewins, and the log names both when they disagree.2.
kv_hit_ratenamespace spelling (traffic_metrics.py)get_avg_kv_hit_ratesanitized the namespace to underscores unconditionally. The router emits it that way, but some scrape paths relabel it back to the hyphenated k8s form (nvidia.com/dynamo-namespace):DeepSeek-V3.2 carries both forms, so neither spelling is universally correct. Fix tries sanitized first, then raw, and names both in the miss log.
Verified live
ITL estimate went 265ms -> 37.9ms against a measured actual of 38.9ms.
Notes
a3e3a561, the commit the deployeddynamo-plannerimage is built from — notmain— so it applies cleanly to what actually runs.FROM <planner image>+COPY), since the planner is pure Python.deepinfra/backendPR [BUG]: dynamographdeployment operator didn't handle multinode spec update operation ai-dynamo/dynamo#4749 (the planner config was also missingnamespaceentirely). All three are needed.🤖 Generated with Claude Code
https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw