Skip to content

planner: fix model-name casing and kv_hit_rate namespace so engine metrics resolve - #33

Open
Pernekhan wants to merge 2 commits into
feat-deepinfra-runtime-07-09from
claude/planner-model-name-casing
Open

Pernekhan wants to merge 2 commits into
feat-deepinfra-runtime-07-09from
claude/planner-model-name-casing

Conversation

@Pernekhan

Copy link
Copy Markdown
Collaborator

Two fixes that together make the planner's Prometheus metrics resolve. Both were silent — no error, just n/a on every tick.

1. Model-name casing (worker_info.py)

resolve_worker_info preferred the MDC-derived model name over the operator-supplied config.model_name. Discovery normalizes that name to lowercase, but engine metrics carry the model's real casing:

model=deepseek-ai/DeepSeek-V4-Flash-0731-roce-disagg -> accept_length 3.67
model=deepseek-ai/deepseek-v4-flash-0731-roce-disagg -> None

One character-case difference emptied every engine-metric query. _clamp_accept_length then falls back to 1.0, so the ITL estimator divides forward-pass time by 1 instead of the real ~3.9 — inflating the estimate ~4x (265ms against a measured 39ms), breaching the ITL SLA every cycle and pinning the planner at SCALE_UP.

Fix: config model_name wins, and the log names both when they disagree.

2. kv_hit_rate namespace spelling (traffic_metrics.py)

get_avg_kv_hit_rate sanitized the namespace to underscores unconditionally. The router emits it that way, but some scrape paths relabel it back to the hyphenated k8s form (nvidia.com/dynamo-namespace):

dynamo_namespace=deepseek_ai__DeepSeek_V4_Flash_0731_roce_disagg -> EMPTY
dynamo_namespace=deepseek-ai--DeepSeek-V4-Flash-0731-roce-disagg -> 5 series

DeepSeek-V3.2 carries both forms, so neither spelling is universally correct. Fix tries sanitized first, then raw, and names both in the miss log.

Verified live

before:  kv_hit_rate: n/a     accept_length: n/a     summary: SCALE_UP (every cycle)
after:   kv_hit_rate: 0.362   accept_length: 3.079   summary: HOLD
         "No prometheus data" warnings: ZERO

ITL estimate went 265ms -> 37.9ms against a measured actual of 38.9ms.

Notes

🤖 Generated with Claude Code

https://claude.ai/code/session_017rS1uFFKT2xV4x63Amfvtw

claude added 2 commits August 29, 2026 03:46
resolve_worker_info took the MDC-derived model name ahead of the operator
supplied config.model_name. Discovery normalizes that name to lowercase, but
engine metrics carry the model's real casing, so every engine-metric query
silently returned empty:

    model=deepseek-ai/DeepSeek-V4-Flash-0731-roce-disagg -> accept_length 3.67
    model=deepseek-ai/deepseek-v4-flash-0731-roce-disagg -> None

With accept_length missing, _clamp_accept_length falls back to 1.0 and the ITL
estimator divides forward-pass time by 1 instead of the real ~3.9, inflating the
estimate ~4x. On DeepSeek-V4-Flash that read 265ms against a 39ms actual, so the
ITL arm breached its SLA forever and the planner recommended SCALE_UP every cycle
regardless of load.

After the fix, accept_length resolves to ~3.96 and the ITL estimate lands at
37.9ms against a measured 38.9ms.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
get_avg_kv_hit_rate sanitized the namespace to underscores unconditionally. The
router emits it that way, but some scrape paths relabel it back to the hyphenated
k8s form (nvidia.com/dynamo-namespace), so the query silently returned no data:

    dynamo_namespace=deepseek_ai__DeepSeek_V4_Flash_0731_roce_disagg -> EMPTY
    dynamo_namespace=deepseek-ai--DeepSeek-V4-Flash-0731-roce-disagg -> 5 series

DeepSeek-V3.2 carries both forms, so neither spelling is universally right. Try
the sanitized form first, then the raw one, and name both in the miss log.

With this and the model-name casing fix, the planner's Observed line goes from
"kv_hit_rate: n/a accept_length: n/a" to real values (0.36 / 3.08) and every
"No prometheus data" warning disappears.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@Pernekhan
Pernekhan temporarily deployed to external_collaborator August 31, 2026 20:24 — with GitHub Actions Inactive
@Pernekhan
Pernekhan changed the base branch from main to feat-deepinfra-runtime-07-09 August 31, 2026 21:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants