Skip to content

perf(agent): reuse observed pod for non-DRA restores - #357

Draft
hutm wants to merge 1 commit into
ai-dynamo:mainfrom
hutm:review/reuse-observed-pod
Draft

hutm wants to merge 1 commit into
ai-dynamo:mainfrom
hutm:review/reuse-observed-pod

Conversation

@hutm

@hutm hutm commented Sep 17, 2026

Copy link
Copy Markdown

Problem

The restore controller already has the scheduled Pod from its informer, but GPU discovery performs another Pod GET before deciding whether DRA is relevant.

Why

For Pods without resource claims, the immutable Pod spec already proves that DRA lookup is unnecessary. Removing the redundant GET reduces API traffic and avoids an extra request on the restore path.

Change

Reuse the observed Pod when its identity matches and it has no resource claims. Pods that may use DRA retain the live Pod lookup because generated claim names arrive through Pod status. GPU assignment still comes from kubelet PodResources.

Local phase measurements showed a smaller GPU-discovery preflight, but no statistically significant end-to-end latency improvement.

Testing

  • Added coverage verifying that a matching Pod without resource claims performs no Pod GET.
  • Ran the complete agent test suite.

Fixes #352

Signed-off-by: Maksim Khadkevich <mkhadkevich@nvidia.com>
@copy-pr-bot

copy-pr-bot Bot commented Sep 17, 2026

Copy link
Copy Markdown

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

@coderabbitai

coderabbitai Bot commented Sep 17, 2026

Copy link
Copy Markdown

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

  • Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

Comment @coderabbitai help to get the list of available commands.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Reuse observed Pod metadata on non-DRA restores

1 participant