perf(agent): publish normal events off restore path

[hutm]

Problem

Every Kubernetes Event write currently blocks checkpoint or restore reconciliation. A slow Event API request can delay the workload even though Normal events are best-effort telemetry.

Why

Informational event delivery should not be part of the restore critical path. Warning events still need synchronous error reporting.

Change

Publish Normal events through a bounded worker queue. Keep Warning events synchronous, copy queued Pod data, and drop a Normal event with a diagnostic when the queue is full.

This removes a blocking dependency from the critical path, but local end-to-end measurements did not show a statistically significant latency improvement.

Testing

• Verified Normal event publication returns while the API request is blocked.
• Verified Warning event publication waits for the API request.
• Ran the complete agent test suite and event publisher race tests.

Fixes #351

[copy-pr-bot]

This pull request requires additional validation before any workflows can run on NVIDIA's runners.

Pull request vetters can view their responsibilities here.

Contributors can view more details about this message here.

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

---

_{Comment @coderabbitai help to get the list of available commands.}