fix(agentruntime): degrade gracefully when target workload is missing

[mrsabath]

Summary

Fixes the operator reconcile error-loop described in rossoctl/rossoctl#2490.

When an AgentRuntime's spec.targetRef workload (a Sandbox for agents, a Deployment for tools) is deleted while the parent AgentRuntime CR remains, the operator previously:

• logged an Error and emitted a Warning event every ~30s, forever, and
• left a stale Ready: True from the last successful reconcile.

This treats a genuinely-missing target as a recoverable degraded state instead.

What changed

• resolveTargetRef wraps the IsNotFound case with a sentinel error (errTargetNotFound, via %w).
• Reconcile branches on errors.Is: for a missing target it now
  • sets Ready=False and TargetResolved=False (reason TargetNotFound) — fixing the stale Ready: True,
  • clears the now-stale status.Card (it was discovered from the absent workload),
  • logs at V(1) instead of Error,
  • emits the TargetNotFound Warning event only on transition into the degraded state (deduped via the pre-existing condition), and
  • requeues at 60s as a backstop.
• Genuine (non-IsNotFound) API errors keep the original loud Error path, now with a distinct reason TargetResolveError (so a transient API error can't suppress a later legitimate degraded-transition event).
• Target-resolution reasons extracted to constants (ReasonTargetFound / ReasonTargetNotFound / ReasonTargetResolveError).

Recovery is automatic and prompt: the existing SetupWithManager workload watches enqueue a reconcile the moment the target reappears (it is not bound by the 60s requeue). No owner references or finalizers are added to the child workload — the operator does not own it (linkage is via spec.targetRef).

Tests

New envtest coverage in agentruntime_controller_test.go (context "When the target Deployment does not exist"):

• degraded reconcile sets Ready=False/TargetResolved=False with reason TargetNotFound;
• the TargetNotFound Warning event is emitted at most once across consecutive degraded reconciles;
• status.Card is cleared on degrade;
• recovery to Ready=True/Configured (and TargetResolved=True/TargetFound) when the target is recreated.

The context's AfterEach now drains the kagenti.io/cleanup finalizer (drives reconciles until the object is gone) so specs don't leak state into each other.

Full controller suite passes (go test ./internal/controller/). Verified on a local Kind cluster: on target deletion the operator emits exactly one TargetNotFound Warning (with the exact wrapped message) instead of the previous per-30s spam, and recovers cleanly when the workload returns.

Notes

• Docs steering users to delete the parent AgentRuntime CR are handled separately (Docs: Fix advanced weather-agent demo (partial, in-progress run) cortex#819).

Assisted-By: Claude (Anthropic AI) noreply@anthropic.com

[cwiklik]

Clean, well-reasoned fix for the rossoctl#2490 reconcile error-loop: a missing spec.targetRef workload is now a recoverable degraded state (Ready=False/TargetResolved=False, reason TargetNotFound, stale status.Card cleared, V(1) log) instead of per-30s Error+Warning spam, with the Warning emitted only on transition (deduped via the pre-existing condition) and recovery driven by the existing workload watch.

The logic is idiomatic and thoughtful:

• errTargetNotFound sentinel + %w + errors.Is — correct.
• The transition-dedup reads the pre-reconcile condition, so the Warning fires only on entry into degraded — and the at most once across cycles test proves it via a FakeRecorder.
• Using a distinct TargetResolveError reason for genuine API errors is a nice detail: it stops a transient API error from setting alreadyDegraded and suppressing a later legitimate degrade event.
• status.Card cleared on degrade (tested); recovery asserts RequeueAfter==0 + Ready=True/Configured.

Tests are assertive and cover every claim (degrade conditions, at-most-once event, Card clearing, recovery), with AfterEach draining the finalizer to avoid state leakage between specs.

One non-blocking observation (not introduced here): the degraded path's unconditional every-60s Status().Update mirrors the pre-existing updateErrorStatus pattern, and For() has no generation predicate — a possible future optimization (guard the update on an actual change), but pre-existing and out of scope for this fix.

Areas reviewed: Go (reconciler control flow, status/condition handling, error wrapping), envtest coverage
Commits: 1, signed-off: yes (Assisted-By trailer correct)
CI: passing (Unit, Integration, E2E all green)

LGTM.