feat(oncall-agent): read-only Slack-triggered cluster triage agent - #293
Draft
manan164 wants to merge 38 commits into
Draft
feat(oncall-agent): read-only Slack-triggered cluster triage agent#293manan164 wants to merge 38 commits into
manan164 wants to merge 38 commits into
Conversation
Scaffolds an Agentspan agent that triages Orkes SaaS health-check alerts. It listens on the Slack alert channel, reads the failing health_check execution by id, and runs READ-ONLY agent-handler commands against the ah5r-prod Conductor API to investigate, then replies in-thread with a root-cause hypothesis. Advisory/dry-run only — no remediating actions. - sql_guard: deterministic SELECT-only guard (not LLM-trusted) for the run_sql_select tool; rejects DML/DDL, multi-statement, comment-smuggling. - conductor_client: dispatch read-only agent-handler workflows + poll; reads org/cluster/cloudEnvironmentTag off the failing execution so only the executionId is threaded into tools. - tools: 9 read-only investigation tools mapped to agent-handler commands. - agent: Claude triage loop with a component->investigation runbook. - slack_app: Socket Mode listener -> triage -> threaded reply. - tests: sql_guard + alert parsing, deterministic (no LLM), validated by proving each fails before passing per CLAUDE.md. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ention Follows the pattern in sdk/python/examples/91_slack_autofix_agent.py (per PR #135): poll the alert channel with conversations.history + reply via chat.postMessage using a bot token only — no slack_bolt / Socket Mode, no app-level token. Run-once or --loop, dedup via a local state file. Slack I/O lives in a deterministic poller; the triage agent stays pure (investigates a given execution id). Adds test_poller.py covering alert-only triage, cross-poll dedup, and failure reporting (fakes, no network/LLM; validated fail-then-pass per CLAUDE.md). Config: drop SLACK_APP_TOKEN; add SLACK_ALERT_CHANNEL (required), ONCALL_POLL_INTERVAL, ONCALL_STATE_FILE. requirements: drop slack-bolt, add requests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
- test_tools_readonly: source-level invariant that no mutating/privileged agent-handler command is wired into tools.py, and SQL goes through the SELECT guard. Validated by adding DELETE_POD and confirming failure. - scripts/smoke_dispatch.py: LLM-free live check against ah5r-prod — reads the failing execution's cluster context, dispatches read-only commands (get_pods_data, get_cluster_metrics, SELECT 1), asserts COMPLETED + output shape. The L1 verification step; run with the Conductor app key. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r domain Two bugs found during the first live run against ah5r-prod (viz-stage): 1. cloudEnvironmentTag is NOT in the health_check workflow input (it's produced by prepare_agent_handler's output), but sql_conductor reads it from workflow.input — derive it as c<orgId[:5]>-<clusterName> when absent. 2. Dispatched commands set no task_to_domain, so customer-cluster tasks (e.g. collect_metrics) sat in the default queue and the in-cluster agent never polled them -> TIMED_OUT. Mirror the control plane: wildcard "*" -> the cluster domain (orgId#-#clusterName), with orchestration tasks pinned to NO_DOMAIN. Switched dispatch to StartWorkflowRequest to carry task_to_domain. Verified live: GET_PODS_DATA and PULL_LOGS now COMPLETE end-to-end with real viz-stage data. Adds deterministic regression tests (fake client) for both. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Validated end-to-end against ah5r-prod (viz-stage Redis-critical alert): the agent now autonomously reads the health-check data and pulls server + worker logs to reach a root-cause hypothesis. Fixes found during the live run: - runtime_compat: on macOS, run conductor tool-workers as THREADS, not forked processes. Forked children segfault in getaddrinfo (Network.framework is not fork-safe) and 'spawn' can't pickle the worker's thread lock. agentspan ships a thread shim but gates it to Windows; reuse it on macOS. No-op on Linux (where fork is safe — how this runs in prod). Wired into triage + slack paths. - get_incident_details: surface parse_conductor_cluster_data (redis.usage, decider_queue_size = running workflows, indexer_queue_size, heap, cpu, postgres) so the agent reads the queue numbers from the health-check JSON instead of deriving them via SQL. - runbook: treat queue/usage as the symptom; find the cause in CONDUCTOR SERVER and WORKER pod logs (+ pod events). Explicitly forbid ad-hoc SQL on large tables like `workflow` (decider_queue_size already is the running-workflow count). run_sql_select is a last resort, not the primary tool. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…check alerts - agent.py: add a compact alert-type playbook covering all 17 HealthIssue types (resource saturation, component-down, pod, networking, self-describing). Preserves the existing Redis/CPU/heap guidance; routes the model symptom -> evidence -> cite. - tools.py: add read-only get_ingress_info (GET_INGRESS_INFO) for DNS / domain resolution / reachability alerts (empty ingress address = LB not provisioned). - test_tools_dispatch.py: assert each tool dispatches its expected agent-handler command (incl. get_ingress_info) via a fake dispatcher. Validated fail-then-pass. - test_tools_readonly.py: fix command names to match the AgentHandlerCommand enum (ROLLOUT_RESTART, KUBECTL_UNRESTRICTED, ...) so the guard actually catches them; add heavy/disruptive commands (DOWNLOAD_HEAP_DUMP, etc.) to the deny list.
…verage Verified against the source workflow/worker (not assumed): - get_incident_details ref "issues" matches health_check.json taskReferenceName, and the issues task output carries the per-issue severity+description text the playbook matches on (HealthCheckIssuesWorker) — the 17-type playbook is actually reachable. - alert.py: the real Slack text is markdown (*`[CRITICAL]`* … _Org's_ env *`cluster`*) + emoji + appended execution URL. The old _CLUSTER_RE choked on the italic/bold markers and returned org/cluster=None. Strip *,` decoration and tolerate italic _ so org/cluster parse; execution id + severity were already fine. Added a test built from the exact worker+notify format; validated fail-then-pass. - test_playbook_coverage.py: deterministic guard that all 17 HealthIssue types stay in the playbook; validated it fails when a type is dropped.
…ai.agents The main merge into this branch (cd689ae) renamed the Python SDK package from `agentspan` to `conductor` (dist `conductor-agent-sdk`), import root `conductor.ai.agents`. oncall-agent still imported `agentspan.agents`, so the module could not import its SDK at all on the post-merge branch. Swap the five SDK imports (Agent, tool, AgentRuntime, worker_manager shim) to the new namespace. Left the local `agentspan_server_url` config field + AGENTSPAN_SERVER_URL env var as-is — they're our own naming, not SDK symbols, and the env var is documented. Full suite passes (37) against the new namespace.
…spatcher Runs the actual conductor.ai.agents AgentRuntime + agent reasoning + tool-calling against the local server, with the Conductor dispatcher replaced by canned fixtures so it never touches ah5r-prod. Validation is deterministic (CLAUDE.md): asserts every dispatched command is read-only and the runtime returned output — does NOT LLM-judge the hypothesis. Verified live: agent reads the incident first, then dispatches only read-only commands (GET_CLUSTER_METRICS, GET_PODS_DATA, GET_POD_EVENTS, PULL_LOGS x2), follows the Redis->decider-queue->logs playbook, and emits the Issue/Findings/Root-cause/Next-step summary. Proves the SDK migration runs end-to-end and the read-only guard holds at runtime.
…DOMAIN_REACHABILITY playbook Two-step logic for the 'domain X is down' (reachability/502) alert: first rule out Conductor itself (server pod crashloop/OOM via pods/events/logs); if the server is healthy, treat the 502 as the network/ingress layer. Encodes the ingress-nginx stale-endpoint failure mode (pod restart -> new IP -> a controller replica keeps routing to the dead IP, inconsistent across replicas), with graceful degradation if the ingress namespace isn't visible to the read-only tools. Fixes the prior blind spot where a provisioned-LB reachability alert was reflexively called 'external'. No coverage-test change: 'domain X is down' anchor preserved.
…CHRONIC The agent triaged every alert as a fresh incident, missing that a chronic flapper (this one fired on ~30% of recent one-staging health-checks) is a standing capacity problem, not a page. Adds get_alert_recurrence: one search filtered by the unique clusterId UUID (NOT the fuzzy cluster name), classified locally by a pure, retention-aware summarize_recurrence(). Instructions now run the recurrence check early and lead the summary with NEW/RECURRING + the retention caveat (true onset may predate the search window -> Slack/Prometheus). Tests: pure classifier, validity proven by mutation; ISO-8601 start_time parsing locked (caught live).
The server returns agent output as {result, finishReason, context,
rejectionReason}; the Slack reply path concatenated that dict into the
message header and crashed (TypeError) at the last step of the live
e2e run. Extract the text via runtime_compat.summary_text (accepts both
the dict shape and the older plain-string output) in both the poller
and the CLI triage path. Repro test: DictOutputRuntime in test_poller,
failed with the exact live TypeError before the fix.
Live e2e showed 'I have all the evidence needed. Let me compile...' leaking into the Slack post; the final message is posted verbatim.
The instruction-only fix didn't hold — the second live run still leaked 'I have all the data I need...' into the Slack reply. Enforce the output contract in summary_text(): drop everything before the mandated *Issue*: header. Test extended with the real leaked-preamble shape; failed before.
…r reported NEW Live miss (2026-07-22): agent passed signature 'Pod Failed' (pod id correctly stripped per instructions) but the reason text is 'Pod orkes-agent-…-shldc Failed' — substring match found nothing, so a 100/100-runs chronic flapper was reported first-seen/NEW. Match every significant signature word as a token in any position; drop numeric tokens on both sides (percentages vary per firing). Live re-check now yields RECURRING/CHRONIC matched=100/100.
The digest channel posts one message per (cluster, alert-type) incident and edits it in place with an occurrence counter — polling it gives flapper dedup for free. The headline text carries no execution URL; the original alert is quoted inside a section block. Flatten text + block mrkdwn (message_text) before parsing, so both raw-channel and digest messages parse with the same parser, and the occurrence count reaches the triage prompt. Fixture is the live At-Bay HEAP_HIGH digest.
…scorable report Production gate: human-scored hypotheses over real alerts. eval_select dedupes the flapper-dominated stream to unique incidents (cluster + number-stripped token set, same normalization as recurrence matching; poll-timeout noise excluded). scripts/eval_batch.py replays each through the agent (one shared runtime) and writes a markdown report with a Useful/Partly/Wrong score line per incident. Reports are generated output and are not committed.
…ifests Two-container pod: agentspan/server:latest sidecar (ANTHROPIC_API_KEY from secret at boot — the known cold-start requirement) + poller image (python:3.11-slim, SDK installed from sdk/python). Entrypoint waits for the sidecar's /health so the SDK never tries to auto-install a CLI in the container. Digest channel is the default source; Recreate strategy keeps the dedup state single-writer (PVC); kill switch = scale to 0. Image builds and imports verified (11 tools registered).
… not -Xmx bumps Team guidance from Manan: raising the heap ceiling by default masks leaks. The playbook now mandates heap dump -> dominant retainers (MAT) -> map to recently deployed changes; rolling restart only as short-term relief; limit increase only after the dump justifies it.
…P_DUMP Team decision: for heap alerts the agent should not tell the engineer to capture a dump — it dispatches ah5r-prod's download_heap_dump on the single highest-heap pod (once per incident; jmap is stop-the-world, so the tool and playbook both forbid multi-pod dumps and non-memory use) and reports the stored dump paths for MAT + recent-changes analysis. DOWNLOAD_HEAP_DUMP moved off the banned list with the rationale recorded in the guard test. Dispatch contract covered by deterministic tests.
SLACK_ALERT_CHANNEL now accepts a comma-separated list; dedup state is kept per channel (ts values are channel-scoped in Slack), the legacy single-channel state shape migrates onto the first configured channel, and a failing channel cannot starve the others. k8s manifest watches both the raw alert channel and the aggregator digest channel.
…or CPU triage download_thread_dump(execution_id, pod) dispatches ah5r-prod's download_thread_dump workflow (jstack — cheap, near-zero pause) and returns the stored dump paths. Playbook CPU EVIDENCE RULE: when a CPU alert's cause isn't visible in logs, dump the hottest pod's threads and include the paths in the summary. DOWNLOAD_THREAD_DUMP moved off the banned list alongside DOWNLOAD_HEAP_DUMP with rationale in the guard test; DOWNLOAD_ALL_POD_LOGS stays banned.
…e it If the draft next step tells the engineer to check/count/verify something the agent's own read-only tools can answer, the agent must do it and move the answer into Findings; the final next step may contain only actions it cannot take (remediation, offline analysis, business decisions). When a check is impossible read-only, say exactly why instead of delegating it.
…window Live: the raw channel fired the same shailesh-test-gcp TIMED_OUT alert 4x in <1h (fresh execution id each firing) and each got a full LLM triage + thread posts. alert_signature() = sorted word tokens with URLs stripped and number/hex tokens dropped — stable across firings of one incident, distinct across clusters/types. Within ONCALL_SIGNATURE_COOLDOWN (default 3600s) a repeated signature is marked processed but not re-triaged and posts nothing; suppression is cross-channel, so an incident seen in both raw and digest channels is triaged once.
…fail playbook run_kubectl_read dispatches KUBECTL_UNRESTRICTED behind a deterministic allowlist guard (kubectl_guard: get/describe/logs/top/events/explain/ auth can-i/rollout history|status; shell metacharacters rejected) — same philosophy as sql_guard, validated live on orkes-wvuf-prod (namespaced reads work; the agent SA is namespace-scoped so -A is Forbidden, documented). Playbook: TIMED_OUT/lost-telemetry alerts now fast-fail to AGENT_DOWN after one hung probe — every tool executes through the in-cluster agent, so when the agent is down there is no read-only path (kubectl_unrestricted included: RunKubectlWorker runs in orkes-saas-agent) and a human with kubectl/cloud-API access is required.
…ture Live: one-staging fired the same CPU-100% alert twice in 10 min naming a different conductor pod each time (…-65pkn vs …-pv6m5) — the 5-char k8s pod suffix survived the hex-only filter, so the cooldown saw two incidents. Drop every mixed digit+letter token (identifiers by nature: uuid/hex fragments, ReplicaSet hashes, pod suffixes); subsumes the old hexish rule.
…t wording Live (orkes-prod): the same CPU condition fired as 'following issue' naming pod …-q9h9d, then as 'following 2 issues' adding pod …-pvxjg — an all-letter k8s suffix the digit+letter rule misses, plus plural boilerplate. k8s suffixes are vowel-free by design, so drop short vowel-less tokens; normalize issues->issue. Repro test from the real message pair failed before.
…g triage once Live: the raw channel's TIMED_OUT message and the digest channel's aggregated message referenced the same execution but tokenize differently, so signature suppression missed the pair and the same execution was triaged twice. An execution id, once triaged, is never triaged again (state['executions'], newest-500 cap); signature cooldown still handles fresh executions of a flapping incident.
Live failure (2026-07-23, twice): a transient network blip left
conductor-python's shared httpx client with a dead socket ('Bad file
descriptor') and a poisoned auth token; every retry reused the broken
client, tools failed indefinitely, and the sequential poll loop wedged
for ~4h. ConductorDispatcher now routes every client call through
_call(): on failure it rebuilds the workflow client (fresh pool + fresh
token exchange) and retries once. Injectable client_factory for tests.
…runs Live: the arm was set in memory before a multi-minute triage but only saved after it; an exception escaping mid-triage (failed Slack post) is absorbed by the per-channel guard and silently discarded the arm — the state file stayed 30+ min stale while duplicates of orkes-prod and zweorksksp001 re-triaged. Save immediately after arming; the post-triage save still records processed/last_ts. Crash-mid-triage now trades a duplicate triage for a possible dangling 'starting' marker (visible, rare, cheaper).
…on, no queue-number-only attributions Manan's review: CPU alerts were converging on 'decider backlog' with empty ERROR greps as the only log step — sweeper churn logs at INFO, so the grep found nothing and the agent inferred causation from the queue size. Now mandatory on CPU alerts: unfiltered 300-line tail of the hottest pod with dominant-pattern-by-volume cited as evidence, INFO-marker greps (sweeper/decider/timed-out/S3-abort/broken-pipe), then thread dump; queue-number-only attribution is forbidden — the summary must cite log volume or state the logs did not confirm.
Manan's review: 'fired 11 of the last 100 health-checks (11%)' counts against ALL runs — most of which pass — and buries the ratio on-call actually needs: of the checks that FAILED, how many were this alert? RecurrenceReport now carries failing_count and fraction_of_failing, and the RECURRING summary reads 'N of the M failing checks (X% of failures)' alongside the window count.
…ing them Manan's review: every repeat firing of a known incident burned a full LLM triage. Now the first firing runs the full investigation and its *Likely root cause* is remembered per signature (state['incidents']); repeat firings within ONCALL_FULL_TRIAGE_INTERVAL (default 6h) get a deterministic in-thread update built from memory — prior diagnosis, firing count, span — with zero LLM tokens. A full re-investigation runs after the interval WITH the prior diagnosis in the prompt (verify + report delta, not rediscover). Signature changes (severity escalation, new issue set) bypass memory by design. Chronic clusters drop from ~24 to ~4 LLM triages/day.
…nt, one memory Live: endpoint-dev's memory was seeded from the raw-channel form, then the digest form of the same incident signed differently (headline + occurrence tokens) and bought a duplicate full triage. signable_text() extracts the blockquoted original alert from digest messages (raw messages pass through), so raw and digest forms share a signature — cross-channel dedup and incident memory now cover both.
…ent top-level Live 2026-07-27: twilio-non-prod-us paged 12 times (24 agent pods Failed) and both triages posted an invented root cause — "the cluster has been deregistered from the Orkes control plane" — because get_context saw clusterId AND clusterName as null and the model reasoned from the nulls. Twilio's health_check schedulers pass only organizationId top-level and put the cluster under agentHandlerRequest.clusterName; Ocean's pass clusterName/clusterId top-level, which is why this never surfaced. clusterName is load-bearing: dispatch builds the agent routing domain from it, so it became "<org>#-#None", prepare_agent_handler FAILED, and every read-only tool returned nothing. Verified against the live execution — with the fallback, prepare_agent_handler completes and get_pods_data / kubectl reads return real cluster data. clusterId stays unresolved for these clusters (it appears nowhere in the execution, all 28 tasks scanned), so get_alert_recurrence still reports no_cluster_id and GET_CLUSTER_METRICS times out for them — the incident's own clusterData covers the metrics. Fixing that belongs in the scheduler definitions.
…26-07-30 outage - never diagnose from a single execution (compare prior runs; invariant signature = persistent, not transient/GC) - Conductor-has-failed + healthy pods -> probe serving path (:5000 UI front-end vs :8080 API), not the JVM - capture evidence (events, kill -3 stacks, live ingress logs) before recommending restarts; restarts destroyed evidence and fixed nothing - repeated identical failure after a restart = retry-storm re-wedge: recommend load-shed before any further restart - always surface FailedScheduling / zero-headroom when pods churn
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(oncall-agent): read-only Slack-triggered cluster triage agent
What this is
An Agentspan agent that triages Orkes SaaS cluster health-check alerts. It polls the Slack
alert channel (Web API, no Socket Mode), parses the
executionIdfrom the failinghealth_checkexecution URL, runs read-only agent-handler commands against the ah5r-prodConductor API to investigate, and replies in-thread with a root-cause hypothesis.
It is strictly read-only and advisory — it never takes a remediating action. SQL is gated by
a deterministic SELECT-only guard (
sql_guard.py), not by trusting the model. The agent readsorganizationId/clusterName/cloudEnvironmentTagoff the failing execution, so the LLMonly threads the
executionIdinto each tool — it cannot target the wrong cluster and no secretspass through tool args.
Done ✅
Slack ingestion — Web API poller (
conversations.history+chat.postMessage), bot tokenonly, run-once /
--loop, state-file dedup. Matchessdk/python/examples/91_slack_autofix_agent.py.Alert parsing (
alert.py) — extracts execution id + severity + org/cluster from the alert text.Conductor dispatch (
conductor_client.py) — starts agent-handler command workflows onah5r-prod via conductor-python (app key/secret), polls to completion, derives + caches cluster context.
Read-only tool set (
tools.py) —get_incident_details,get_cluster_metrics,get_infrastructure_metrics,get_pods_data,get_deployments_info,get_pod_events,get_top_output,pull_pod_logs,get_ingress_info,run_sql_select(SELECT-guarded).SQL safety guard (
sql_guard.py) — SELECT/WITH/EXPLAIN/SHOW only; rejects every mutation,multi-statement, and comment-smuggling case before it reaches the DB.
Per-alert-type triage playbook for all 17 health-check alert types (
agent.py) — symptom →evidence-to-gather → what-to-cite, matched off the
issuestext. Keeps the strong Redis /decider-queue / CPU / heap guidance and extends to component-down, pod, networking, and
self-describing alerts (table below).
Tests (deterministic, no LLM in the assertion path, per
CLAUDE.md) —test_sql_guard.py,test_alert.py,test_poller.py,test_conductor_client.py,test_tools_readonly.py(read-only safety guard, pinned to the real
AgentHandlerCommandenum names),test_tools_dispatch.py(each tool dispatches its expected agent-handler command). 45 passing.Local run path (
python -m oncall_agent.main [triage <execId>]),.env.example, README.Live end-to-end run (2026-07-22) — the FULL production path: poller read a real MAJOR
alert from a Slack channel (
collective-staging,Pod orkes-agent-deployment-* Failed,exec
d552305e-85ce-11f1), parsed it, ran the triage agent with real read-only agent-handlerdispatches to ah5r-prod (~80s, recurrence check + pods + events + logs + metrics), and posted
a correct root-cause hypothesis in-thread (stale Failed pod superseded by a 2026-07-20
rollout; cluster already self-healed). The first run caught a real bug at the last step —
the runtime returns
result.outputas a dict ({result, finishReason, ...}) and the Slackpost crashed concatenating it; fixed via
runtime_compat.summary_text()with a repro testthat failed with the exact live TypeError first. An earlier CLI-path run (2026-07-03,
triage <execId>) had validated the investigation loop.Digest-channel support (2026-07-22) — polls the alert-aggregator channel (one message per
(cluster, alert-type) incident, edited in place with an occurrence counter → flapper dedup for
free). Block-text flattening (
alert.message_text); validated live (At-Bay HEAP_HIGH triagedin-thread). Token-based recurrence matching fixed the flapper-reported-as-NEW miss.
Eval batch tooling (
scripts/eval_batch.py+eval_select.py) — dedupes the raw stream tounique incidents and replays them into a human-scorable markdown report. First run: 6/6 unique
48h incidents triaged clean (2× pod-failed, 3× CPU, 1× heap), ~90s each.
Containerization (
Dockerfile,deploy/k8s.yaml) — two-container pod (agentspan serversidecar + poller), digest channel default, single-writer state on PVC, kill switch = scale to 0.
Image build + in-container import smoke verified.
To do 🚧
kubectl apply.HITL approval gate.
The 17 health-check alert types — all now have a playbook
Source of truth:
HealthIssueenum inorkes-saas/.../worker/HealthCheckIssuesWorker.java. "Approach" = howagent.pytriages it.REDIS_CRITICAL_USAGEREDIS_HIGH_USAGECONDUCTOR_HIGH_HEAP_USAGECONDUCTOR_HIGH_CPU_USAGECONDUCTOR_ERROR_LOGS_COUNT_EXCEEDED_THRESHOLDCONDUCTOR_WARN_LOGS_COUNT_EXCEEDED_THRESHOLDCONDUCTOR_HEALTHY(failed)WORKERS_HEALTHY(failed)PROMETHEUS_NOT_RUNNINGPOD_NOT_RUNNINGPOD_RESTARTEDDNS_HEALTHY(failed)get_ingress_info— no address ⇒ LB unprovisioned; else external → infraDOMAIN_RESOLUTIONget_ingress_info→ resolve vs escalate to infraDOMAIN_REACHABILITYget_ingress_info→ reachable vs escalate to infraAUTH_STALEDOMAIN_CERTIFICATE_WILL_EXPIRERESPONSE_TIMEOn testing the playbook: the playbook is prompt text — by
CLAUDE.mdrule 1 it can't beLLM-judged in unit tests and isn't deterministically assertable, so it has no unit test by design.
What is tested deterministically: the read-only safety guard and the per-tool dispatch contract
(
get_ingress_info→GET_INGRESS_INFO, mutating commands stay unreachable). Playbook quality isvalidated in the eval step above.
Testing
Out of scope (v1)
Triage + read-only investigation only. No remediation (restart/scale/rollback).