AI-assisted root-cause analysis for OSDC step failures (inline in job logs)

[huydhn]

Proposal: AI-assisted root-cause analysis for OSDC step failures

Problem

When a job step fails on OSDC, the in-pod hook wrapper currently emits a generic message:

[OSDC] Step script exited with code 1. This is a script/workflow error, not an infrastructure issue. Check the step logs above for the actual failure.

(Source: osdc/modules/arc-runners/templates/runner.yaml.tpl, the wrapper.js ConfigMap.)

It correctly tells users "this is your workflow, not OSDC infra," but it doesn't help them find the actual cause — they still have to scroll the log. Example: https://github.com/pytorch/pytorch/actions/runs/26852479075/job/79190579763#step:11:57114

Idea

When a step fails, have the wrapper send the failing step's log tail to an Anthropic Claude model on Bedrock (same models/pattern as claude-autorevert-advisor.yml), get a short root-cause analysis, and print it inline in the same job log, right under the existing [OSDC] error — so the likely cause + suggested fix show up exactly where the user is already looking.

Scope / intent

• Trigger on all non-zero step failures, fleet-wide.
• Analysis is a strictly optional, fail-open side effect: a bounded timeout, and any failure (no creds, network, model error, parse error, timeout) silently skips it and returns the job's real exit code unchanged. It must never block, delay unduly, or fail a job.
• Reuse the autorevert Bedrock approach (Opus 4.6).

A possible design + the feasibility constraints discovered so far are in a comment below. This is a discussion starter — details still TBD.

[huydhn]

Possible design (high-level — for discussion)

Where it runs

Inline in the existing wrapper.js hook (runner.yaml.tpl), at the result.code !== 0 branch. No changes needed in pytorch/pytorch workflows — the analysis appears in the same failing job's logs via a ::notice:: workflow command, just after the current [OSDC] error.

Sketch of the flow

1. Capture the log tail. spawnReal currently uses stdio: ['pipe', 'inherit', 'inherit'], so the wrapper never sees the step output. Switch to piped stdio and tee stdout/stderr — re-emit live (logs keep streaming) while keeping a small capped ring buffer (last N KB) in memory.
2. On non-zero exit, emit the existing message, then (if enabled + creds present) call Bedrock with the captured tail.
3. Emit the analysis as a short ::notice:: block (likely cause + suggested fix), or emit nothing on any error/timeout.

Model

Match autorevert: Claude Opus 4.6 on Bedrock (global.anthropic.claude-opus-4-6-v1).

Key feasibility constraints found (the hard parts)

• No AWS creds on the runner pod today. The arc-runner ServiceAccount has no IRSA. Need a new IAM role granting only bedrock:InvokeModel, annotated onto arc-runner (canonical pattern exists in modules/pypi-cache).
• No AWS SDK/CLI in the runner image, and wrapper.js is a single self-contained ConfigMap file. The Bedrock call must be pure Node stdlib — STS AssumeRoleWithWebIdentity (IRSA token) → SigV4-signed InvokeModel over https/crypto. Also runs in a tight heap (--max-old-space-size=128).
• Pods are IPv6-only, and bedrock-runtime has no public dualstack endpoint (unlike S3, which is why pypi-cache works over IPv6). This is the biggest open question — likely needs a Bedrock dualstack VPC interface endpoint (PrivateLink) per cluster, or NAT64/DNS64. Needs a spike to confirm dualstack support in the target regions.
• Latency on the critical path. The runner waits for the hook to exit, so Opus adds time to every failing job. Mitigate with a bounded timeout + fail-open, modest max_tokens, and likely no extended thinking.
• Per-cluster config. Bedrock region + model id templated via clusters.yaml → generate_runners.py. Clusters span us-west-1 / us-east-1 / us-east-2 (us-west-1 has limited Bedrock — cross-region or endpoint choice matters).

Open questions to discuss

• IPv6 → Bedrock egress path (interface endpoint vs NAT64).
• Cost at fleet-wide scale (Opus per failure) — do we want gating/sampling later?
• Is per-failure inline the right surface, or also persist (artifact/ClickHouse) like autorevert?

[huydhn]

cc @jeanschmidt