API auth failure detection and auto-recovery

[FrederikHandberg]

Problem

Two authentication_failed errors hit on Mar 21, 10 seconds apart. Root cause was accepted as 'transient' without evidence. If auth fails mid-job, the job fails silently — no notification to Frederik, no retry mechanism.

Auth failures during running jobs → status: failed, no alert.

Fix

1. Pattern detection: If 2+ auth failures occur within 1 hour, automatically:

   • Create a GH issue with diagnostics (time, job IDs, error text)
   • Send ntfy notification
   • Stop queueing new Claude jobs until manual clearance

2. Auth preflight: Before every job, verify Claude API auth:

   # Quick API health check
   curl -s -o /dev/null -w '%{http_code}'      -H 'x-api-key: $ANTHROPIC_API_KEY'      'https://api.anthropic.com/v1/models' | grep -q '200'

3. Retry with backoff: On single auth failure mid-job, wait 30 seconds and retry once before marking failed.

4. Morning report: Include API auth status (last successful call, any recent failures)

Acceptance Criteria

• Auth failure count tracked per hour
• 2+ failures in 1h → GH issue auto-created
• Single failure → retry once with 30s backoff
• Morning report shows auth health
• Preflight check before every job

Context

From Jensen limitation audit (2026-03-22). Workstream: WS-001.

[FrederikHandberg]

Partial fix: morning-check.sh now scans recent job YAMLs for authentication_failed errors and includes the count in the morning report health context. Full fix (pattern detection, auto-GH-issue, retry backoff) is deferred as it requires worker.py changes with more invasive logic. Commit: 68cb5ed

[FrederikHandberg]

Partial implementation in commit fda69df (tutoria-hub/jensen):

Completed (4/5 ACs):

• AC#1: Auth failure count tracked per hour in logs/auth-failures.log
• AC#2: 2+ failures in 1h → GH issue auto-created via _maybe_create_auth_issue() in worker.py
• AC#3: Single auth failure → 30s backoff retry via _pane_has_auth_failure() detection
• AC#4: Morning report now reads auth-failures.log for accurate 24h count

Not implemented:

• AC#5: Preflight check before every job — skipped because ANTHROPIC_API_KEY is managed internally by the claude binary (not available as env var for curl). Detection happens post-failure via pane scan instead.

Root cause fix also included: SHELL_IDLE_GRACE increased from 30s → 120s. The 30s idle detection was firing false positives during Claude API wait times (60-90s between tool calls is normal), causing jobs to be marked failed while Claude was still actively running.