Production-readiness pass: observability, store-backed approvals, retention, shutdown, config validation

[shadowhunter-92]

What this PR does

Closes the gaps flagged in docs/ROADMAP.md as demand-gated / known limitations. The maintainer's own roadmap listed these as the work needed before the project is production-ready; this PR ships all of it.

Test suite: 157 passing, 7 skipped (was 136; +21 new tests). Skips need external resources (Postgres / redis).

The 8 things this PR ships

1. Observability (was roadmap #1)

• /metrics Prometheus endpoint: call counter (agentbridge_calls_total{src,dst,capability,decision}), latency histograms (agentbridge_call_duration_seconds, agentbridge_translate_duration_seconds), audit-entry gauge, per-agent budget gauges, pending-approvals gauge, HTTP request counter + duration histogram, rate-limit-hit counter, auth-failure counter.
• OpenTelemetry tracing (optional, lazy-initialized): set OTEL_EXPORTER_OTLP_ENDPOINT to ship spans. Gateway opens a span around every route_call.
• Structured JSON logging (AGENTBRIDGE_LOG_JSON=1, auto-on in production / k8s): one JSON object per line with ts, level, logger, msg, request_id, plus any extra= fields.
• Correlation IDs: every request gets an X-Request-ID (echoed in the response), surfaced in logs via a ContextVar.

2. Store-backed ApprovalQueue (was roadmap #2 — last piece of in-process state)

• Approvals now live in the durable store (InMemory / SQLite / Postgres) instead of in-process state. Multi-worker safe — no instance pinning required for approval traffic.
• New store.consume_approval() atomically transitions approved -> consumed, so two workers can't double-consume a one-shot grant.
• Schema additions: approvals table in SQLite + Postgres with indexes on status and agent_id.

3. JWKS auto-fetch for OIDC (was roadmap #3)

• When no static signing key is configured, the verifier fetches <issuer>/.well-known/openid-configuration to discover jwks_uri, then fetches + caches JWKS keys (TTL 15min, refresh on kid miss).
• Explicit AGENTBRIDGE_OIDC_JWKS_URL also supported.

4. Audit retention + legal hold

• POST /control/audit/checkpoint — sign the current audit head with Ed25519 so a third party can later prove the log wasn't truncated before this point.
• POST /control/audit/retention —
  • {"action":"truncate","seq":N} removes entries with seq < N (safe after a checkpoint).
  • {"action":"legal_hold","on":true} freezes truncation (subsequent truncate attempts return 409).
• New store.truncate_audit_before(seq) on all 3 backends.

5. Graceful shutdown + split health probes

• lifespan installs SIGTERM/SIGINT handlers; flips readiness to False; drains in-flight requests up to AGENTBRIDGE_SHUTDOWN_GRACE (default 10s); then closes.
• /health — liveness (always 200, even during drain, so k8s doesn't restart the pod mid-shutdown).
• /ready — readiness (503 during drain OR if the governance store is unreachable).
• /version — build info (version, Python, store type).
• CLI passes graceful-shutdown timeout through to uvicorn.

6. Retry/backoff on transient store errors

• append_audit_chained and mutate_budget now retry on SQLite database is locked / psycopg OperationalError (up to 4 attempts, exponential + jitter, capped at 0.5s). Permanent errors bubble immediately.

7. Config validation at startup

• New src/config.py validates env vars before any state is created.
• Production requires AGENTBRIDGE_ADMIN_KEY and AGENTBRIDGE_DB; rate-limit and shutdown-grace values are range-checked; OIDC issuer must be a URL; psycopg must be importable when a postgres URL is configured.
• Errors raise ConfigError (fail-fast at boot, not first request).
• Warnings emitted when admin key is missing/short, when in-memory store is used, when OIDC has no signing key.

8. Production-safety polish

• FastAPI docs (/docs, /redoc) suppressed when AGENTBRIDGE_ENV=production unless AGENTBRIDGE_DOCS=1.
• CLI serve improvements: --workers N, --log-level, disables uvicorn's noisy access log (we have our own structured middleware).
• prometheus-client added as a runtime dependency; [otel] optional extra added for OpenTelemetry.

File changes (17 files, +2072/-214)

New files (5):

• src/observability/__init__.py — Prometheus metrics + OpenTelemetry tracing
• src/observability/logging.py — structured JSON logging with correlation IDs
• src/config.py — startup config validation (fail-fast)
• src/governance/resilience.py — retry/backoff on transient store errors
• tests/test_production_readiness.py — 21 new tests

Modified files (12):

• src/api/control_plane.py — middleware rewrite, /metrics + /ready + /version, lifespan, validate_config, audit retention endpoints, AUTH_FAILURES / approvals gauges
• src/api/auth_oidc.py — JWKS auto-discovery + cache + key rotation on kid miss
• src/governance/store.py — approval table + methods, consume_approval, truncate_audit_before, retry on atomic ops
• src/governance/approvals.py — store-backed rewrite
• src/governance/audit.py — set_legal_hold, truncate_before
• src/governance/gateway.py — span + metrics instrumentation
• src/protocols/registry.py — translation-duration metric
• src/cli.py — --workers, --log-level, graceful shutdown
• pyproject.toml — prometheus-client runtime dep, [otel] extra
• CHANGELOG.md, docs/ROADMAP.md, docs/ENTERPRISE.md — updated to reflect what's now done

Test results

157 passed, 7 skipped in ~10s

Skips:

• 6 Postgres integration tests (need AGENTBRIDGE_TEST_PG=postgres://...)
• 1 conformance test (needs redis)

New test coverage in tests/test_production_readiness.py:

• Store-backed approvals: persistence, one-shot consume, idempotent approve/deny
• Audit retention + legal hold + checkpoint signing + verification
• Prometheus metrics rendering
• Structured JSON logging
• Config validation (5 scenarios: clean dev, prod-missing-key, prod-missing-db, bad rate-limit, bad OIDC issuer)
• Retry/backoff (3 scenarios: succeeds-after-transient, gives-up-after-max, no-swallow-permanent)
• /health + /ready + /version + /metrics endpoints
• Request-ID echo
• Full audit-retention HTTP round-trip

What's NOT in this PR (still genuinely demand-gated)

• Streaming/multi-turn canonical model (would change wire bytes; needs live-agent re-validation)
• SIEM push connectors (the export + checkpoint primitives are there; the shippers are a small future addition)
• Engine/mesh consolidation (same — changes wire bytes)
• ANP support (correctly deferred to identity/discovery plane)
• Managed hosting / SOC 2 (operations, not code)

Reviewer notes

• The PR is large (~2K LOC) but each file's changes are tightly scoped to its feature.
• Suggested review order: src/observability/ → src/governance/resilience.py → src/governance/store.py (approvals + truncate) → src/governance/approvals.py + audit.py → src/config.py → src/api/control_plane.py (the biggest single file) → src/api/auth_oidc.py → tests.
• All existing tests pass unchanged (no behavioral regression on existing endpoints).
• The ApprovalQueue(store=store) change in control_plane.py is the only line that changes existing runtime behavior — approvals now persist across restarts.

Disclosure

This PR was produced with AI assistance (Claude) under human direction. The author reviewed every change, ran the test suite, and verified the end-to-end smoke test (see CHANGELOG for the smoke-test checklist).

[shadowhunter-92]

Reviewed file-by-file and fixed all findings (commit d4a2a4b — CI now green on 3.11 + 3.12):

• requirements.txt was missing prometheus-client (CI red) → added
• CLI graceful-shutdown timeout was ×1000 (~2.8h) → seconds
• audit truncation broke verify_chain/verify_durable → require_genesis + auto-detect; truncated chain stays verifiable
• JWKS resolution was fragile/untested → rewritten to cache the key object + a round-trip test
• gateway copied the whole audit log per call to count it → AuditLog.count() (O(1))
• resilience retried permanent errors (caught sqlite3.DatabaseError) → OperationalError only
• HTTP metrics labeled by raw path (cardinality bomb) → route template
• tests strengthened; 8 files normalized to mode 644

Decision: parking, not merging. Solid, green, mergeable work — but it's demand-gated production infra (observability / audit-retention / JWKS) and there are no production users yet. Keeping this branch ready; will merge the moment a design-partner needs metrics or audit-retention. main stays lean.