Production-readiness pass: observability, store-backed approvals, retention, shutdown, config validation

CHANGELOG.md

@@ -5,36 +5,92 @@ All notable changes to AgentBridge will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
-## [Unreleased]
-
-### Added — production/repo hygiene
-- GitHub Actions **CI** workflow: runs the test suite on Python 3.11 and 3.12 (`.github/workflows/ci.yml`).
-- **`pyproject.toml`** packaging with optional extras (`[test]`, `[postgres]`, `[dev]`).
-- Root **`Dockerfile`** + **`docker-compose.yml`** (one-command run; healthcheck on `/health`).
-- Reference **Kubernetes** manifests (`k8s/`) — a starting point; validate against your own cluster.
-- **`agentbridge` CLI** (`python -m src` or `src.cli`): `serve`, `mcp`, `translate`, `demo`, `quickstart`, `--version`.
-- Static **status dashboard at `/dashboard`** — shows live `/health` and `/control/protocols` (no mock data).
-- README **badges** (CI, Python 3.11/3.12, license).
-- `Makefile`, `.pre-commit-config.yaml`, `.editorconfig`, `.env.example`, `MANIFEST.in`.
-- `docs/FAQ.md`, `SECURITY.md`, `CODE_OF_CONDUCT.md`, GitHub issue/PR templates, Dependabot config.
-- `Dockerfile.dev` for development; one-line setup scripts (`scripts/install.sh|.ps1`, `scripts/run_demo.sh|.ps1`).
-- `examples/README.md` (examples guide); rendered `docs/architecture.png`; `docs/adr/` (architecture decision records).
-- `RELEASING.md` release checklist; `.github/FUNDING.yml` (GitHub Sponsors).
-- Additional GitHub Actions — `publish.yml` (PyPI on release), `docker-publish.yml` (Docker Hub on release), `ci-manual.yml` (manual): all **dormant** (release/manual-triggered, require secrets) so they can't produce a red badge before they're configured.
-- `tests/conftest.py` with reusable fixtures using the **real** governance API (replaces a hallucinated draft).
-
-> Deliberately NOT wired yet (needs an external account/secret to function): Codecov coverage upload + badge.
+## [Unreleased] — production-readiness pass
 
-### Changed
-- Dependency floors bumped (pydantic ≥2.13.4, httpx ≥0.28.1) and GitHub Actions updated (checkout v7, setup-python v6) via grouped Dependabot PRs.
-- **Dependabot** reconfigured to **monthly + grouped** minor/patch updates (one PR instead of ~15 at once) and to **ignore `a2a-sdk` major bumps** (1.x breaks the 0.3.x conformance tests).
-- CI now prints a **coverage report** (`pytest-cov`).
+### Added — observability
+- **Prometheus metrics** at `/metrics`: call counter (`agentbridge_calls_total{src,dst,capability,decision}`),
+  call latency histogram (`agentbridge_call_duration_seconds`), translation latency histogram
+  (`agentbridge_translate_duration_seconds`), audit-entry gauge, per-agent budget gauges,
+  pending-approvals gauge, HTTP request counter + duration histogram, rate-limit-hit counter,
+  auth-failure counter. Uses a private `CollectorRegistry` so it never collides with other libs.
+- **OpenTelemetry tracing** (optional): set `OTEL_EXPORTER_OTLP_ENDPOINT` or
+  `AGENTBRIDGE_OTEL_ENABLED=1` to ship spans. The governance gateway opens a span around
+  `route_call` with `agent_id`/`src`/`dst`/`cost` attributes. Lazy-initialized, no-op safe
+  when the OTel SDK isn't installed.
+- **Structured JSON logging** (`AGENTBRIDGE_LOG_JSON=1`, on by default in production / k8s):
+  one JSON object per line with `ts`, `level`, `logger`, `msg`, `request_id`, plus any
+  `extra=` fields. Plain-text fallback for dev.
+- **Correlation IDs**: every request gets an `X-Request-ID` (echoed in the response), and
+  the log formatter picks it up via a `ContextVar`. Slow-request warnings logged above
+  `AGENTBRIDGE_SLOW_LOG_SECONDS` (default 2s).
+
+### Added — reliability
+- **Graceful shutdown**: `lifespan` installs SIGTERM/SIGINT handlers that flip readiness
+  to False, drain in-flight requests up to `AGENTBRIDGE_SHUTDOWN_GRACE` (default 10s),
+  then close. The CLI passes the same value to uvicorn's `--timeout-graceful-shutdown`.
+- **Split health probes**:
+  - `/health` — liveness (always 200, even during drain, so k8s doesn't restart the pod mid-shutdown).
+  - `/ready`  — readiness (503 during drain OR if the governance store is unreachable).
+- **`/version`** endpoint (build + Python + store type).
+- **Retry with backoff** on transient store errors: `append_audit_chained` and
+  `mutate_budget` now retry on SQLite `database is locked` / psycopg `OperationalError`
+  (up to 4 attempts, exponential + jitter, capped at 0.5s). Permanent errors bubble immediately.
+- **Store-backed `ApprovalQueue`**: approvals now live in the durable store (InMemoryStore
+  for tests, SQLite/Postgres in prod) instead of in-process state. Multi-worker safe —
+  the last piece of in-process runtime state is gone. Atomic `approved -> consumed`
+  transition via `consume_approval` so two workers can't double-consume a one-shot grant.
+- **JWKS auto-fetch for OIDC**: when no static signing key is configured, the verifier
+  fetches `<issuer>/.well-known/openid-configuration` to discover `jwks_uri`, then
+  fetches + caches JWKS keys (TTL 15min, refresh on `kid` miss). Explicit
+  `AGENTBRIDGE_OIDC_JWKS_URL` also supported.
+- **Config validation at startup** (`src/config.py`): checks env vars before any state is
+  created. Production requires `AGENTBRIDGE_ADMIN_KEY` and `AGENTBRIDGE_DB`; rate-limit
+  and shutdown-grace values are range-checked; OIDC issuer must be a URL; psycopg must be
+  importable when a postgres URL is configured. Errors raise `ConfigError` (fail-fast at boot).
+- **Audit retention + legal hold**:
+  - `POST /control/audit/checkpoint` — sign the current audit head with Ed25519 so a third
+    party can later prove the log wasn't truncated before this point.
+  - `POST /control/audit/retention` — `{"action":"truncate","seq":N}` removes entries with
+    `seq < N`; `{"action":"legal_hold","on":true}` freezes truncation (returns 409 on
+    subsequent truncation attempts). Backed by `store.truncate_audit_before` (InMemory/SQLite/Postgres).
+- **CLI `serve` improvements**: `--workers N`, `--log-level`, disables uvicorn's noisy
+  access log (we have our own structured middleware), passes graceful-shutdown timeout
+  through to uvicorn.
 
-### Fixed
-- **No test can hang the suite.** Added `pytest-timeout` (90s, thread method) and hardened the threaded concurrency test (daemon workers + bounded `join`) — previously a worker stalling before the barrier could deadlock `t.join()` indefinitely.
+### Added — production safety
+- FastAPI docs (`/docs`, `/redoc`) are suppressed when `AGENTBRIDGE_ENV=production`
+  unless `AGENTBRIDGE_DOCS=1` is set.
+- Warnings emitted (not just logged) when admin key is missing/short, when in-memory
+  store is used, when OIDC has no signing key configured.
 
 ### Tests
-- `tests/test_cli.py` — CLI smoke tests (`--version`, help, a live `openai → mcp` translation). Suite now **153 passing (159 with a Postgres DB)**.
+- `tests/test_production_readiness.py` — 21 new tests covering: store-backed approvals,
+  audit retention + legal hold + checkpoint signing, Prometheus metrics rendering,
+  structured JSON logging, config validation (5 scenarios), retry/backoff (3 scenarios),
+  `/health` + `/ready` + `/version` + `/metrics` endpoints, request-ID echo, full
+  audit-retention HTTP round-trip.
+
+  Plus a JWKS end-to-end round-trip test (previously untested). Suite green in CI on
+  Python 3.11 + 3.12; the only skips are the Postgres integration tests (need `AGENTBRIDGE_TEST_PG`).
+
+### Changed
+- `pyproject.toml`: added `prometheus-client>=0.20.0` as a runtime dependency; added
+  `[otel]` optional extra (`opentelemetry-sdk`, OTLP exporter, FastAPI instrumentation).
+
+### Fixed (post-review hardening)
+- `requirements.txt` now lists `prometheus-client` (CI installs from it — the `/metrics` tests
+  were red because it was only in `pyproject.toml`).
+- CLI `serve`: graceful-shutdown timeout passed to uvicorn in **seconds** (was `×1000` → ~2.8h).
+- **Audit retention now keeps the chain verifiable**: `verify_chain(..., require_genesis=False)`
+  + auto-detection in `verify_integrity`/`verify_durable`, so a truncated log no longer reports as
+  "tampered". Removed a docstring claim about a "truncate pseudo-entry" that was never written.
+- `resilience.retry_transient` no longer catches `sqlite3.DatabaseError` (parent of
+  `IntegrityError`/`ProgrammingError`) — only `OperationalError`, so permanent errors fail fast.
+- Gateway stopped copying the whole audit list per call to count it (`AuditLog.count()`, O(1)).
+- HTTP metrics label by the route **template**, not the raw path (prevents Prometheus cardinality blow-up).
+- OIDC JWKS resolves the cryptography key object directly (PyJWT accepts it) instead of a brittle
+  JWK→PEM round-trip.
+- Normalized 8 source files back to mode 644 (the pass had flipped them to 755).
 
 ## [1.0.0]
 

README.md

@@ -69,8 +69,9 @@ Add identity, budgets, and a tamper-evident audit trail **only when you want the
 ```bash
 # Run the meta-bridge control plane (mesh + governance) — CLI or uvicorn
 python -m src serve                        # = uvicorn src.api.control_plane:app
-#   docs at /docs · status dashboard at /dashboard · health at /health
+#   docs at /docs · dashboard at /dashboard · liveness /health · readiness /ready · Prometheus /metrics
 #   set AGENTBRIDGE_ADMIN_KEY for operator endpoints; AGENTBRIDGE_DB=/path.db (or a postgres:// URL)
+#   scale out: python -m src serve --workers 4   (needs a durable AGENTBRIDGE_DB)
 
 # Or with Docker (healthcheck on /health, persistent SQLite volume)
 docker compose up
@@ -247,6 +248,44 @@ like at runtime. Reproduce with `python examples/policy_guardrails_demo.py`.*
 [shadowhunter-92.github.io/agentbridge/media/explainer.html](https://shadowhunter-92.github.io/agentbridge/media/explainer.html)
 — source: [`media/explainer.html`](media/explainer.html).
 
+## Production & operations
+
+The control plane is built to run unattended — everything here is on by default or one env var away.
+
+**Observability**
+- `GET /metrics` — Prometheus exposition: call/latency/translation histograms, audit-entry and
+  per-agent budget gauges, pending-approvals, HTTP request/duration, rate-limit and auth-failure counters.
+- **OpenTelemetry tracing** — set `OTEL_EXPORTER_OTLP_ENDPOINT` (or `AGENTBRIDGE_OTEL_ENABLED=1`) to ship
+  spans; the gateway traces every `route_call`. No-op when the OTel SDK isn't installed.
+- **Structured JSON logs** with a per-request `X-Request-ID` (echoed in the response) via
+  `AGENTBRIDGE_LOG_JSON=1`. Requests slower than `AGENTBRIDGE_SLOW_LOG_SECONDS` (default 2s) are flagged.
+
+**Health & lifecycle (Kubernetes-ready)**
+- `GET /health` — liveness (always 200, even mid-drain, so the pod isn't restarted during shutdown).
+- `GET /ready` — readiness (503 while draining or if the governance store is unreachable).
+- `GET /version` — build + Python + store type.
+- **Graceful shutdown** — SIGTERM flips readiness to 503 and drains in-flight requests up to
+  `AGENTBRIDGE_SHUTDOWN_GRACE` (default 10s). Scale out with `agentbridge serve --workers N` (needs a
+  durable `AGENTBRIDGE_DB`).
+
+**Audit retention & compliance**
+- `POST /control/audit/checkpoint` — Ed25519-sign the current audit head so a third party can later prove
+  the log wasn't truncated or rewound past that point.
+- `POST /control/audit/retention` — `{"action":"truncate","seq":N}` drops entries before `N`;
+  `{"action":"legal_hold","on":true}` freezes truncation. A truncated chain **stays verifiable** from its
+  earliest retained entry (pair it with the signed checkpoint to vouch for the truncation point).
+
+**Operator SSO (OIDC)** — point `AGENTBRIDGE_OIDC_ISSUER` at your IdP (Okta / Auth0 / Azure AD / Keycloak);
+signing keys are auto-discovered via JWKS (`<issuer>/.well-known/openid-configuration`), cached, and
+refreshed on key rotation. A role claim maps to RBAC (admin / operator / viewer).
+
+**Fail-fast config** — the environment is validated at boot: production requires a stable
+`AGENTBRIDGE_ADMIN_KEY` and a durable `AGENTBRIDGE_DB`; bad rate-limit / OIDC / numeric values abort startup
+instead of failing on the first request.
+
+> The latency cost of all governance (identity + budget + policy + hash-chained audit) is sub-millisecond
+> in-process — see [`docs/BENCHMARKS.md`](docs/BENCHMARKS.md). Durable stores add one indexed insert per audited call.
+
 ## Editions & pricing (direction)
 
 Open-core: the mesh + basic governance are free and self-hostable (Apache 2.0). Monetization is

docs/API_REFERENCE.md

@@ -23,7 +23,10 @@ OIDC env: `AGENTBRIDGE_OIDC_ISSUER`, `AGENTBRIDGE_OIDC_AUDIENCE`, and one of
 
 | Method | Path | Body | Returns |
 |--------|------|------|---------|
-| GET | `/health` | — | `{status, protocols}` |
+| GET | `/health` | — | `{status, version, protocols, store}` — liveness (always 200, even while draining) |
+| GET | `/ready` | — | `{status, store}` — readiness; **503** while draining or if the store is unreachable |
+| GET | `/version` | — | `{version, python, store}` |
+| GET | `/metrics` | — | Prometheus exposition (text) |
 | GET | `/control/protocols` | — | `{protocols: [...]}` |
 | POST | `/control/translate/call` | `{src, dst, wire}` | `{wire}` — request translated src→dst |
 | POST | `/control/translate/result` | `{src, dst, wire}` | `{wire}` — result translated src→dst |
@@ -44,6 +47,8 @@ Malformed wires (non-object, or empty/unroutable) return **400** with a clear re
 | POST | `/control/approvals/{id}/approve` · `/deny` | `approvals:write` | Resolve an approval |
 | GET | `/control/audit` | `audit:read` | Audit entries + integrity check |
 | GET | `/control/audit/export` | `audit:export` | Audit log as JSONL (for SIEM/auditors) |
+| POST | `/control/audit/checkpoint` | `audit:export` | Ed25519-sign the current audit head — a third party can later prove the log wasn't truncated/rewound past this point |
+| POST | `/control/audit/retention` | `audit:export` | `{action:"truncate", seq}` drops entries before `seq` (chain stays verifiable); `{action:"legal_hold", on}` freezes truncation (**409** while a hold is active) |
 | POST | `/control/policy/rules` | `policy:write` | Add a declarative policy rule (see below) |
 | GET | `/control/policy/rules` | `policy:read` | List active policy rules |
 

docs/DEPLOYMENT.md

@@ -45,7 +45,18 @@ docker run -p 8000:8000 \
 | `AGENTBRIDGE_OIDC_ISSUER` | unset (OIDC off) | Enable OIDC operator SSO: your IdP issuer URL |
 | `AGENTBRIDGE_OIDC_AUDIENCE` | `agentbridge` | Expected `aud` claim |
 | `AGENTBRIDGE_OIDC_PUBLIC_KEY_PEM` / `_FILE` | unset | IdP signing public key (inline PEM or file path) |
+| `AGENTBRIDGE_OIDC_JWKS_URL` | unset (auto-discover) | Explicit JWKS URL; otherwise discovered from `<issuer>/.well-known/openid-configuration` |
 | `AGENTBRIDGE_OIDC_ROLE_CLAIM` | `role` | Token claim mapped to the RBAC role (admin/operator/viewer) |
+| `AGENTBRIDGE_ENV` | unset | Set to `production` to require admin key + durable DB at boot and suppress `/docs` |
+| `AGENTBRIDGE_LOG_JSON` | unset | `1` → structured JSON logs (with `X-Request-ID`); else plain text |
+| `AGENTBRIDGE_SLOW_LOG_SECONDS` | `2.0` | Log a warning for requests slower than this |
+| `AGENTBRIDGE_SHUTDOWN_GRACE` | `10` | Seconds to drain in-flight requests on SIGTERM before close |
+| `OTEL_EXPORTER_OTLP_ENDPOINT` | unset (tracing off) | Ship OpenTelemetry spans to this OTLP/HTTP endpoint |
+
+> **Health probes for k8s:** liveness → `GET /health` (always 200), readiness → `GET /ready`
+> (503 while draining or if the store is unreachable). Metrics → `GET /metrics` (Prometheus).
+> Config is validated at boot — a misconfigured production env (no admin key / no durable DB)
+> aborts startup instead of failing on the first request.
 
 ## 4. Persistence backends