Neurowealth · Abidoyesimze · Sep 2, 2026 · Sep 1, 2026 · Sep 1, 2026
diff --git a/deploy/monitoring/prometheus/alert-rules.yaml b/deploy/monitoring/prometheus/alert-rules.yaml
@@ -74,6 +74,15 @@ groups:
           summary: "Severe network congestion"
           description: "Congestion level severe for 5 minutes — LOW ops deferring"
 
+      - alert: AgentBreakerGlobalOpen
+        expr: agent_breaker_state{scope="GLOBAL"} == 2
+        for: 1m
+        labels:
+          severity: critical
+        annotations:
+          summary: "Agent circuit breaker is OPEN globally"
+          description: "All agent rebalancing is halted"
+
   - name: neurowealth_warning
     interval: 30s
     rules:
@@ -139,3 +148,12 @@ groups:
         annotations:
           summary: "HTTP requests slow"
           description: "P95 HTTP request duration is {{ $value }}s (> 5s)"
+
+      - alert: AgentBreakerOpen
+        expr: agent_breaker_state{scope!="GLOBAL"} == 2
+        for: 2m
+        labels:
+          severity: warning
+        annotations:
+          summary: "Agent circuit breaker OPEN for a protocol or user"
+          description: "{{ $labels.scope }} {{ $labels.scopeKey }} is halted (state={{ $value }})"
diff --git a/docs/OBSERVABILITY.md b/docs/OBSERVABILITY.md
@@ -48,6 +48,24 @@ The backend exposes Prometheus-compatible metrics through the `/metrics` endpoin
 - `agent_rebalances_triggered_total` - Counter
 - `agent_snapshot_duration_seconds` - Histogram
 
+### Agent Circuit Breaker Metrics (#345)
+
+- `agent_breaker_state` - Gauge, labels `scope` (`GLOBAL`|`PROTOCOL`|`USER`) and `scopeKey`; 0 = CLOSED, 1 = HALF_OPEN, 2 = OPEN. Written every agent breaker evaluation tick and on every transition.
+- `agent_breaker_trips_total` - Counter, labels `scope`, `rule` (`abnormal_loss`|`depeg`|`oscillation`|`stale_data`|`manual`). Increments whenever a breaker trips or re-trips.
+
+Query examples:
+
+```
+# Any circuit breaker open right now
+agent_breaker_state == 2
+
+# Global halt (stops all agent rebalancing)
+agent_breaker_state{scope="GLOBAL"} == 2
+
+# Breaker trip rate by rule
+sum(rate(agent_breaker_trips_total[15m])) by (rule)
+```
+
 ### Database Metrics
 
 - `db_operation_duration_seconds` - Histogram with label: `operation`
@@ -74,6 +92,7 @@ The backend exposes Prometheus-compatible metrics through the `/metrics` endpoin
 | `cursor_lag_ledgers` | `> 100` | Critical | Event processing lagging significantly |
 | `dlq_size` | `> 50` | Critical | Dead Letter Queue critically large |
 | `failures_total` (rate) | `> 10 per minute` for 5m | Critical | High failure rate |
+| `agent_breaker_state{scope="GLOBAL"}` | `== 2` for 1m | Critical | Global circuit breaker OPEN — all rebalancing halted |
 
 ### Warning Alerts (Investigate Within 1 Hour)
 
@@ -85,6 +104,7 @@ The backend exposes Prometheus-compatible metrics through the `/metrics` endpoin
 | `events_processing_duration_seconds` (p95) | `> 2 seconds` | Warning | Event processing slow |
 | `db_operation_duration_seconds` (p95) | `> 1 second` | Warning | Database operations slow |
 | `http_request_duration_seconds` (p95) | `> 5 seconds` | Warning | HTTP requests slow |
+| `agent_breaker_state{scope!="GLOBAL"}` | `== 2` for 2m | Warning | Protocol or user breaker OPEN — affected rebalancing halted |
 
 ### Info Alerts (Monitor Trend)
 

diff --git a/docs/RUNBOOK.md b/docs/RUNBOOK.md
@@ -482,3 +482,85 @@ psql "$DATABASE_URL" -c "SELECT \"sponsorAccount\", count(*), sum(\"xlmReserved\
 ```
 
 No auto top-up — operational runbook only. Reconciliation job (`reserveReconciliation` hourly) flags drift where on-chain sponsor ≠ ledger.
+
+## 9. Agent Circuit Breaker (#345)
+
+The agent circuit breaker halts agent-initiated rebalancing when the market,
+a protocol, or a user's account shows risk. It never touches withdrawals.
+Scopes: `GLOBAL` (halt everything), `PROTOCOL` (halt a target protocol +
+batches leaving it), `USER` (halt that user's batches). Breakers are
+`CLOSED → OPEN → HALF_OPEN → CLOSED`; an `OPEN` breaker auto-probes after its
+cooldown and needs `BREAKER_DEPEG_SUSTAINED_CHECKS` clean evaluations before
+recovering to `HALF_OPEN`, then one clean probe to close.
+
+### Rules
+
+| Rule | Env | Default | Trips when |
+|---|---|---|---|
+| abnormal_loss | `BREAKER_LOSS_PCT`, `BREAKER_LOSS_WINDOW_HOURS` | 5% / 24h | mark-to-market drawdown over the window exceeds the pct |
+| depeg | `BREAKER_DEPEG_ENABLED` (=`false`) | off | reported stablecoin price deviates > `BREAKER_DEPEG_BPS` (150) from $1 |
+| oscillation | `BREAKER_MAX_FLIPS`, `BREAKER_FLIP_WINDOW_HOURS` | 3 / 24h | same batch rebalances ≥ N times in the window |
+| stale_data | `BREAKER_STALE_MINUTES`, `BREAKER_STALE_CONSECUTIVE_FAILURES` | 120m / 3 | APY table older than limit, never scanned, or ≥ N consecutive failures |
+
+`BREAKER_COOLDOWN_MS` (1h) is the base cooldown; a repeated HALF_OPEN trip
+doubles it up to `BREAKER_MAX_COOLDOWN_MS` (24h).
+
+### Known limitation — de-peg price feed
+
+The de-peg rule is a pure consumer of a stablecoin spot price. As of this
+change no live price feed exists in this codebase: the fee oracle
+(`src/stellar/feeOracle.ts`) publishes only fees, and `src/stellar/routing.ts`
+is a stub. `getStablecoinPrice()` (`src/agent/breakerService.ts`) is the single
+integration point and currently returns `null` (fails safe — the rule never
+trips); the rule is disabled by default. Wire the oracle there, keep the pure
+rule unchanged, flip `BREAKER_DEPEG_ENABLED=true`, and re-run the de-peg unit
+tests.
+
+### Inspector
+
+```bash
+# All breakers with current state
+curl -H "Authorization: Bearer $ADMIN_API_TOKEN" http://localhost:3001/api/v1/admin/agent/breakers | jq
+
+# Agent status (incl. cached global breaker summary)
+curl -H "X-Internal-Token: $INTERNAL_SERVICE_TOKEN" http://localhost:3001/api/v1/agent/status | jq
+```
+
+### Manual trip
+
+`POST /api/v1/admin/agent/breakers` — body `{"scope":"PROTOCOL","scopeKey":"blend","reason":"..."}`.
+`GLOBAL` needs no `scopeKey`; `USER`/`PROTOCOL` require it. `reason` is always
+required. Written to the admin audit log (`TRIP_AGENT_BREAKER`).
+
+```bash
+curl -X POST http://localhost:3001/api/v1/admin/agent/breakers \
+  -H "Authorization: Bearer $ADMIN_API_TOKEN" -H "Content-Type: application/json" \
+  -d '{"scope":"PROTOCOL","scopeKey":"blend","reason":"incident-4821 protocol outage"}'
+```
+
+A manual trip can only be cleared manually (`rule=manual` breakers never
+auto-reset). Manual resets are audit-logged too:
+
+```bash
+curl -X POST http://localhost:3001/api/v1/admin/agent/breakers/<id>/reset \
+  -H "Authorization: Bearer $ADMIN_API_TOKEN" -H "Content-Type: application/json" \
+  -d '{"reason":"incident-4821 resolved, APY table verified fresh"}'
+```
+
+A breaker skips its tick when evaluation itself fails: the agent alerts
+(critical, `agent-breaker:eval-failed`) and halts **all** rebalancing for that
+tick rather than trading blind.
+
+### Response
+
+1. **Global halt**: investigate the trip rule (`agent_breaker_trips_total{scope="GLOBAL"}`), check the `[Breaker]` logs for the `lastEvaluation` detail, fix root cause, then either wait for auto-recovery or reset manually.
+2. **Protocol halt**: verify the protocol's APY/status independently before resetting; `compareProtocols` already refuses it as a target while OPEN.
+3. **User halt**: confirm with the user before resetting.
+4. **Evaluation-failed halt**: the breaker could not decide — check DB connectivity and the logged error before the next tick.
+
+### Verify
+
+```bash
+curl -s http://localhost:3001/metrics | grep -E "agent_breaker_(state|trips_total)"
+psql "$DATABASE_URL" -c "SELECT scope, \"scopeKey\", state, \"trippedRule\" FROM agent_circuit_breakers ORDER BY \"updatedAt\" DESC;"
+```
diff --git a/prisma/migrations/20260901000000_add_agent_circuit_breaker/migration.sql b/prisma/migrations/20260901000000_add_agent_circuit_breaker/migration.sql
@@ -0,0 +1,27 @@
+-- Migration: add_agent_circuit_breaker (#345)
+-- Agent circuit breaker: a pre-trade kill switch that halts agent-initiated
+-- rebalancing at GLOBAL / PROTOCOL / USER scope when abnormal loss, de-peg,
+-- oscillation or stale data is detected. Requires an explicit, audited reset.
+-- User-initiated withdrawals are never blocked by these rows (enforced in code).
+
+CREATE TYPE "BreakerScope" AS ENUM ('GLOBAL', 'PROTOCOL', 'USER');
+CREATE TYPE "BreakerState" AS ENUM ('CLOSED', 'OPEN', 'HALF_OPEN');
+
+CREATE TABLE "agent_circuit_breakers" (
+    "id"           TEXT NOT NULL,
+    "scope"        "BreakerScope" NOT NULL,
+    "scopeKey"     TEXT NOT NULL,          -- "" for GLOBAL, protocolName, or userId
+    "state"        "BreakerState" NOT NULL DEFAULT 'CLOSED',
+    "trippedRule"  TEXT,                   -- abnormal_loss | depeg | oscillation | stale_data | manual
+    "trippedAt"    TIMESTAMP(3),
+    "detail"       JSONB,
+    "resetBy"      TEXT,                   -- admin identity on manual reset
+    "resetAt"      TIMESTAMP(3),
+    "autoResetAt"  TIMESTAMP(3),           -- earliest time HALF_OPEN is allowed
+    "updatedAt"    TIMESTAMP(3) NOT NULL DEFAULT CURRENT_TIMESTAMP,
+
+    CONSTRAINT "agent_circuit_breakers_pkey" PRIMARY KEY ("id")
+);
+
+CREATE UNIQUE INDEX "agent_circuit_breakers_scope_scopeKey_key" ON "agent_circuit_breakers"("scope", "scopeKey");
+CREATE INDEX "agent_circuit_breakers_state_idx" ON "agent_circuit_breakers"("state");
diff --git a/prisma/migrations/20260901000000_add_agent_circuit_breaker/rollback.sql b/prisma/migrations/20260901000000_add_agent_circuit_breaker/rollback.sql
@@ -0,0 +1,13 @@
+-- rollback.sql — reverse of 20260901000000_add_agent_circuit_breaker/migration.sql
+-- Drops the agent circuit breaker table and its enum types (#345).
+-- WARNING: DATA LOSS — all breaker state (OPEN/HALF_OPEN), trip details and
+-- admin reset audit rows are lost. Revert app code BEFORE running.
+-- Indexes are dropped with the table (explicit drops for idempotency).
+-- Safe to run multiple times.
+-- Run with: psql $DATABASE_URL -f prisma/migrations/20260901000000_add_agent_circuit_breaker/rollback.sql
+
+DROP INDEX IF EXISTS "agent_circuit_breakers_state_idx";
+DROP INDEX IF EXISTS "agent_circuit_breakers_scope_scopeKey_key";
+DROP TABLE IF EXISTS "agent_circuit_breakers" CASCADE;
+DROP TYPE IF EXISTS "BreakerState";
+DROP TYPE IF EXISTS "BreakerScope";
diff --git a/prisma/schema.prisma b/prisma/schema.prisma
@@ -1906,3 +1906,34 @@ model ProtocolLiquiditySnapshot {
   @@index([fetchedAt])
   @@map("protocol_liquidity_snapshots")
 }
+
+// --- Issue #345: Agent Circuit Breaker ---
+enum BreakerScope {
+  GLOBAL
+  PROTOCOL
+  USER
+}
+
+enum BreakerState {
+  CLOSED
+  OPEN
+  HALF_OPEN
+}
+
+model AgentCircuitBreaker {
+  id           String       @id @default(uuid())
+  scope        BreakerScope
+  scopeKey     String       // "" for GLOBAL, protocolName, or userId
+  state        BreakerState @default(CLOSED)
+  trippedRule  String?      // "abnormal_loss" | "depeg" | "oscillation" | "stale_data" | "manual"
+  trippedAt    DateTime?
+  detail       Json?        // the measurements that tripped it
+  resetBy      String?      // admin identity on manual reset
+  resetAt      DateTime?
+  autoResetAt  DateTime?    // earliest time HALF_OPEN is allowed
+  updatedAt    DateTime     @updatedAt
+
+  @@unique([scope, scopeKey])
+  @@index([state])
+  @@map("agent_circuit_breakers")
+}