Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
189 changes: 189 additions & 0 deletions docs/INCIDENT_RESPONSE.md
Original file line number Diff line number Diff line change
Expand Up @@ -130,6 +130,195 @@ This document outlines procedures for responding to security alerts detected by
>
> Please investigate immediately and update the alert status.

## Automated response (circuit breaker)

The procedures above are what a human does. This section covers what the system
does on its own, before anyone reads the alert.

When a security scan (`/api/cron/security-scan` or `/api/admin/security/scan`)
raises enough CRITICAL alerts against a single pool, the circuit breaker pauses
that pool so no further money moves until an admin has looked at it. Every
decision it takes, including the ones it decides against, is recorded in the
`incidents` table and surfaced in the admin audit log.

### What "auto-pause" means, and what it cannot do

The pause has two halves, and only the first can be automatic.

| Half | Automatic? | Effect |
|------|-----------|--------|
| Platform pause | Yes | `pools.status` becomes `paused` with a reason and timestamp. The app stops offering deposits and payouts immediately. Reversible from the admin endpoint. |
| On-chain pause | Yes, when pre-authorised | Submitted by the platform using an authorization the admin signed in advance. Without one, the admin signs `rotational::pause` themselves. |

The contract asserts `admin.require_auth()` and that the caller equals the pool's
stored admin, which is the creator's wallet. The platform holds no key that
satisfies it today: `SPONSOR_SECRET_KEY` only pays network fees, and a fee bump
authorises nothing inside the transaction. So an executed incident is recorded
with `onchain_status = 'pending'` and the admin signs the contract call from the
review screen, after which the hash is recorded against the incident.

That is a key-custody gap rather than a contract limitation, and it is closed
with **pre-signed authorization entries**, with no contract change.

### How the automatic on-chain pause works

A `SorobanAuthorizationEntry` is signed independently of the transaction
envelope, so the party who authorises a call and the party who submits it can be
different. The admin signs one entry covering exactly `pause(admin)` on exactly
their pool's contract. The platform stores it and, when the breaker trips, wraps
it in a transaction it pays for and signs the envelope of.

Two signatures, two jobs: the admin authorises the call, the platform authorises
the fee. The platform never holds the admin's key, and the credential it does
hold can do one thing.

```
admin's wallet platform
| |
| signs pause(admin) entry |
|------------------------------->| stored, single use, expires
| |
| breaker trips
| wraps entry in a tx, pays the fee
|------------------> Soroban
```

An alternative exists and was deliberately not taken: `require_auth` for a
classic `G` address uses Stellar multisig at the medium threshold, so an admin
could add a platform signer with enough weight instead. That is simpler to
operate but a far wider grant, since the weight applies to the account in
general rather than to one call.

### Authorising it

```
GET /api/admin/pause-authorizations?poolId=<id>&callerAddress=<address>
POST /api/admin/pause-authorizations { admin_address, pool_id, entry_xdr }
POST /api/admin/pause-authorizations { action: "revoke", id, signature, signed_at }
```

`lib/pause-authorization.ts` builds and signs the entry in the browser through
the wallet kit. The server validates what actually arrived rather than trusting
the client: the entry must be address-credentialed, invoke `pause`, take the
signer as its only argument, and carry no sub-invocations, so it cannot smuggle a
second call. It is also matched against the pool's contract and admin, and
refused if it expires too soon to be useful.

The entry XDR is never returned by `GET`, and the table has no read policy for
anyone but the service role. It is a bearer credential: whoever holds it can
pause the pool, which would be a griefing vector against the pool's own members.

### Revoking needs a signature, not an address

Registering an authorization is self-validating: an entry that was not signed by
the pool real admin is refused by the inspector no matter who posted it, and the
contract would reject it anyway. Revoking is different, because revoking disarms
the automatic pause. An attacker preparing to drain a pool could otherwise switch
off the defence using only public data, since a pool id and its creator address
are both readable.

So revocation asks the wallet to sign a short, timestamped message naming the
exact authorization, and the server verifies it under SEP-53 against the pool
admin as recorded, not against any address in the request. A captured proof stops
working within minutes and does not transfer to another authorization.
`lib/pause-authorization.ts` has the client side, `lib/server/wallet-proof.ts`
the verification.

### What happens when there is no authorization

The platform pause still happens, immediately. The incident is recorded with
`onchain_status = 'pending'`, the admin is told why in their notification, and
they sign the contract call themselves from the review screen. The pool is
protected either way; pre-authorising only removes the wait.

Entries are single-use and expire, so an admin who wants the automatic pause to
keep working re-signs one occasionally. `GET` reports `armed: true` while a
usable one exists.

### emergency_withdraw is never automatic

Nothing in the automated path can move funds. The breaker's action type has
exactly two values, `pause` and `none`, and a unit test asserts that set has not
grown. `emergency_withdraw` stays a manual, admin-only contract call.

### Thresholds and cooldown

| Setting | Default | What it does |
|---------|---------|--------------|
| `INCIDENT_AUTO_PAUSE_ENABLED` | unset (dry-run) | Arms the breaker. Only the exact string `true` arms it. |
| `INCIDENT_CRITICAL_THRESHOLD` | 2 | Critical alerts against one pool needed to trip it. |
| `INCIDENT_THRESHOLD_WINDOW_MS` | 3600000 (1h) | How far back alerts count towards the threshold. |
| `INCIDENT_MAX_PAUSES_PER_WINDOW` | 1 | Auto-pauses allowed per pool inside the cooldown window. |
| `INCIDENT_COOLDOWN_WINDOW_MS` | 86400000 (24h) | The cooldown window. |

The cooldown is what prevents pause-flap. With the defaults, a pool is
auto-paused at most once a day; if it trips again it stays paused and waits for
an admin instead of oscillating. The gate is checked before the action, so a
pool in cooldown is never paused and then reverted. Only pauses that actually
happened count towards it, so a dry-run period does not silently consume a
pool's allowance.

### Rolling it out with dry-run

Dry-run is the default and is the intended rollout mechanism, not a switch to
skip past. In dry-run the breaker still decides, still writes the incident and
still notifies the admin. It just does not pause anything.

Both scan endpoints report `incidentResponse`, which always answers whether an
action *would* have fired, independently of dry-run:

```jsonc
{
"incidentResponse": {
"dryRun": true,
"wouldFire": 2, // pools that met the thresholds
"paused": 0, // pools actually paused
"cooldownBlocked": 1, // held back by the cooldown
"decisions": [ /* one per pool, with the reason */ ],
"incidentIds": ["..."]
}
}
```

Run it that way for a while, read the incidents it would have created, and only
then set `INCIDENT_AUTO_PAUSE_ENABLED=true`.

### Admin review and recovery

```
GET /api/admin/incidents?poolId=<id>&callerAddress=<address>
POST /api/admin/incidents/<incidentId>
```

The POST body takes `admin_address` and an `action`:

| Action | What it does |
|--------|--------------|
| `resolve` | Closes the incident with a required note. The pool stays paused. |
| `resume` | Closes it and returns the pool to `active`. |
| `record_onchain` | Attaches the hash of the `pause` or `unpause` transaction the admin signed. |

Both endpoints verify the caller against the pool's `creator_address`
server-side, the same check `/api/admin/audit-log` uses.

`resume` lifts the platform pause only. If the admin had already signed an
on-chain pause, the response returns `onchainUnpauseRequired: true` and the
contract stays paused until they sign `unpause` themselves.

### Where to look

| Piece | File |
|-------|------|
| Decision logic (pure, unit tested) | `frontend/lib/incident-response.ts` |
| Tests | `frontend/lib/incident-response.test.ts` |
| Execution against Supabase | `frontend/lib/server/incident-actions.ts` |
| Admin review and recovery | `frontend/app/api/admin/incidents/` |
| On-chain pause submission | `frontend/lib/server/pause-onchain.ts` |
| Signing an authorization (browser) | `frontend/lib/pause-authorization.ts` |
| Authorization endpoints | `frontend/app/api/admin/pause-authorizations/` |
| Schema | `supabase/migrations/20260827120000_incident_response.sql` |
| Authorization schema | `supabase/migrations/20260827130000_pause_authorizations.sql` |

## Review and Post-Incident

After resolving any CRITICAL or WARNING alert:
Expand Down
33 changes: 33 additions & 0 deletions frontend/.env.example
Original file line number Diff line number Diff line change
Expand Up @@ -73,3 +73,36 @@ CRON_SECRET=
# Full public URL of the deployed app (no trailing slash). Used by cron routes
# to call sibling API endpoints (e.g. the push dispatch endpoint).
NEXT_PUBLIC_APP_URL=https://joint-save.vercel.app

# ── Automated incident response (security circuit breaker) ────────────────────
# When a security scan flags enough critical alerts against one pool, the
# breaker pauses that pool so no further money moves until an admin reviews it.
# See docs/INCIDENT_RESPONSE.md.

# Arms the breaker. Left unset (or anything other than the exact string "true")
# it stays in dry-run: decisions are recorded, incidents are written and admins
# are notified, but no pool is ever paused. Run it that way first and read the
# incidents it would have created before turning it on.
INCIDENT_AUTO_PAUSE_ENABLED=false

# Critical alerts against one pool needed to trip the breaker. Default 2.
# INCIDENT_CRITICAL_THRESHOLD=2

# How far back critical alerts count towards the threshold, in ms. Default 1h.
# INCIDENT_THRESHOLD_WINDOW_MS=3600000

# Anti-flap: auto-pauses allowed per pool inside the window below. Default 1,
# so a pool that trips again stays paused for an admin instead of flapping.
# INCIDENT_MAX_PAUSES_PER_WINDOW=1

# The cooldown window, in ms. Default 24h.
# INCIDENT_COOLDOWN_WINDOW_MS=86400000

# Network the contracts live on, used when the breaker submits the on-chain
# pause. Defaults to testnet, matching components/web3-provider.tsx.
# STELLAR_NETWORK_PASSPHRASE=Test SDF Network ; September 2015

# Submitting the on-chain pause needs SPONSOR_SECRET_KEY (above) to pay the fee.
# It never authorises the pause itself: that comes from an authorization the
# pool admin pre-signed. Without it the breaker still pauses at the platform
# level and asks the admin to sign the contract call.
Loading
Loading