Skip to content

feat(contracts): emergency circuit breaker with scoped function guards & time-bound auto-expiry - #46

Merged
meshackyaro merged 6 commits into
workman-labs:developmentfrom
oss-dw:feat/emergency-circuit-breaker
Aug 20, 2026
Merged

feat(contracts): emergency circuit breaker with scoped function guards & time-bound auto-expiry#46
meshackyaro merged 6 commits into
workman-labs:developmentfrom
oss-dw:feat/emergency-circuit-breaker

Conversation

@bbjiggy

@bbjiggy bbjiggy commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Closes #42

Release notes: soroban-contracts/CHANGELOG.md
— the ## [Unreleased] section carries this change's entry (Added / Changed /
Notes), ready for a releaser or release automation to pick up. The changelog
links back to this PR and to the benchmark table, so the two stay paired.

Key docs: Emergency circuit breaker
· Events / wire format
· Hot-path cost

What this adds

A shared emergency circuit breaker across escrow, reputation,
loyalty-token and loyalty-emissions. Until now none of them had any halt
mechanism — the only responses to a live exploit were "watch it drain" or
"upgrade under fire", and the upgrade path itself is still being built out.

The primitive lives in contracts/governance-guard's new pausable module,
next to the upgrade guard and for the same reason: every contract that needs it
already depends on that crate.

The design constraint

A plain bool paused flag would be a rug pull waiting to happen. Three
properties are load-bearing, each closing a failure mode that a boolean leaves
open.

1. Scoped, not global. A pause names a bitmask, so entry paths halt
independently of exit paths.

Scope Bit Guarded entrypoints
SCOPE_INTAKE 1 escrow: create_appointment, create_milestone_escrow, add_milestone · loyalty-token: mint · loyalty-emissions: create_schedule
SCOPE_SETTLEMENT 2 escrow: confirm_completion, approve_milestone, release_milestone_funds · loyalty-emissions: claim
SCOPE_ATTESTATION 4 reputation: submit_attestation

A contract with no entrypoint in some scope ignores a pause naming it, so an
operator can broadcast one mask to every contract without special-casing.

2. Fund-recovery paths are never guarded. This is the half the issue is
really about, and it is enforced by omission at the call sites — so it's
stated explicitly in the module docs and pinned by tests rather than left
implicit:

Contract Always callable, even while paused Why
escrow cancel_appointment, raise_dispute, resolve_dispute, raise_milestone_dispute, resolve_milestone_dispute Every route by which an escrowed balance reaches whoever is entitled to it.
loyalty-token transfer, transfer_from, approve, burn These move already-held balances. Only mint — the sole path creating supply that doesn't yet exist — is guarded.
loyalty-emissions reclaim Mints and burns nothing, so it can't take a balance anyone holds.
all every read-only view A consumer must be able to see the state that prompted the halt.

There is no scope value, ALL_SCOPES included, that reaches any of these.

3. Time-bound, enforced on read. A pause carries an expires_at ledger
timestamp capped at MAX_PAUSE_DURATION (7 days), and expiry is evaluated on
every guard consultation — so a pause lapses with no unpause transaction, no
live admin, no working key
.

Two judgement calls worth flagging in review

Pausing SCOPE_SETTLEMENT withholds a payout, so it deserves an argument
rather than an assumption. It's a delay, not a seizure: it changes no party's
power relative to the others — a client could always cancel a Funded
appointment for a full refund, and either party could always force
raise_disputeresolve_dispute; both stay open throughout. And
release_milestone_funds is permissionless, which is precisely why it has to
be haltable, or the scope is decorative. On emissions, vesting is a pure
function of the ledger clock and keeps accruing through a pause, so a halted
claim mints the identical amount afterwards, just later.

Authorization is any single governance signer, not the M-of-N threshold and
deliberately not each contract's own admin. Gathering a threshold takes time
an incident doesn't give you, and the action is bounded on every axis that
matters — this mirrors cancel_upgrade, which is unilateral for the same
reason. The signer set rather than admin because it's M-of-N, rotatable via
the timelocked flow from #27, and survives the loss of one key; an incident is
exactly when a single non-rotatable key is least trustworthy. unpause is
unilateral in the same way, so a responder who pauses and then goes offline
can't wedge it in place.

What this does not promise: an admin who is present and hostile can
re-pause each time the window lapses. That isn't closable here — the same
signer set can already replace all of a contract's code via the upgrade flow.
The honest guarantee is narrower: an unattended pause always clears, and no
pause of any duration can stop a user recovering funds they already own
. The
second half is what keeps the residual risk a liveness problem for new
business rather than a custody problem for existing balances.

Interface

Added to all four contracts:

  • pause(caller, scopes, duration_secs, reason) -> PauseStatereason is
    operator context capped at 64 UTF-8 bytes (not characters), may be empty
  • unpause(caller, scopes) -> u32 — clears only the named scopes and returns
    what's still halted, without touching the deadline. Re-calling pause
    with a narrower mask would also lift scopes but restarts the clock on
    everything left, so partial unpause is how you bring the system back a
    piece at a time.
  • get_pause_state() -> Option<PauseState> (None once expired),
    paused_scopes() -> u32, is_paused(scope) -> bool

Events for off-chain monitoring: Paused { caller, scopes, expires_at, reason }
and Unpaused { caller, scopes, remaining_scopes }. Topic ordering and the
Map<Symbol, Val> data shape are documented in the README and pinned by
assertion
in contracts/escrow/src/test.rs, so they cannot drift silently. Auto-expiry emits nothing —
it's a read-time evaluation with no transaction behind it, so monitors should
treat Paused.expires_at as the authoritative end of the window unless an
Unpaused arrives sooner.

A dedicated OperationPaused error per contract, not a reused
InvalidStatus: "the protocol is halted, retry later" and "this request was
never valid" call for opposite reactions from a client. Plus
InvalidPauseScope, InvalidPauseDuration, NotPaused, each appended at
whatever offset came next in that contract's existing enum so no existing code
moved.

Storage & safety

One new instance key, GovernanceDataKey::PauseState, holding
PauseState { scopes, expires_at, paused_by, paused_at } — appended after
PendingRotation so already-deployed contracts' key encodings stay put. Only
one record exists at a time; a second pause replaces the first outright
rather than layering, so halted scopes and deadline are always readable from
one place. paused_by is recorded for attribution and grants no rights.

Guards run as the first statement of each entrypoint, ahead of auth and any
storage read: a halted call costs nothing and reveals nothing beyond the
already-public pause state.

Tests — 271 total, all passing

66 new unit tests in governance-guard (scope isolation, expiry boundaries
including the exact expires_at instant, the duration cap, masked unpause,
event emission, adversarial auth), plus per-contract integration tests for the
wiring. The ones tied directly to the acceptance criteria:

  • a_client_can_still_cancel_and_be_refunded_while_everything_is_paused
    balances asserted, and the pause verified still in force afterwards
  • disputes_can_still_be_raised_and_resolved_while_everything_is_paused
  • milestone_disputes_resolve_while_everything_is_paused
  • holders_keep_full_control_of_existing_balances_while_everything_is_paused
    transfer, transfer_from, burn all succeed under ALL_SCOPES
  • intake_resumes_on_its_own_once_the_pause_expires — only the clock moves
  • a_non_signer_cannot_pause_the_escrow, the_admin_arbiter_is_not_a_pause_authority
  • a_pause_longer_than_the_cap_is_refused_outright
  • pausing_the_token_alone_stops_claims_at_the_mint_boundary — cross-contract
  • a_lapsed_pause_can_be_placed_again_by_a_present_admin — pins the limit of
    the guarantee so nobody later reads it as stronger than it is

CI

cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings,
cargo test --workspace and cargo build --release --target wasm32v1-none all
run clean locally. Clippy already blocks warnings on PRs, satisfying the
"warnings automatically blocked" criterion.

Cargo dependency caching was already in place (Swatinem/rust-cache@v2); I
added cache-on-failure: true, since a run that fails on fmt or clippy has
still built the whole dependency tree and the retry shouldn't recompile
soroban-sdk from scratch.

Noted, not fixed here

soroban-contracts/README.md's "Upgrade governance" section and its
"Notes / follow-ups" list still describe the signer set as immutable after
initialize — stale since signer rotation (#27) landed, and unrelated to this
change. Left alone to keep the diff to one concern; happy to fix in a
follow-up if you'd prefer.

Adds a `pausable` module to the shared governance-guard crate: the
emergency circuit breaker the core contracts have had no equivalent of.
Until now the only responses to a live exploit were "watch it drain" or
"upgrade under fire".

A plain `bool paused` flag would be a rug pull waiting to happen, so the
primitive is built around three properties instead:

* **Scoped, not global.** A pause names a bitmask — SCOPE_INTAKE,
  SCOPE_SETTLEMENT, SCOPE_ATTESTATION — so entry paths can be halted
  independently of exit paths. Halting value entering escrow is a safety
  measure; halting value leaving it is a hostage situation, and only a
  scoped guard can express the first without the second.

* **Fund-recovery paths are never guarded.** Enforced by omission at the
  call sites rather than by a runtime check, so it is stated explicitly
  in the module docs: there is no scope value, ALL_SCOPES included, that
  can reach a refund or a dispute. That wiring lands in the next commit.

* **Time-bound, enforced on read.** A pause carries an `expires_at`
  ledger timestamp capped at MAX_PAUSE_DURATION (7 days), and expiry is
  evaluated on every guard consultation. A pause lapses with no unpause
  transaction, no live admin and no working key.

Authorization is `require_signer` — any *single* governance signer, not
the M-of-N threshold, mirroring `cancel_upgrade`. Gathering a threshold
takes time an incident does not give you, and the action is bounded on
every axis that matters. Using the signer set rather than each contract's
own `admin` matters because the set is M-of-N, rotatable, and survives
the loss of one key.

`unpause` takes a mask and leaves the deadline alone, so an operator can
bring the system back a piece at a time; re-calling `pause` with a
narrower mask would also lift scopes but restarts the clock on the rest.

Adds four GovernanceError variants (OperationPaused, InvalidPauseScope,
InvalidPauseDuration, NotPaused) — OperationPaused is deliberately
distinct from any status error, since "halted, retry later" and "this was
never valid" call for opposite reactions from a client. State lives in
one new instance key, GovernanceDataKey::PauseState, appended last so
deployed contracts' key encodings stay put.

66 unit tests cover scope isolation, expiry boundaries, the duration cap,
masked unpause, event emission, and adversarial auth.
…acts

Exposes pause/unpause/get_pause_state/paused_scopes/is_paused on escrow,
reputation, loyalty-token and loyalty-emissions, and places the scope
guards. Each contract gains the four pause error variants at whatever
offset came next in its own enum, so existing codes are untouched.

What is guarded:

  SCOPE_INTAKE      escrow: create_appointment, create_milestone_escrow,
                    add_milestone; loyalty-token: mint;
                    loyalty-emissions: create_schedule
  SCOPE_SETTLEMENT  escrow: confirm_completion, approve_milestone,
                    release_milestone_funds; loyalty-emissions: claim
  SCOPE_ATTESTATION reputation: submit_attestation

What is *not*, and why — this is the load-bearing half:

  escrow            cancel_appointment, raise_dispute, resolve_dispute,
                    raise_milestone_dispute, resolve_milestone_dispute.
                    Every route by which an escrowed balance reaches
                    whoever is entitled to it. No scope reaches them.
  loyalty-token     transfer, transfer_from, approve, burn — these move
                    already-held balances. A holder's points are their
                    property; only mint creates supply that doesn't yet
                    exist, so only mint is guarded.
  loyalty-emissions reclaim — mints and burns nothing, so it cannot take
                    a balance anyone holds.
  all               every read-only view, so a consumer can always see
                    the state that prompted the halt.

Pausing settlement withholds a payout, so it gets an explicit argument
rather than an assumption: it changes no party's power relative to the
others (a client could always cancel a Funded appointment, either party
could always force a dispute — both stay open), and
release_milestone_funds is permissionless, which is exactly why it has to
be haltable. On emissions, vesting is a pure function of the ledger clock
and keeps accruing through a pause, so a halted claim mints the identical
amount afterwards, just later.

Guards run as the first statement of each entrypoint, ahead of auth and
any storage read: a halted call should cost nothing and reveal nothing
beyond the already-public pause state.

Tests, per contract, covering the criteria in the issue: refund and
dispute resolution succeed with ALL_SCOPES halted and money actually
lands; intake blocked while no funds move; scope isolation in both
directions; auto-expiry restoring service with no unpause transaction;
non-signer and admin-key pause attempts rejected with NotASigner; the
duration cap refused; partial unpause. Plus the cross-contract case where
halting the token's mint stops emission claims underneath a healthy
engine.
Adds an "Emergency circuit breaker" section to the contracts README
covering the scope table, the never-guarded recovery paths and why each
one is exempt, the auto-expiry guarantee and its honest limit (an
unattended pause always clears; a present admin can re-arm, which the
upgrade flow already implies), the authorization model, the new
entrypoints and events, per-contract error codes, storage layout, and
CLI usage.

Replaces the "Upgrade path exists; pause does not" bullet in the
security notes — now false — with a statement of what compromising one
signer key actually buys: a bounded liveness attack on new business, and
no reach into any existing balance.

Sets cache-on-failure on the Soroban CI cache. A run that fails on
fmt or clippy has still built the whole dependency tree; caching it keeps
the retry from recompiling soroban-sdk from scratch, which is most of the
job's wall time. (Cargo caching itself was already in place.)

Notes for review, not fixed here: the "Upgrade governance" section and
the "Notes / follow-ups" list still describe the signer set as immutable
after initialize, which signer rotation made stale before this change.
Left alone to keep this diff to one concern.
@meshackyaro

Copy link
Copy Markdown
Contributor

Closes #42

What this adds

A shared emergency circuit breaker across escrow, reputation, loyalty-token and loyalty-emissions. Until now none of them had any halt mechanism — the only responses to a live exploit were "watch it drain" or "upgrade under fire", and the upgrade path itself is still being built out.

The primitive lives in contracts/governance-guard's new pausable module, next to the upgrade guard and for the same reason: every contract that needs it already depends on that crate.

The design constraint

A plain bool paused flag would be a rug pull waiting to happen. Three properties are load-bearing, each closing a failure mode that a boolean leaves open.

1. Scoped, not global. A pause names a bitmask, so entry paths halt independently of exit paths.

Scope Bit Guarded entrypoints
SCOPE_INTAKE 1 escrow: create_appointment, create_milestone_escrow, add_milestone · loyalty-token: mint · loyalty-emissions: create_schedule
SCOPE_SETTLEMENT 2 escrow: confirm_completion, approve_milestone, release_milestone_funds · loyalty-emissions: claim
SCOPE_ATTESTATION 4 reputation: submit_attestation
A contract with no entrypoint in some scope ignores a pause naming it, so an operator can broadcast one mask to every contract without special-casing.

2. Fund-recovery paths are never guarded. This is the half the issue is really about, and it is enforced by omission at the call sites — so it's stated explicitly in the module docs and pinned by tests rather than left implicit:

Contract Always callable, even while paused Why
escrow cancel_appointment, raise_dispute, resolve_dispute, raise_milestone_dispute, resolve_milestone_dispute Every route by which an escrowed balance reaches whoever is entitled to it.
loyalty-token transfer, transfer_from, approve, burn These move already-held balances. Only mint — the sole path creating supply that doesn't yet exist — is guarded.
loyalty-emissions reclaim Mints and burns nothing, so it can't take a balance anyone holds.
all every read-only view A consumer must be able to see the state that prompted the halt.
There is no scope value, ALL_SCOPES included, that reaches any of these.

3. Time-bound, enforced on read. A pause carries an expires_at ledger timestamp capped at MAX_PAUSE_DURATION (7 days), and expiry is evaluated on every guard consultation — so a pause lapses with no unpause transaction, no live admin, no working key.

Two judgement calls worth flagging in review

Pausing SCOPE_SETTLEMENT withholds a payout, so it deserves an argument rather than an assumption. It's a delay, not a seizure: it changes no party's power relative to the others — a client could always cancel a Funded appointment for a full refund, and either party could always force raise_disputeresolve_dispute; both stay open throughout. And release_milestone_funds is permissionless, which is precisely why it has to be haltable, or the scope is decorative. On emissions, vesting is a pure function of the ledger clock and keeps accruing through a pause, so a halted claim mints the identical amount afterwards, just later.

Authorization is any single governance signer, not the M-of-N threshold and deliberately not each contract's own admin. Gathering a threshold takes time an incident doesn't give you, and the action is bounded on every axis that matters — this mirrors cancel_upgrade, which is unilateral for the same reason. The signer set rather than admin because it's M-of-N, rotatable via the timelocked flow from #27, and survives the loss of one key; an incident is exactly when a single non-rotatable key is least trustworthy. unpause is unilateral in the same way, so a responder who pauses and then goes offline can't wedge it in place.

What this does not promise: an admin who is present and hostile can re-pause each time the window lapses. That isn't closable here — the same signer set can already replace all of a contract's code via the upgrade flow. The honest guarantee is narrower: an unattended pause always clears, and no pause of any duration can stop a user recovering funds they already own. The second half is what keeps the residual risk a liveness problem for new business rather than a custody problem for existing balances.

Interface

Added to all four contracts:

  • pause(caller, scopes, duration_secs) -> PauseState
  • unpause(caller, scopes) -> u32 — clears only the named scopes and returns
    what's still halted, without touching the deadline. Re-calling pause
    with a narrower mask would also lift scopes but restarts the clock on
    everything left, so partial unpause is how you bring the system back a
    piece at a time.
  • get_pause_state() -> Option<PauseState> (None once expired),
    paused_scopes() -> u32, is_paused(scope) -> bool

Events for off-chain monitoring: Paused { caller, scopes, expires_at } and Unpaused { caller, scopes, remaining_scopes }. Auto-expiry emits nothing — it's a read-time evaluation with no transaction behind it, so monitors should treat Paused.expires_at as the authoritative end of the window unless an Unpaused arrives sooner.

A dedicated OperationPaused error per contract, not a reused InvalidStatus: "the protocol is halted, retry later" and "this request was never valid" call for opposite reactions from a client. Plus InvalidPauseScope, InvalidPauseDuration, NotPaused, each appended at whatever offset came next in that contract's existing enum so no existing code moved.

Storage & safety

One new instance key, GovernanceDataKey::PauseState, holding PauseState { scopes, expires_at, paused_by, paused_at } — appended after PendingRotation so already-deployed contracts' key encodings stay put. Only one record exists at a time; a second pause replaces the first outright rather than layering, so halted scopes and deadline are always readable from one place. paused_by is recorded for attribution and grants no rights.

Guards run as the first statement of each entrypoint, ahead of auth and any storage read: a halted call costs nothing and reveals nothing beyond the already-public pause state.

Tests — 248 total, all passing

66 new unit tests in governance-guard (scope isolation, expiry boundaries including the exact expires_at instant, the duration cap, masked unpause, event emission, adversarial auth), plus per-contract integration tests for the wiring. The ones tied directly to the acceptance criteria:

  • a_client_can_still_cancel_and_be_refunded_while_everything_is_paused
    balances asserted, and the pause verified still in force afterwards
  • disputes_can_still_be_raised_and_resolved_while_everything_is_paused
  • milestone_disputes_resolve_while_everything_is_paused
  • holders_keep_full_control_of_existing_balances_while_everything_is_paused
    transfer, transfer_from, burn all succeed under ALL_SCOPES
  • intake_resumes_on_its_own_once_the_pause_expires — only the clock moves
  • a_non_signer_cannot_pause_the_escrow, the_admin_arbiter_is_not_a_pause_authority
  • a_pause_longer_than_the_cap_is_refused_outright
  • pausing_the_token_alone_stops_claims_at_the_mint_boundary — cross-contract
  • a_lapsed_pause_can_be_placed_again_by_a_present_admin — pins the limit of
    the guarantee so nobody later reads it as stronger than it is

CI

cargo fmt --check, cargo clippy --workspace --all-targets -- -D warnings, cargo test --workspace and cargo build --release --target wasm32v1-none all run clean locally. Clippy already blocks warnings on PRs, satisfying the "warnings automatically blocked" criterion.

Cargo dependency caching was already in place (Swatinem/rust-cache@v2); I added cache-on-failure: true, since a run that fails on fmt or clippy has still built the whole dependency tree and the retry shouldn't recompile soroban-sdk from scratch.

Noted, not fixed here

soroban-contracts/README.md's "Upgrade governance" section and its "Notes / follow-ups" list still describe the signer set as immutable after initialize — stale since signer rotation (#27) landed, and unrelated to this change. Left alone to keep the diff to one concern; happy to fix in a follow-up if you'd prefer.

Assessment — Request changes.

This is an excellent, well-explained, and carefully-scoped design for a shared emergency circuit breaker. The scoped bitmask, time-bound expiry, and explicit omission of fund-recovery paths are the right trade-offs for safety and recoverability. Conceptually I approve the approach, but before merging I want a few clarifications, a couple of security / testing gaps closed, and some documentation to be added. But a couple of bottle-necks and things to take note of:

Blocking / required changes

  1. Tests: add explicit unit/integration tests for pause expiry and scope semantics

    • Test that pause expires automatically: create pause with a short expires_at in tests, fast-forward time past expires_at, then confirm guard allows previously-blocked entrypoints.
    • Test enforcement of MAX_PAUSE_DURATION (attempt to create a pause longer than MAX_PAUSE_DURATION and assert revert).
    • Test that fund-recovery functions remain callable while scopes are paused (for each contract: escrow cancel/raise_dispute/resolve_dispute, loyalty-token transfer/approve/burn, loyalty-emissions reclaim).
    • Test cross-contract broadcast semantics: applying a pause bitmask to a contract that has no entrypoints in that scope should be ignored — add a test that broadcasts a mask to all contracts and assert only intended entrypoints are blocked.
    • Test permissioning: only a governance signer can pause/unpause; non-governance addresses revert.
    • Test reentrancy/race cases around pause/unpause called concurrently where possible.
  2. Events & observability

    • Emit an event on pause/unpause (e.g., PauseSet(address operator, uint256 mask, uint256 expiresAt), PauseCleared(address operator, uint256 mask) or single PauseChanged). Include operator, mask, expires_at, and a reason optional string.
    • Add events so off-chain indexers and operators can track active pauses and expiries.
  3. Documentation & changelog

    • Add or update module-level docs inside contracts/governance-guard/pausable explaining:
      • The three design constraints (scoped masks, fund-recovery omission, time-bound expiry).
      • The semantics of "checked on read" and what exactly constitutes a guard consultation.
      • The single-governance-signer authorization rationale (short note and link to Governance signer rotation for the upgrade guard #27/timelock).
      • Example flows: how to pause/unpause a specific scope; how to broadcast to many contracts; what happens to vesting and accrued rights during pause.
    • Add a short changelog entry and mention that this closes Emergency Circuit Breaker with Scoped Function Guards & Time-Bound Auto-Expiry #42 (you already note it in the PR body — copy into CHANGELOG.md or releases notes).
  4. Authorization & governance clarity

    • Add an explicit comment in code and module docs clarifying why a single governance signer is used, and a pointer to the timelock/rotation flow that governs signer rotation.
    • Add an on-chain check test showing that a non-governance key fails and a governance signer succeeds.
  5. Gas and hot-path cost

    • Guard checks run on every guarded entrypoint. Please add a short gas benchmark or tests demonstrating the incremental gas cost on hot paths (create_appointment, release_milestone_funds, mint, claim). If the guard adds significant overhead, consider micro-optimizations (bitmask operations, minimize storage reads, cache patterns).
    • If the expiry is read from storage on every guard consult, consider documenting why that storage read is acceptable or optimize to reduce storage loads.
  6. Edge cases & safety questions to confirm in code/comments

    • Clarify which timestamp is used (block.timestamp / ledger timestamp). Document the attacker model around miner/validator manipulation for short windows and why MAX_PAUSE_DURATION = 7 days is chosen.
    • Confirm that evaluating expiry "on read" (i.e., checks at entry) cannot permanently lock funds because expiry is enforced from guards; add tests for a scenario where pause expires while a transaction is pending.
    • Ensure pause/unpause functions are non-reentrant and protected with appropriate modifiers / checks.
  7. Small API / UX improvements

    • Add a view helper to list active pauses for a contract (return mask + expires_at) so operators and on-chain watchers can query active pause state easily.
    • Consider adding a reason field (bytes32 or string limited length) for operator-provided context when creating a pause — optional but useful off-chain.
    • Consider adding helper scripts (owner-maintainers repo or examples) that broadcast a pause mask to all contracts; at minimum mention the recommended approach in the docs.

Minor nits (non-blocking)

  • Naming: scan for consistent use of SCOPE_{NAME} and ALL_CAPS constants. Minor rename suggestions are inline in code (I'll point them out if you want line-level comments).
  • Commit history: this is a large feature; consider splitting future PRs so review is easier, but not required here.
  • Tests: add more negative tests for unexpected inputs (e.g., mask=0 behavior).
  • Add unit tests for serialization/deserialization if your pause entries are complex structs stored in mapping arrays.

Suggested PR review text to paste into GitHub
(you can paste this as a single review comment)

"Summary

  • Great work: adds a scoped, time-bound emergency circuit breaker shared across multiple contracts. The design addresses fund safety and preserves recovery paths.

Requested changes before merge

  1. Add unit/integration tests for expiry behavior, MAX_PAUSE_DURATION enforcement, fund-recovery function availability, cross-contract broadcast semantics, and permissioning.
  2. Emit events when pauses are created/cleared (operator, mask, expires_at).
  3. Add module-level docs and a CHANGELOG entry describing design rationale, examples, and governance authorization (single signer) rationale.
  4. Provide gas impact numbers or microbenchmarks for hot paths and consider micro-optimizations if needed.
  5. Clarify timestamp choice and attacker model for block.timestamp manipulation; add tests for expiry while transactions are pending.
  6. Add a view helper to query active pauses and consider an optional reason field for operator context.

Once those items are addressed (tests + docs + events + gas notes), I’ll re-review and we can merge. Conceptually this is a strong approach and I’m excited to get it in — thanks for the thorough design and tests so far."

Next steps from me

  • I’ve reviewed the code and created this list of required changes and suggestions. Please address the test, docs, event, and gas items above and push an update; I’ll re-review the updates ASAP and sign off.

If you want, I can:

  • Add exact unit-test scaffolding (I can draft a test file showing time-fast-forward + expiry assertions).
  • Draft the PauseSet/PauseCleared event signatures and example emit sites.
  • Review the gas benchmark results if you post them.

Overall: conceptually approved, but request changes until the test, observability, and docs gaps are closed.

Addresses review feedback on workman-labs#46.

Adds a length-capped `reason` to `pause`, stored with the record and
emitted with the event, so "why is this halted?" is answerable from chain
state rather than from a chat log nobody can find at 3am. Capped at 64
bytes because the record lives in instance storage that every subsequent
invocation pays to load — an incident ticket reference belongs on-chain,
the write-up does not. Empty is allowed, so the field never stands between
a responder and a halt. New `InvalidPauseReason` error, appended per
contract so nothing existing moved.

Adds hot-path cost measurement using the SDK's budget metering, since the
guard runs on every guarded entrypoint and its cost should be measured
rather than assumed:

  trivial instance-storage view                       51,549 CPU insns
  paused_scopes(), no pause record                    51,717   (+168)
  paused_scopes(), live pause record                  77,023   (+25,474)
  create_appointment, no pause record                336,299
  create_appointment, live record on another scope   365,219   (+8.6%)
  create_appointment rejected while paused            78,216   (23%)

In normal operation — no pause ever set, which is the state the contracts
are in essentially always — the guard costs ~168 instructions against a
336k booking. The ~25k figure is deserializing the record and is paid only
while an incident is in progress. A rejected call costs ~23% of the work
it replaces, because the guard sits ahead of auth and every other storage
access, which is what makes a pause a usable response to an entrypoint
being hammered rather than an amplifier. No micro-optimization is
warranted at these numbers; the cap on `reason` is what keeps the
incident-time figure bounded too. Native-test metering underestimates
compiled Wasm, so the tests fence relative behaviour, not absolute cost.

New tests: reason round-trip through storage and event, cap boundary at
exactly 64 and 65 bytes, full PauseState serialization round-trip,
broadcast no-ops on escrow and loyalty-token for scopes they don't
implement, a genuine two-contract sweep with independent records and
independent signer sets, authority not leaking between deployments, and
ordering cases — a second signer's pause observing the first, and repeated
guard consultations within one ledger all agreeing. 267 tests total.

Documents what was previously left implicit: what "enforced on read"
means and what a guard consultation is; that expiry cannot land
mid-transaction, since the ledger timestamp is fixed for the whole
invocation; which clock is used and why validator influence points the
harmless way (forward only ends a pause sooner, and recovery paths consult
no clock at all); why wall-clock seconds rather than ledger sequence; that
Soroban forbids reentrancy and pause/unpause make no cross-contract calls
at all; worked example flows; and an explicit pointer to the workman-labs#27 rotation
flow, including the deliberate asymmetry that rotating who may pause is
timelocked while pulling the alarm is not.

Adds scripts/broadcast-pause.sh for sweeping a mask across all four
contracts, and CHANGELOG.md.
@bbjiggy

bbjiggy commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks — pushed f26fee2 addressing this. Taking the items in order, and flagging up front where something was already in the branch, since a few of the requests are already satisfied and I'd rather point you at the code than silently re-add it.

One general note: several items are framed in Solidity terms (block.timestamp, uint256, non-reentrant modifiers, mapping arrays, gas). These are Soroban/Rust contracts, so I've answered the underlying concern in each case rather than the literal mechanism — where the Soroban answer is genuinely different, I've said so and documented it in the code.


1. Tests

Most of these were already in the branch. Concretely, by request:

Request Where
Pause expires automatically; fast-forward past expires_at, confirm previously-blocked entrypoints work a_pause_lapses_on_its_own_with_no_unpause_transaction, a_pause_is_still_in_effect_one_second_before_its_deadline, plus per-contract intake_resumes_on_its_own_once_the_pause_expires, attestations_resume_on_their_own_…, minting_resumes_on_its_own_…
MAX_PAUSE_DURATION enforcement a_duration_past_the_cap_is_rejected, expiry_holds_for_a_maximum_length_pause_too, and a per-contract a_pause_longer_than_the_cap_is_refused in all four
escrow cancel/raise_dispute/resolve_dispute callable while paused a_client_can_still_cancel_and_be_refunded_while_everything_is_paused, disputes_can_still_be_raised_and_resolved_while_everything_is_paused, milestone_disputes_resolve_while_everything_is_paused — balances asserted, and the pause verified still in force afterwards so it's clearly exemption and not expiry
loyalty-token transfer/approve/burn callable while paused holders_keep_full_control_of_existing_balances_while_everything_is_paused
loyalty-emissions reclaim callable while paused reclaim_stays_open_while_settlement_is_halted
Only a governance signer can pause/unpause a_non_signer_cannot_pause/_unpause at the primitive level, plus a_non_signer_cannot_pause_the_escrow/_the_token/_the_engine and the_admin_arbiter_is_not_a_pause_authority / the_config_admin_is_not_a_pause_authority / the_emissions_admin_is_not_a_pause_authority

Genuinely missing, now added:

  • Broadcast semantics. Only reputation had a no-op test. Added a_scope_escrow_has_no_entrypoints_for_is_a_well_formed_no_op and scopes_the_token_has_no_entrypoints_for_are_well_formed_no_ops, plus a real two-contract sweep — one_scope_mask_broadcast_to_two_contracts_halts_only_the_intended_paths — using the emissions fixture, which registers the token and the engine as separate deployments with separate storage and separate signer sets. Also the_two_contracts_pause_records_are_fully_independent and a_signer_of_one_contract_cannot_pause_the_other, since authority leaking across deployments would be the nastier failure.
  • Serialization. the_whole_pause_record_survives_a_storage_round_trip reads the raw instance-storage entry back and asserts every field.
  • mask = 0. Already covered by an_empty_scope_mask_is_rejected_rather_than_silently_pausing_nothing — empty is rejected rather than treated as a no-op, because during an incident a mask that halts nothing is a mistake you want to hear about. Also an_unknown_scope_bit_is_rejected, including a valid bit mixed with a bogus one.

Reentrancy / races. There is nothing to race here, and I've documented why rather than writing a test that would only look reassuring:

  • Soroban's host forbids reentrancy outright — a contract already on the call stack cannot be re-entered.
  • pause/unpause make no cross-contract calls at all. They touch one instance-storage key and publish an event, so they can't yield control mid-update even in principle.
  • Transactions within a ledger are strictly ordered, so "two signers at once" is always one after the other.

What is testable is that the second actor sees the first's write, so I added a_second_signers_pause_observes_and_replaces_the_first and unpause_immediately_after_pause_in_the_same_ledger_leaves_nothing_halted.

Total: 267 tests, all passing.

2. Events & observability

Already implemented — Paused { caller, scopes, expires_at } and Unpaused { caller, scopes, remaining_scopes }, with topics ["gov_pause","paused",caller] / ["gov_pause","unpaused",caller]. Defined in pausable.rs and asserted by pause_and_unpause_each_emit_an_event, a_rejected_pause_emits_nothing, and a_rejected_unpause_emits_nothing_and_leaves_the_pause_standing. So the operator/mask/expiry fields you asked for were all there.

Added the optional reason you suggested (item 7), on both the event and the stored record. It's String, capped at 64 bytes, and may be empty. I capped it rather than leaving it free-form because the record lives in instance storage that every subsequent invocation pays to load — an incident ticket reference belongs on-chain, the write-up doesn't. Empty is allowed so the field never stands between a responder and a halt.

One thing worth stating explicitly, now in the docs: auto-expiry emits nothing. It's a read-time evaluation with no transaction behind it, so there's no execution context to emit from. Indexers should treat Paused.expires_at as the authoritative end of a window unless an Unpaused arrives sooner. That's inherent to expiry needing no transaction — which is the property that makes it trustworthy.

3. Documentation & changelog

Module docs already covered the three constraints and the authorization rationale. Added the parts that were genuinely missing:

  • What "enforced on read" means, with "guard consultation" defined precisely as a call to require_not_paused / is_paused / paused_scopes / get_pause_state.
  • Worked example flows — halt, widen, partial lift, expiry — as a walkthrough.
  • Broadcast guidance: it's N transactions, not one; they needn't land in the same ledger; a partial sweep is a valid state, not a corrupt one, because each contract's guard reads only its own record.
  • What a pause does to accrued rights: nothing. Vesting keeps accruing, escrowed balances stay reachable, recorded scores stay readable. A pause delays an action, never a right.
  • CHANGELOG.md added, with the Emergency Circuit Breaker with Scoped Function Guards & Time-Bound Auto-Expiry #42/feat(contracts): emergency circuit breaker with scoped function guards & time-bound auto-expiry #46 entry. I seeded a short "Before this changelog" section from commit history rather than reconstructing earlier PRs in detail, so it doesn't claim more precision than it has.

4. Authorization & governance clarity

Expanded the module docs with an explicit pointer to the #27 rotation flow (propose_signer_rotationapprove_signer_rotationexecute_signer_rotation), and named the asymmetry that I think is the actual design point: rotating who may pause is timelocked and needs threshold agreement; exercising the pause is instant and unilateral. Changing the guest list is a governance decision; pulling the alarm is not.

The on-chain check tests you asked for already existed (§1 above) — including that each contract's own admin is specifically not a pause authority, which is the case most likely to regress.

5. Gas / hot-path cost

Measured with the SDK's budget metering rather than estimated. Tests are in contracts/escrow/src/test.rs under "hot-path cost":

CPU instructions
trivial instance-storage view (get_storage_version) 51,549
paused_scopes(), no pause record 51,717 — +168
paused_scopes(), live pause record 77,023 — +25,474
create_appointment, no pause record 336,299
create_appointment, live record on another scope 365,219 — +8.6%
create_appointment rejected while paused 78,216 — 23%

The shape that matters: in normal operation — no pause ever set, which is the state the contracts are in essentially always — the guard costs ~168 instructions against a 336k booking. That's a rounding error. The ~25k figure is deserializing the PauseState struct and is paid only while an incident is in progress, which is exactly when you've accepted degraded throughput. It's also why I capped reason: the cap is what keeps the incident-time number bounded.

A rejected call costs ~23% of the work it replaces, because the guard is the first statement of each entrypoint, ahead of auth and every other storage access. That's deliberate — it makes a pause a usable response to an entrypoint being hammered rather than an amplifier.

On your "if significant, consider micro-optimizations": I don't think any are warranted at these numbers, and I'd rather not add caching complexity to a safety mechanism to save 168 instructions. On the storage read specifically — the instance entry is already in the footprint of any invocation that touches admin or governance state, so it's a hit on an entry the host has loaded regardless, not an extra ledger read. The tests fence relative behaviour (a loose "must not approach doubling") rather than absolute numbers, since native-test metering underestimates compiled Wasm and I don't want a brittle CI failure on an SDK bump.

6. Edge cases

  • Which timestamp. env.ledger().timestamp() — Stellar's ledger close time in seconds, agreed by SCP consensus and required to be monotonic, not a value any single validator picks. So the Solidity block.timestamp threat model doesn't transfer directly. More to the point, the manipulation surface points the harmless way: nudging the clock forward can only end a pause sooner, backward isn't possible, and no fund-recovery path consults a clock at all. There's no clock manipulation that extends a halt or reaches a refund. Now documented, along with why I chose wall-clock seconds over ledger sequence (which the upgrade timelocks use): a pause duration is negotiated between humans mid-incident — "give us six hours" — and seconds say that directly, where a ledger count only approximates it at an assumed close rate.
  • 7 days. Documented as the shortest duration that still covers triage → patch → review → upgrade through the M-of-N flow across a weekend and time zones. Longer stops being "we are responding" and starts being "we have forgotten", which is the state auto-expiry exists to recover from.
  • Expiry while a transaction is pending. Can't happen: env.ledger().timestamp() is the close time of the ledger the transaction executes in and is constant for the whole invocation, including across cross-contract calls. A single call can never see the guard paused at one point and open at another. Documented, and pinned by repeated_guard_consultations_within_one_ledger_all_agree, which checks several consultations inside one invocation both before and after the deadline.
  • Permanent lock. Not reachable — expiry is enforced from the guards with no transaction required, so losing every key still clears the pause. And independently of expiry, recovery paths are unguarded, which is the stronger guarantee.

7. API / UX

  • View helper for active pauses. get_pause_state() already returns exactly { scopes, expires_at, paused_by, paused_at, reason } in one read, and None once expired — deliberately, so a view can never disagree with require_not_paused. I've made that more prominent in the README; I think it's what you were after, but happy to add a different shape if you had something else in mind.
  • Reason field. Added, per §2.
  • Broadcast helper. Added soroban-contracts/scripts/broadcast-pause.sh with pause / unpause / status subcommands, client-side validation of the duration cap and reason length, and per-contract failure isolation (one failure warns and continues, since a partial sweep is valid). Documented in the README, including the caveat that each contract has its own signer set.

Minor nits

  • NamingSCOPE_* / MAX_PAUSE_* / ALL_SCOPES are consistent throughout. Happy to take line-level suggestions if you still see drift.
  • Commit history — split into four focused commits rather than one.

Still not fixed, deliberately

README.md's "Upgrade governance" section and "Notes / follow-ups" list still describe the signer set as immutable after initialize. That's been stale since #27 landed and is unrelated to this PR, so I've left it out of this diff. Say the word and I'll do it here or in a follow-up.

CI is green: fmt --check, clippy -D warnings, cargo test --workspace (267 passing) and the release WASM build.

@meshackyaro

Copy link
Copy Markdown
Contributor

Approved — minor nits.

Minor follow-ups I'd like before merging

  1. Clarify the reason cap semantics in the docs and tests

    • In docs and the function API comment, state explicitly that the 64 limit is bytes (UTF-8), and add a short note that callers should limit to ASCII or otherwise account for multibyte characters. Your tests check 64/65, but make that byte-vs-char assertion explicit in the README so indexers/CLI authors know what to validate client-side.
  2. Event signature / topics documented in README

    • You mentioned topics ["gov_pause","paused",caller] / ["gov_pause","unpaused",caller]. Please add the exact event signature (field names + types) and the topic ordering to the module README (or the events subsection) so off-chain indexers can rely on it. If you already have this, point me to the exact file/lines and I’ll mark as done.
  3. Benchmarks: small follow-up

    • Add the benchmark table (or a short summary) to the docs (maybe in the pausable README section) and note that numbers are SDK-test metered, not absolute Wasm costs. That makes the tradeoff visible to future reviewers and operations folks.
  4. CHANGELOG placement

    • The changelog entry is great. For release hygiene, put a short note in the PR description linking to CHANGELOG.md (or vice versa) so release automation (or a releaser) can pick it up easily.

Optional / future ideas (no block)

  • Consider adding a small example event JSON in the README showing paused/unpaused with reason, expires_at, and caller for indexers to copy.
  • If you want to harden the on-chain storage cost predictability further in future, a fixed-size byte array type for reason could be revisited; not necessary now given the cap and rationale.

Suggested review text to post (paste-ready)
"Thanks — excellent follow-up. You’ve implemented the tests, events, docs, reason field, benchmarks, and ops tooling I requested; CI is green and test coverage is comprehensive. I have three small requests before merging: (1) explicitly document that the 64 limit on reason is bytes (UTF-8) and add that byte-vs-char note to tests/docs, (2) add the exact event signature and topic ordering to the pausable README so indexers can rely on it, and (3) add the benchmark table or a short note to docs clarifying SDK-test metering vs compiled Wasm. After those tiny docs clarifications I’m happy to approve and merge."

Next steps from me

  • I’m happy to merge after you push a tiny docs commit covering the three follow-ups above.

Thanks — this is a solid, well-tested safety primitive and a great improvement to core contract resilience.

…able

Addresses the four follow-ups on workman-labs#46. Two of them turned out to be
statements about behaviour rather than prose, so they are pinned by
assertion instead of described and hoped for.

Event wire format. Documented the exact topic ordering and data shape, and
added tests that assert it against real emitted events, so the README table
cannot drift from what is emitted without a test failing:

  topics  Symbol("gov_pause"), Symbol("paused"|"unpaused"), Address(caller)
          — prefix symbols first, then each #[topic] field in declaration
          order
  data    Map<Symbol, Val> keyed by field name, since contractevent's
          data_format defaults to "map". Field *names* are part of the
          contract, their order is not — indexers must key, not position.

Verified rather than assumed: the map format is the macro default, which is
worth stating because "data" reads as a tuple if you only see the struct.
Added an example event JSON for indexers to copy, per the optional
suggestion.

The reason cap is bytes, not characters. MAX_PAUSE_REASON_LEN bounds what
String::len() reports, which is the UTF-8 byte length. For ASCII the
distinction is invisible, which is exactly why it needed pinning — a client
validating character count would submit reasons the contract rejects. Two
tests make it executable rather than assertable-by-comment: 32 characters of
U+00E9 is exactly 64 bytes and is accepted; 33 characters is 66 bytes and is
rejected despite being half a 64-character budget. Documented on the
constant, in PauseState, on `pause`, and in the README with the byte-count
idioms for JS and Python.

Benchmark table moved into the README with the metering caveat made
prominent rather than parenthetical: these are SDK-test-metered numbers that
underestimate compiled Wasm and carry no instantiation cost, so they are a
relative comparison, not a fee estimate.

Also promoted the circuit-breaker section's bold labels to real headings
with a jump list, since it had grown past the point where prose bolding was
navigable. Heading names are deliberately distinctive (Pause errors, Pause
storage layout) so their anchors don't collide with the per-contract Errors
and CLI usage sections further down and shift existing ones.

271 tests, all passing.
 and PR workman-labs#46

Pairs with the release-notes link now in the PR description, so a releaser
can get from either end to the other.
@bbjiggy

bbjiggy commented Aug 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks! All four pushed — 7e5e51d (docs + tests) and caa1145 (cross-links).

Two of these turned out to be claims about behaviour rather than prose, so I pinned them with assertions instead of just writing them down. If the wire format or the byte cap ever drifts, a test fails rather than the README quietly becoming wrong.


1. reason cap is bytes, not characters

Stated explicitly on MAX_PAUSE_REASON_LEN, on PauseState::reason, on pause, and in the README — and made executable, because this is exactly the kind of thing a comment can be wrong about:

/// 32 × U+00E9, two bytes each: 32 characters, exactly 64 bytes.
const REASON_32_CHARS_64_BYTES: &str = "éé…";   // accepted, len() == 64

/// 33 × U+00E9: 33 characters — comfortably under a 64-*character* limit —
/// but 66 bytes, which is over the cap.
const REASON_33_CHARS_66_BYTES: &str = "éé…";   // InvalidPauseReason

a_multibyte_reason_is_measured_in_bytes_and_accepted_at_exactly_the_cap and a_multibyte_reason_over_the_byte_cap_is_rejected_despite_a_short_char_count, in contracts/governance-guard/src/pausable_test.rs. The second is the one that matters for client authors: 33 characters would pass any character-count check, and is rejected. The README carries the byte-count idioms — Buffer.byteLength(s, 'utf8') in JS, len(s.encode('utf-8')) in Python — in a callout rather than buried in a sentence.

(For the record, this is String::len()'s own semantics — it calls the host's string_len, which is bytes. I verified it rather than assuming, which is how the 32-vs-33 boundary got picked.)

2. Event signature and topic ordering

I only had this as an inline mention, so: now a proper Events subsection in the README with exact types, plus doc comments on both event structs.

Event Topics (in order) Data map
Paused Symbol("gov_pause"), Symbol("paused"), Address(caller) expires_at: u64, reason: String, scopes: u32
Unpaused Symbol("gov_pause"), Symbol("unpaused"), Address(caller) scopes: u32, remaining_scopes: u32

One thing worth flagging for indexer authors, which I'd left implicit and which surprised me when I checked: the data is a Map<Symbol, Val> keyed by field name, not a tuple — contractevent's data_format defaults to "map". So field names are part of the contract but their order is not; index by key, never by position. Easy to get wrong from reading the struct definition alone.

Topic ordering is: the prefix symbols from the attribute first, then each #[topic] field in declaration order.

Both shapes are asserted against real emitted events in contracts/escrow/src/test.rspaused_event_has_the_documented_topics_and_data_shape and its unpaused counterpart. Also added the example event JSON you suggested, so an indexer author can copy the shape rather than derive it.

3. Benchmark table in the docs

Moved into the README under Hot-path cost, with the metering caveat promoted from a parenthetical to a blockquote right under the table:

⚠️ These are SDK-test-metered numbers, not absolute Wasm costs or fees. Per the SDK's own documentation, native Rust test execution underestimates both CPU and memory relative to the compiled Wasm, and this harness charges no Wasm instantiation. They are meaningful as a relative comparison of the same call with and without a pause record — and should not be used to size a transaction fee.

The changelog links to the table, and the table points at the tests, so an operator or future reviewer lands on the caveat before the numbers either way.

4. CHANGELOG placement

PR description now opens with a Release notes link to soroban-contracts/CHANGELOG.md, noting the entry lives under ## [Unreleased] with Added / Changed / Notes. The changelog entry links back to #42 and this PR (caa1145), so a releaser can get from either end to the other. I also refreshed the description's own details that the follow-up round had made stale — test count, the pause signature, and the events line.

Optional items

  • Example event JSON — added, per above.
  • Fixed-size byte array for reason — agreed it's not needed now, and I'd lean against it later too: BytesN<32> would force callers to hash or truncate, which costs the human-readability that's the whole point of the field, and the cap already bounds the storage cost. Worth revisiting only if incident-time cost ever shows up as a real constraint, which the ~25k figure suggests it won't. Happy to be overruled if you'd rather have the predictability.

Also promoted the circuit-breaker section's bold labels to real headings with a jump list, since it had outgrown prose bolding. The heading names are deliberately distinctive (Pause errors, Pause storage layout) so their anchors don't collide with the per-contract Errors / CLI usage sections further down and silently shift those.

271 tests passing; fmt, clippy -D warnings, cargo test --workspace and the release WASM build all green.

The only thing still deliberately untouched is the stale "signer set is immutable after initialize" claim in the Upgrade governance section, left over from #27. Say the word and I'll fold it in here or open a follow-up.

@meshackyaro meshackyaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve — great work.

This PR adds a well-reasoned, thoroughly-tested scoped emergency circuit breaker and wires it into the four core contracts. Tests (267), events (with a capped reason), docs, benchmarks, ops tooling (broadcast script), and CHANGELOG are all in place and CI is green. The design balances safety (fund-recovery paths unguarded), operability (single-signer pause/unpause, automatic expiry), and observability (events + reason + view helpers). I’m happy to approve and merge this PR — thanks @bbjiggy for working on it.

@meshackyaro
meshackyaro merged commit 28a20c1 into workman-labs:development Aug 20, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Emergency Circuit Breaker with Scoped Function Guards & Time-Bound Auto-Expiry

2 participants