Protocol pause switch is in-process: pausing one worker leaves others live

[YaronZaki]

Labels / Complexity: bug, incident, redis, Backend, reliability · Extremely High — 500

Problem

The incident-response pause switch in quantara/web_app/api/pausable.py is a module-level, in-process flag:

pause_controller = PauseController()  # threading.Lock + bool, no persistence

protocol_pause_middleware reads pause_controller.status().paused directly. Because the state lives only in the memory of the process that handled the admin request, pausing does not propagate to other processes.

Consequences:

• In any multi-worker or multi-replica deployment (the README and devops/ compose files run the backend as a service that can be scaled), POST /api/admin/pause pauses only the single worker that served the request. All other workers keep accepting deposits, closes, and withdrawals.
• On a restart, the pause is silently lost, so an operator who paused the protocol to stop a live incident comes back after a deploy with the protocol serving again.
• The token comparison in verify_admin_token uses != rather than hmac.compare_digest, which is a minor timing side channel on the admin credential (secondary to the core defect).

The project already has durable, cross-process state primitives this should reuse: the Redis-backed session_store (quantara/web_app/api/session.py), the wallet-connect Redis session (quantara/web_app/api/walletconnect.py), and the notification dedupe (quantara/web_app/telegram/dedupe.py). The pause switch is the one incident control that is not durable.

Root cause

pause_controller = PauseController()          # ← process-local bool
# ...
async def protocol_pause_middleware(request, call_next):
    if (... and pause_controller.status().paused):  # ← only sees this worker's state
        return JSONResponse(status_code=503, content={"detail": PROTOCOL_PAUSED_DETAIL})

Why this is architecturally hard

1. The shortcut — "store the bool in Redis" — is directionally right but underspecified: the middleware now has to read Redis on every /api/* request, which needs a shared client, a read path that fails open or closed, and a caching strategy so the pause check does not add a Redis round-trip to every request.
2. There is no established config/feature-flag abstraction to copy; the contributor must decide whether this is a Redis key (protocol:paused), a small dataclass backed by Redis, or a shared "incident controller" with an in-memory cache plus TTL.
3. Fail-closed vs fail-open on Redis outage is a real safety decision: fail-open re-enables a broken circuit breaker during an incident, fail-closed takes the whole protocol down when Redis blips. The choice must be documented.
4. The admin token check and the pause endpoints currently bypass the standard APIError envelope (quantara/web_app/api/errors.py), so aligning them with the rest of the API is a secondary but necessary cleanup.

Proposed design

# durable pause backed by Redis, cached briefly in-process
class PauseController:
    async def is_paused(self) -> bool:
        return await redis.get("protocol:paused") == b"1"

async def protocol_pause_middleware(request, call_next):
    if is_api_path(request) and not is_exempt(request.url.path) and await pause_controller.is_paused():
        return JSONResponse(status_code=503, content={"detail": PROTOCOL_PAUSED_DETAIL})
    return await call_next(request)

Offer the fail-open/fail-closed choice and cache TTL explicitly in the PR description.

Downstream impact

No ABI or public API change beyond the admin endpoints' behaviour; the frontend does not call /api/admin/pause. Operators' runbooks and any incident automation that expects an in-process toggle must be updated.

Acceptance criteria

Service

• Pausing via POST /api/admin/pause is visible to every backend worker/replica, not just the serving process.
• The pause state survives a worker restart.
• Admin token comparison uses a constant-time comparison.
• The Redis-outage behaviour (fail-open vs fail-closed) is implemented and documented.

Tests

• Tests prove the pause is durable (set pause, simulate a second process/reader, assert 503) and that unpause restores service.
• Tests cover the Redis-unavailable path with the chosen behaviour.
• Tests run via cd quantara && poetry run pytest web_app/tests.

Out of scope

Do not add role-based admin auth or audit logging in this issue; only make the pause switch durable and constant-time.

Getting started

Files in scope: quantara/web_app/api/pausable.py, quantara/web_app/api/main.py (middleware registration). Verify with:

cd quantara && poetry run pytest web_app/tests -k pausable

Good first files to read: quantara/web_app/api/pausable.py, quantara/web_app/api/session.py, quantara/web_app/telegram/dedupe.py.

[Sensei-Victor]

THIS IS MY DOMINANT FIELD

[daRk8238]

Hello,
I can work on this issue, kindly assign it to me
Thank you

[Caerlower]

Allow me to do this, I can fix this.

[tolulopedd26]

can i handle this?

[grantfox-oss]

🦊 GrantFox — @tolulopedd26 has been assigned to this issue as part of the Third Campaign campaign!

Next steps:

1. Open a Pull Request referencing this issue (e.g., Closes #416)
2. Your PR will be reviewed by the Quantarq maintainers

Good luck! Track your progress on GrantFox.