Job dispatcher: single-worker loop parks inline on awaiting_signature → head-of-line blocking

[TaprootFreak]

Summary

The background job dispatcher processes the mpsc job queue serially and inline and parks on the awaiting_signature wait inside that same loop. While any one send job is parked waiting for its commit, every subsequent queued job — mints included — is stuck in queued/progress=0 until the parked job resolves or its park times out. On the shared DEV node under concurrent load this surfaces as mints/sends frozen in queued for up to the 10-minute awaiting_signature_timeout, then eventually completing once the queue drains.

Mechanism (from code)

node/src/job_dispatcher.rs (spawn, ~L232-250):

while let Some(env) = rx.recv().await {
    // Process serially: one prove at a time ... We do NOT
    // tokio::spawn here — that would defeat the single-worker invariant.
    process_envelope(&job_store, &app_state, &notify_map, timeout, env).await
}

For a send, process_send_initial → after the prove leg → wait_for_commit (~L567):

let outcome = tokio::select! {
    _ = notifier.commit_wake.notified() => Signaled,
    _ = tokio::time::sleep(awaiting_signature_timeout) => TimedOut, // default 10 min
};

This .await runs inside the single-consumer loop, so the loop does not call rx.recv() again — and no other queued job advances — until the wallet's POST /api/jobs/:id/commit fires commit_wake, or the 10-minute park times out.

The single-worker invariant is correct for the prove leg (Plonky2 saturates all cores; parallel proves only thrash cache — see the module doc-comment and project invariant 3). But the awaiting_signature park is pure waiting, not compute, and should not occupy the single worker.

Impact

• One slow or abandoned commit from any client starves all other users' jobs (mints and sends) on the same node — a head-of-line-blocking / shared-resource fairness hazard.
• Most visible during E2E bursts against the shared DEV node: jobs sit queued/progress=0 for ~180s+ while a foreign send is parked, then complete normally once unblocked. The HTTP layer stays healthy throughout (read-only endpoints answer in ~80ms), which distinguishes this from a node-down condition.

Evidence

• A single mint against an idle queue completes in ~6s (queued → proving → completed).
• Under concurrent E2E load, fresh probe mints observed stuck at status: queued, progress: 0 for 180s while /api/info and /api/balance answered 200 in ~80ms — i.e. the worker was parked, not the process.

Proposed direction (not in scope of the E2E test PR)

Decouple the awaiting_signature wait from the single prove-worker: when a send job reaches awaiting_signature, return the consumer to rx.recv() and let the commit handler re-enqueue a broadcast-leg envelope. The dispatcher already re-enters via process_send_resume for the (Send, AwaitingSignature) state on a fresh envelope, so the wake path can drive the broadcast leg without holding the worker during the wait. The prove (single-flight) and the broadcast (cheap) stay serialized; only the idle park moves off the hot loop.

Context

Surfaced while migrating the api_remote E2E suite to the async Job-API (PR for branch test/api-remote-async-job-api). The E2E suite itself is correct (it commits each send promptly); a clean single CI run on a freshly-deployed node with an empty queue is not expected to hit this. Filing separately because the fix is a node-side architectural change, not a test change.