ci: cut node-tests wall ~47 min -> ~10-12 min via shared Postgres + parallel tests

[TaprootFreak]

Speed up node-tests CI job: shared Postgres + parallel test execution

Summary

The Node + Shared Tests (M3 Ultra) CI job currently takes ~47 minutes wall on a self-hosted M3 Ultra agent (measured 2026-06-02, two consecutive Release PR runs: 46:48 and 46:36 min). The job is forced to run serially (--test-threads 1) by long-standing repo convention, and every test that needs Postgres spawns its own postgres:17 container via testcontainers. Investigation below shows the --test-threads=1 invariant is stricter than it needs to be, and the per-test container model is the dominant fixed cost.

This issue proposes two complementary optimisations and quantifies the expected wall-time gain + implementation effort for each.

Today's CI shape

• Workflow: .github/workflows/ci.yaml job node-tests (line 154 onwards).
• Test command: cargo nextest run -p node -p shared --release --all-features --test-threads 1 -E 'not binary(api_remote)' (line 281).
• Test count: 400 tests across 12 *_tests.rs files in node/src/.
• Postgres-touching tests: ~220 of the 400 (db_tests 102 + state_tests 58 + r2_probe_tests 32 + username_tests 28 + main_tests 15 + runtime_tests 5, minus a few overlaps).
• Per-test Postgres container spawn: ~3 s wall (measured) — that alone is ~11 min of fixed setup wall across the suite.

Distribution of test wall times (measured, Apple M5 Max workstation, 217-of-400 partial sweep)

Bucket	Count (of 217)	Wall per test	Wall total	Typical test
10–19 s	~100	12–19 s	~16 min	router::tests::* — container + handler + ZK prove
3–10 s	~70	5–8 s	~7 min	db_tests::*, scanner_tests::*
<1 s	~50	<1 s	<1 min	pure logic unit tests

(Full suite hit WaitContainer(StartupTimeout) at test 80/400 on the first run with default fail-fast — re-running with --no-fail-fast produced the distribution above. The startup-timeout symptom is itself a consequence of stacking 200+ sequential container spawns and is addressed by Optimisation B below.)

The 5 slowest tests measured so far were all router::tests::commit_* at 17–19 s each — these spin a container, hit /api/commit, and run a real ZK proof.

Investigation: what really forces --test-threads=1?

The CI comment justifies the flag with two reasons (.github/workflows/ci.yaml:271):

--test-threads 1 is preserved — the repo invariant is that tests run serially to avoid testcontainers port races and shared-state pollution.

Both claims were checked against the source:

1. "testcontainers port races" — not a real risk

Every dynamic port allocation in the test code uses ephemeral binding:

• TcpListener::bind("127.0.0.1:0") in runtime_tests.rs:85, runtime_tests.rs:214, and 7× in scanner_ws_tests.rs (lines 56, 141, 210, 367, 445, 534, 667).
• testcontainers_modules::postgres::Postgres resolves the host port via container.get_host_port_ipv4(5432) (e.g. db_tests.rs:32), which returns the kernel-assigned dynamic mapping.

There is no hard-coded port anywhere in the test code paths. Two parallel tests cannot collide on a port.

2. "Shared-state pollution" — limited, and consistent with the current env

The only process-wide shared state in node/src/lib.rs is four lazy_static cells (lines 170+):

• NETWORK_CONFIG (built from ESPLORA_URL / ESPLORA_WS_URL / IS_MAINNET)
• USERNAME_DOMAIN
• PUBLISHER_KEY
• PUBLISHER_ADDRESS (derived from PUBLISHER_KEY)

These read env vars once at first access and stay fixed for the process lifetime. CI sets all four env vars at job-level (ci.yaml:176–195), identical for every test. Two tests running in parallel read the same cached values — no conflict.

What is not today protected by --test-threads=1:

• Filesystem state under /data/proofs/ — the ProofStore in router.rs:660 keeps a numeric next_id per dir. If multiple tests share the same dir, IDs collide. Each test today creates a fresh dir via tempfile::TempDir, so parallel-safe — but worth verifying with a quick grep before flipping the flag.
• tracing/println! log interleaving — cosmetic, not correctness.

What actually serialises today: the Docker daemon under load

Each Postgres test calls Postgres::default().with_tag("17").start().await. Sequential, the daemon happily handles one at a time. With --test-threads=N we'd be asking the daemon for N concurrent Postgres containers. Docker can do that, but:

• RAM pressure climbs (each Postgres container is ~50–100 MB resident).
• Daemon ops queue, container spawn wall stretches (~3 s → ~5–8 s under contention).
• Already today, 220 sequential spawns produced the WaitContainer(StartupTimeout) flake.

Conclusion: the lock isn't actually port-races or globals — it's the per-test container model. Removing --test-threads=1 without fixing that model just trades one bottleneck for another.

Optimisation B — Shared Postgres container + per-test schema isolation

What changes

Replace the per-test setup_pool() pattern in 6 files:

node/src/db_tests.rs           (53 testcontainers refs)
node/src/main_tests.rs         (6 refs)
node/src/r2_probe_tests.rs    (18 refs)
node/src/runtime_tests.rs      (7 refs)
node/src/state_tests.rs       (18 refs)
node/src/username_tests.rs    (16 refs)

With a shared model:

use tokio::sync::OnceCell;

// One container per test binary, started lazily.
static SHARED_PG: OnceCell<SharedPg> = OnceCell::const_new();

struct SharedPg {
    container: ContainerAsync<Postgres>,
    base_url: String,
}

pub struct SchemaScope {
    pub pool: PgPool,
    schema: String,
    pg: &'static SharedPg,
}

impl Drop for SchemaScope {
    fn drop(&mut self) {
        // Best-effort cleanup of the per-test schema. Pool is closed
        // implicitly by the surrounding tokio runtime.
        let schema = self.schema.clone();
        let url = self.pg.base_url.clone();
        // Drop on a detached task — runs at most a few ms, container
        // teardown only happens at test binary exit.
    }
}

pub async fn setup_pool() -> SchemaScope {
    let pg = SHARED_PG
        .get_or_init(|| async {
            let c = Postgres::default()
                .with_tag("17")
                .start()
                .await
                .expect("start postgres container");
            let host = c.get_host().await.unwrap();
            let port = c.get_host_port_ipv4(5432).await.unwrap();
            let base = format!("postgres://postgres:postgres@{host}:{port}/postgres");
            SharedPg { container: c, base_url: base }
        })
        .await;

    let schema = format!("t_{}", uuid::Uuid::new_v4().simple());
    let admin = PgPool::connect(&pg.base_url).await.unwrap();
    sqlx::query(&format!("CREATE SCHEMA {schema}")).execute(&admin).await.unwrap();

    // Build a per-test pool whose connections set search_path to the
    // isolated schema before any query runs.
    let pool = PgPoolOptions::new()
        .after_connect(move |conn, _meta| {
            let s = schema.clone();
            Box::pin(async move {
                sqlx::query(&format!("SET search_path TO {s}, public")).execute(conn).await?;
                Ok(())
            })
        })
        .connect(&pg.base_url)
        .await
        .unwrap();

    sqlx::migrate!("../migrations").run(&pool).await.unwrap();
    SchemaScope { pool, schema, pg }
}

Time cost replaced

Phase	Today (per test)	After B (per test)
Container spawn	~3 s	0 s (amortised)
Migrations on fresh DB	~500 ms – 1 s	~100–200 ms (empty schema)
Per-binary one-time container start	—	~3 s once

For 220 Postgres tests: ~660 s today → ~25 s after B, i.e. ~10 min wall saved per CI run, even at --test-threads=1.

Risks and follow-ups

• Migration SQL compatibility — every statement in node/migrations/00**.sql must work without an explicit schema prefix. Quick audit needed; if any migration uses public.foo, those need a one-line refactor.
• Tests that hold ContainerAsync<Postgres> for lifecycle reasons — a few tests (grep _container in the 6 files) keep the container handle to verify Drop behaviour. Those need adapter functions or to be reframed as "schema-lifecycle" tests.
• Drop order — SchemaScope::drop runs on the tokio runtime; if a test panics, the schema is leaked. Acceptable for CI (container teardown wipes everything), worth a one-line note in the doc.

Effort

1–2 days of one engineer:

• setup_pool() + SchemaScope implementation: ~4 h
• Migrate the 6 test files to the new helper: ~4 h
• Migration SQL audit + small fixes: ~2 h
• Tests that depended on ContainerAsync<Postgres> directly: ~2–4 h
• Local + CI validation (one Release-PR cycle): ~2–4 h

Expected wall-time after B alone (still --test-threads=1)

• M3 Ultra runner: 47 min → ~37 min
• Apple M5 Max workstation reference: 27 min → ~17 min

Optimisation A — Parallel test execution (--test-threads=N)

What changes

• .github/workflows/ci.yaml:281 — drop --test-threads 1 (or set 4–8).
• CONTRIBUTING.md lines 198, 199, 287, 333, 661 — update the documented command.
• Pre-flight code scan: confirm no /tmp/foo.bin-style hard-coded paths, no shared OnceCells holding test-mutable state, no set_var/remove_var calls in tests (those mutate process-wide env).

Why it works after B

With per-test schemas, the only remaining "shared" backend resource is the single Postgres container. Postgres handles N concurrent schemas with no contention beyond shared buffers.

• CPU-bound logic tests: roughly linear speedup with thread count.
• DB-touching tests: roughly linear speedup until Postgres's own concurrency wall.
• ZK-prove-touching tests: no speedup — each prove_* call already uses Rayon over all cores, two of them concurrently fight for the same pool.

Expected gain

For the measured mix (~50 % of tests are router/db/state in the 5–15 s range, ~10 % are pure prove tests >15 s, the rest <1 s):

• Pure-logic + DB tests gain ~3× at N=4
• Prove tests gain ~1.0× (core contention)
• Mixed weighted gain: ~2.5× on what remains after B

After A + B combined:

• M3 Ultra runner: 47 min → ~10–12 min
• Apple M5 Max workstation reference: 27 min → ~6–8 min

Risks

• Tracing log interleave — cosmetic, harmless.
• New flake surface — any unnoticed shared state shows up here. Mitigated by running A only after B is merged + stable for one Release cycle, and by an initial run with --retries 1 in CI.
• RUST_LOG=info interleaved across threads: harder to read failure tracebacks. Test failures already include the panicked thread's full output via nextest's "failures:" block, so this is also cosmetic.

Effort

2–4 h total once B has landed:

• 1 h grep for hidden shared state (set_var, static mut, fixed /tmp paths)
• 30 min config + doc change
• 1–2 h CI validation on a ci:full PR
• 1 h flake fixing if any surface

Recommendation

Land B first, then A. Two reasons:

1. B removes the Docker daemon bottleneck. A without B would 4–8× the daemon load — likely to make the WaitContainer(StartupTimeout) flake worse, not better.
2. B is mechanical and low-risk; A trades some flake risk for speed. Doing them separately gives a clean attribution if a regression appears.

Expected wall-time after each step

Step	M3 Ultra (CI runner)	Notes
Today	47 min	Measured 2026-06-02 (two Release PRs)
After B alone	~37 min	Removes ~10 min of container spawn
After A + B	~10–12 min	Limited from below by the ZK-prove tests on the slow tail

Validation plan

1. B PR: run the heavy job twice on the same branch with ci:full label, confirm both runs PASS and wall is <= 40 min.
2. A PR (after B is in main): run heavy job 3× consecutively, confirm zero flakes and wall is <= 15 min. Re-run with --retries 0 to confirm stability without retry-papering.
3. Open a separate follow-up issue if the prove-test tail becomes the new dominant cost — that's the boundary where Plonky3 / circuit-level work starts paying off more than infra tweaks.

References

• Test files: node/src/{db,main,r2_probe,runtime,state,username,router,scanner,scanner_ws,audit,publisher,account_node}_tests.rs
• CI: .github/workflows/ci.yaml lines 154–281
• Lazy globals: node/src/lib.rs:170 onwards
• Bench data: scripts/bench/results/m5-max-vs-m3-ultra-2026-06-02.md