Skip to content

Flaky test suite: auth.test.ts and pairs.test.ts fail intermittently (~1 in 5 runs) #151

Description

@Miracle656

Problem

The vitest suite fails intermittently. Same commit, same machine, no code change between runs:

run 1:  Tests  2 failed | 255 passed | 1 skipped (258)
          FAIL src/__tests__/auth.test.ts   > API key auth hook > returns 401 when no key is provided
          FAIL src/__tests__/pairs.test.ts  > POST /pairs > adds a new pair and returns 201 with pairKey
run 2:  Tests  257 passed | 1 skipped (258)
run 3:  Tests  257 passed | 1 skipped (258)
run 4:  Tests  257 passed | 1 skipped (258)

Roughly 1 run in 5. Observed on at least two different branches today, and an earlier occurrence on a different pair (auth.test.ts + networkVenueConfig.test.ts), so the set of victims varies.

Both files pass in isolation. Only the full run fails. That points at cross-file state rather than a bug in either test.

Why this is worth fixing rather than living with

It has already produced two false alarms today. In one, a full run showed 2 failures against a green main, which looks exactly like "this PR broke something" — and the natural next step is to send a contributor away to fix a bug that does not exist. Only re-running four times distinguished it.

A suite that is wrong 20% of the time trains everyone to re-run rather than investigate, which is how a real failure eventually gets waved through. Related in effect to #146: between them, a red result here currently carries almost no information.

Likely cause

Cross-file shared state. Candidates, in rough order:

  • process.env mutation. Several suites set env vars (networkVenueConfig.test.ts, soroswapEnabled.test.ts, the x402 tests) and vitest runs files in parallel within one process by default, so one file's beforeEach can land while another is asserting.
  • Module-level config caching. src/config.ts computes at import; whichever file imports first wins, and the order varies between runs.
  • Shared Redis / API-key registry state between auth.test.ts and pairs.test.ts — both exercise the API-key hook.

Suggested execution

  1. Reproduce deterministically first — run with a fixed seed / single thread and a repeated loop until it fails, so you can tell a fix from luck.
  2. Find the shared state. vitest --no-file-parallelism passing while the default fails is strong evidence and takes one command.
  3. Fix the isolation properly: snapshot and restore process.env per file, or make config resolution lazy rather than import-time. Prefer that over serialising the whole suite — fileParallelism: false hides the bug and slows CI.

Acceptance criteria

  • The root cause is identified and named in the PR — which state, shared how — not just "added a reset"
  • 20 consecutive full runs pass. Paste the evidence; this is the criterion that actually matters and it cannot be satisfied by inspection
  • The fix does not simply disable parallelism, unless the PR argues why isolation is impossible here
  • Any test mutating process.env restores it, and this is enforced (a shared setup file, not a convention people must remember)

Notes


Drips Wave · Complexity: Intermediate · 100 points


Required: Before submitting, join the contributor Telegram so your work can be tracked and counted toward the Stellar Wave: https://t.me/+fxHXq8f1SwlkZDBk

Metadata

Metadata

Assignees

No one assigned

    Labels

    Stellar WaveIssues in the Stellar wave programarea:testsTesting and coveragebugSomething isn't workingdifficulty:intermediate~half a dayhelp wantedExtra attention is neededpoints:100Easy tier — 100 Wave points

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions