Problem
Multiple fathomdb test paths create /tmp/fathomdb-* files without cleanup, leaking state between runs and eventually flaking unrelated tests via disk pressure on /tmp.
Leak sources observed during the 0.3.1 release cycle:
typescript/apps/sdk-harness/src/skip.ts:27 — tempDbPath() mints /tmp/fathomdb-harness-<scenario>-<timestamp>-<random>.db paths per scenario and never removes them. Every npm run test -w @fathomdb/sdk-harness leaks one .db plus WAL/SHM/lock sidecars per scenario.
python/examples/harness/app.py:84 — the harness app opens the given --db path as-is without recreating it, and many scenarios call HarnessContext.sibling_db() (models.py:31) to spin up sibling engines at derived paths like /tmp/fathomdb-harness-baseline-trace-and-excise.db. Preflight ran these twice against fixed paths without cleanup.
go/fathom-integrity/test/e2e/… — Go e2e tests leak /tmp/fathomdb-e2e-toolchain-* directories.
During 0.3.1 release prep, /tmp accumulated 380+ harness .db files plus dozens of fathomdb-e2e-toolchain-* directories (over 900 total filesystem entries). That disk pressure eventually caused a vitest afterEach cleanup hook in typescript/packages/fathomdb/test/search_builder.test.ts to time out at 10 seconds, on a test that normally completes in under 100ms. The failure was unrelated to the test itself — it was contention on /tmp.
Short-term fix (shipped in 0.3.1)
scripts/preflight-CI.sh now does a wholesale rm -rf /tmp/fathomdb-harness-* /tmp/fathomdb-e2e-toolchain-* at the top of the script, plus per-invocation globbed cleanup before each examples.harness.app run. This keeps preflight re-runs deterministic but does not address the underlying leak — a dev running cargo test or npm test individually still leaks files.
Proposed long-term fix
All fathomdb test tooling (Rust integration tests, Python harness, TypeScript sdk-harness, Go e2e) should mint their temp state under a dedicated per-session root directory like /tmp/fathomdb-<session-id>/ that is:
- Created fresh at the start of the test session
- Empty at creation time (no leftover state possible)
- Removed wholesale at the end of the session (trap, defer,
atexit, etc.)
- Never shared across sessions
Concretely:
- Introduce a small helper (one per language) that returns the session root and registers cleanup at exit. Example shapes:
- Rust:
test_session_root() -> PathBuf using tempfile::TempDir::new_in("/tmp") with a pinned prefix, held for the lifetime of the test binary.
- Python:
session_tmp_root() fixture using tempfile.TemporaryDirectory(prefix="fathomdb-").
- TypeScript (sdk-harness): replace
skip.ts:tempDbPath() with one that takes a session root and path.join(sessionRoot, scenarioName). Session root minted once per npm test invocation and removed via a process.on('exit', ...) handler.
- Go:
t.TempDir() in e2e tests (standard library), or a shared package-level helper.
- Migrate all call sites off the bare
/tmp/fathomdb-harness-… pattern.
- Delete the
rm -rf /tmp/fathomdb-harness-* in preflight-CI.sh once every test path goes through the session root.
Acceptance criteria
Not in scope
- Changing the public
--db CLI surface of examples.harness.app. The app should still accept a caller-provided path for the scripted preflight case; the fix is that the scenarios inside the harness should not leak their own sibling state outside the session root.
Engine.open() itself — engines legitimately write to caller-provided paths, that's by design.
Discovered
During the 0.3.1 release cycle. See commit da91fc2 for the short-term preflight cleanup and dev/notes/release-checklist.md "known pre-existing issues" appendix for the history.
Problem
Multiple fathomdb test paths create
/tmp/fathomdb-*files without cleanup, leaking state between runs and eventually flaking unrelated tests via disk pressure on/tmp.Leak sources observed during the 0.3.1 release cycle:
typescript/apps/sdk-harness/src/skip.ts:27—tempDbPath()mints/tmp/fathomdb-harness-<scenario>-<timestamp>-<random>.dbpaths per scenario and never removes them. Everynpm run test -w @fathomdb/sdk-harnessleaks one.dbplus WAL/SHM/lock sidecars per scenario.python/examples/harness/app.py:84— the harness app opens the given--dbpath as-is without recreating it, and many scenarios callHarnessContext.sibling_db()(models.py:31) to spin up sibling engines at derived paths like/tmp/fathomdb-harness-baseline-trace-and-excise.db. Preflight ran these twice against fixed paths without cleanup.go/fathom-integrity/test/e2e/…— Go e2e tests leak/tmp/fathomdb-e2e-toolchain-*directories.During 0.3.1 release prep,
/tmpaccumulated 380+ harness.dbfiles plus dozens offathomdb-e2e-toolchain-*directories (over 900 total filesystem entries). That disk pressure eventually caused a vitestafterEachcleanup hook intypescript/packages/fathomdb/test/search_builder.test.tsto time out at 10 seconds, on a test that normally completes in under 100ms. The failure was unrelated to the test itself — it was contention on/tmp.Short-term fix (shipped in 0.3.1)
scripts/preflight-CI.shnow does a wholesalerm -rf /tmp/fathomdb-harness-* /tmp/fathomdb-e2e-toolchain-*at the top of the script, plus per-invocation globbed cleanup before eachexamples.harness.apprun. This keeps preflight re-runs deterministic but does not address the underlying leak — a dev runningcargo testornpm testindividually still leaks files.Proposed long-term fix
All fathomdb test tooling (Rust integration tests, Python harness, TypeScript sdk-harness, Go e2e) should mint their temp state under a dedicated per-session root directory like
/tmp/fathomdb-<session-id>/that is:atexit, etc.)Concretely:
test_session_root() -> PathBufusingtempfile::TempDir::new_in("/tmp")with a pinned prefix, held for the lifetime of the test binary.session_tmp_root()fixture usingtempfile.TemporaryDirectory(prefix="fathomdb-").skip.ts:tempDbPath()with one that takes a session root andpath.join(sessionRoot, scenarioName). Session root minted once pernpm testinvocation and removed via aprocess.on('exit', ...)handler.t.TempDir()in e2e tests (standard library), or a shared package-level helper./tmp/fathomdb-harness-…pattern.rm -rf /tmp/fathomdb-harness-*inpreflight-CI.shonce every test path goes through the session root.Acceptance criteria
typescript/apps/sdk-harness/src/skip.ts:tempDbPath()is replaced with a session-rooted helper that cleans up on exit.python/examples/harnesseither uses a session-scoped temp dir for its main DB by default, or documents that callers must pass a fresh path (pytest wrappers already do the right thing via fixtures — the issue is the CLI entry point).go/fathom-integritye2e tests uset.TempDir()or equivalent rather than/tmp/fathomdb-e2e-toolchain-*.ls /tmp/fathomdb*returns nothing (or only files owned by other currently-running processes).rm -rflines inscripts/preflight-CI.shcan be removed without breaking preflight re-run determinism.Not in scope
--dbCLI surface ofexamples.harness.app. The app should still accept a caller-provided path for the scripted preflight case; the fix is that the scenarios inside the harness should not leak their own sibling state outside the session root.Engine.open()itself — engines legitimately write to caller-provided paths, that's by design.Discovered
During the 0.3.1 release cycle. See commit
da91fc2for the short-term preflight cleanup anddev/notes/release-checklist.md"known pre-existing issues" appendix for the history.