Skip to content

Move all test tooling to a dedicated per-session /tmp/fathomdb-<unique>/ root #40

Description

@coreyt

Problem

Multiple fathomdb test paths create /tmp/fathomdb-* files without cleanup, leaking state between runs and eventually flaking unrelated tests via disk pressure on /tmp.

Leak sources observed during the 0.3.1 release cycle:

  1. typescript/apps/sdk-harness/src/skip.ts:27tempDbPath() mints /tmp/fathomdb-harness-<scenario>-<timestamp>-<random>.db paths per scenario and never removes them. Every npm run test -w @fathomdb/sdk-harness leaks one .db plus WAL/SHM/lock sidecars per scenario.
  2. python/examples/harness/app.py:84 — the harness app opens the given --db path as-is without recreating it, and many scenarios call HarnessContext.sibling_db() (models.py:31) to spin up sibling engines at derived paths like /tmp/fathomdb-harness-baseline-trace-and-excise.db. Preflight ran these twice against fixed paths without cleanup.
  3. go/fathom-integrity/test/e2e/… — Go e2e tests leak /tmp/fathomdb-e2e-toolchain-* directories.

During 0.3.1 release prep, /tmp accumulated 380+ harness .db files plus dozens of fathomdb-e2e-toolchain-* directories (over 900 total filesystem entries). That disk pressure eventually caused a vitest afterEach cleanup hook in typescript/packages/fathomdb/test/search_builder.test.ts to time out at 10 seconds, on a test that normally completes in under 100ms. The failure was unrelated to the test itself — it was contention on /tmp.

Short-term fix (shipped in 0.3.1)

scripts/preflight-CI.sh now does a wholesale rm -rf /tmp/fathomdb-harness-* /tmp/fathomdb-e2e-toolchain-* at the top of the script, plus per-invocation globbed cleanup before each examples.harness.app run. This keeps preflight re-runs deterministic but does not address the underlying leak — a dev running cargo test or npm test individually still leaks files.

Proposed long-term fix

All fathomdb test tooling (Rust integration tests, Python harness, TypeScript sdk-harness, Go e2e) should mint their temp state under a dedicated per-session root directory like /tmp/fathomdb-<session-id>/ that is:

  • Created fresh at the start of the test session
  • Empty at creation time (no leftover state possible)
  • Removed wholesale at the end of the session (trap, defer, atexit, etc.)
  • Never shared across sessions

Concretely:

  • Introduce a small helper (one per language) that returns the session root and registers cleanup at exit. Example shapes:
    • Rust: test_session_root() -> PathBuf using tempfile::TempDir::new_in("/tmp") with a pinned prefix, held for the lifetime of the test binary.
    • Python: session_tmp_root() fixture using tempfile.TemporaryDirectory(prefix="fathomdb-").
    • TypeScript (sdk-harness): replace skip.ts:tempDbPath() with one that takes a session root and path.join(sessionRoot, scenarioName). Session root minted once per npm test invocation and removed via a process.on('exit', ...) handler.
    • Go: t.TempDir() in e2e tests (standard library), or a shared package-level helper.
  • Migrate all call sites off the bare /tmp/fathomdb-harness-… pattern.
  • Delete the rm -rf /tmp/fathomdb-harness-* in preflight-CI.sh once every test path goes through the session root.

Acceptance criteria

  • typescript/apps/sdk-harness/src/skip.ts:tempDbPath() is replaced with a session-rooted helper that cleans up on exit.
  • python/examples/harness either uses a session-scoped temp dir for its main DB by default, or documents that callers must pass a fresh path (pytest wrappers already do the right thing via fixtures — the issue is the CLI entry point).
  • go/fathom-integrity e2e tests use t.TempDir() or equivalent rather than /tmp/fathomdb-e2e-toolchain-*.
  • After running the full test matrix, ls /tmp/fathomdb* returns nothing (or only files owned by other currently-running processes).
  • The wholesale rm -rf lines in scripts/preflight-CI.sh can be removed without breaking preflight re-run determinism.

Not in scope

  • Changing the public --db CLI surface of examples.harness.app. The app should still accept a caller-provided path for the scripted preflight case; the fix is that the scenarios inside the harness should not leak their own sibling state outside the session root.
  • Engine.open() itself — engines legitimately write to caller-provided paths, that's by design.

Discovered

During the 0.3.1 release cycle. See commit da91fc2 for the short-term preflight cleanup and dev/notes/release-checklist.md "known pre-existing issues" appendix for the history.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions