Vacuum Pinning Lab

An interactive observability experiment about an MVCC cleanup failure that can hide behind a completely healthy vacuum scheduler.

An update creates an obsolete row version, but vacuum can reclaim that version only when no active snapshot still needs to see it. One long-running reporting transaction can therefore hold the visibility horizon in place while vacuum jobs keep starting and finishing. Dead tuples accumulate, the physical table grows, the hot working set escapes cache, and read latency rises.

The deterministic model and HTTP server are written in V, adding another compiled language to the telemetry.sh labs. The interactive client uses dependency-free HTML, CSS, and JavaScript.

The four responses

• Short transactions are the control. The cleanup horizon advances and ordinary vacuum capacity reclaims every dead version.
• Pinned snapshot holds one reporting transaction open for the selected lifetime. Vacuum runs, but those row versions remain ineligible.
• 3× vacuum workers adds cleanup capacity without moving the horizon. It produces the same peak bloat, then creates an I/O burst when the snapshot finally releases.
• Bounded snapshots enforces the selected transaction-age budget, terminates the stale report, and lets ordinary vacuum catch up.

Every response sees the same write churn. The experiment reports dead and pinned tuples, physical table size, buffer-cache hit rate, read p99, vacuum progress, vacuum I/O, oldest snapshot age, and terminated reports.

Run it

Requirements:

• V 0.5.1 or newer
• jq, curl, and Node.js for checks

make run

Open http://localhost:8080. Moving a control recomputes all four policies through the V API.

Run the native suite:

make check

Or run the production container:

docker build -t vacuum-pinning-lab .
docker run --rm -p 8080:8080 vacuum-pinning-lab

The final Alpine image runs as nobody:nobody; the V binary performs its own container health check.

Query the model

curl --silent \
  'http://localhost:8080/api/simulate?write_tps=650&update_percent=72&snapshot_minutes=70' |
  jq '.strategies[] | {
    label,
    final_dead_tuples,
    peak_pinned_tuples,
    max_table_gb,
    min_cache_hit_percent,
    max_read_p99_ms,
    reclaim_percent
  }'

For a machine-readable default run without starting the server:

./build/vacuum-pinning-lab --json | jq

What to look for

The default workload creates 28,080 dead row versions per minute. Ordinary vacuum can reclaim 39,000 versions per minute, so the control has real headroom. A 70-minute snapshot nevertheless pins 1.97 million versions—about 7.5 GB of extra heap.

1. The control ends with zero dead tuples. Cache hit remains at 98.7% and read p99 stays at 14 ms.
2. The pinned snapshot grows the table from 12 GB to 19.5 GB. Cache hit falls to about 65% and read p99 exceeds 47 ms even though all 91 modeled vacuum passes run.
3. Tripling workers does not change peak bloat or peak latency. It cannot cross the visibility boundary and creates a larger cleanup I/O burst after release.
4. The 15-minute age budget bounds the damage. One report is terminated, the horizon advances, and ordinary vacuum catches up by the end of the experiment.

Increase write rate or update percentage to raise version churn. Increase snapshot lifetime to retain more history. Adding vacuum workers helps only after the oldest snapshot releases; lowering transaction age attacks the actual constraint.

Why telemetry.sh

No single database graph proves vacuum pinning:

• vacuum run counts look healthy;
• worker CPU can be low because most versions are ineligible;
• dead-tuple and relation-size growth show retained mass, not ownership;
• the oldest transaction identifies the blocking boundary;
• application route, deployment, and session dimensions identify its owner;
• cache hit and request p99 connect engine retention to user impact.

telemetry.sh makes application transactions, database internals, storage, deployment, and request latency queryable in one investigation. Instead of adding cleanup workers to a permission problem, an operator can align the old snapshot with flat reclamation, heap growth, cache dilution, and read latency.

Project structure

src/vacuum_model/model.v deterministic MVCC and vacuum model
src/main.v               V HTTP server, API, and CLI
public/                  responsive interactive experiment
tests/model_test.v       model invariants
tests/http-test.sh       API, assets, methods, and security headers
tests/container-test.sh  production image and health checks
.github/workflows/ci.yml

Safety

The workload is synthetic. It does not connect to a database, kill real transactions, need credentials, or ingest telemetry.

License

MIT