Summary
During a quarter-tile (0p5x0p5) e2e_verify run against prod on
2026-07-30, I observed two precheck containers running simultaneously on the
harken checker. The e2e flow cancels a precheck at step 9 and then resubmits
at step 10, so the most likely explanation is that the step-7 container that was
cancelled was still running alongside the step-12 (resubmitted) container —
i.e. cancelling a check in the UI may not stop/remove the underlying Docker
container on the checker VM.
This needs verification: does cancelling a check actually stop and remove the
container on the remote checker, or does it leave it running?
Why it matters
Each precheck container now reserves 32 GiB soft / 64 GiB hard and occupies
one of a checker's limited concurrent-check slots (prod = 5 per checker after
the recent 32 GiB-per-check resize). A container that survives cancellation:
- holds ~32–64 GiB of RAM until it exits on its own, and
- consumes a concurrency slot the dispatcher believes is free,
so a few leaked containers could push a checker into memory pressure
(systemd-oomd / kernel OOM) or block new dispatches. This is a bigger deal under
the new 32 GiB-per-check sizing than it was at 24 GiB.
How to reproduce / investigate
- Dispatch a precheck to a checker; confirm the container is running
(docker ps on the checker VM).
- Cancel it from the UI (or via the same path
e2e_verify step 9 uses).
- On the checker VM, check whether the container is stopped and removed, or
still Up.
Expected
On cancel, the platform should stop and remove the container on the checker so
its RAM and concurrency slot are released promptly.
Context
Observed while validating the 32 GiB-per-check memory work end-to-end. Not yet
confirmed as a leak vs. a legitimate brief overlap — hence "verify" — but worth
closing out given the container memory footprint.
Summary
During a quarter-tile (
0p5x0p5)e2e_verifyrun against prod on2026-07-30, I observed two precheck containers running simultaneously on the
harkenchecker. The e2e flow cancels a precheck at step 9 and then resubmitsat step 10, so the most likely explanation is that the step-7 container that was
cancelled was still running alongside the step-12 (resubmitted) container —
i.e. cancelling a check in the UI may not stop/remove the underlying Docker
container on the checker VM.
This needs verification: does cancelling a check actually stop and remove the
container on the remote checker, or does it leave it running?
Why it matters
Each precheck container now reserves 32 GiB soft / 64 GiB hard and occupies
one of a checker's limited concurrent-check slots (prod = 5 per checker after
the recent 32 GiB-per-check resize). A container that survives cancellation:
so a few leaked containers could push a checker into memory pressure
(systemd-oomd / kernel OOM) or block new dispatches. This is a bigger deal under
the new 32 GiB-per-check sizing than it was at 24 GiB.
How to reproduce / investigate
(
docker pson the checker VM).e2e_verifystep 9 uses).still
Up.Expected
On cancel, the platform should stop and remove the container on the checker so
its RAM and concurrency slot are released promptly.
Context
Observed while validating the 32 GiB-per-check memory work end-to-end. Not yet
confirmed as a leak vs. a legitimate brief overlap — hence "verify" — but worth
closing out given the container memory footprint.