fix(ci): cap mutation-test memory so a runaway mutant cannot kill the runner - #665
Merged
Merged
Conversation
… runner A mutant that negates a loop bound turns a bounded padding loop into one that allocates without bound. The test process exhausts the runner VM's memory within seconds, well before gremlins' per-mutant timeout, and the VM is torn down mid-run: gremlins takes the shutdown signal, exits 0 and writes no report, and the check reports the package as producing no usable result. The nightly full run dies the same way every night. Running the mutation step inside a memory-capped cgroup with swap denied turns that failure into the right one: the kernel OOM killer takes out the runaway test process, gremlins records the mutant as killed, and the run completes with a report. Reproduced and verified locally: mutating internal/cli/status uncapped OOMs the machine at the padding-loop negation; capped, the same run completes with that mutant killed.
This was referenced Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
The
mutation-test-prcheck has been failing on any PR whose changed packages carry heavy test suites, and the nightlymutation-test-pagesrun has failed every night for weeks. All of these deaths share one log signature: gremlins is mid-run when the runner logs "The runner has received a shutdown signal" and the job is killed.The mechanism, reproduced locally: a mutant can negate a loop bound and turn a bounded padding loop (for example the panel-height padding in
internal/cli/status/agentview.go) into one that appends forever. The test process allocates gigabytes within seconds, faster than gremlins' per-mutant timeout can fire, and exhausts the VM. The runner is torn down, gremlins takes the shutdown signal, exits 0, and writes no report, which the check correctly classifies as "no usable result".Fix
Run the mutation step inside a memory-capped cgroup with swap denied, in both workflows. When a mutant runs away, the kernel OOM killer now takes out the ballooning test process inside the cgroup; gremlins observes a failed test, records the mutant as killed (which is the semantically right verdict for a mutant that would crash the program), and the run completes with a report.
Verified locally by running gremlins v0.6.0 on
internal/cli/status:memory.max+memory.swap.max=0, OOM-continue semantics): the same run completes in 40 seconds with a valid report, and the runaway mutant is recorded as KILLED.The 12 GB cap is sized against the observed baseline of a normal run (2 to 6 GB aggregate across gremlins' four workers) while leaving the 16 GB runner room for the runner agent. If a legitimate suite ever exceeds the cap, the cost is a falsely killed mutant in an advisory score, not a dead runner.
Complementary to #653, which confines each gremlins invocation to the invoked package: that shrinks runs, this makes a runaway mutant in any run survivable.