feat: report OOM kills from the scenario's memory cgroup

Cargo.toml

@@ -78,7 +78,7 @@ version = "2.0.104"
 features = ["full", "extra-traits", "visit"]
 
 [target.'cfg(unix)'.dependencies]
-nix = { version = "0.31", features = ["process", "signal"] }
+nix = { version = "0.31", features = ["process", "resource", "signal"] }
 
 # reflink is disabled on musl for the time being due to
 # <https://github.com/nicokoch/reflink/issues/11> and

NEWS.md

@@ -2,6 +2,12 @@
 
 ## Unreleased
 
+- New: Outcome lines, scenario logs, and `outcomes.json` now say when a phase's process was killed by a signal, when the kernel OOM-killed something in the scenario's memory cgroup, and when processes left running by the tests had to be reaped, so that (for example) an OOM-caught mutant can be told apart from one caught by a failing assertion. The caught / missed / unviable / timeout classification is unchanged.
+
+- New: `--max-memory SIZE` (and the `max_memory` config key) bounds how much memory each scenario's cargo process tree may use, so that a mutant that turns a loop into an unbounded allocator is stopped by the kernel rather than taking the machine down with it. On Linux this uses a cgroup v2 `memory.max` where a writable cgroup is available, and otherwise `setrlimit(RLIMIT_AS)`; the mechanism in use is logged. macOS does not enforce `RLIMIT_AS`, so the option is a no-op there. If the option is given and neither mechanism can be applied, cargo-mutants fails before testing any mutant rather than running with no limit. Scenarios stopped by the limit are counted in the run summary.
+
+- Fixed: Each cargo invocation now runs as the leader of its own process group, and that group is swept after every phase, not only after a timeout. Processes left running by a scenario's tests are `SIGTERM`ed, given a short grace period, and then `SIGKILL`ed, so they can't keep consuming memory or CPU while later mutants are tested. Unix only.
+
 - New: `#[mutants::exclude_re("pattern")]` attribute to exclude specific mutations by regex, without disabling all mutations on the function. The attribute can be placed on functions, `impl` blocks, `trait` blocks, modules, files, and on expressions that can carry an attribute (such as `match`, struct literals, call expressions, method calls, and unary expressions). Multiple patterns can be applied. Also supported within `cfg_attr`. Requires the [mutants](https://crates.io/crates/mutants) crate version `0.0.5` or later.
 
 - Fixed: `#[mutants::skip]` (and `#[cfg_attr(..., mutants::skip)]`) is now honoured when placed on `const` and `static` items, including associated constants in `impl` and `trait` blocks. Previously the attribute was silently ignored on these items and operator mutants inside the initializer expression were still generated ([#508](https://github.com/sourcefrog/cargo-mutants/issues/508)).

book/src/timeouts.md

@@ -45,6 +45,122 @@ In this case you can use the `--build-timeout` or `--build-timeout-multiplier` o
 
 You might also choose to skip mutants that can cause long-running const evaluation.
 
+## Memory limits
+
+A timeout is not always enough. A mutant can turn a bounded loop into an unbounded
+allocator — flipping `+=` to `-=` on a parser's cursor, say — and a test that grows at
+hundreds of megabytes per second can exhaust the machine long before the test timeout
+arrives. On a CI runner the usual result is that the whole VM is torn down, with no log
+and no record of which mutants had been tested.
+
+`--max-memory SIZE`, or the `max_memory` key in the configuration file, puts a ceiling on
+each scenario's cargo process tree instead, so that the kernel stops the scenario rather
+than the machine. Sizes may be plain byte counts, or carry a `K`, `M`, `G`, or `T`
+suffix, which are binary multiples: `1K` is 1024 bytes. The smallest accepted value is
+1M: unlike `--timeout=0`, `--max-memory=0` is not a way to turn the limit off, so it is
+rejected rather than silently stopping every scenario.
+
+```shell
+cargo mutants --max-memory 8G
+```
+
+```toml
+# .cargo/mutants.toml
+max_memory = "8G"
+```
+
+The limit is off by default, and applies to every phase of every scenario, builds
+included, so leave room for the compiler as well as for the tests.
+
+Two mechanisms can enforce it, and they are not equivalent:
+
+* **cgroup v2** `memory.max`, on a cgroup created for each scenario. This limits
+  *resident* memory for the whole process tree, and the kernel reports what it did through
+  `memory.events`, so an OOM-killed mutant can be told apart from one caught by a failing
+  assertion. This is preferred whenever a writable cgroup is available. Swap is also
+  capped, where the kernel accounts for it; on kernels that do not, the limit covers
+  resident memory only.
+
+* **`setrlimit(RLIMIT_AS)`** on the cargo process, inherited by everything it spawns.
+  This limits *address space*, which is a much cruder proxy: allocators and rustc reserve
+  far more address space than they ever make resident, so a limit that is comfortable as
+  a resident-memory ceiling can fail builds outright when applied this way. If
+  cargo-mutants falls back to this mechanism, set the limit generously.
+
+Which one is in use is reported at startup, for example:
+
+```
+INFO Limiting each scenario to 8589934592 bytes of memory using cgroup v2 memory.max
+```
+
+For the cgroup mechanism, cargo-mutants needs somewhere it may create child cgroups with
+`memory.max`. It looks at its own cgroup first, writing `+memory` to that cgroup's
+`cgroup.subtree_control` if it isn't set already. Failing that it looks at the parent,
+which works when something has already put a `memory.max` fence around cargo-mutants — a
+CI shard running under a memory-limited systemd scope or container, for instance. As a
+last resort it moves itself into a `cargo-mutants-supervisor` cgroup of its own so that
+its original cgroup can delegate the memory controller.
+
+> **The parent fallback escapes an enclosing limit.** Scenario cgroups made under the
+> parent are *siblings* of cargo-mutants' own cgroup, so a `memory.max` set on
+> cargo-mutants does not contain them: with `--jobs N` the run can use up to N ×
+> `--max-memory` in total, whatever that outer fence says. cargo-mutants warns when it
+> takes this path.
+
+On macOS, `RLIMIT_AS` is accepted by the kernel and then ignored, and cgroups do not
+exist, so `--max-memory` has no effect there; cargo-mutants warns and carries on. On any
+platform where *neither* mechanism can be applied, giving `--max-memory` is an error,
+reported before any mutant is tested, rather than a run that quietly had no limit.
+
+This option does not change how mutants are classified. A mutant whose tests are
+OOM-killed fails its tests and so is caught, in just the same way as one that panics. So
+that those are not invisible, the run summary counts them separately:
+
+```
+40 mutants tested in 3m 2s: 40 caught (3 stopped by the --max-memory limit)
+```
+
+## Leftover processes
+
+A test can leave processes running after it returns: a daemon it started, a helper it
+forgot to wait for, or a test binary that was not reaped. Those processes keep running
+— and keep allocating — while cargo-mutants moves on to the next mutant, so they can
+exhaust the machine's memory in a window where no cargo phase is running at all.
+
+To prevent this, cargo-mutants starts each cargo invocation as the leader of its own
+process group, and sweeps that group after *every* phase, not only after a timeout.
+Once the cargo process itself exits, anything left in the group is sent `SIGTERM`, given
+a short grace period, and then `SIGKILL`ed. What was reaped is recorded in the
+scenario's log, and the pids are shown at `--level=debug`.
+
+The same escalation applies to the cargo process itself on a timeout: it is sent
+`SIGTERM`, and `SIGKILL`ed if it has not exited by the end of the grace period, so a
+child that ignores `SIGTERM` cannot stall the run.
+
+This has no effect on how a mutant is classified; it only stops work from one scenario
+leaking into the next. Windows has no process groups, and cargo-mutants does not yet use
+job objects, so this sweep is Unix-only.
+
+## Why a scenario died
+
+A mutant caught because the kernel killed its tests looks, in the summary counts, exactly
+like a mutant caught by a failing assertion. When there is more to say, cargo-mutants says
+it in parentheses on the outcome line:
+
+```
+caught   src/parse.rs:41:9: replace += with -= in Cursor::advance (test OOM-killed by the kernel (1 process) for exceeding the memory limit) in 3s build + 1s test
+caught   src/server.rs:88:5: replace listen -> bool with false (test killed by SIGABRT; test left 1 stray process behind (SIGKILLed: 30411)) in 2s build + 9s test
+```
+
+Three things get reported this way: the signal that killed a phase's cargo process, if it
+died by one; the kernel's `oom_kill` count from the scenario's cgroup, when the cgroup
+memory limit is in use; and anything the process group sweep had to clean up. The same
+information is written to the scenario's log and, in `mutants.out/outcomes.json`, to a
+`report` field on each phase result.
+
+None of this changes the caught / missed / unviable / timeout classification. It only
+makes the reason visible.
+
 ## Exceptions
 
 The multiplier timeout options cannot be used when the baseline is skipped

examples/custom_config.toml

@@ -30,6 +30,11 @@ timeout_multiplier = 2.0
 # Minimum test timeout in seconds
 minimum_test_timeout = 60.0
 
+# Maximum memory for each scenario, so that a mutant that allocates without bound is
+# stopped by the kernel rather than by the machine running out. Suffixes are binary
+# multiples: "1K" is 1024 bytes.
+max_memory = "8G"
+
 # Copy VCS directories (.git, etc.) to build directories
 copy_vcs = true
 

src/cargo.rs

@@ -20,6 +20,7 @@ use crate::options::{Options, TestTool};
 use crate::outcome::{Phase, PhaseResult};
 use crate::output::ScenarioOutput;
 use crate::package::PackageSelection;
+use crate::process::memory::MemoryLimit;
 use crate::process::{Exit, Process};
 
 // Allowed nextest codes (those will be considered a mutation caught / ignored without a warning)
@@ -37,6 +38,7 @@ pub fn run_cargo(
     packages: &PackageSelection,
     phase: Phase,
     timeout: Option<Duration>,
+    memory_limit: Option<&MemoryLimit>,
     scenario_output: &mut ScenarioOutput,
     options: &Options,
     console: &Console,
@@ -55,12 +57,13 @@ pub fn run_cargo(
         debug!(?encoded_rustflags);
         env.push(("CARGO_ENCODED_RUSTFLAGS".to_owned(), encoded_rustflags));
     }
-    let process_status = Process::run(
+    let (process_status, report) = Process::run(
         &argv,
         &env,
         build_dir.path(),
         timeout,
         jobserver,
+        memory_limit,
         scenario_output,
         console,
     )?;
@@ -79,6 +82,7 @@ pub fn run_cargo(
         duration: start.elapsed(),
         process_status,
         argv,
+        report,
     })
 }
 

src/config.rs

@@ -64,6 +64,8 @@ pub struct Config {
 
     /// Space or comma separated list of features to activate.
     pub features: Vec<String>,
+    /// Maximum memory for each scenario, e.g. "4G"; suffixes are binary multiples.
+    pub max_memory: Option<String>,
     /// Minimum test timeout, in seconds, as a floor on the autoset value.
     pub minimum_test_timeout: Option<f64>,
     /// Do not activate the `default` feature.

src/console.rs

@@ -90,6 +90,10 @@ impl Console {
             style_outcome(outcome),
             style_scenario(scenario, true),
         );
+        let death_reasons = outcome.death_reasons();
+        if !death_reasons.is_empty() {
+            write!(s, " ({})", death_reasons.join("; ")).expect("format death reasons");
+        }
         if options.show_times {
             let prs: Vec<String> = outcome
                 .phase_results()

src/lab.rs

@@ -18,8 +18,8 @@ use tracing::{debug, debug_span, error, trace, warn};
 use crate::{
     BaselineStrategy, BuildDir, Console, Context, Mutant, Options, Phase, Result, Scenario,
     ScenarioOutcome, cargo::run_cargo, options::TestPackages, outcome::LabOutcome,
-    output::OutputDir, package::Package, package::PackageSelection, timeouts::Timeouts,
-    workspace::Workspace,
+    output::OutputDir, package::Package, package::PackageSelection, process::memory::MemoryLimit,
+    timeouts::Timeouts, workspace::Workspace,
 };
 
 /// Run all possible mutation experiments.
@@ -38,6 +38,9 @@ pub fn test_mutants(
 ) -> Result<LabOutcome> {
     let start_time = Instant::now();
     console.set_debug_log(output_dir.open_debug_log()?);
+    // Fail here, before the tree is copied, rather than after a long run that silently
+    // had no limit.
+    let memory_limit = options.max_memory.map(MemoryLimit::new).transpose()?;
     if options.shuffle {
         fastrand::shuffle(&mut mutants);
     }
@@ -62,6 +65,7 @@ pub fn test_mutants(
     let lab = Lab {
         output_mutex,
         jobserver,
+        memory_limit,
         tests_for_mutant,
         options,
         console,
@@ -164,6 +168,7 @@ fn join_threads(threads: Vec<thread::ScopedJoinHandle<'_, Result<()>>>) -> Resul
 struct Lab<'a> {
     output_mutex: Mutex<OutputDir>,
     jobserver: Option<jobserver::Client>,
+    memory_limit: Option<MemoryLimit>,
     tests_for_mutant: TestsForMutant,
     options: &'a Options,
     console: &'a Console,
@@ -207,6 +212,7 @@ impl Lab<'_> {
             build_dir,
             output_mutex: &self.output_mutex,
             jobserver: self.jobserver.as_ref(),
+            memory_limit: self.memory_limit.as_ref(),
             tests_for_mutant: &self.tests_for_mutant,
             options: self.options,
             console: self.console,
@@ -222,6 +228,7 @@ struct Worker<'a> {
     build_dir: &'a BuildDir,
     output_mutex: &'a Mutex<OutputDir>,
     jobserver: Option<&'a jobserver::Client>,
+    memory_limit: Option<&'a MemoryLimit>,
     tests_for_mutant: &'a TestsForMutant,
     options: &'a Options,
     console: &'a Console,
@@ -289,6 +296,7 @@ impl Worker<'_> {
                 test_packages,
                 phase,
                 timeout,
+                self.memory_limit,
                 &mut scenario_output,
                 self.options,
                 self.console,

src/main.rs

@@ -291,6 +291,13 @@ pub struct Args {
     #[arg(long, help_heading = "Execution")]
     list: bool,
 
+    /// Maximum memory for each scenario, e.g. 4G: a mutant that exceeds it is stopped
+    ///
+    /// Sizes may be given in bytes, or with a `K`, `M`, `G`, or `T` suffix, which are
+    /// binary multiples: `1K` is 1024 bytes. Enforced on Linux only; see the manual.
+    #[arg(long, help_heading = "Execution", value_name = "SIZE")]
+    max_memory: Option<String>,
+
     /// List source files, don't run anything.
     #[arg(long, help_heading = "Execution")]
     list_files: bool,