feat: prefer cgroup v2 memory.max for --max-memory

Cargo.toml

@@ -78,7 +78,7 @@ version = "2.0.104"
 features = ["full", "extra-traits", "visit"]
 
 [target.'cfg(unix)'.dependencies]
-nix = { version = "0.31", features = ["process", "signal"] }
+nix = { version = "0.31", features = ["process", "resource", "signal"] }
 
 # reflink is disabled on musl for the time being due to
 # <https://github.com/nicokoch/reflink/issues/11> and

NEWS.md

@@ -2,6 +2,8 @@
 
 ## Unreleased
 
+- New: `--max-memory SIZE` (and the `max_memory` config key) bounds how much memory each scenario's cargo process tree may use, so that a mutant that turns a loop into an unbounded allocator is stopped by the kernel rather than taking the machine down with it. On Linux it uses a cgroup v2 `memory.max` where a writable cgroup is available, and otherwise `setrlimit(RLIMIT_AS)`; the mechanism in use is logged. macOS does not enforce `RLIMIT_AS`, so the option is a no-op there. If the option is given and it cannot be applied, cargo-mutants fails before testing any mutant rather than running with no limit.
+
 - New: `#[mutants::exclude_re("pattern")]` attribute to exclude specific mutations by regex, without disabling all mutations on the function. The attribute can be placed on functions, `impl` blocks, `trait` blocks, modules, files, and on expressions that can carry an attribute (such as `match`, struct literals, call expressions, method calls, and unary expressions). Multiple patterns can be applied. Also supported within `cfg_attr`. Requires the [mutants](https://crates.io/crates/mutants) crate version `0.0.5` or later.
 
 - Fixed: `#[mutants::skip]` (and `#[cfg_attr(..., mutants::skip)]`) is now honoured when placed on `const` and `static` items, including associated constants in `impl` and `trait` blocks. Previously the attribute was silently ignored on these items and operator mutants inside the initializer expression were still generated ([#508](https://github.com/sourcefrog/cargo-mutants/issues/508)).

book/src/timeouts.md

@@ -45,6 +45,74 @@ In this case you can use the `--build-timeout` or `--build-timeout-multiplier` o
 
 You might also choose to skip mutants that can cause long-running const evaluation.
 
+## Memory limits
+
+A timeout is not always enough. A mutant can turn a bounded loop into an unbounded
+allocator — flipping `+=` to `-=` on a parser's cursor, say — and a test that grows at
+hundreds of megabytes per second can exhaust the machine long before the test timeout
+arrives. On a CI runner the usual result is that the whole VM is torn down, with no log
+and no record of which mutants had been tested.
+
+`--max-memory SIZE`, or the `max_memory` key in the configuration file, puts a ceiling on
+each scenario's cargo process tree instead, so that the kernel stops the scenario rather
+than the machine. Sizes may be plain byte counts, or carry a `K`, `M`, `G`, or `T`
+suffix, which are binary multiples: `1K` is 1024 bytes. The smallest accepted value is
+1M: unlike `--timeout=0`, `--max-memory=0` is not a way to turn the limit off, so it is
+rejected rather than silently stopping every scenario.
+
+```shell
+cargo mutants --max-memory 8G
+```
+
+```toml
+# .cargo/mutants.toml
+max_memory = "8G"
+```
+
+The limit is off by default, and applies to every phase of every scenario, builds
+included, so leave room for the compiler as well as for the tests.
+
+Two mechanisms can enforce it, and they are not equivalent:
+
+* **cgroup v2** `memory.max`, on a cgroup created for each scenario. This limits
+  *resident* memory for the whole process tree, which is what you actually care about.
+  It is preferred whenever a writable cgroup is available. Swap is also capped, where the
+  kernel accounts for it; on kernels that do not, the limit covers resident memory only.
+
+* **`setrlimit(RLIMIT_AS)`** on the cargo process, inherited by everything it spawns.
+  This limits *address space*, a much cruder proxy: allocators and rustc reserve far more
+  address space than they ever make resident, so a limit that is comfortable as a
+  resident-memory ceiling can fail builds outright when applied this way. If cargo-mutants
+  falls back to this mechanism, set the limit generously.
+
+Which one is in use is reported at startup, for example:
+
+```
+INFO Limiting each scenario to 8589934592 bytes of memory using cgroup v2 memory.max
+```
+
+For the cgroup mechanism, cargo-mutants needs somewhere it may create child cgroups with
+`memory.max`. It looks at its own cgroup first, writing `+memory` to that cgroup's
+`cgroup.subtree_control` if it isn't set already. Failing that it looks at the parent,
+which works when something has already put a `memory.max` fence around cargo-mutants — a
+CI shard running under a memory-limited systemd scope or container, for instance. As a
+last resort it moves itself into a `cargo-mutants-supervisor` cgroup of its own so that
+its original cgroup can delegate the memory controller.
+
+> **The parent fallback escapes an enclosing limit.** Scenario cgroups made under the
+> parent are *siblings* of cargo-mutants' own cgroup, so a `memory.max` set on
+> cargo-mutants does not contain them: with `--jobs N` the run can use up to N ×
+> `--max-memory` in total, whatever that outer fence says. cargo-mutants warns when it
+> takes this path.
+
+On macOS, `RLIMIT_AS` is accepted by the kernel and then ignored, and cgroups do not
+exist, so `--max-memory` has no effect there; cargo-mutants warns and carries on. On any
+platform where *neither* mechanism can be applied, giving `--max-memory` is an error,
+reported before any mutant is tested, rather than a run that quietly had no limit.
+
+This option does not change how mutants are classified. A mutant whose tests are stopped
+by the limit fails its tests and so is caught, in just the same way as one that panics.
+
 ## Exceptions
 
 The multiplier timeout options cannot be used when the baseline is skipped

examples/custom_config.toml

@@ -30,6 +30,11 @@ timeout_multiplier = 2.0
 # Minimum test timeout in seconds
 minimum_test_timeout = 60.0
 
+# Maximum memory for each scenario, so that a mutant that allocates without bound is
+# stopped by the kernel rather than by the machine running out. Suffixes are binary
+# multiples: "1K" is 1024 bytes.
+max_memory = "8G"
+
 # Copy VCS directories (.git, etc.) to build directories
 copy_vcs = true
 

src/cargo.rs

@@ -20,6 +20,7 @@ use crate::options::{Options, TestTool};
 use crate::outcome::{Phase, PhaseResult};
 use crate::output::ScenarioOutput;
 use crate::package::PackageSelection;
+use crate::process::memory::MemoryLimit;
 use crate::process::{Exit, Process};
 
 // Allowed nextest codes (those will be considered a mutation caught / ignored without a warning)
@@ -37,6 +38,7 @@ pub fn run_cargo(
     packages: &PackageSelection,
     phase: Phase,
     timeout: Option<Duration>,
+    memory_limit: Option<&MemoryLimit>,
     scenario_output: &mut ScenarioOutput,
     options: &Options,
     console: &Console,
@@ -61,6 +63,7 @@ pub fn run_cargo(
         build_dir.path(),
         timeout,
         jobserver,
+        memory_limit,
         scenario_output,
         console,
     )?;

src/config.rs

@@ -64,6 +64,8 @@ pub struct Config {
 
     /// Space or comma separated list of features to activate.
     pub features: Vec<String>,
+    /// Maximum memory for each scenario, e.g. "4G"; suffixes are binary multiples.
+    pub max_memory: Option<String>,
     /// Minimum test timeout, in seconds, as a floor on the autoset value.
     pub minimum_test_timeout: Option<f64>,
     /// Do not activate the `default` feature.

src/lab.rs

@@ -18,8 +18,8 @@ use tracing::{debug, debug_span, error, trace, warn};
 use crate::{
     BaselineStrategy, BuildDir, Console, Context, Mutant, Options, Phase, Result, Scenario,
     ScenarioOutcome, cargo::run_cargo, options::TestPackages, outcome::LabOutcome,
-    output::OutputDir, package::Package, package::PackageSelection, timeouts::Timeouts,
-    workspace::Workspace,
+    output::OutputDir, package::Package, package::PackageSelection, process::memory::MemoryLimit,
+    timeouts::Timeouts, workspace::Workspace,
 };
 
 /// Run all possible mutation experiments.
@@ -38,6 +38,9 @@ pub fn test_mutants(
 ) -> Result<LabOutcome> {
     let start_time = Instant::now();
     console.set_debug_log(output_dir.open_debug_log()?);
+    // Fail here, before the tree is copied, rather than after a long run that silently
+    // had no limit.
+    let memory_limit = options.max_memory.map(MemoryLimit::new).transpose()?;
     if options.shuffle {
         fastrand::shuffle(&mut mutants);
     }
@@ -62,6 +65,7 @@ pub fn test_mutants(
     let lab = Lab {
         output_mutex,
         jobserver,
+        memory_limit,
         tests_for_mutant,
         options,
         console,
@@ -164,6 +168,7 @@ fn join_threads(threads: Vec<thread::ScopedJoinHandle<'_, Result<()>>>) -> Resul
 struct Lab<'a> {
     output_mutex: Mutex<OutputDir>,
     jobserver: Option<jobserver::Client>,
+    memory_limit: Option<MemoryLimit>,
     tests_for_mutant: TestsForMutant,
     options: &'a Options,
     console: &'a Console,
@@ -207,6 +212,7 @@ impl Lab<'_> {
             build_dir,
             output_mutex: &self.output_mutex,
             jobserver: self.jobserver.as_ref(),
+            memory_limit: self.memory_limit.as_ref(),
             tests_for_mutant: &self.tests_for_mutant,
             options: self.options,
             console: self.console,
@@ -222,6 +228,7 @@ struct Worker<'a> {
     build_dir: &'a BuildDir,
     output_mutex: &'a Mutex<OutputDir>,
     jobserver: Option<&'a jobserver::Client>,
+    memory_limit: Option<&'a MemoryLimit>,
     tests_for_mutant: &'a TestsForMutant,
     options: &'a Options,
     console: &'a Console,
@@ -289,6 +296,7 @@ impl Worker<'_> {
                 test_packages,
                 phase,
                 timeout,
+                self.memory_limit,
                 &mut scenario_output,
                 self.options,
                 self.console,

src/main.rs

@@ -291,6 +291,13 @@ pub struct Args {
     #[arg(long, help_heading = "Execution")]
     list: bool,
 
+    /// Maximum memory for each scenario, e.g. 4G: a mutant that exceeds it is stopped
+    ///
+    /// Sizes may be given in bytes, or with a `K`, `M`, `G`, or `T` suffix, which are
+    /// binary multiples: `1K` is 1024 bytes. Enforced on Linux only; see the manual.
+    #[arg(long, help_heading = "Execution", value_name = "SIZE")]
+    max_memory: Option<String>,
+
     /// List source files, don't run anything.
     #[arg(long, help_heading = "Execution")]
     list_files: bool,

src/options.rs

@@ -13,7 +13,7 @@ use std::env;
 use std::ffi::OsString;
 use std::time::Duration;
 
-use anyhow::Context;
+use anyhow::{Context, bail};
 use camino::{Utf8Path, Utf8PathBuf};
 use clap::ArgAction;
 use globset::GlobSet;
@@ -96,6 +96,9 @@ pub struct Options {
     /// The minimum test timeout, as a floor on the autoset value.
     pub minimum_test_timeout: Duration,
 
+    /// The maximum memory for each scenario's process tree, in bytes, if set.
+    pub max_memory: Option<u64>,
+
     pub print_caught: bool,
     pub print_unviable: bool,
 
@@ -237,6 +240,47 @@ fn join_slices(a: &[String], b: &[String]) -> Vec<String> {
     a.iter().chain(b).cloned().collect()
 }
 
+/// The smallest `--max-memory` worth accepting.
+///
+/// Anything near zero stops every scenario before it can do anything, which is never
+/// what someone means. In particular `--max-memory=0` is not "no limit", unlike `-t 0`.
+const MIN_MAX_MEMORY: u64 = 1 << 20;
+
+/// Parse a memory size like `256M`, `2GiB`, or a plain count of bytes.
+///
+/// Suffixes are binary multiples, as they conventionally are for memory: `1K` is 1024
+/// bytes, not 1000.
+fn parse_size(s: &str) -> Result<u64> {
+    let s = s.trim();
+    let digits_end = s.find(|c: char| !c.is_ascii_digit()).unwrap_or(s.len());
+    let (digits, suffix) = s.split_at(digits_end);
+    let number: u64 = digits
+        .parse()
+        .with_context(|| format!("{s:?} does not start with a number of bytes"))?;
+    let multiple: u64 = match suffix.trim().to_ascii_lowercase().as_str() {
+        "" | "b" => 1,
+        "k" | "kb" | "kib" => 1 << 10,
+        "m" | "mb" | "mib" => 1 << 20,
+        "g" | "gb" | "gib" => 1 << 30,
+        "t" | "tb" | "tib" => 1 << 40,
+        _ => bail!("unrecognized size suffix {suffix:?} in {s:?}"),
+    };
+    number
+        .checked_mul(multiple)
+        .with_context(|| format!("size {s:?} is too large"))
+}
+
+/// Parse and sanity-check a `--max-memory` value.
+fn parse_max_memory(s: &str) -> Result<u64> {
+    let bytes = parse_size(s)?;
+    if bytes < MIN_MAX_MEMORY {
+        bail!(
+            "{s:?} is too small to run anything in: --max-memory must be at least {MIN_MAX_MEMORY} bytes"
+        );
+    }
+    Ok(bytes)
+}
+
 /// Should ANSI colors be drawn?
 #[derive(Debug, Default, Clone, Copy, PartialEq, Eq, Display, Deserialize, ValueEnum)]
 #[strum(serialize_all = "snake_case")]
@@ -366,6 +410,13 @@ impl Options {
             jobserver: args.jobserver,
             jobserver_tasks: args.jobserver_tasks,
             leak_dirs: args.leak_dirs,
+            max_memory: args
+                .max_memory
+                .as_deref()
+                .or(config.max_memory.as_deref())
+                .map(parse_max_memory)
+                .transpose()
+                .context("Failed to parse --max-memory")?,
             minimum_test_timeout,
             no_default_features: args.no_default_features
                 || config.no_default_features.unwrap_or(false),
@@ -570,6 +621,76 @@ mod test {
         assert_eq!(options.build_timeout_multiplier, Some(3.5));
     }
 
+    #[test]
+    fn parse_size_understands_binary_suffixes_and_rejects_nonsense() {
+        // Input -> bytes, where None means it should not parse at all.
+        let cases = [
+            ("0", Some(0)),
+            ("1024", Some(1024)),
+            ("1024B", Some(1024)),
+            ("256M", Some(256 * 1024 * 1024)),
+            ("256MiB", Some(256 * 1024 * 1024)),
+            (" 4g ", Some(4 * 1024 * 1024 * 1024)),
+            ("2T", Some(2 * (1u64 << 40))),
+            ("", None),
+            ("M", None),
+            ("-1", None),
+            ("1.5G", None),
+            ("1 zettabyte", None),
+            ("18446744073709551615K", None),
+        ];
+        for (input, expected) in cases {
+            assert_eq!(parse_size(input).ok(), expected, "input: {input:?}");
+        }
+    }
+
+    /// Zero is a footgun rather than a way to turn the limit off, unlike `-t 0`.
+    #[test]
+    fn parse_max_memory_rejects_uselessly_small_limits() {
+        for tiny in ["0", "1", "1K", "1023K"] {
+            assert!(
+                parse_max_memory(tiny).is_err(),
+                "{tiny:?} should be rejected as too small"
+            );
+        }
+        assert_eq!(parse_max_memory("1M").ok(), Some(1 << 20));
+    }
+
+    #[test]
+    fn options_from_max_memory_arg() -> Result<(), Box<dyn std::error::Error>> {
+        let args = Args::parse_from(["mutants", "--max-memory=256M"]);
+        let options = Options::new(&args, &Config::default())?;
+        assert_eq!(options.max_memory, Some(256 * 1024 * 1024));
+
+        let args = Args::parse_from(["mutants"]);
+        let options = Options::new(&args, &Config::default())?;
+        assert_eq!(options.max_memory, None);
+        Ok(())
+    }
+
+    #[test]
+    fn cli_max_memory_overrides_config() -> Result<(), Box<dyn std::error::Error>> {
+        let config: Config = "max_memory = \"8G\"".parse()?;
+        let options = Options::new(&Args::parse_from(["mutants"]), &config)?;
+        assert_eq!(options.max_memory, Some(8 * 1024 * 1024 * 1024));
+
+        let args = Args::parse_from(["mutants", "--max-memory=1G"]);
+        let options = Options::new(&args, &config)?;
+        assert_eq!(options.max_memory, Some(1024 * 1024 * 1024));
+        Ok(())
+    }
+
+    #[test]
+    fn unparseable_max_memory_is_an_error() {
+        let args = Args::parse_from(["mutants", "--max-memory=lots"]);
+        let err = Options::new(&args, &Config::default())
+            .expect_err("--max-memory=lots should not be accepted");
+        assert!(
+            format!("{err:#}").contains("--max-memory"),
+            "unhelpful error message: {err:#}"
+        );
+    }
+
     #[test]
     fn cli_timeout_multiplier_overrides_config() {
         let config = indoc! { r"