Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 44 additions & 0 deletions .config/nextest.toml
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,50 @@
# `files_read` and `bytes_read`, which count acquisitions at the boundary and so
# cannot see an O(n*m) evaluation over a document already in hand.

# THE HANG GUARD, AND THE RATCHET THAT WILL BECOME A SLOW-TEST BAN (CLOUD-1571).
#
# `slow-timeout.period` is when nextest MARKS a case slow and prints it;
# `period x terminate-after` is when it KILLS it, reporting TIMEOUT — a FAILURE,
# because `on-timeout` defaults to "fail". `policy/nextest-slow.rego` gates the
# PRODUCT, so neither half can be raised alone to weaken it.
#
# WHAT THIS BOUNDS TODAY IS A HANG, NOT A BUDGET, and that retreat is measured
# rather than preferred. Four calibrated kills were each refused by CI:
#
# 90s -> `agentic_record::a_replay_...` TIMEOUT 90.022s, under verify load
# 240s -> the same case TIMEOUT 240.210s, windows runner
# 90s -> the three `symbols` cases TIMEOUT ~90.2s, windows runner
# 300s -> four `mutate` cases TIMEOUT 300.0s, 2-vCPU linux ci
#
# The last is the clearest: those four are ~9s each on this container. A 30x
# spread between the box a threshold is chosen on and the runners it is enforced
# on means no locally-measured per-case number is safe, and every kill DESTROYS
# the measurement that would have set the right one — each line above is the
# threshold the case was stopped at, never what it costs. Five guesses would be
# the same mistake five times.
#
# SO THE NUMBER COMES FROM THE JOB BUDGET INSTEAD, which is a real bound rather
# than a guess. `ci.yml` declares `timeout-minutes: 87` for this job on this
# repository's own convention of measured p95 x3. A 1200s per-test kill sits far
# above every case observed on any runner, well inside that budget so it names
# the failing TEST rather than letting the job die anonymously, and still catches
# the class the ban exists for: a genuine hang. Measured on this branch —
# `land.bats`'s mid-wait race sat 40 MINUTES at 0% CPU holding the landing lease
# (CLOUD-1661), which is what an unbounded case actually costs.
#
# `period = "10s"` is unchanged and is the VISIBILITY half: 11 cases here, 8 on
# windows, 4 on the ci runner are named on every run. That list, gathered across
# platforms over time, is what a future step ratchets the kill down onto — with
# data nobody had when this was first armed.
#
# NO PER-CASE OVERRIDES REMAIN. The three that existed — `agentic_record` and the
# `symbols` trio — were each 1200s, so the default now covers them and a separate
# row would assert a bound it no longer sets. CLOUD-1641 still owns
# `agentic_record`'s real cost and is NOT closed by this: the override went away
# because the floor rose, not because the case got fast.
[profile.default]
slow-timeout = { period = "10s", terminate-after = 120 }

[[profile.default.overrides]]
filter = 'test(deleting_six_governed_paths_costs_a_flat_multiple_of_the_floor)'
threads-required = "num-test-threads"
110 changes: 105 additions & 5 deletions batten.toml
Original file line number Diff line number Diff line change
Expand Up @@ -6331,6 +6331,35 @@ tool = "hyperfine"
version = "1.20.0"
input = "target/release/batten"

# The slow-test ban's declaration, guarded (CLOUD-1571 follow-on).
#
# THE BAN ITSELF IS THE RUNNER'S. `.config/nextest.toml`'s `slow-timeout` marks a
# case slow at `period` and kills it at `period x terminate-after`, reporting
# TIMEOUT — a failure, since `on-timeout` defaults to "fail". Measured on this
# tree: a case under a 50ms period reported TERMINATING, then TIMEOUT, then
# `error: test run failed`, exit 100.
#
# SO THIS ROW GUARDS A DECLARATION RATHER THAN A DURATION, and that division is
# deliberate. `mise-tasks/suite-bench-check.sh` records why a duration gate is the
# wrong instrument — it "would be red on every second run and would be bypassed
# within a day" — and CLOUD-1419 wrote an aggregate ratchet and withdrew it in the
# same branch. nextest measures and decides; batten refuses the weakening.
#
# `lines` NAMES ONE FILE AND THAT IS SAFE HERE, unlike `landing-roster-guarded`
# which had to declare a whole directory. That module's absent-source arm was dead
# because a glob matching zero files leaves the rule SKIPPED rather than evaluated
# — but `.config/nextest.toml` is a literal path, not a glob, so a tree without it
# still acquires the declared source and `nextest-slow-unbounded` fires on the
# empty document. `crates/batten/tests/it/nextest_slow.rs` drives the engine over a
# tree with the file deleted, because that is the only tier that can tell.
[[rule]]
id = "nextest-slow"
kind = "policy"
scope = "tree"
lines = [".config/nextest.toml"]
module = "policy/nextest-slow.rego"
severity = "deny"

# The agentic trials protocol (CLOUD-1116). A completeness predicate over the two
# records in `bench/agentic/`, and nothing else — it decides whether a row that
# CLAIMS an outcome carries the arms, the fixture, the run count and the falsifier
Expand Down Expand Up @@ -7342,10 +7371,10 @@ root = "target"
keep = 2

[prune.warm]
mb = 9472
worst_mb = 9472
mb = 11140
worst_mb = 11140
multiplier = 1
measured = "2026-09-06"
measured = "2026-09-08"

[prune.cold]
mb = 21455
Expand Down Expand Up @@ -7669,14 +7698,49 @@ measured = "2026-09-06"
# warm's basis refreshed to 220 and cold's left at 208, the very next lap refused
# on `[prune.cold]`'s staleness arm with warm never breached. The two arms are
# judged at different times and only one of them waits.
# THE 2026-09-08 MOVE, 220 -> 232, BOTH BASES, AND WARM RE-MEASURED (CLOUD-205).
#
# WHAT MOVED THE BASIS is ordinary growth plus one file from this branch:
# `crates/batten/tests/it/nextest_slow.rs`, the compiled tier for the slow-test
# ban. The tree was already 11 past the basis before it, so this branch finished a
# drift rather than causing one.
#
# WHY IT HAD TO MOVE NOW. `target-prune` refused and `verify` narrated the refusal
# as "not enough disk to run the gate" with **13809 MB free against a 10839 MB
# learned floor**. There was ample disk; the arm that refused was the staleness
# one. That misnaming is filed separately — it is the same class as a loop naming
# a cause it never read — and this entry records only that the cause here was the
# basis and not the disk.
#
# WARM IS RE-MEASURED, by the method the block above prescribes: `du -sm target`
# immediately after a successful prune on this container, **11140 MB at 232
# stems**. That is 48.02 MB per stem against the previous basis's 45.54, so the
# `keep x stems x size` model holds and the figure is a measurement rather than a
# scaling.
#
# AND THE OLD FLOOR WAS UNDER-BUDGETED, which is what this move actually repairs.
# 9472 is 45.54 x 208 — the figure for the basis BEFORE the 2026-09-06 move to 220
# — so warm has been budgeting for 208 stems against a tree of 232. The block above
# says what that costs: "a floor taken against a smaller stem count passes and then
# lets the build write more than it budgeted for", arriving as a rustc IO error
# inside a test run rather than as a disk fault. The engine had already noticed and
# was judging against a LEARNED floor of 10839 MB rather than the declared 9472.
#
# COLD DOES NOT MOVE, obeying the 2026-09-05 entry rather than ignoring it: its
# exact measurement needs a build from an empty `target`, which needs more free
# space than this container has. It cannot be taken here, which is different from
# being skipped, and it stays OWED. Its BASIS moves with warm's because the
# 2026-09-06 entry records what happens otherwise — refreshing one basis and not
# the other made the very next lap refuse on the other arm, with warm never
# breached. The two arms are judged at different times and only one of them waits.
[prune.warm.basis]
glob = "crates/batten/tests/**/*.rs"
count = 220
count = 232
tolerance = 10

[prune.cold.basis]
glob = "crates/batten/tests/**/*.rs"
count = 220
count = 232
tolerance = 10

# THE REGROWABLE ROOTS THE ESCALATION MAY DROP (CLOUD-1157), in the order it drops
Expand Down Expand Up @@ -12535,6 +12599,42 @@ id = "module read first"
kind = "document"
target = "policy/ci-cache-declared.rego"

[[verdict]]
id = "suite bind missing"
gloss = "the runner config declares no slow-timeout this gate can read, so no slow-test ban is in force"
class = """
`period` alone only MARKS a case slow and prints it; `terminate-after` is what kills it, and the TIMEOUT that follows is what fails the run. A declaration carrying one and not the other is not a ban and must not read as one. An absent file, a commented-out declaration, and a period in a unit this module cannot convert all reach the same class deliberately: nextest accepts `2m` and `500ms`, either of which would leave the comparison unreachable and the gate silently green over a bound nobody is enforcing. Fail-closed is the only honest direction for a guard whose subject is whether a guard exists.
"""

[[verdict.route]]
id = "module read first"
kind = "document"
target = "policy/nextest-slow.rego"

[[verdict]]
id = "waiver file missing"
gloss = "a per-test slow-timeout override cites no tracker row, so the exception is written but not filed"
class = """
A per-test override is how a legitimately slow case keeps the ban in force for every other case, so the mechanism is sanctioned and the ceiling deliberately does not reach it. What it owes instead is a reason a reader can follow: an override with no row behind it is the ban switched off for whichever test was inconvenient, and an unexplained exception is the thing that rots while looking like policy.
"""

[[verdict.route]]
id = "module read first"
kind = "document"
target = "policy/nextest-slow.rego"

[[verdict]]
id = "bound edit refused"
gloss = "the declared slow-test period is above the ceiling policy/nextest-slow.rego commits to"
class = """
The ratchet is "never above", not "always exactly", and the asymmetry is the point: lowering the period is free, so a branch that makes the suite faster tightens the bound without negotiating with this gate. Raising it past the ceiling is refused. Lowering the CEILING is a reviewed edit to the module, which is where a reader sees it in the diff rather than inferring it from a number that moved.
"""

[[verdict.route]]
id = "module read first"
kind = "document"
target = "policy/nextest-slow.rego"

[[startup]]
id = "engine-reads-the-authority"
gloss = "the engine that hook registrations invoke by bare name can read this repository's committed config"
Expand Down
70 changes: 69 additions & 1 deletion crates/batten/src/lib.rs
Original file line number Diff line number Diff line change
Expand Up @@ -5626,12 +5626,80 @@ fn admission_anchor(
// exact match widens to those rows and no further. Every typed kind keeps
// the narrow fast path it was given, which is what the ~90s measurement
// above is about.
//
// AND "TO THOSE ROWS" IS A STATEMENT ABOUT WHICH *KIND*, WHICH IS NOT THE
// SAME AS WHICH ROW — the difference cost 2m22s per mint (CLOUD-1571).
//
// Widening to every `policy` row ran every policy module over the whole
// tree, whether or not any of them *could* publish the predicate being
// minted against. Measured on `main` at `6eb08e14`: `override request
// --rule protected-mutation` took **2m22.121s**, against **0.123s** for a
// full adjudication of the same tree, and one test case paying it was 24%
// of the entire suite. `protected-mutation` is an engine-side rule name
// with zero hits under `policy/`, so no bundle could ever have published
// it: 58 modules were evaluated to produce findings the filter below
// discards one line later.
//
// `Bundle::declared` is the module's own published predicate set, so
// asking it is what turns "which kind" into "which row". It is the same
// authority `attribute` resolves a violation's id against and the one
// `lint.rs` reads for `waiver-names-no-rule` (CLOUD-1553), on identical
// reasoning: reading the published set adds no second authority over what a
// bundle declares, where re-deriving it from module source would.
//
// THE LOAD IS NOT THE COST, and that is why this is affordable rather than
// merely narrower. Compiling every bundle is **2.4s** measured here against
// the 2m22s above — the expense is acquiring the tree and evaluating over
// it, which is exactly what this now skips. The load is also the same work
// `run_over` does internally, so no module is compiled that would not have
// been.
//
// CLOUD-1087/CLOUD-1125's property is untouched: a predicate a bundle DOES
// publish still selects that bundle's row and still anchors on its finding.
// This narrows the widening; it does not restore the `declared.id == rule`
// filter that made every policy admission a silent no-op.
let selected: Vec<_> = if exact.is_empty() {
config
let policy_rows: Vec<_> = config
.rules
.iter()
.filter(|declared| declared.kind == rules::RuleKind::Policy)
.cloned()
.collect();
// COULD-NOT-LOOK FALLS BACK rather than refusing, which is the posture
// the `run_over` call below already takes for its own `Err`: a bundle
// set that will not load leaves no honest way to ask who publishes the
// predicate, and `head()` is never weaker than what shipped.
let Ok(bundles) = policy::load(
root,
&policy_rows,
policy::Vocabulary {
patterns: &config.patterns,
verdicts: &config.verdicts,
recorders: &config.recorders,
},
// The same entitlement the run below is given. A mint answers about
// one rule over one subject, so registry equality's exhausted half —
// a property of the whole authority — is not this verb's to assert.
policy::ModuleChecks::RunOverSelection,
None,
) else {
return head();
};
let publishers = policy::publishers_of(&bundles, rule);
// NOBODY PUBLISHES IT, so there is nothing to scan FOR and the scan is
// skipped entirely rather than run and discarded. This is the honest
// answer for an engine-side rule name, which is a legitimate thing to
// mint against — `protected-mutation` has a real override route.
//
// Silent, deliberately: reporting the rule as unknown here would decide
// CLOUD-1551's open question about an anchor that binds `call:` when
// nothing matches, and this row must not settle that by accident.
if publishers.is_empty() {
return head();
}
policy_rows
.into_iter()
.filter(|declared| publishers.contains(declared.id.as_str()))
.collect()
} else {
exact
Expand Down
30 changes: 30 additions & 0 deletions crates/batten/src/policy.rs
Original file line number Diff line number Diff line change
Expand Up @@ -463,6 +463,36 @@ impl Bundle {
}
}

/// The enabling-row ids of every bundle that publishes `predicate` (CLOUD-1571).
///
/// **The narrowing a mint needs, and the reason it is a function rather than a
/// filter written at its one call site.** [`crate::admission`]'s anchor has to
/// re-run the rule a refusal named in order to recover its fingerprint, and
/// `--rule` carries a PREDICATE id — `filed-here` publishes `filed-over-own-diff`
/// — so a row-id match selects nothing and the mint silently binds the head
/// (CLOUD-1087, CLOUD-1125). Widening from there to every `policy` row fixed that
/// and cost 2m22s per mint, because "which KIND of row" is not "which row":
/// `protected-mutation` is an engine-side name no bundle can publish, and 58
/// modules were evaluated over the whole tree to produce findings the caller
/// discarded one line later.
///
/// Asking each bundle's own published set answers the question exactly. An EMPTY
/// result is therefore a real answer — *nothing can raise this predicate, so
/// there is nothing to scan for* — and never a could-not-look: a caller reads it
/// as "skip the scan", which is only sound because [`Bundle::declared`] is the
/// same authority [`Bundle::attribute`] resolves a violation's id against.
///
/// Borrowed rather than owned, so a caller filtering its own rows against the
/// result allocates nothing per row.
#[must_use]
pub fn publishers_of<'a>(bundles: &'a [Bundle], predicate: &str) -> BTreeSet<&'a str> {
bundles
.iter()
.filter(|bundle| bundle.declared().contains(predicate))
.map(Bundle::id)
.collect()
}

impl std::fmt::Debug for Bundle {
/// Names the row, its modules' paths and the ids they publish — and **never
/// a source**, so a policy body cannot reach a log through a derived `Debug`
Expand Down
Loading