Skip to content

test(lint): replace the last flaky wall-clock ratio gate - #94

Merged
takeokunn merged 1 commit into
mainfrom
fix/flaky-cost-ratio-test
Aug 3, 2026
Merged

test(lint): replace the last flaky wall-clock ratio gate#94
takeokunn merged 1 commit into
mainfrom
fix/flaky-cost-ratio-test

Conversation

@takeokunn

Copy link
Copy Markdown
Collaborator

lint-introspection's the_cost_of_each_rule_grows_linearly_with_the_input asserted a wall-clock doubling ratio below 3.0. It failed three CI runs on untouched code — at 4.30× and 9.24× — and passed 3/3 on re-run at load average 69. It was noise on PRs #90, #91 and #92.

Its own docstring claimed "neither is a wall-clock threshold — those are what make a test flaky on a loaded machine", which was wrong about itself. A ratio of two wall-clock measurements normalizes for machine speed but not for load changing between the two measurements, which is exactly what a shared box does.

The split

Following the precedent PR #85 set for lint-documentation, lint-contract-annotation and lint-performance:

  • each_rule_is_dispatched_once_per_head_match keeps the deterministic half — the engine's own invocation counter — and gains an explicit assertion that doubling the definitions doubles the dispatch count. That is the shape the ratio existed to catch (a rule re-walking the file per match), stated as a property of dispatch rather than of nanoseconds.
  • ignored_bench_doubling_ratio keeps the timing table behind #[ignore], with the invocation to run it by hand.

Mutation-checked: perturbing one expected invocation count by 1 fails the new test, so it still discriminates rather than passing vacuously.

Audit of the rest

No others need fixing:

site state why
lint-form-shape already #[ignore]d PR #85
lint-performance already #[ignore]d PR #85
lint-package-hygiene left alone deliberately normalizes against a control rule measured in the same pass, so a loaded machine moves both ratios together; its comment says exactly this, its bounds are 9.0 absolute and 2.2× control, and it has never flaked

That last one is the technique worth keeping: a bare constant can't distinguish "the rule got slower" from "the machine got busier", but a control measured in the same pass can.

Verification

cargo test -p paredit-feature-lint-introspection → 0 (114 passed, 1 ignored), cargo fmt --all --check → 0, cargo clippy -p … --all-targets -- -D warnings → 0.

`lint-introspection`'s `the_cost_of_each_rule_grows_linearly_with_the_
input` asserted a wall-clock doubling ratio below 3.0. It failed three
CI runs on untouched code, at 4.30x and 9.24x, and passed 3/3 on re-run
at load average 69.

Its own docstring claimed "neither is a wall-clock threshold -- those
are what make a test flaky on a loaded machine", which was wrong about
itself. A ratio of two wall-clock measurements normalizes for machine
*speed* but not for load changing *between* the two measurements, which
is exactly what a shared box does.

Split, following the precedent PR #85 set for lint-documentation,
lint-contract-annotation and lint-performance:

- `each_rule_is_dispatched_once_per_head_match` keeps the deterministic
  half -- the engine's own invocation counter -- and gains an explicit
  assertion that doubling the definitions doubles the dispatch count.
  That is the shape the ratio existed to catch (a rule re-walking the
  file per match), stated as a property of dispatch rather than of
  nanoseconds.
- `ignored_bench_doubling_ratio` keeps the timing table behind
  `#[ignore]`, with the invocation to run it by hand.

Mutation-checked: perturbing one expected invocation count by 1 fails
the new test, so it still discriminates.

An audit of the remaining ratio assertions found no others to fix.
`lint-form-shape` and `lint-performance` are already `#[ignore]`d.
`lint-package-hygiene`'s is deliberately left alone -- it normalizes
against a control rule measured in the same pass, which absorbs machine
load, and its comment says so; it has never flaked.
@takeokunn
takeokunn merged commit b61fe5d into main Aug 3, 2026
10 checks passed
@takeokunn
takeokunn deleted the fix/flaky-cost-ratio-test branch August 3, 2026 09:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant