Put the authored rule shapes in CI, where they cannot rot - #144
Merged
Conversation
The authored rule shapes were verified once, by hand, in a local working directory no CI runs. They are the only evidence that the shipped rules are not inert — and that failure mode is the whole reason the verification exists: a rule the engine silently rejects, or one whose match never fires, is indistinguishable from protection. It is in the bundle, it shows in the dashboard, it blocks nothing. Four of these were inert at some point for exactly that kind of reason: an inline-flag regex, a phase left at the default, a parameter source that does not exist. The five shapes now run through the real guard on every build, each payload paired with a benign request, because a rule that blocks the attack and also blocks ordinary traffic has mitigated nothing either. `rule-corpus.test.ts` already proves the engine can express one canonical rule per class; this asserts that these shapes, as authored — AND-ed gadget spellings, mutations, extra carriers — execute as written. Shapes are keyed by mechanism, and the fixture carries payload shapes only: no advisory identifiers, no affected version ranges, no mapping from a shape to what it covers. That boundary is asserted rather than left to review, because this repository is public and the coverage mapping is the product. Per-advisory duplication collapses with it: several advisories share one shape byte for byte, and what actually differs between them is the version scope. Writing this found a drift in the source corpus worth recording: one shape was missing the third condition its own prose described — the carrier that reaches a cookie-borne payload — while the rule that had been verified included it. The shape was weaker than both its documentation and the artifact the assertions covered, and regenerating from it would have quietly dropped that coverage. Both mutation directions are now checked: weakening a rule fails a blocking assertion, broadening one fails a false-positive assertion.
|
Comprehensive tests ensure all authored rule shapes execute as intended. 🎯 Quality: 86% Excellent · 📦 Size: Large — consider splitting if possible 🛡️ Standards: no pre-flight fit check ran for this change — wire 📈 This month: Your 80th PR — above team average · Averaging Good |
Contributor
Author
|
/review |
mariojgt
approved these changes
Aug 19, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Independent of #142 and #143 — this touches only
tests/protect/, so it can merge in any order.Why
The authored rule shapes were verified once, by hand, in a local working directory no CI runs. They are
the only evidence that the shipped rules are not inert — and that is the failure mode the verification
exists for: a rule the engine silently rejects, or one whose match never fires, is indistinguishable from
protection. It is in the bundle, it shows in the dashboard, it blocks nothing. Four of these were inert at
some point for exactly that class of reason: an inline-flag regex, a phase left at the default, a
parameter source that does not exist.
rule-corpus.test.tsalready proves the engine can express one canonical rule per vulnerability class.This is a different claim about a different artifact: these are the shapes as authored — the AND-ed
gadget spellings, the mutations, the extra carriers — and the question is whether the engine executes them
as written.
What lands
Five shapes, each run through the real guard, each payload paired with a benign request — a rule that
blocks the attack and also blocks ordinary traffic has mitigated nothing either. Plus structural checks
for the three ways one of these went inert before: undelimited regexes, unknown parameter sources, and a
rule that screens egress while defaulting to the request phase.
Both mutation directions are checked, so neither half can rot into a tautology:
The boundary
Shapes are keyed by mechanism, and the fixture is payload shapes only: no advisory identifiers, no
affected version ranges, no mapping from a shape to what it covers. A test asserts that, rather than
leaving it to review — this repository is public, and the coverage mapping is the product, so it stays
where the advisories are triaged.
Per-advisory duplication collapses with the same cut: several advisories share one shape byte for byte,
and what actually differs between them is the version scope, which is not a payload shape.
A drift this surfaced
One shape was missing the third condition its own prose described — the carrier that reaches a cookie-borne
payload — while the rule that had actually been verified included it. So the recorded shape was weaker than
both its documentation and the artifact the assertions covered, and regenerating instances from it would
have quietly dropped that coverage. The verified form is what landed here.
Verification
Full suite 1083 passed, typecheck clean.