Compile member-selectors once instead of per resolution - #3
Merged
Merged
Conversation
A regex member-selector was stored as a string and recompiled with the regex crate on every value it was tested against, on every request. The Core Rule Set carries 162 rules with a `!REQUEST_COOKIES:/__utm/` or `/_pk_ref/` exclusion, and each recompiles its selector once per resolved cookie, per rule, per request. Isolated, that recompilation measured at 1.4 ms for a single-cookie request and 3.3 ms for five cookies, against a whole-inspection budget of about 2 ms: the dominant cost, hiding in the exclusion path. Compilation now turns each parsed `Target` into a `CompiledTarget` whose regex selector is a compiled automaton, built once. Resolution matches against it directly and rebuilds nothing. Caching the same compiled forms drops the isolated cost by about 99%. The parsed `Target`/`Selector` are unchanged, so the sealed rule-set format is untouched; only the in-memory compiled program gains the new types. This also subsumes the selector validation added for the fail-closed fix: compiling the selector is what validates it, so an uncompilable selector still fails the build with the same `InvalidSelector` error. The Core Rule Set still compiles to 590 rules with the same 4 detectSQLi/detectXSS refusals, and cookie exclusions still exclude: a request carrying `__utmz` is allowed by a rule that a `session` cookie trips.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
(Supersedes #2, which GitHub auto-closed when its stacked base branch #1 was deleted on merge. Same commit, now rebased onto main.)
The lever
A regex member-selector (
REQUEST_HEADERS:/^X-/,!REQUEST_COOKIES:/__utm/) was kept as a string and recompiled with theregexcrate on every value it was tested against, on every request. CRS carries 162 rules with a!REQUEST_COOKIES:/__utm/or/_pk_ref/exclusion, and the exclusion closure recompiles the selector once per resolved cookie, per rule, per request.Measured in isolation:
Against the ~2 ms the gateway measures for a whole inspection, this was the dominant cost, hiding in the exclusion path.
The change
Compilation turns each parsed
Targetinto aCompiledTargetwhose regex selector is a compiledregex::Regex, built once. Resolution matches the compiled form directly and rebuilds nothing (grepconfirms noregex::Regex::newremains in the resolve path, was 2).The parsed
Target/Selectorin the AST are untouched, so the sealed rule-set format is unchanged — only the in-memory compiled program gainsCompiledTarget/CompiledSelector. This also subsumes the selector validation merged in #1: compiling the selector is what validates it, socheck_targetsbecamecompile_targets, and an uncompilable selector still fails the build with the sameInvalidSelectorerror.Verification
__utmz=attackis allowed by a rule thatsession=attacktrips.