Skip to content

Compile member-selectors once instead of per resolution - #3

Merged
ndreno merged 1 commit into
mainfrom
perf/precompile-selectors
Sep 11, 2026
Merged

ndreno merged 1 commit into
mainfrom
perf/precompile-selectors

Conversation

@ndreno

@ndreno ndreno commented Sep 11, 2026

Copy link
Copy Markdown
Contributor

(Supersedes #2, which GitHub auto-closed when its stacked base branch #1 was deleted on merge. Same commit, now rebased onto main.)

The lever

A regex member-selector (REQUEST_HEADERS:/^X-/, !REQUEST_COOKIES:/__utm/) was kept as a string and recompiled with the regex crate on every value it was tested against, on every request. CRS carries 162 rules with a !REQUEST_COOKIES:/__utm/ or /_pk_ref/ exclusion, and the exclusion closure recompiles the selector once per resolved cookie, per rule, per request.

Measured in isolation:

cookies= 1   selector evals=  162   recompile-each= 1.359 ms   cached= 0.004 ms   saved 99.7%
cookies= 5   selector evals=  810   recompile-each= 3.251 ms   cached= 0.014 ms   saved 99.6%
cookies=20   selector evals= 3240   recompile-each=12.519 ms   cached= 0.061 ms   saved 99.5%

Against the ~2 ms the gateway measures for a whole inspection, this was the dominant cost, hiding in the exclusion path.

The change

Compilation turns each parsed Target into a CompiledTarget whose regex selector is a compiled regex::Regex, built once. Resolution matches the compiled form directly and rebuilds nothing (grep confirms no regex::Regex::new remains in the resolve path, was 2).

The parsed Target/Selector in the AST are untouched, so the sealed rule-set format is unchanged — only the in-memory compiled program gains CompiledTarget/CompiledSelector. This also subsumes the selector validation merged in #1: compiling the selector is what validates it, so check_targets became compile_targets, and an uncompilable selector still fails the build with the same InvalidSelector error.

Verification

  • 169 unit tests pass; clippy and fmt clean.
  • Real CRS 4.9.0 still compiles to 590 rules, same 4 detectSQLi/detectXSS refusals, zero unexpected errors.
  • Exclusion semantics preserved: a request carrying __utmz=attack is allowed by a rule that session=attack trips.
  • Both fail-closed behaviors from Refuse invalid selectors and backward skipAfter at compile time #1 survive.
  • CI (including the FTW regression job) green on this commit.

A regex member-selector was stored as a string and recompiled with the regex
crate on every value it was tested against, on every request. The Core Rule Set
carries 162 rules with a `!REQUEST_COOKIES:/__utm/` or `/_pk_ref/` exclusion,
and each recompiles its selector once per resolved cookie, per rule, per
request. Isolated, that recompilation measured at 1.4 ms for a single-cookie
request and 3.3 ms for five cookies, against a whole-inspection budget of about
2 ms: the dominant cost, hiding in the exclusion path.

Compilation now turns each parsed `Target` into a `CompiledTarget` whose regex
selector is a compiled automaton, built once. Resolution matches against it
directly and rebuilds nothing. Caching the same compiled forms drops the
isolated cost by about 99%.

The parsed `Target`/`Selector` are unchanged, so the sealed rule-set format is
untouched; only the in-memory compiled program gains the new types. This also
subsumes the selector validation added for the fail-closed fix: compiling the
selector is what validates it, so an uncompilable selector still fails the
build with the same `InvalidSelector` error.

The Core Rule Set still compiles to 590 rules with the same 4
detectSQLi/detectXSS refusals, and cookie exclusions still exclude: a request
carrying `__utmz` is allowed by a rule that a `session` cookie trips.
@ndreno
ndreno merged commit 0d24e28 into main Sep 11, 2026
8 checks passed
@ndreno
ndreno deleted the perf/precompile-selectors branch September 11, 2026 07:48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant