Measure library code coverage in CI, at a 98% floor - #5
Merged
Merged
Conversation
The CI checked rule-language coverage (parse, operator and @rx coverage across CRS) but never measured how much of the engine's own code the tests exercise. Adds a coverage job using cargo-llvm-cov, source-based instrumentation, run once over the library with all features and formatted three ways: a summary posted to the run page, an lcov artifact, and a floor check. The floor is 88, just under the current 90.5% line coverage, so ordinary variation does not fail a build. Raise it when the real number rises; do not lower it to turn a red build green.
The coverage job landed at 90.5% and an 88% floor. This adds the tests that were missing to reach 98.6% line coverage, and raises the floor to 98. Most of the gap was code with no direct test rather than untestable code: - collections.rs had no test module at all. Added one that resolves every collection and scalar, the count and *_NAMES forms, combined-size, the XML XPath forms, regex and whole-collection exclusions, and the macro context. - transaction.rs gained tests for the response phases, the accessors, every ctl: directive (ruleRemoveById/ByTag, ruleRemoveTarget*, requestBodyProcessor, ruleEngine Off/DetectionOnly, auditEngine), and the urlencoded/cookie parsers. - action.rs, operator.rs, engine.rs, parse.rs gained tests for the parse error branches and the metadata/disruptive/ctl arms. - matcher.rs gained the DirDataLoader read path, the v4-mapped IP folding, the bracketed-v6 and host:port address forms, and empty-operand handling. - transform, xml, multipart, rule gained their remaining decoder and parser edge cases. 242 tests pass. What remains uncovered is defensive arms and a few unreachable-by-construction branches, not behaviour.
ndreno
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
You asked whether we check code coverage. We didn't — the existing CI "coverage" jobs (parse / operator / @rx) measure rule-language coverage across CRS, not how much of the engine's own code the tests run. Then you asked for more than 98%.
What
coveragejob usingcargo-llvm-cov(source-based instrumentation): runs the instrumented library tests once with all features, posts a summary to the run page, uploadslcov.info, and enforces a line-coverage floor.Coverage
Measured locally with the exact CI command sequence; the 98 floor passes with headroom. 242 unit tests (was 172).
Most of the gap was code with no direct test, not untestable code:
collections.rs*_NAMESforms, combined-size, XML XPath forms, name/regex/whole-collection exclusions, macro context.transaction.rsctl:directive, the urlencoded/cookie parsers.action.rs/operator.rs/parse.rs/engine.rsmatcher.rsDirDataLoaderread path, v4-mapped IP folding, bracketed-v6 andhost:portaddress forms, empty operands.transform/xml/multipart/ruleWhat remains uncovered is defensive arms and a few unreachable-by-construction branches, not behaviour.
Matches the coverage setup added to barbacane, so both repos measure code coverage the same way.