sec (4/9): scope benchmark requirement preflight to runnable scanners#28
Open
jesse-merhi wants to merge 1 commit into
Open
sec (4/9): scope benchmark requirement preflight to runnable scanners#28jesse-merhi wants to merge 1 commit into
jesse-merhi wants to merge 1 commit into
Conversation
A benchmark validates required env for every requested scanner before running. A url-only user-defined scanner would abort a skill benchmark it would never run against, because its required credentials look missing. Filter requirement validation to scanners that support the per-case target kind.
This was referenced Jul 23, 2026
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: sec (4/9): scope benchmark requirement preflight to runnable scanners This is item 1/1 in the current shard. Shard 0/1. This placeholder means the worker is alive and reading the current context. I will edit this same comment with the actual review when the claws are done clicking. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changes
Benchmark requirement preflight now only considers scanners that can
actually run on the benchmark's target kind. A scanner that does not support the
target kind (e.g. a plugin-only scanner during a skill benchmark) no longer has
its required env vars demanded before the run starts.
Why it matters
Preflight fails fast when a required credential is missing — good. But it was
checking every configured scanner, including ones that would be skipped anyway
because they don't support the target kind. That meant a benchmark could abort
demanding, say, a token for a scanner that was never going to execute. Scoping
preflight to the runnable set makes the "fail before partial work" check match
the work that will actually happen.
Before / after
New helper
runnableBenchmarkScanners(opts, kind)filters the scanner set viathe registry's
SupportsTargetKindbefore requirement resolution;RunBenchmarkfeeds that filtered set into preflight.
Verify
TestRunnableBenchmarkScannersFiltersUnsupportedTargetKindasserts a scannerthat does not support the target kind is dropped from the preflight set.