benchmark: add cve-agent model benchmark and fix what it exposed - #104
Merged
Merged
Conversation
- Add AgentConfig.no_knowledge (default False) and thread it through __main__.py's arg parsing, config building, and main() - main() passes knowledge_base=None to process_single_cve() when the flag is set, instead of constructing a real KnowledgeBase - Guard save_knowledge_pattern() with an early return when config.no_knowledge or knowledge_base is None, so no pattern is written on success - Widen knowledge_base's type to Optional[KnowledgeBase] through orchestrator.py's resolution pipeline (_resolution_loop, _run_single_resolution_attempt, _finalize_resolution, _handle_not_applicable, _handle_clean_apply, process_single_cve, _run_cve_pipeline) so a None knowledge base flows through safely without every call site needing its own guard - Document the flag in README.md's AI-assisted backporting section - Add tests/agent/test_no_knowledge.py: end-to-end no-KB-write/read checks, a control case confirming the pattern IS saved without the flag, direct unit checks on the guarded write path, and CLI-wiring tests (default false, flag sets true, threads into AgentConfig, main() skips constructing KnowledgeBase entirely) Useful for benchmarking a model's unaided backporting performance without knowledge-base assistance skewing the result. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The prior two commits (--no-knowledge flag, tests/benchmark suite)
left several docs stale:
- AGENTS.md: add tests/benchmark to the directory map (was entirely
missing from the tests/{...} listing)
- .agents/summary/interfaces.md: add --no-knowledge to cve-agent's CLI
flags reference
- .agents/summary/codebase_info.md: add tests/benchmark/ to the test
structure listing
- CHANGELOG.md: add both changes under [Unreleased]
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Add git checkout -- .*, git format-patch .*, find \. .*, and grep .* to match the allowedCommands list in yocto-cve-backport-interactive.json - Keeps both agents at parity per their documented "same capabilities" relationship Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
AGENT_INSTRUCTIONS.md explicitly documents bare 'git checkout <path>' (without --ours/--theirs) as unavailable, but both interactive manifests (.kiro/agents/ and cve_agent/agents/) allowed '^git checkout -- .*$' since the agent's initial commit. Remove it from all four manifest copies so behavior matches the documented policy. - Drop the 'git checkout -- .*' allowedCommands entry from both yocto-cve-backport-interactive.json copies - Propagate git format-patch/find/grep parity additions to cve_agent/agents/yocto-cve-backport.json (the packaged fallback used when .kiro/agents/ is absent) to match the .kiro/ copy - Add matching Bash(git format-patch:*), Bash(find .:*), and Bash(grep:*) entries to ClaudeBackend._ALLOWED_TOOLS so the Claude backend keeps parity with the kiro manifest, per test_claude_kiro_parity.py - Drop the now-obsolete skip in test_destructive_command_is_rejected_interactive that carved out an exception for the removed bare-checkout allowance Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
conclusion.json lives in the persistent agent dir, which survives across resolution attempts within a single CVE run. If an early attempt escalated (needs_human) or marked not_applicable without touching the tree, and a later attempt then actually resolved the conflict without writing a fresh conclusion, the orchestrator re-read the stale file and reported the CVE as escalated/skipped -- discarding a good, building resolution. Clear conclusion.json at the start of every resolution attempt so the orchestrator only ever observes the verdict of the session that just ran. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
A CVE fix is often referenced by the pull request's merge SHA rather than the commit that changed the code (e.g. setuptools' CVE-2024-6345 fix 88807c70, "Merge pull request #4332"). Git treats merges specially in two ways that both broke the backport flow: - `git cherry-pick <merge>` is refused before it starts ("is a merge but no -m option was given") and leaves no CHERRY_PICK_HEAD and no unmerged entries. find_least_conflict_commit() measured conflicts right after and therefore scored the merge as "0 conflicts" -- the best possible candidate -- and _handle_no_clean_apply() reported EXIT_CONFLICT over a pristine workspace. - `git show` prints no diff (and no --name-only file list) for a merge, so the agent's file scope came out empty. context.md advertised an empty Allowed Files list, the pre-commit scope guard rejected every write, and the model could only escalate after a paid session. - Add is_merge_commit(), cherry_pick_command() (adds `-m 1` for merges) and has_conflict_state() to cve_corrector.git_ops - Score least-conflict candidates only when the pick actually started; discard picks git refused outright - Verify real conflict state before raising ConflictError, else raise PatchError (exit 5) so no resolver is sent to a clean workspace - Add merge_diff_flags() (`-m --first-parent`) to shared.git_runner and use it for the agent's file scope, context.md stat/diff hints, review diff and interdiff - Compute the session file scope once in compute_allowed_files() so context.md and the scope guard cannot disagree - Escalate before launching a session when the scope is still empty - Add regression tests for both tools over real merge-commit repos Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Git resolves a relative `.gitmodules` URL against the superproject's
remote. In a cve-corrector workspace that remote is a local bare mirror
when --mirror-dir/--mirror-path is used (and otherwise unset, so git
falls back to the workspace path), so glib's
[submodule "subprojects/gvdb"]
url = ../../GNOME/gvdb.git
resolved to a nonexistent sibling directory such as
<workspace>/GNOME/gvdb.git and submodule init failed. For glib the
pre-patch build then failed too, since meson.build bootstraps the
submodule itself:
meson.build:2128:0: ERROR: git submodule failed to init
ERROR: glib-2.0-1_2.78.6-r0 do_configure: meson failed
The corrector reported that as exit 10 (pre-existing build failure), so
every glib CVE came out as "skipped" without the agent ever running.
- Add resolve_relative_submodule_url(), mirroring git's own algorithm
(each `..` drops the base's last path component, `.` is skipped),
for https/git/ssh, scp-like and local base URLs
- Anchor resolution on the upstream remote when it is a real remote URL,
else on the canonical URL deduced from the CVE's fix-commit URLs
- Prefer a local mirror of the submodule when --mirror-dir holds one,
matching how the superproject is fetched, and pass
protocol.file.allow=always since git blocks local transports for
submodules by default (CVE-2022-39253)
- Split submodule setup into init -> URL overrides -> update, because
`submodule init` never overwrites an existing submodule.<name>.url
- Forward hash_details/mirror_dir through prepare_cve_branch
- Add unit and real-git integration tests for the glib/gvdb layout
Assisted-by: kiro:claude-opus-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Runs cve-agent across a fixed roster of CVEs and a selection of models, then an AI judge pass on diffs that came out moderately/majorly different from the human reference backport. - bench_lib.py: pure-Python helpers with no I/O beyond the paths passed in (score_tier, is_mirror_gap_only, count_conflict_markers, classify_diff_bucket, resolve_models, relative_cost_weight, observed_avg_credits/project_remaining_cost, total_spent, count_tool_calls, filter_for_judging, judge_diff) - run_benchmark.sh: orchestrator sourcing tests/integration/ test_common.sh for the OE tree lifecycle (reset_oe_tree, setup_cve_branch, run_cve_corrector, compare_patches_detailed). Always passes --no-knowledge to cve-agent so the benchmark measures unaided model performance. Single confirmation prompt covers both the agent-run and judge phases; prints a relative cost weight and (on resume) a projected remaining cost, both explicitly labelled as not credit predictions - benchmark-roster.json: a fixed, committed 7-CVE roster (1 easy, 1 medium, 5 hard spread across real conflict complexity) so every run tests the same CVEs. --retier re-probes and refreshes the roster's recorded stats in place with cve-corrector only (no AI cost); it never adds, removes, or reorders roster CVEs - generate_benchmark_report.py: markdown report generator (per-model summary, per-tier bucket distribution, meaningful-vs-stylistic split for the judged subset) - The fixed judge model (claude-opus-4.8) is deliberately not part of the benchmarked roster - test_bench_lib.py / test_generate_benchmark_report.py: full unit coverage for the above Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Adds a 2-commit dependent series (shared fix with CVE-2026-26157) that produces a genuine conflict plus a post-apply ptest failure, cheap to rebuild -- a useful hard-tier addition to the fixed benchmark roster. - tests/benchmark/benchmark-roster.json: new CVE-2026-26158 entry (tier hard, recipe busybox, series_len 2, conflict_markers 5) - tests/integration/test-cve-metadata-agent.json: fixture metadata for the new CVE's two dependent fix commits - tests/integration/test_common.sh: exclude the benchmark's per-model generated_<cve>_<model>_*.patch archives from compare_patches_detailed's original-patch discovery, so they aren't mistaken for a reference patch on a later model's comparison - AGENTS.md, CHANGELOG.md: bump roster count references 7 -> 8 Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Adds bench_lib.is_agent_env_failure(), which recognizes log markers meaning the AI agent never actually ran because its runtime environment was unusable (missing/unauthenticated backend CLI, or agent configs that failed to install/refresh). Such a failure is not a model-quality signal -- it would recur identically for every model and CVE -- so the benchmark run loop can use this to abort early instead of recording a wall of identical failures. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Adds bench_lib.ordered_roster_cases(), which enumerates the roster in the same canonical run order run_benchmark.sh uses (tiers easy->medium->hard, alphabetical by CVE within a tier) and numbers them 1-based as 'cases', and select_cases(), which resolves a set of 1-based case numbers back to roster entries (de-duplicated, order preserved, out-of-range/empty input rejected with a clear message). These back a future --list-cases/--run-case CLI in run_benchmark.sh, letting a user list the roster as numbered cases and re-run a subset (e.g. to skip an expensive recipe or re-run a single case) without touching benchmark-roster.json. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Previously a run whose generated patch touched a different (but
overlapping) set of files than the reference backport was classified
as 'file-mismatch' and never sent to the judge — even when most files
matched and only a handful were missing/extra.
- bench_lib.classify_diff_bucket(): a missing/extra fileset now
resolves to the new 'partial' bucket when the two patch sets still
share at least one file (via _common_file_count()), falling back to
'file-mismatch' only for fully disjoint filesets
- bench_lib.scope_diff_to_common_files(): strips the one-sided
missing/extra blocks out of a compare_patches_detailed diff patch,
keeping only the two-sided (shared-file) blocks that are a
meaningful backport-vs-reference comparison
- bench_lib.count_diff_changed_lines(): counts +/- lines in a unified
diff, used to report a 'partial' row's diff_lines scoped to the
shared files rather than the whole-patch divergence
- JUDGEABLE_BUCKETS now includes 'partial'; a partial overlap whose
shared files come out byte-identical is recorded as a
'structural-only' judge_results.csv verdict (no judge call) instead
of being left unjudged
- judge_results.csv gains a 'scope' column ('full' vs 'partial') so a
verdict is self-describing without cross-referencing agent_results.csv
- generate_benchmark_report.py: bucket distribution table and the
meaningful-vs-stylistic split account for 'partial'/'structural-only'
- README.md: document the new bucket, CSV columns, and per-model
artifact naming
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Follow-up to the CVE-2026-26158 roster addition: bump the remaining 7-CVE/7-roster-CVEs references to 8 and add the busybox row to the fixed-roster table, which were missed when the roster entry itself was committed. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Wires the library additions (agent-env-failure detection, roster case ordering/selection, partial-fileset judge scoping) into the run loop, plus operational robustness the benchmark lacked: - --list-cases / --run-case <N...>: list the roster as numbered cases (run order) and optionally scope the agent run, cost estimate, and --retier to a subset — handy to skip an expensive recipe or re-run a single case - Forward Ctrl+C to the agent: the agent runs under setsid so a per-run timeout can reap its whole process tree, which also means a terminal SIGINT never reached it. A trap now kills the agent's process group directly, waits briefly, force-kills if needed, resets the OE tree, and exits 130 - Preflight-abort when kiro-cli is missing/unusable (bench_lib.is_agent_env_failure): such a failure would recur identically for every model and CVE, so the run now aborts with a clear message and --resume hint instead of recording a wall of identical failures - Archive each run's generated patch(es) to generated_<cve>_<model>_<file>.patch before the OE tree reset, and rename compare_patches_detailed's per-CVE differences report/diff patch to per-model filenames, so multiple models' comparisons for the same CVE no longer overwrite each other - Judge phase: scope a 'partial' bucket's diff to the shared files before judging (bench_lib.scope_diff_to_common_files), record a 'structural-only' verdict when nothing common differs, and write the new judge_results.csv 'scope' column - Upgrade an old 4-column judge_results.csv header in place on --resume when it has no data rows yet (avoids appending 5-field rows under a stale 4-field header) - README.md: document --list-cases/--run-case and the per-model artifact naming scheme Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
After the fixes, it doesn't need the cve-agent to generate a patch Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
AGENT_INSTRUCTIONS.md caps each per-file stanza of the AI's `Conflicts Resolved:` block at a few bullets and a few dozen words, but nothing enforced it, so the notes grew into multi-paragraph narration of the investigation instead of stating the adaptation. - Add cve_agent/commit_notes.py: a pure parser and budget checker (no git calls, no package imports) so the hook, the orchestrator gate and review.py can share one definition of the rules - Cap at 3 bullets and 48 words per file, warning from 40 words: guidance can stay softer than enforcement, and a stanza a few words over must not cost a whole session - Count bullets, not physical lines: a 40-word note wrapped at 72 columns spans 5 lines, so counting those would make the word budget unreachable; continuation lines fold into their bullet - Charge anything the parser cannot attribute to a file to one synthetic stanza, so a malformed header, prose after the trailers, or text split across two blocks cannot switch enforcement off - Recognise decorated and misspelled headings (`#### `, `**...**`, lowercase, blockquote) without mistaking an unrelated `## Description` in a preserved upstream body for notes - Exempt `<file>: omitted (...)` one-liners wherever they appear, including wrapped reasons, but treat the same text as a bullet inside a stanza as prose - Restrict trailer detection to known keys, so a body line like `Reason: ...` no longer truncates the block and hides later stanzas - Signal a rejection with exit code 3, not 1: the interpreter itself exits 1 on `-m <missing module>` and 2 on a usage error, and callers must be able to tell "over budget" from "the checker broke" - Have review.py import the note-block markers from the new module, with a deliberately broader set for de-duplication than for the budget Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The budget checker only helps if something runs it while the AI is still in a position to fix the message. A commit-msg hook fires on both `git cherry-pick --continue` and `git commit --amend`, and a rejection is non-destructive: the cherry-pick stays in progress and .git/MERGE_MSG survives, so the AI can shorten the notes and re-run the same command. - Add install_notes_hook()/remove_notes_hook(), mirroring the existing pre-commit scope guard: same hooks dir, same backup handling, same session lifecycle - Resolve the interpreter and package root at install time and quote them with shlex, since the workspace is an unrelated repository - Fail open: only the checker's dedicated rejection status blocks a commit. A missing interpreter or unimportable package warns and allows, because the AI is instructed never to bypass the hook and a check that cannot run would otherwise deadlock the session - Install and remove alongside the scope guard in guarded_session(), so the hook is gone before review.amend_commit_with_summary() appends its own change summary - Cover the real recovery path end to end: reject on --continue, shorten MERGE_MSG, --continue again Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Both guards write into a user's workspace and restore what they found, so the edge cases matter: a lost backup means a developer silently loses their own hook, and a leaked guard means later commits are policed by a session that already ended. - Skip the backup when the existing hook carries our own marker: a second install without an intervening removal (a crashed session) previously backed up our hook over the user's real backup, then restored ours as if it were theirs - Read a pre-existing hook with errors='replace': a hook in another encoding raised UnicodeDecodeError, which is not an OSError, aborting the session and leaving the first guard installed - Install both guards inside the try, so a failure between the two installs cannot leave the first one behind - Warn when core.hooksPath is set: git then ignores .git/hooks entirely and both the file-scope guard and the note budget are inert while appearing installed - Isolate the real-git tests from the developer's global git config, which would otherwise disable the hook under test Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The hook only sees commits the AI makes during a session. A commit created before the hook was installed, or edited by hand, reaches HEAD unchecked — so validate once more before the change is shown for approval. - Add validate_commit_notes() and check HEAD's notes after the session, before request_approval() - On a hard violation, write the overage to human_feedback.txt so the next session gets the exact counts and the instruction to amend the message only, appending to any feedback a human already left - Cap the bounces at 2, then accept with a warning: discarding a technically correct backport over commit-message prose is the worse failure - Do not spend a resolution attempt on a prose bounce. It previously consumed one, so an always-verbose AI exhausted max_retries and was reported ESCALATED — exactly the outcome the cap exists to prevent. total_attempts still counts bounces, so --max-total-attempts bounds a pathological loop - Reset the bounce allowance on a phase change, so notes added during a later build or ptest amend are still checked - Record every verdict, including non-blocking warnings, in the session audit log Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The feedback section prefixed only the first line with '> ', which was fine for a one-line human note but breaks a multi-line report out of the markdown blockquote — now that the note-budget gate writes a six-line verdict there, most of it rendered as unquoted prose. - Prefix every line of human_feedback.txt, keeping blank lines as '>' Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Several documents contradicted the budget they were supposed to describe. The worst offender was ptest.md, which asked for exactly the content AGENT_INSTRUCTIONS.md forbids -- which cases failed, and the pass/fail counts -- and which cannot fit in 40 words. Since a ptest retry amends the commit, that instruction reliably pushed the notes over budget. - State the enforced numbers instead of "at most 2-3 lines (~40 words)": 3 bullets, ~40 words, rejected past 48, and note that wrapping is free - Document the recovery path, and that `git commit --amend --no-edit` must never be used for it: it resubmits the identical rejected message, so the hook rejects it again forever. Flag the same trap next to the build-retry instruction that legitimately uses --amend --no-edit - Distinguish the two hooks in Scope Rules: pre-commit rejects out-of-scope files, commit-msg rejects over-long notes - Require every line in the block to sit under a file header, and document `<file> (0 conflicts):` for a build or ptest fix to a file that had no merge conflict -- previously there was no valid header shape for that case - Unify the two `<file>: omitted (...)` spellings and state that they are exempt from the budget - Replace ptest.md's failing-case narration with a one-bullet rule, and point conflict.md at the Commit Note column as the target length - Add a test that fails if the documented numbers and the enforced constants ever diverge Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
`^git show .*$` in both agent manifests and `Bash(git show:*)` for the Claude backend already permit this form; only the AI-facing list in AGENT_INSTRUCTIONS.md never mentioned it, so the agent had no reason to know it could read a file as committed. No permission is granted here. - Document `git show HEAD:<path>` and `git show <sha>:<path>`, and when to prefer them over the file-reading tool - Lock the form in with permission tests for both manifests and a parity test for the Claude allow-list, so a future narrowing of the `git show` pattern fails the suite Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The benchmark and integration harnesses compared a generated backport against the reference patch as two sets of +/- lines, which reported differences that are not differences. oe-core's less CVE-2024-32487.patch is whitespace-damaged (tab-indented context lines lost their leading space) and ends its git signature '--' rather than '-- ', so the phantom '--' removal was counted as the sole divergence while the per-file diff handed to the AI judge was pure whitespace noise. Compare the two patch sets with interdiff instead: reduce each side to its diff body and diff the effects, so commit metadata, hunk offsets, reordered hunks and signature spelling stop registering as changes. - Repair whitespace-damaged context lines before invoking interdiff, which otherwise refuses the input ("Whitespace damage detected in input") and loses the comparison entirely; hunk boundaries come from the @@ line counts so metadata and the signature are untouched - Add allow_empty to generate_interdiff so an empty delta ("the patches are equivalent") is distinguishable from "could not compare" - Separate files touched by only one side behind ONE_SIDED_MARKER, which scope_diff_to_common_files truncates at; reports in the old format still use the legacy span heuristic - Fall back to the line-set comparison when patchutils is absent, and stop counting a '--' signature as a removed diff line there too - Fix _extract_files_touched mangling paths via lstrip('b/'), a character set strip that turned 'b/bin/x.c' into 'in/x.c' Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
judge_results.csv stored a bare 'meaningful'/'stylistic' verdict, so a
surprising classification could not be audited without re-reading the diff
by hand. Comment churn also counted as divergence: a backport that only
reworded a comment was sent to the judge and could come back 'meaningful'.
- Ask the judge for one or two sentences naming the construct behind its
decision, and store them in a new 'reason' column. Split sentences only on
a period followed by whitespace, so C identifiers such as '!S_ISLNK' are
not cut in half, and drop kiro-cli's credits footer
- Write judge rows with csv.writer rather than echo, since the reason is
free-form prose containing commas and quotes
- Strip comment-only changed lines before judging and answer 'comment-only'
with no model call when nothing else remains. Comment syntax is chosen per
file from the diff headers ('//' and '/* */' for C-like sources, '#' only
for shell/Python/make, never for C where '#if' is meaningful), and
block-comment state is tracked per diff side so pointer code such as
'*p++ = *s++;' is not mistaken for a comment continuation
- Count 'comment-only' in the report and add a Judge Reasoning table,
suppressed for results dirs judged before the column existed
diff_lines and diff_bucket in agent_results.csv still count comment lines;
only the judge ignores them.
Assisted-by: kiro:claude-opus-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Re-probed the 7 roster CVEs with cve-corrector only (--skip-build --skip-ptest, no AI cost) against openembedded-core scarthgap. The roster membership is unchanged; only the recorded stats move. - conflict_markers: CVE-2024-32487 3->2, CVE-2024-6345 0->2, CVE-2025-1153 52->45, CVE-2025-47183 6->1, CVE-2025-47203 12->7, CVE-2026-26158 5->3 - diff_lines: CVE-2026-0990 18->32 - Every tier and exit_code is unchanged, so the easy/medium/hard split the benchmark runs is identical The wholesale reordering in the diff is not a content change: --retier rewrites the file with json.dump(sort_keys=True), which alphabetizes the CVE entries and moves the leading "_comment" to the end. Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The _comment entry duplicated the README and had gone stale: it still claimed "the exact same 8 CVEs" after CVE-2025-4373 was removed, and --retier's json.dump(sort_keys=True) kept relocating it to the end of the file on every refresh. - Move the one fact it carried that the README lacked -- that every tier/exit_code/diff_lines/series_len/conflict_markers value is measured data from a live cve-corrector probe plus the historical bulk run -- into the roster section of tests/benchmark/README.md - Correct the same 8-CVE staleness in the README and in run_benchmark.sh's --help text: the roster is 7 CVEs (1 medium, 6 hard), no easy entry - Update the documented conflict_markers spread to the refreshed range (0 to 45, was 0 to 52) The '_comment' filters in bench_lib.ordered_roster_cases and run_benchmark.sh are kept: they cost nothing and let a comment be reintroduced without breaking the roster readers. Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Add three nested rosters: default (7 CVEs), balanced (20, 6/6/8), extended (40, 6/10/24). Each is a superset of the previous so results stay comparable. All fields are measured from the 289-CVE bulk run in tests/integration/test-results/bulk_20260626_081658/ - Add --roster default|balanced|extended|<path> flag to run_benchmark.sh with a startup log line recording which roster ran - Add tools/plot_benchmark_results.py (matplotlib) with 6 PNG charts: outcome by model, raw bucket distribution, cost, quality-vs-cost scatter, effort, and per-CVE x per-model outcome matrix - Swap minimax-m2.5 out of the default model set (21.12 credits per usable backport in bench_20260828_145923) and add claude-sonnet-4.8 - Add tests/benchmark/test_benchmark_roster.py: schema, tier consistency, nesting chain, composition guards (44 tests across all three files) - Add tests/tools/test_plot_benchmark_results.py: outcome classification, aggregation, ranking, matrix (33 tests, no matplotlib dependency) - Document roster selection rules, run-cost comparison table, and the per-CVE credit cap (20, non-binding on current data) in README Assisted-by: kiro:claude-sonnet-4.9 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Allow read-only git ref/object inspection (git branch, describe,
show-ref, ls-tree, cat-file -t/-s/-p, grep) and a single-line-range
sed -n form, in AGENT_INSTRUCTIONS.md, both agent manifests, and the
Claude backend's allowed-tools list
- Let git status --porcelain -- <path>, git restore --staged <path>,
and git checkout --ours/--theirs <path> take several paths per call
instead of one
- Loosen the echo allowlist from two hardcoded exit-code messages to any
double-quoted text ending in $? or ${PIPESTATUS[0]}
- Add 'Undoing a Bad Cherry-Pick' to AGENT_INSTRUCTIONS.md: four cases
(abort/skip in progress, amend a wrong commit, recover pre-image
content from original-version, escalate on a wrong-commit
cherry-pick) now that git reset/revert/update-ref/branch -f remain
unavailable
- Point instructions/build.md and instructions/conflict.md at the new
section instead of leaving the agent to discover on its own that
reset/revert cannot be used
- Extract the kiro-cli execute_bash guard model (file-redirect
detection, compound-command splitting, allowedCommands matching) out
of test_security.py into tests/agent/allowlist_model.py so
test_agent_permissions.py can reuse it instead of duplicating it
- Add tests/integration/test_enrich_metadata_from_oe.py and extend
enrich_metadata_from_oe.py's docstring to explain why URL-derived
hashes from Upstream-Status: Backport are tried before the less
reliable git format-patch From-header SHA
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- score_tier() now derives easy/medium/hard from conflict_markers and distinct files_involved (git's own CONFLICT (content) lines), not diff_lines/series_len -- restricted to a recoverable exit code (conflict/ptest/build), and raises otherwise - Add count_conflicted_files() alongside count_conflict_markers() - Purge every clean-exit entry from the three resolution rosters (default/balanced/extended): a clean apply has no conflict to size, so it was being scored 'easy'/'medium' on diff size instead of measuring anything about conflict-resolution difficulty - Re-probe and rebuild default (6), balanced (8), and extended (22) against the current OE-Core (scarthgap) with the new metric; nesting and shared-entry equality preserved - Add benchmark-roster-clean-apply.json (5 CVEs): a separate, non-nested roster for CVEs whose cherry-pick applies with no conflict, using phase: 'clean_apply' instead of tier since score_tier has nothing to measure there. Exercises cve-agent's mandatory-analysis path (_handle_clean_apply), not conflict resolution - Wire --roster clean-apply through case listing, cost estimate, and the phase-2 run loop (ROSTER_TIERS), and give retier_roster() a symmetric guard: only accepts a recoverable exit for the three resolution rosters, only a clean exit for clean-apply -- either guard leaves cached stats untouched and warns instead of overwriting real data with a non-signal - Rewrite test_benchmark_roster.py and test_bench_lib.py for the new schema/thresholds; fix generate_benchmark_report.py's per-tier loop to not silently drop a non-easy/medium/hard tier - Backfill 15 CVE-metadata-extractor entries used while building the new rosters (test-cve-metadata-agent.json), and rename the diff- bucket threshold classify_diff_bucket() actually uses to MODERATE_DIFF_LINES_THRESHOLD so it reads as unrelated to score_tier's own thresholds Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Add 'minor' to JUDGEABLE_BUCKETS, so a small (but nonzero) divergence from the reference patch gets an actual meaningful/stylistic verdict instead of being assumed stylistic by bucket alone. 'identical' stays excluded (a zero-line diff has nothing to judge); 'file-mismatch' stays excluded (disjoint filesets leave no shared-file diff to judge) - Update README, report header text, and generate_benchmark_report.py's docstring/section header to say minor/moderate/major/partial - Rename/rewrite the affected tests in test_bench_lib.py and test_generate_benchmark_report.py for the new split Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
claude-sonnet-4.8 is not a model kiro-cli recognizes -- every invocation failed instantly with 'Model does not exist', escalating after 3 retries for every CVE/model combination in the default set. Rename it to claude-sonnet-4.6, which was already present (and valid) as a full-tier entry; drop the now-duplicate full-tier claude-sonnet-4.6 entry since the default-tier one replaces it. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The strip-level retry loop reused one am_result variable, so the PatchError it raised always described the final '-p3 --3way' attempt. Since stripping three path components from a normal -p1 patch produces 'git diff header lacks filename information', every failure in this path was reported with that message regardless of its actual cause, sending debugging down the wrong track. - Record each attempt's stderr as (label, stderr) and report the failure at the *detected* strip level, listing the other attempts for context - Log why each git apply variant (-p1, -C0, --3way) refused the patch; these were discarded entirely, which made an over-broad conflict resolution that no longer applies to the target tree look identical to a malformed patch - Add regression tests for both: the strip-level test fails against the old single-variable behavior Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
A wholesale 'git checkout --theirs' on a heavily diverged file imports upstream's whole file, which can reference APIs the stable branch lacks and break the build on hundreds of unrelated errors. Recovering from that needed the stable file back, and no available command could do it: tee is confined to agent-dir .log files, so a 15k-line file could not be routed back into the tree, and rewriting it through the file-editing tool is not viable at that size. In bench_20260831_140123 this cost two escalations on CVE-2025-1176 (binutils bfd/elflink.c), both models correctly identifying the small surgical fix they wanted to apply but unable to undo the wholesale import first. Allow 'git checkout original-version -- <path>...', pinned to that one ref so it can only roll a tracked file back to its pre-cherry-pick state and cannot become a general checkout primitive. Arbitrary revisions (HEAD~1, branch names, original-version~1), flag injection and the bare pathless form all stay denied. Update AGENT_INSTRUCTIONS.md 'Undoing a Bad Cherry-Pick' case 3 and the conflict.md fragment to prescribe it over the tee + rewrite route. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The exit status collapses four different results into two, and the two
collisions point in opposite directions:
skipped exits 0 -- the model declared the CVE not applicable and
produced no patch. A wrong verdict dismisses a live
vulnerability while still scoring as a clean pass.
escalated exits 14 -- the model declined to guess and asked for a
human, which is the correct outcome when the fix cannot be
made within the allowed file scope, yet it scores as a
failure identical to a real breakage.
Ranking models on the exit code therefore rewards a confident wrong
dismissal over an honest escalation. In bench_20260831_140123 that
inverted the leaderboard: claude-opus-5 showed 4 'failures' and
claude-sonnet-4.6 only 1, but split by outcome opus-5 has zero outright
failures (4 escalations, 2 dismissals) against sonnet-4.6's 4
dismissals -- and three models marked CVE-2024-6387 (regreSSHion, on
openssh 9.6p1, inside the affected 8.5p1-9.7p1 range) not applicable.
- Add bench_lib.parse_agent_outcome, reading the ResultStatus line that
cve_agent/__main__.py already prints; validated against all 110 logs
of that run (exit 0 -> 74 conflict_resolved + 17 skipped, exit 14 ->
14 escalated + 5 failed)
- Add an 'outcome' column to agent_results.csv and log it per run
- Report: add a per-model outcome table and a 'Not-Applicable Verdicts
(verify these)' audit showing how many models agreed on each
dismissal, so a lone dismissal among successful backports is visible
- Older results dirs without the column still report; those rows are
counted as 'no outcome' rather than assumed successful
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
A commit reachable from HEAD shipped in this version, so it cannot be the fix being backported. Replaying one is worse than a no-op: git pits the stale change against whatever superseded it, which surfaces as a large, entirely plausible conflict. CVE-2024-6387 (regreSSHion) is the case that exposed this. Its metadata carries four hashes and the first, 752250caa "revised log infrastructure" (2020), is the commit Qualys identified as *introducing* the regression. Verified against the openssh mirror: it is an ancestor of V_9_6_P1, the recipe's own version. Cherry-picking it produced 30 conflicts across 7 files, and resolving toward the incoming side would have reverted later hardening (faf2b86a4). All five benchmarked models were handed that commit; three concluded from the resulting mess that the CVE was not applicable, which for a live pre-auth RCE is the worst possible outcome. The existing guard only matched a short hash against the output of 'git log --oneline -10', so anything older than ten commits slipped through. - Add git_ops.is_ancestor_of_head using 'git merge-base --is-ancestor', treating only exit 0 as a positive answer so a failed probe (unknown object, no HEAD) never silently discards a viable candidate - Skip ancestors in apply_single_commits and in find_least_conflict_commit, where an ancestor is especially harmful: superseded code can score *fewer* conflicts than the real fix's genuine adaptation work and so be actively preferred Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
A commit touching only the top-level Makefile is a release bump, not a fix -- upstreams like u-boot cut releases by editing the VERSION and PATCHLEVEL variables there. It was not recognised as metadata-only, so find_least_conflict_commit could rank it ahead of a real candidate. CVE-2025-24857 is the case: its sole metadata hash, c253573f3e2 "Prepare v2017.11", changes one Makefile line and is dated 2017, eight years before the CVE. Verified against the u-boot mirror; the actual fix is 87d85139a96 "fs: fat: Perform sanity checks on getsize in get_fatent()", which touches fs/fat/fat.c and names the CVE in its own commit message. Two of five benchmarked models concluded from the resulting nonsense that the CVE did not apply. Add Makefile, Makefile.am and Makefile.in to _METADATA_ONLY_FILES. The check requires *every* file in the commit to match, so a genuine fix that also edits a Makefile is unaffected. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Allowed-file scope is computed from the paths an upstream commit touches, with prefix-variant fallbacks for src/lib/source and monorepo subprojects/. Nothing covered a file the stable branch simply keeps somewhere else, so scope could name a path that does not exist in the tree being patched -- leaving the agent unable to edit the very file it had to fix. libsoup is the case: 3.x holds websocket sources in libsoup/websocket/, while the 2.4 branch behind the libsoup-2.4 recipe keeps them flat in libsoup/. Verified against the libsoup mirror -- the metadata's commits are all 3.x, and the CVE-2024-52532 backport commit on the stable branch uses libsoup/soup-websocket-connection.c. Four of five benchmarked models escalated it, one of them asking for mkdir so it could recreate the upstream directory (which would have been the wrong fix). expand_path_variants now falls back to searching the tree for the basename when the upstream path is absent, accepting only an unambiguous single match -- an ambiguous name such as Makefile or meson.build must never widen scope to the wrong file. The walk skips .git and build output directories and is capped, since scope resolution runs per session. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
apply_single_commits returns the first candidate that cherry-picks, and an irrelevant commit is *more* likely to apply cleanly than a real fix that needs adaptation. The metadata-only check existed but was consulted only by find_least_conflict_commit, the fallback path -- so in the primary path a trivial commit sitting ahead of the real fix won outright. CVE-2024-6387 shows how bad that gets. Its metadata carries four hashes from three repositories (openssh-portable twice, plus the openela-main and hpn-ssh forks). Verified against the openssh mirror at V_9_6_P1: 752250caabda skipped, ancestor of HEAD (the introducing commit) e1f438970e5a skipped, bad object (openela-main fork) 81c1099d22b8 the genuine 9.8 fix; conflicts against 9.6p1 6518797401f2 a 2006 ChangeLog-only commit on the V_4_4 branch, not an ancestor of 9.6p1, and applies trivially Walking that list in order, the real fix conflicts and the 2006 documentation commit applies -- so the corrector reported success having "backported" a twenty-year-old changelog edit as the fix for a pre-auth RCE. Confirmed by replaying both orderings against the mirror. Partition candidates into substantive and metadata-only, and try the substantive ones first. Metadata-only commits are kept as a last resort rather than dropped, preserving the "unless no better option" behaviour find_least_conflict_commit already documented. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
ResultStatus.SKIPPED covers several unrelated situations and only one of
them is a claim by the model, so the not-applicable audit added in the
previous commit accused models of dismissing CVEs they were never asked
about. Of the 17 skipped runs in bench_20260831_140123:
6 ai_not_applicable the model's own reasoning -- the only kind worth
auditing
6 empty_cherry_pick the cherry-pick produced no changes, so the fix
looked already present
5 build_preexisting libpng did not build even unpatched (corrector
exit 10); no model was ever consulted
CVE-2025-64505 accounted for all five environmental skips and so appeared
in the audit as a unanimous 5-of-5 dismissal, which was simply wrong. Two
of the three CVE-2024-6387 dismissals were empty cherry-picks rather than
judgement calls, which is itself a symptom of handing over a commit
already reachable from HEAD -- the case is_ancestor_of_head now filters.
- Add bench_lib.parse_skip_reason and a skip_reason column. The
corrector's exit code outranks the printed AI wording, because the
already-applied path prints the same "Agent concluded ... is not
applicable" line and trusting that string alone misreports a mechanical
skip as a model judgement
- Audit only ai_not_applicable rows (and rows with no recorded reason, so
older results dirs are not silently dropped); report mechanical skips in
their own section
- With this, every remaining genuine dismissal is a lone model out of five,
which is the strongest available signal that each one is an error
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
enrich_metadata_from_oe.py only ever filled CVEs that had *no* hashes, so an entry holding confidently wrong ones was never questioned. But the recipes already carry the answer: a maintainer wrote "Upstream-Status: Backport [<url>]" specifically to name the upstream commit they backported, which makes it the one authoritative source available offline. Audited the 107-entry fixture against the 765 CVE patches in OE-Core (scarthgap): 19 entries did not contain the commit that was actually backported. Verified against local mirrors, the failure modes are CVE-2025-24857 u-boot only "Prepare v2017.11", a 2017 release bump; real fix 87d85139a96 touches fs/fat/fat.c and names the CVE in its message CVE-2024-3596 wpa-suppl. 28 commits from FreeRADIUS and pam_radius, different projects entirely; the recipe needs 9 hostap commits CVE-2024-5569 zipp a merge commit rather than the fix itself CVE-2025-1153 binutils one commit of a three-commit chain Add --correct-existing to prepend the missing ground-truth commits, keeping the tracker hashes behind them as fallbacks, and apply it. Where more than one patch exists the commits are recorded as a `series`, not appended to `hashes`: cve-corrector treats hashes as alternatives and stops at the first that applies, which for a dependent chain leaves a partial fix. CVE-2025-1153 is the clearest case -- three patches whose third reverts part of the first, so applying only the first is worse than applying none. Order comes from the patch filenames, matching how OE applies them (0019-CVE-2025-1153-1.patch .. 0021-...-3.patch, in that SRC_URI order). Eight entries gained a series this way; header-derived SHAs stay out of chains, since they are often local or rebased commits that do not exist upstream. Note that this changes what cve-corrector will cherry-pick for several roster CVEs, so the rosters' recorded conflict_markers/files_involved and tiers are now stale -- run --retier before drawing conclusions from a new benchmark run. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
libpng fails cve-corrector's pre-patch build check in a stock scarthgap environment (EXIT_BUILD_PREEXISTING, exit 10), so the recipe is already broken before any patch is applied. In bench_20260831_140123 all five models were recorded as having "skipped" it without ever being consulted, which is how it came to look like a unanimous not-applicable verdict. A case that fails identically for every model measures nothing about model quality -- the same reasoning selection rule 2 already applies to mirror gaps -- and it cost five agent invocations per run to learn that. extended is now 21 CVEs (10 easy, 2 medium, 9 hard). It remains a superset of balanced and default, which never contained this CVE, so the nesting chain and the 2-per-recipe cap are unaffected. The metadata entry is kept, so re-adding it is a one-line change once the recipe builds. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Re-probed the extended roster against the corrected metadata. Five entries shifted, none across a tier boundary: CVE-2024-6387 openssh 23/14 -> 18/9 no longer replays the CVE-introducing commit CVE-2025-1153 binutils 45/28 -> 52/28 now applies all three commits of the real series CVE-2026-26007 cryptography 16/13 -> 10/7 CVE-2025-47273 setuptools 6/5 -> 7/5 CVE-2026-24049 wheel 2/2 -> 3/2 CVE-2025-1153 is the only one shared with default/balanced; its refreshed stats are copied there verbatim, as test_shared_entries_are_identical requires. CVE-2025-24857 now applies cleanly, so it moves to the clean-apply roster. It previously failed with an empty cherry-pick because its only recorded fix commit was c253573f3e2 "Prepare v2017.11" -- a 2017 release bump, and an ancestor of the recipe's own 2024.01. With the real fix in place (87d85139a96, recovered from the Upstream-Status header of OE-Core's own patch) and ancestors now skipped, the retier log shows it auto-merging fs/fat/fat.c and reproducing the reference backport exactly: "Differences: 0 lines / Patches are equivalent". The retier guard left the stale entry untouched rather than recording exit 0 in a resolution roster, which is what surfaced the move. extended is now 20 (9 easy, 2 medium, 9 hard) and clean-apply 6. Nesting, the no-overlap rule between clean-apply and extended, and the 2-per-recipe cap all still hold. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
CodeQL flagged the substring check as py/incomplete-url-substring-sanitization (high): the host may sit at an arbitrary position in the string. It is a test assertion over a hardcoded input rather than a sanitizer, so it was not exploitable, but a substring match is also a weaker assertion than this test wants -- it would pass for a value that merely mentions the host somewhere, which is exactly the case the test exists to rule out. Assert the exact derived base instead. This subsumes both dropped assertions: a precise match already implies the value is non-None and does not start with a local path separator. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
42 commits adding a model benchmark suite for
cve-agentand fixing thecorrectness problems that building it exposed. The benchmark work came
first; using it as an instrument is what surfaced the rest.
The changes fall into four groups.
1. Benchmark suite for
cve-agent(tests/benchmark/,tools/)Runs a fixed roster of real CVEs across a set of models, then has an AI
judge compare each backport against the human reference patch. Four
committed rosters, with
default ⊆ balanced ⊆ extendedso runs staycomparable, plus a separately-schema'd
clean-applyroster:defaultbalancedextendedclean-applyEvery roster field is measured by a live
cve-correctorprobe(
--retier), never estimated.score_tierkeys off the number ofconflicts and files involved, with thresholds terciling the pool's real
0–52 marker distribution. Resolution rosters contain only CVEs that
actually need resolution (recoverable exits 1/3/4); clean cherry-picks
live in the separate roster so the two measure different things.
Also adds
tools/plot_benchmark_results.pyfor charts andtests/integration/test_patch_compare.py(interdiff-based comparison).2. Scoring correctness
exit_statuscollapsed four distinct outcomes into two, and it did so inthe wrong direction: a not-applicable skip exits 0 and scored as
success, while an honest escalation exits 14 and scored as failure.
Adds an
outcomecolumn parsed from the agent's own status line,validated against all 110 logs of a completed run.
skippedthen turned out to have several causes, so a follow-up addsskip_reason, treating the corrector's exit code as authoritative ratherthan trusting the model's claim. Of 17 skips in that run, only 6 were
genuine model verdicts — 6 were empty cherry-picks and 5 were a recipe
that does not build unpatched. After the split, every genuine dismissal is
a lone model out of five.
3.
cve-correctorpicking the wrong commitFour bugs that all let a plausible-looking non-fix through:
recorded hash is the CVE-introducing commit and a confirmed ancestor of
the recipe's own tag; picking it produced 30 conflicts across 7 files.
Adds a real
git merge-base --is-ancestorcheck (the old guard was ashort-hash substring match against
git log --oneline -10).metadata-only check was only consulted by the least-conflict fallback,
never by the primary path. For CVE-2024-6387 that path returned a 2006
ChangeLog-only commit on an unrelated branch as the fix for a pre-auth
RCE, because it applied trivially. Verified by replaying both orderings
against a real mirror.
Makefile-only commit as a version bump. CVE-2025-24857'sonly recorded hash was "Prepare v2017.11".
git amfailure. The strip-level retry loop reusedone result variable, so every failure was reported as the last
-p3 --3wayattempt regardless of cause.4. Metadata ground truth from OE-Core
Audited the metadata fixture against 765 OE-Core CVE patches, using their
own
Upstream-Status: Backport [url]headers as ground truth. 19 entriesdid not contain the commit that was actually backported. Adds
--correct-existingto the enrichment script (it previously only filledempty entries).
Where a CVE needs more than one commit the result is recorded as an ordered
series, not flathashes—hashesare alternatives that stop at thefirst that applies, whereas a series must apply in full. 8 series and 11
single-commit prepends.
The clearest case is CVE-2025-1153 (binutils, in all three resolution
rosters): the metadata held one commit, but the real fix is a three-commit
chain whose third commit reverts part of the first. That explains why all
five models failed it.
Testing
ruff check .— cleanmypy cve_agent cve_corrector cve_metadata_extractor shared— clean, 52 filespytest --cov— 2058 passed, 6 skipped, 82.67 % coverage (threshold 65 %)bash -n tests/benchmark/run_benchmark.sh— cleanno overlap between
clean-applyandextended, per-recipe capgit amone was confirmed tofail against the pre-fix code before being kept
--list-casesNotes for review
interdiff(patchutils) andmatplotlibare optional and degrade gracefully.
tests/benchmark/README.mddocuments roster selection, tieringthresholds, and per-roster run cost.
for being hard: one whose recipe fails its pre-patch build check, and one
that now applies cleanly and moved to the
clean-applyroster.