cve-agent: native OpenAI-compatible backend + corrector/agent reliability fixes - #105
Merged
Merged
Conversation
- validate typed file operations against exact authorized roots - bound inspection, search, mutation, and model-visible results - share a session deadline across filesystem operations - cover path traversal, symlink, race, size, and policy failures Assisted-by: codex:gpt-5
- expose fixed inspection and cherry-pick operations without arbitrary argv - constrain revisions, pathspecs, environment, output, and deadlines - validate staged and changed paths against the authorized source scope - cover repository state, conflict, rollback, and hostile input paths Assisted-by: codex:gpt-5
- compose file and Git runtimes behind one closed host contract - run devtool builds with bounded logs, process groups, and deadlines - require interactive approval for side effects when configured - create host-owned terminal conclusions only after state validation Assisted-by: codex:gpt-5
- resolve CLI, environment, and portable endpoint defaults explicitly - require a model and validate API-key indirection without exposing secrets - gate remote endpoints and insecure transport behind separate opt-ins - keep endpoint availability checks local and network-free Assisted-by: codex:gpt-5
- send portable non-streaming requests through the existing requests dependency - validate response headers, sizes, JSON shape, messages, and tool calls - bound retries by one shared deadline and classify transport failures - redact bearer and configured credentials from diagnostics Assisted-by: codex:gpt-5
- preserve validated assistant and tool messages across bounded turns - dispatch only advertised typed tools and validate argument trees - retain a protected bounded JSONL transcript for every session - terminate on protocol, deadline, budget, and no-progress failures Assisted-by: codex:gpt-5
- register the backend and expose validated CLI configuration - assemble backend-specific tool instructions without changing Kiro or Claude - connect the client, loop, typed runtime, transcript, and guarded session - cover configuration, end-to-end runtime, compatibility, and live opt-in paths Assisted-by: codex:gpt-5
- run hook removal, scope reversion, and audit generation on errors - retain the primary exception after all cleanup attempts - surface cleanup failures instead of silently skipping guards Assisted-by: codex:gpt-5
- Exercise the native backend through a loopback HTTP server and real Git - Cover conflict resolution, protocol failures, cleanup, and redaction - Separate session tool budgets from single-response client bounds - Isolate the optional live Ollama smoke test in a disposable workspace Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- Add copy-pasteable local and remote endpoint configuration - Describe portable API, security, approval, and transcript contracts - Reconcile maintainer summaries and backend plugin guidance - Test documented options, precedence, examples, and response shape Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- Add constrained follow-up commit and amend operations - Preserve build validity when recording already-built source changes - Reject unsafe index state, conflicts, messages, and Git metadata - Cover repair workflows, failures, approval, and adversarial inputs Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- reserve and validate openai-<profile> backend selectors - securely read strict bounded profile files - validate portable chat and Ollama profile settings Assisted-by: codex:gpt-5
- inspect and prepare dedicated model aliases through fixed APIs - verify context and tool capabilities under one session deadline - require approval for model creation and recreation Assisted-by: codex:gpt-5
- resolve profile selectors through CLI and context generation - apply allowlisted chat settings to portable requests - prepare configured Ollama aliases before model sessions Assisted-by: codex:gpt-5
- add a strict schema-v2 outcome and failure taxonomy - migrate legacy JSON and integration CSV results without inventing verification - expose unverified workflow completion in logs, summaries, and resume gates Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- create restrictive result directories and transcripts before preflight - mirror bounded provider, tool, mutation, build, and lifecycle events - centralize redaction, telemetry, secret scanning, and artifact hashes Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- capture complete NUL-safe repository state before backend handoff - classify initialization failures with stable actionable codes - reproduce and resolve large dirty-status initialization failures Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- bind agent scope to a versioned manifest with repository fingerprints - require explicit merge mainlines and narrowly restore generated files - reject drift and out-of-scope dirt before provider invocation Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- map source paths with exact, configured, or strong content anchors - apply bounded text adaptations with rollback and exact path postchecks - retain transfer manifests and propagate mapped paths into agent scope Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- validate commit, amend, and cherry-pick parent/tree transitions before advancing trusted state - keep build freshness tied to content mutations rather than staging or commit IDs - bind later-session baselines to validated handoff provenance and durable audit artifacts Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- add SHA-256-guarded unique-context edits without raising the full-file write limit - enforce strict path, text, size, line, race, atomicity, and postcondition checks - cover interactive approval and the deterministic HTTP-to-finish workflow Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
Add trusted offline reference manifests and conservative semantic validation for completed agent repairs. Persist bounded review artifacts, gate release acceptance on verified or equivalent outcomes, and cover exact fixes, adaptations, prerequisites, omitted tests, path mappings, merge parents, and false-positive patterns. Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
Replace call-tuple progress heuristics with canonical host evidence digests, bounded state summaries, staged no-progress handling, and classified budget failures. Preserve terminal validation budget after late mutations and add hashed targeted context with bounded build diagnostics and explicit security prerequisites. Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- describe an allowlisted portable provider dialect - retain bounded credential-free failure evidence - version and digest the capability contract Assisted-by: codex:gpt-5
- encode only profile-declared request fields - classify bounded HTTP and protocol failure evidence - preserve reasoning replay and shared loop counters Assisted-by: codex:gpt-5
- exercise fixed chat, tool-call, continuation, and final turns - avoid repository data and arbitrary provider-specific fields - report stable capability error codes Assisted-by: codex:gpt-5
- validate capability and conformance-probe sections - constrain fallback selectors to one distinct named profile - reject nested and unsupported fallback policy values Assisted-by: codex:gpt-5
- preserve one deadline, transcript, tool runtime, and loop budget - validate trusted Git state before a declared fallback - limit fallback to classified provider failures and profile policy Assisted-by: codex:gpt-5
Run baseline health, full backend, crossover, fallback-policy, and stability campaigns from immutable manifests and independent clean snapshots. Keep legacy resume imports explicitly unverified and prevent rows from being copied across backend cohorts. Aggregate decomposed trusted telemetry into deterministic security-first JSON, CSV, and Markdown reports with strict comparison guards and separate baseline denominators. Assisted-by: codex:gpt-5 Signed-off-by: Anders Heimer <anders.heimer@est.tech>
- document deterministic hostile-input and provider checks - connect evaluation evidence to release acceptance - align component and workflow summaries with the gate Assisted-by: codex:gpt-5
test_merge_paths_require_and_verify_explicit_mainline built a merge by checking out "master", but _repo runs a plain "git init" -- so the branch is whatever init.defaultBranch says. On a host configured with "main" the checkout failed in setup and the test errored before reaching any assertion about mainline selection, which is what it exists to cover. Ask git for the name instead, as the sibling merge test in this file already does. Verified passing with init.defaultBranch set to both main and master. Assisted-by: kiro:claude-opus-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Detect a rewritten commit subject directly, not only via a recognized Conflicts Resolved: marker, so an AI-fabricated commit with no marker at all is still caught - Reset --author and GIT_AUTHOR_DATE to the upstream commit's identity when restoring the message, since git commit --amend never changes authorship on its own - Without this, an AI backend that creates a brand-new commit instead of amending the cherry-picked one keeps its own author/date/subject in the exported patch's From:/Date: headers even after message-text restoration Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Issue one 'git checkout devtool -- <files...>' call per chunk of up to 200 files instead of one subprocess per missing file, and the same for the unstaging 'git reset HEAD' step - Trees with hundreds of missing generated files (autotools output, secondary-tarball payloads) previously paid one process spawn per file; this collapses that to a handful of calls - Chunked at 200 files per call to stay under typical platform ARG_MAX limits for very large file sets Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- _verify_and_commit() recreates the transferred change as a brand-new commit via 'git commit -F' on the target branch, which (unlike cherry-pick or am) always authors as the ambient git identity — every transfer was silently discarding the original contributor's name, email, and date regardless of what came before it in the pipeline - Set GIT_AUTHOR_NAME/EMAIL/DATE from the source commit before creating the transferred commit, so the exported patch's From:/Date: headers match the upstream author instead of whoever ran the corrector - Committer identity is left untouched — only authorship of the transferred change is restored This is the actual root cause behind the CVE-2025-6021/libxml2 case: the CVE-branch commit's author was already correct by the time cherry_pick_to_devtool() ran, but transfer_commits() then recreated the commit on the devtool branch — the branch devtool finish exports the patch from — discarding it every time. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Add AIBackend.verify() -> VerifyResult, defaulting to an is_available() presence check so extra/ plugin backends keep working unmodified - Implement real checks for kiro/claude: a bare CLI invocation (no agent config, tool allow-lists, or workspace) asking for a fixed marker string back, run in a hermetic temp directory - Implement openai backend's check by reusing the existing ProviderConformanceProbe instead of a new minimal probe, since that backend runs tools in-process and tool-call conformance is its meaningful health check - Wire --verify-backend into the existing required --cve-id/--cve-list mutually exclusive group so argparse enforces exactly one of the three with no new validation code; prints a pass/fail line and exits 0/EXIT_AGENT_ERROR without touching the CVE workflow - Document the flag in README.md and the AIBackend.verify() plugin hook in extra/README.md Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Resolve and seed GIT_AUTHOR_*/GIT_COMMITTER_* from the operator's global git config at cve-agent startup (shared.git_runner. resolve_git_identity, cve_agent.__main__._seed_git_identity_env), fixing 'Committer identity unknown' failures in sandboxed openai- backend sessions that otherwise have no usable Git identity. Explicit env vars set by the caller are never overwritten, and the sandboxed executor still never reads global/system git config directly. - Add a revert_to_baseline typed tool (openai_git_tools.py) that lets the model discard its own typed file changes back to the session baseline before escalating via finish(needs_human/not_applicable), reusing the existing _discard_typed_mutations rollback internals instead of duplicating them. - Document both in AGENT_INSTRUCTIONS.md, docs/openai-compatible- backend.md, and README.md. - Add an autouse fixture in tests/conftest.py to snapshot/restore Git identity env vars around every test, preventing the new startup seeding from leaking a real global identity into unrelated tests. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Add cve_agent/openai_console.py with a pure, total format_console_line() dispatching a small subset of transcript events (tool_request, tool_result, terminal_result, session_end, session_error, progress_warning, retry, http_failure) to terse [#sequence]-prefixed lines - Add an optional console writer to JSONLTranscript, delegated to from write() after the mandatory audit write succeeds and wrapped in contextlib.suppress(OSError) so console output can never affect the audit path; default None preserves existing behavior - Wire console=None if interactive else <stdout writer> into both the primary and fallback transcript_factory() call sites in OpenAICompatibleBackend, so non-interactive sessions stream live tool-call progress while interactive sessions stay silent on this path, with no new flag - Document the new streaming behavior and list the new module in AGENTS.md's directory map - Add unit and end-to-end tests covering line formatting, transcript byte-identity with/without a console writer, OSError suppression, and live stdout streaming for non-interactive/interactive/fallback sessions Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- Add an assistant_response formatter to openai_console.py that
prints the model's visible commentary text ("model: <text>") for
turns where it produced any alongside its tool calls; tool-call-only
turns (empty or missing content) are skipped without printing an
empty line
- The full, untruncated text is printed, matching what is already
recorded in the JSONL transcript's assistant_response.content field
- The provider's hidden/opaque reasoning-replay field remains
unlogged and unstreamed; only the model's already-visible content
is surfaced
- Update docs/openai-compatible-backend.md to describe the expanded
streamed subset
- Extend unit and end-to-end tests to cover the new formatter,
including the tool-call-only skip case and silence during
interactive sessions despite commentary being present
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Upstream commit 3fb6b31c7166 itself has a defect: it strips ".." from every link_target, including symlink targets, which legitimately may start with "../" or "/". The only correct fix for the resulting ptest regression requires gating the strip to hardlinks only (typeflag '1') -- a real, necessary deviation from the literal upstream diff. Semantic validation correctly cannot auto-verify a deviated patch against upstream, so the release gate reports SECURITY_REVIEW_REQUIRED and cve-agent exits 14 (EXIT_AGENT_ERROR) rather than 0, even though the backport is correct and fully built/tested. Update the test's expected exit code and the README test table to match this by-design outcome instead of treating it as a failure. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
cherry_pick_to_devtool() cleans and checks out the superproject before calling transfer_commits(), but git clean/checkout never touch a submodule's own working tree. Recipes whose upstream vendors dependencies as submodules (e.g. coreutils' gnulib) can leave the submodule checked out at the wrong commit or with leftover untracked content after switching to devtool, so transfer_commits()'s strict clean-tree check aborts with TRANSFER_APPLY_FAILED: target tree is not clean -- an unrecoverable exit that never gives the agent a chance to retry. Add reset_submodules() (git_ops.py), a no-op unless .gitmodules registers submodules, that re-inits and force-resets them to match the superproject. Call it in cherry_pick_to_devtool() right after checking out devtool. Add a regression test with a real git submodule fixture reproducing the coreutils/CVE-2024-0684 scenario; verified it fails without the fix and passes with it. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
…lure transfer_commits() computed 'git status --porcelain=v2 -z' to check the target tree is clean before planning a transfer, but discarded the output entirely -- a failure surfaced only as the bare string 'target tree is not clean', with no path to start debugging from. This made diagnosing dirty-submodule cases (see the reset_submodules fix) purely guesswork from the corrector's log alone, since git_clean_workspace/ git checkout calls are never logged. Add _summarize_dirty_status() to parse the porcelain v2 record shapes (ordinary changes, untracked/ignored, and submodule S... entries) and include the offending paths (bounded by the existing MAX_TRANSFER_DIAGNOSTIC_PATHS) in the TransferError message. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The only conflicting line when cherry-picking the openssh fix is OpenBSD's "$OpenBSD: clientloop.c,v ..." CVS version-stamp tag -- present on essentially every upstream openssh commit and always diverging from the stable branch's own stamp. No conflict resolution can reproduce the literal upstream diff exactly here: keeping upstream's stamp value differs from what the branch's own pre-image expected, and keeping the branch's value differs from what upstream's diff explicitly sets. Semantic validation correctly flags this as a real (if functionally inert) byte-level deviation and reports SECURITY_REVIEW_REQUIRED, so cve-agent exits 14 rather than 0 even though the backport is correct and builds cleanly (devtool finish succeeds, patch exported). Update the test's expected exit code and the README test table to match this by-design outcome, same as the CVE-2026-26157 case. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The corrector applies this monorepo commit cleanly with zero conflicts and zero AI involvement, but devtool finish deletes the workspace directory as its last step on EXIT_SUCCESS. By the time the agent's get_workspace_path() checks for it, there is nothing left to run semantic validation against. Per docs/result-schema.md this is by design: build success alone is never proof the security fix is preserved, so the outcome is security_status=not_evaluated (WORKFLOW_COMPLETED_UNVERIFIED) rather than verified/equivalent, and cve-agent exits 14 (EXIT_AGENT_ERROR) under the default --security-gate=equivalent even though the backport is a byte-for-byte clean upstream cherry-pick with no deviation at all. Update the test's expected exit code and the README test table to match, same pattern as the CVE-2026-26157 and CVE-2024-39894 cases. Note: unlike those two cases, nothing here required a real deviation from upstream -- the unverified status is purely an artifact of workspace-cleanup timing. Whether get_workspace_path's already- finalized fallback should preserve enough evidence for a genuinely clean, zero-deviation corrector-only apply to reach 'equivalent' automatically is a separate design question, out of scope for this test-expectation fix. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The header scenario list (tests 6, 11, 13) still described the old, now-incorrect exit-0 expectations after the inline comments and run_test calls were updated to exit 14 for the CVE-2026-26157, CVE-2024-39894, and CVE-2024-44331 agent cases. Add a short pointer from each header entry to its inline comment/rationale so the two stay consistent. Also flag test 7 (CVE-2024-0684 / coreutils) in the header: its expected exit is still under investigation (TRANSFER_APPLY_FAILED / exit 5 from a dirty gnulib submodule), so the header no longer implies 0 is confirmed correct for it. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The previous reset_submodules() fix (81a4fe8) gated all cleanup behind '.gitmodules exists in the current working tree'. That check reads .gitmodules from whatever branch is currently checked out -- but by the time reset_submodules() runs, cherry_pick_to_devtool() has already switched to devtool. Real-world coreutils/CVE-2024-0684 confirmed the actual failure mode: devtool's branch (built from a release tarball) has no .gitmodules and no gnulib gitlink at all, while the CVE branch's upstream history does vendor gnulib as a real submodule. The old check saw no .gitmodules on devtool and no-opped, leaving the already-initialized gnulib/ directory on disk. git status then reports it as a plain untracked directory (?? gnulib/), not a submodule status, and ordinary 'git clean -fdx' silently leaves it because git refuses to descend into a directory containing its own .git unless forced twice -- so transfer_commits' clean-tree precheck still failed with the exact message seen live: 'target tree is not clean: gnulib/'. Fix: keep resetting submodules that *are* registered on the current branch (the original wrong-commit case), and unconditionally run 'git clean -ffdx' (double force) afterwards to remove any leftover nested-repository directories the ordinary single-force clean cannot touch, regardless of whether the current branch declares them. Add test_transfer_survives_submodule_untracked_on_devtool, reproducing this exact scenario (devtool branch with no gnulib path at all). Confirmed it fails with the old implementation with the identical 'target tree is not clean: gnulib/' message, and passes with the fix. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
With the gnulib submodule fix (31e5b88) in place, a live rerun confirms cve-corrector now completes CVE-2024-0684 cleanly end to end: the upstream commit cherry-picks and transfers with zero conflicts ('Transfer verified: 1 mapped entries, 1 final paths'), the build and before/after ptest both pass (504 passed, 0 failed), and devtool finish exports the patch successfully (cve-corrector exit 0). Because the corrector needed no AI help at all, devtool finish removes the workspace before the agent's get_workspace_path() can find it to run semantic validation, so the outcome is security_status=not_evaluated (WORKFLOW_COMPLETED_UNVERIFIED) rather than verified/equivalent, and cve-agent exits 14 (EXIT_AGENT_ERROR) under the default --security-gate=equivalent. This is the identical by-design pattern already fixed for the CVE-2024-44331 case (test 13). Update the test's expected exit code, its header comment, and the README table to match, and drop the 'under investigation' note now that the underlying submodule bug is confirmed fixed. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
…test Test 7 targets the gnulib submodule dirty-tree fix (cve_corrector/git_ops.py's reset_submodules), which lives entirely in cherry_pick_to_devtool/transfer_commits -- a step that runs unconditionally regardless of --skip-build/--skip-ptest. Those flags only gate devtool build and ptest execution, and cve-agent has no --skip-build flag at all, so the previous agent-runner form of this test paid for a full build+ptest cycle (or, before that, a full build+ptest cycle that never even completed the corrector's success path) without exercising anything extra relevant to the fix under test. It also meant the test's real outcome was cve-agent's post-hoc 'workspace already finalized' exit 14, one hop removed from the actual regression being verified. Switch to invoking cve-corrector directly with --skip-build --skip-ptest, matching the pattern already used by tests 1, 9, and 12 for corrector-only regression cases. This exercises the identical cherry_pick_to_devtool/transfer_commits path in a few seconds with no bitbake invocation at all, and asserts cve-corrector's own exit code (0) directly instead of cve-agent's semantic-validation exit code. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
When cve-corrector applies a CVE fix with no conflict, it runs
devtool finish itself before cve-agent gets a chance to intervene,
removing the source workspace (and its git history) before
_run_cve_pipeline's get_workspace_path() check ever runs. That branch
returned SUCCESS with no outcome argument, so CveResult.__post_init__
fell back to the legacy-status shim migrate_legacy_status("success",
build_evidence=True), which reports SecurityStatus.NOT_EVALUATED with
no failure_class/failure_code at all.
This is indistinguishable, to any downstream reader of the durable
result.json (in particular tests/benchmark/run_benchmark.sh's
benchmark_artifact_outcome and cve_agent/__main__.py's
security_gate_satisfied), from a run that never attempted semantic
validation for an unknown reason. Since the default --security-gate
is 'equivalent', every one of these clean, zero-conflict,
zero-AI-involvement successes exits 14 (EXIT_AGENT_ERROR) -- the same
exit code as a genuine build failure or a rejected fix -- even though
nothing went wrong and no model was ever consulted. A recent benchmark
run showed this exact pattern identically across all 5 models for
CVE-2025-47183, confirming it is a deterministic corrector-path gap,
not a model-quality signal.
Route this branch through the same optional-evidence machinery the
other completed paths already use: _record_semantic_validation(None,
None, ...) naturally falls through validate_semantic_result's
existing manifest-is-None branch, since building a real
ReferenceManifest/GeneratedSnapshot is structurally impossible once
the workspace is gone. This produces an honest
NOT_EVALUATED/comparison_artifact_missing outcome (with a real
failure_class/failure_code) instead of the unreasoned legacy shim, so
it's distinguishable from a genuine rejection/failure without
fabricating comparison evidence that can't exist.
Add test_clean_apply_already_finalized_reports_honest_outcome, which
fails without the fix (failure_code is None) and passes with it.
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The runner selected the per-run artifact directory by open-coding
"$artifact_data_root/results/cases/$cve_id", but
shared.paths.data_dir() appends the application component
('yocto-security-tools') to CVE_TOOLS_DATA_DIR, and
cve_agent.artifacts.RunArtifacts creates runs under
data_dir()/results/cases/<cve>. The probed path therefore never
existed: iterdir() raised FileNotFoundError, the 'exactly one run
directory' check saw zero, and artifact_dir was silently set to empty
for every run.
Both consumers of artifact_dir then degraded without any trace:
* benchmark_artifact_outcome() was never called, so durable_summary
stayed '-' and the exit-status override never fired -- rows kept
the raw process exit code, which is 14 for any outcome the default
--security-gate does not accept, including completed workflows.
* candidate_ready stayed false, so the generated-vs-reference patch
comparison only ran for rows whose raw exit was already 0.
In bench_20260904_165741 that left 93 of 100 rows at
diff_bucket='-'. Since filter_for_judging() only judges
minor/moderate/major/partial, the judge phase evaluated exactly one
row out of 100. The 7 rows that did get a bucket correlate perfectly
with the 7 whose raw exit status was 0, confirming candidate_ready was
never true for any row.
Derive the case root through data_dir() so the layout cannot drift
from the agent again, and report a diagnostic on stderr (appended to
the run log) naming the case root, walking the ancestor chain, and
listing the deepest directory that does exist. stdout stays the sole
channel for the resolved path, since anything printed there is
captured into artifact_dir itself.
Add two tests that execute the runner's own extracted selection
program rather than reimplementing it: one asserts it resolves a real
RunArtifacts directory (fails with the open-coded join restored), the
other asserts a miss is attributable from stderr alone.
Assisted-by: kiro:claude-sonnet-5
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
run_benchmark.sh compares each generated backport against the human reference at the end of every run, while the patch is still in the OE tree. When that comparison does not happen the row keeps diff_bucket='-', and since filter_for_judging() only judges minor/moderate/major/partial, the judge phase silently skips it. In bench_20260904_165741 that left 1 judged row out of 100. Re-running the agent to recover those rows costs the full model spend again, and it is unnecessary: every attempt already writes its final candidate commits to a durable host-owned artifact directory (final-commits.patch), which survives reset_oe_tree. Add backfill_diffs.py, which replays only the comparison step from those artifacts. It needs no Yocto environment, no OE tree and no model calls: the reference side comes from the bench_<cve>_*.patch copies the run already saved, and the generated side from the artifact directory. It writes the per-model *_differences.txt and *_differences_diff.patch files the judge phase reads, updates agent_results.csv in place (keeping the original as .csv.orig), and leaves rows that already have a bucket untouched unless --force. Bucket rules mirror the runner rather than inventing new ones: a durable SKIPPED state has no candidate, a completed built attempt is compared even when the release gate rejected it, and any other non-comparable outcome (escalated, failed) keeps '-'. Artifact provenance is validated with the same manifest binding the live runner uses, so an unrelated directory is never consumed. On bench_20260904_165741 this recovers 55 judgeable rows from 1. The 26 rows that stay '-' account exactly for the durable outcomes that cannot be compared: 12 escalated, 7 with an empty artifact, 5 failed, and 2 without a valid artifact directory. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
The outcome column was empty for all 100 rows of bench_20260904_165741. parse_agent_outcome() matches the old '<check> <cve>: <status>' line, but cve_agent/__main__.py now prints '<cve>: <summary_state>' (e.g. 'CVE-2024-7537: SECURITY_VERIFIED'), so the regex never matched and every row recorded ''. That silently disabled the safeguard the column exists for. As bench_lib already documents, the process exit status conflates 'skipped' (exit 0, but no backport -- possibly a live vulnerability dismissed as not-applicable) with a real success, and 'escalated' (exit 14, the correct answer for an out-of-scope fix) with a genuine breakage. With the column blank, generate_benchmark_report.py could not tell them apart, and its not-applicable audit -- gated on outcome == 'skipped' -- never ran at all. summary_state cannot be reverse-mapped to recover it: it returns SECURITY_REVIEW_REQUIRED for both a completed-but-review-required run and a WorkflowStatus.ESCALATED one, which is exactly the conflation being guarded against. The durable result.json already carries the ResultStatus verbatim in legacy_status, so read it there instead, following the same optional-artifact pattern count_tool_calls() uses. The log stays a fallback for runs old enough to print the legacy line, and an unreadable artifact yields '' rather than a guess. Also recover the column for finished runs in backfill_diffs.py, and keep a row's existing bucket while doing so: that comparison ran against the real OE tree and stays authoritative, whereas the outcome was never populated for any row. On bench_20260904_165741 this recovers 76 conflict_resolved, 12 escalated, 5 success and 5 failed, matching the legacy_status distribution in the artifacts exactly; the remaining 2 have no valid artifact. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
classify_outcome() returned 'failed' for any exit_status other than '0'. That column no longer holds a pass/fail flag: run_benchmark.sh records the durable ResultOutcome.summary_state there whenever one exists, so a run that completed and produced a comparable patch still reports e.g. SECURITY_REVIEW_REQUIRED when the release gate declined to accept it. In bench_20260904_165741 that wrote off 93 of 100 rows as failures, 45 of which had completed and produced a patch -- including CVE-2024-7537/claude-sonnet-4.6, whose backport differs from the human reference by 5 lines. Read the run's disposition from the outcome column instead, which now carries cve-agent's own ResultStatus from result.json, and reserve 'failed' for the states that really produced nothing to compare (WORKFLOW_FAILED, HOST_INITIALIZATION_ERROR, PROVIDER_TIMEOUT, AGENT_NO_PROGRESS, TIMEOUT, SETUP_FAILED, plus a bare non-zero exit code when no durable outcome was recorded). Completed runs are scored on patch equivalence, which is what run_benchmark.sh already intends by keeping them comparable. Add two categories rather than folding them into existing ones: * escalated -- the model declined to guess and asked for a human. bench_lib documents this as the correct answer for an out-of-scope fix, so scoring it identically to a crash punishes the honest result. * gate-rejected -- the host's semantic validation rejected the result. This can accompany a patch textually identical to the reference (rejection grounds such as a missing prerequisite are invisible to a diff), so 'equivalent' would hide a real security finding while 'failed' would misreport a correct patch. Four rows in this run are exactly that shape. The 'broken' column in the text summary stays failed + no-patch. Rows without an outcome column keep their previous classification, so older result directories read unchanged. Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
This reverts commit 16449d7.
- CVE-2025-47183 (gstreamer1.0-plugins-good) was tiered as an "easy" conflict-resolution case (conflict_markers: 1, exit_code: 1) in benchmark-roster-extended.json, but bench_20260904_165741 shows every model's cve-corrector run actually exits 0 with a clean cherry-pick apply and no conflict at all ("Clean apply (workspace already finalized)") - Per bench_lib.py's own documented design, a clean exit "is not a resolution case at all" and belongs in the clean-apply roster instead, so no per-model diff/judge comparison was ever generated for it in the extended roster, silently excluding it from judge_results.csv - Move the entry to benchmark-roster-clean-apply.json with the correct schema (phase: "clean_apply", diff_lines: 0, series_len: 1) and drop it from benchmark-roster-extended.json - Update tests/benchmark/README.md's roster tables/counts (extended: 20 -> 19 CVEs, 9 -> 8 easy; clean-apply: 6 -> 7 CVEs) and test_benchmark_roster.py's EXPECTED_SIZES to match Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
- _extract_files_touched() only recognized diff --git a/<path> b/<path> headers, so a reference patch using plain diffutils -p / Index: style (no diff --git line, --- / +++ paths carrying a version-numbered tarball directory instead of git's a/ / b/) returned an empty file set - This made compare_patches_detailed() misreport every file the reference patch touches as "missing in generated" relative to a generated patch that in fact touches the exact same file, forcing classify_diff_bucket() to bucket the run as file-mismatch and skip it entirely from judge_results.csv - Confirmed against the real CVE-2025-1176 (binutils) reference patch from bench_20260904_165741: before the fix, _extract_files_touched() returned an empty set for bench_CVE-2025-1176_CVE-2025-1176.patch (an Index:-style patch imported from Ubuntu's debian/patches/); after the fix it correctly returns {'bfd/elflink.c'}, matching the generated patch's diff --git header - Fall back to the +++ header for patches without a diff --git line, stripping a/ / b/ when present or otherwise the leading tarball directory component (patch -p1 semantics); skip /dev/null so a deleted file's +++ header isn't reported as a bogus path - Add regression tests for the Index:-style case and for /dev/null headers in tests/integration/test_patch_compare.py Assisted-by: kiro:claude-sonnet-5 Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
Signed-off-by: Daniel Turull <daniel.turull@ericsson.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a native OpenAI-compatible backend for
cve-agent(typed tool loop, bounded Chat Completions client, Git/filesystem/build tools, named provider profiles, Ollama support) and a large set of relatedcve-corrector/cve-agentreliability fixes and test coverage built on top of it:openaibackend: bounded chat client, typed Git/filesystem/build/commit tools, agent tool loop, named profile parsing (etc/openai-<profile>.cfg), provider capability dialects/probe/fallback, Ollama provisioningSize note
This branch is 67 commits / ~39k lines across 143 files (
git diff --stat origin/main..feature/openai+fixes). The work is highly sequential — later commits (fixes, tests) depend on backend/infra introduced earlier in the same branch — so it is submitted as a single PR rather than split arbitrarily, which would risk non-buildable intermediate states. Happy to split into smaller reviewable PRs along natural seams (e.g. native OpenAI backend core vs. evaluation/benchmark tooling vs. corrector reliability fixes) if maintainers prefer — let me know which boundaries you'd like.Testing
Existing test suite covers the new code (
tests/agent,tests/corrector,tests/benchmark,tests/integration); see individual commits for per-change test additions.