Traces GitHub Security Advisories to the commits and PRs that introduced and fixed them.
- Python 3.12
- uv
- A GitHub personal access token (public read access is enough): https://github.com/settings/tokens
- An Anthropic API key (the tracing agent runs on Claude): https://platform.claude.com/
uv syncCreate a .env file from the template and fill in your keys:
cp .env.example .env| Variable | Purpose |
|---|---|
GITHUB_TOKEN |
Authenticates GitHub API requests |
ANTHROPIC_API_KEY |
Authenticates the Claude agent that completes the trace |
Both variables are required. Exported environment variables take precedence over .env.
The rest of this README uses uv run python -m code_audit ... for every example, which is
the development workflow set up above. Two additional ways to run the CLI without cloning
the repository are described next; both expose the exact same commands.
Install the package directly from this repository with pip (a virtual environment is
recommended):
pip install git+https://github.com/sinachaichi/code-audit.gitOr, from a local checkout:
pip install .Either way, this installs a code-audit console script that behaves identically to
uv run python -m code_audit:
code-audit trace GHSA-jfh8-c2jp-5v3q
code-audit trace-many GHSA-jfh8-c2jp-5v3q GHSA-36p3-wjmg-h94x
code-audit evalGITHUB_TOKEN and ANTHROPIC_API_KEY still need to be set in the environment (or a .env
file in the current directory); installing the package does not configure them.
Build the image from the repository root:
docker build -t code-audit .The image's entrypoint is the code-audit console script, so a container behaves like the
installed CLI directly. Credentials are never baked into the image; pass them at run time:
docker run --rm -e GITHUB_TOKEN=... -e ANTHROPIC_API_KEY=... code-audit trace GHSA-jfh8-c2jp-5v3qor with an env file:
docker run --rm --env-file .env code-audit trace GHSA-jfh8-c2jp-5v3q--debug writes its transcript to debug/ inside the container's filesystem, which
disappears with --rm unless a host directory is mounted over it. Mount a local ./debug
directory to read the transcripts afterward:
docker run --rm --env-file .env -v $(pwd)/debug:/app/debug code-audit trace GHSA-jfh8-c2jp-5v3q --debugThe same applies to trace-many --from-file: the file has to be visible inside the
container, so mount its directory and refer to it by the mounted path:
docker run --rm --env-file .env -v $(pwd):/app/host code-audit trace-many --from-file /app/host/advisories.txtFetch an advisory and print it as JSON:
uv run python -m code_audit advisory GHSA-jfh8-c2jp-5v3qAdd --pretty to print a short human-readable summary (the GHSA id, summary, severity,
and source location, one per line) instead of the full JSON:
uv run python -m code_audit advisory GHSA-jfh8-c2jp-5v3q --prettyThe default (no --pretty) output is unchanged.
Trace an advisory to its introducing and fixing changes. Deterministic tracing from the advisory references runs first; a Claude agent then investigates only the fields that remain unknown, and every finding is verified against the GitHub API:
uv run python -m code_audit trace GHSA-jfh8-c2jp-5v3qExpected output (trimmed):
{
"ghsa_id": "GHSA-jfh8-c2jp-5v3q",
"introducing_commit": {
"sha": "f1a0cac60f1e41347c9bced7c1470be488840344",
"message": "LOG4J2-313 - Add JNDILookup",
"files": ["..."]
},
"introducing_pull_request": null,
"fixing_commit": {
"sha": "c77b3cb39312b83b053d23a2158b99ac7de44dd3",
"message": "Restrict LDAP access via JNDI (#608)",
"files": ["..."]
},
"fixing_pull_request": {
"number": 608,
"title": "Restrict LDAP access via JNDI",
"state": "closed",
"merged": true
}
}Fields stay null when no confirmed answer exists, for example an introducing change
that predates pull requests.
Add --pretty to print a short human-readable summary instead of the full JSON: one
line per field (introducing_commit, introducing_pull_request, fixing_commit,
fixing_pull_request) with a short SHA or PR number, the first line of the commit
message or PR title, and a file or state count, or not found when a field is null:
uv run python -m code_audit trace GHSA-jfh8-c2jp-5v3q --prettyintroducing_commit: f1a0cac LOG4J2-313 - Add JNDILookup (1 file)
introducing_pull_request: not found
fixing_commit: c77b3cb Restrict LDAP access via JNDI (#608) (1 file)
fixing_pull_request: #608 Restrict LDAP access via JNDI (closed, merged)
The default (no --pretty) output is unchanged: full JSON, byte for byte the same as
before, so anything scripted against it keeps working without passing --pretty.
Add --metrics to print execution metrics as a second JSON document after the trace
result. It reports timings for the deterministic and LLM phases, API request counts,
token usage, and the estimated API cost:
uv run python -m code_audit trace GHSA-jfh8-c2jp-5v3q --metrics{
"total_seconds": 74.312,
"deterministic_seconds": 1.208,
"llm_seconds": 72.535,
"github_requests": 14,
"anthropic_requests": 9,
"input_tokens": 118423,
"output_tokens": 5210,
"cache_creation_tokens": 0,
"cache_read_tokens": 0,
"estimated_cost_usd": 0.722365
}Add --debug to record the agent's investigation as JSON Lines under a debug/
directory, one line per model turn (thinking summary, tool calls, truncated tool
results, and the final findings), named by advisory and timestamp. This is useful for
understanding why a field came back null or wrong:
uv run python -m code_audit trace GHSA-jfh8-c2jp-5v3q --debugTrace several advisories concurrently. Pass IDs as arguments, or --from-file with one
ID per line, or both. Each advisory runs through the same pipeline as trace and prints
as it finishes, so progress stays visible during a long batch; a failure in one advisory
does not stop the others, and a final line summarizes how many succeeded versus failed
and the total batch wall-clock time:
uv run python -m code_audit trace-many GHSA-jfh8-c2jp-5v3q GHSA-36p3-wjmg-h94x
uv run python -m code_audit trace-many --from-file advisories.txt--concurrency (default 5) bounds how many advisories are traced at once, since running
too many in parallel would trip the GitHub and Anthropic rate limits and their retry
backoff rather than finishing faster. --debug, --metrics, and --pretty work as they
do for trace, writing one transcript, one metrics block, or one pretty summary per
advisory in place of its JSON block. With --metrics,
each advisory's block also reports started_at_seconds and ended_at_seconds as offsets
from the batch start, so overlapping windows show the advisories ran concurrently rather
than one after another. The GitHub and Anthropic clients are shared across threads, so
their request counts cannot be attributed to a single advisory; those totals are reported
once in the final summary instead of in each advisory's metrics block.
Run the tracer against a curated set of advisories with known-correct answers and print
a per-field success rate. Because the agent varies from run to run, each case runs
several times (--runs, default 3) and each field reports how many runs were correct.
A field is correct when it matches the expected value or is correctly null; it is
wrong when it reports a non-null identifier that does not match. The command exits
non-zero if any wrong value is reported, since a wrong SHA or PR number is worse than an
honest null:
uv run python -m code_audit eval --runs 3The cases live in src/code_audit/eval_cases.json; each entry records the expected
fields and a source URL for every confirmed value. A field's expected value may be a
list when several identifiers are acceptable (for example the same fix cherry-picked to
different branches with different SHAs). A case marked "disputed": true (a
vulnerability whose origin is genuinely ambiguous) is still reported but excluded from
the exit code. Point --cases at another file to evaluate a different set.
Add --debug to also write one JSON Lines transcript per run under debug/, named by
advisory, timestamp, and run number, in the same format as trace --debug:
uv run python -m code_audit eval --debuguv run ruff format --check . # formatting
uv run ruff check . # lint
uv run mypy # type check
uv run pytest # tests