ProofGate

Trust-but-verify gate for AI-authored code. It reads a change and the claims its author made about that change, then checks whether the claims hold up against reality before the code can merge.

Most review tools comment on the diff. ProofGate asks a different question: did you actually do what you said? It catches the failure modes that AI codegen produces and that look fine in a diff:

• "Added tests" that are empty, skipped, or assert nothing.
• "All tests pass" when the suite was never run or actually fails.
• "Fixed the bug" with no code change, or no regression test.
• "No breaking changes" that quietly removed a public export.
• Hardcoded secrets, not-implemented stubs, security checks stubbed to return true.
• Imports of modules and packages that do not exist.

The design principle (what blocks vs what warns)

A merge-blocking gate must only block on facts it can verify by structure or execution, never on what it infers from prose. This line is the spine of the whole tool:

• Blockers (fail the gate): a committed secret, a broken build, a test that is literally empty or always-true, an import that resolves nowhere, a security function whose only logic is a constant return, a configured test command that actually exited non-zero. All of these are checked against code or command exit codes.
• Warnings (advisory, never block): anything inferred from the PR description. "You wrote 'added tests' but I see no test files" is a warning, because reading authorship intent out of English is not reliable enough to stop an honest merge. A human decides.

This principle was earned: ProofGate was hardened through seven rounds of adversarial audit, and the recurring failure across the first five was an NL-inferred finding wrongly blocking honest PRs. Moving that class to warnings closed it for good.

How it works

diff + PR description -> parse claims -> [optional] run build/test -> detectors -> verdict

The core is a deterministic engine (src/) with zero runtime dependencies. The GitHub App and (planned) sandboxed runner are thin adapters on top; the engine is the product and is fully unit-tested (346 tests).

Install / run

npm install            # dev tooling only; the engine has zero runtime deps
npm test               # 346 tests
npx tsx src/cli.ts run --help

Against a range of commits:

npx tsx src/cli.ts run --base main --head HEAD --pr-body "$(cat pr-body.md)"

Against a saved diff:

npx tsx src/cli.ts run --diff change.diff --pr-body-file pr-body.md --format markdown

Exit code is 0 when the gate passes and 1 when a blocker is found, so it slots straight into CI. See examples/github-action.yml for a required-check setup that needs no app install. A missing --diff file is a hard error (exit 2), never a silently-clean pass.

Detectors

Detector	What it catches	Blocks the gate?
runtime	Configured build/test/smoke commands that fail	yes (per-check severity)
secrets	API keys, tokens, private keys, inline connection-string and assigned credentials (credential-shaped values only)	yes (warning in test/fixture/doc paths)
stubs	Not-implemented throws, security functions stubbed to a constant	yes; TODO/FIXME and fake-data identifiers warn
tests	Empty, pending, assertion-free, or always-true tests on added lines	yes (warning if pre-existing)
imports	Relative imports that resolve nowhere (JS/TS)	yes; undeclared packages warn
scope	Files changed outside the PR's stated scope	warning only
claims	"tests pass" / "refactor only" contradicted by a real failing test run	yes for those; "added tests" with no test files is a warning

Configuration

proofgate.config.json in the repo root:

{
  "checks": [
    { "name": "build", "command": "npm run build" },
    { "name": "test", "command": "npm test", "failSeverity": "blocker" }
  ],
  "disable": ["scope"],
  "severityOverrides": { "stubs": "warning" },
  "ignore": ["src/generated/", "*.min.js"]
}

checks run before detectors so a real broken build or failing suite fails the gate, and they let claims verify a "tests pass" claim for real. An invalid failSeverity fails closed (defaults to blocker). severityOverrides set a detector's severity absolutely and can lower it (mapping to info/warning stops it from gating) — use deliberately.

Trust model

The runtime step executes your configured commands. Locally that runs in your environment, same trust level as a CI script, so only point it at commands you trust; commands run in their own process group and are reaped on timeout. The hosted runner sandboxes this step in an ephemeral microVM. The static detectors never execute the code under review. Note that runtime checks read their command list from proofgate.config.json in the inspected tree; in CI, pin the config to a trusted ref so a PR cannot introduce a command in the same change.

Status and roadmap

Shipped: the verification engine, all seven detectors, the CLI, a CI integration via job-summary output, a self-hostable GitHub App (webhook → check run + inline annotations, config read from the base ref, run:false hardcoded), and a measured false-positive / false-negative feedback loop (proofgate metrics). 346 tests, zero runtime dependencies, hardened through many rounds of adversarial audit plus continuous self-dogfooding (ProofGate gates its own commits).

Honest about what is not built yet:

1. Sandboxed runner: today the GitHub App runs static-only (run:false); executing a PR's build/test in an ephemeral microVM (Firecracker / Vercel Sandbox) so untrusted code runs safely is the next adapter, not a shipped one.
2. Fix-forward: propose the patch for a finding, not just flag it.

Honest limitations

The detectors are heuristic, and a heuristic gate has a long tail. The design keeps that tail safe rather than pretending it does not exist:

• The added_tests claim is parsed from English and is therefore a warning, not a blocker. It surfaces "you said tests but changed no test files" for a human; it will not block, and it will miss some phrasings.
• secrets only blocks on credential-shaped values, so a real secret that is literally a common dev default (admin, postgres, password) is suppressed to avoid false positives on dev connection strings, and prose/env-var-name values are skipped. This is a deliberate precision tradeoff.
• tests and imports cover JS/TS and Python; other languages are best-effort.
• Static analysis cannot decide value-level tautologies (assert.equal(a, a)) or run the code; the runtime checks and a human reviewer cover what it cannot.

None of this replaces human review. It removes the floor of obvious, repeated AI-codegen mistakes so reviewers spend their attention on what matters.

License

MIT. See LICENSE. Free to use, self-host, and modify.