Quality gates for AI coding agents. Structural enforcement built in, semantic evaluation via LLM.
AI coding tools make you fast. Nobody built the quality layer that keeps fast from becoming reckless.
| Finding | Source |
|---|---|
| Iterative LLM refinement increases vulnerabilities monotonically | Shukla, Joshi & Syed 2025 (arXiv:2506.11022) |
| AI-assisted developers 19% slower while believing 20% faster | Becker et al. / METR 2025 (arXiv:2507.09089) |
| Copilot vulnerable in ~40% of security-relevant scenarios | Pearce et al. / IEEE S&P 2022 (doi:10.1109/SP46215.2023.10179324) |
| Pass@1 doesn't correlate with code quality | Sabra et al. 2025 (arXiv:2508.14727) |
| Greedy self-improvement provably suboptimal without external signal | Tang et al. / NeurIPS 2024 (arXiv:2405.17503) |
| Models trained on own output collapse | Shumailov et al. / Nature 2024 (doi:10.1038/s41586-024-07566-y) |
Quality gates are the missing external signal.
Copy one file. Claude Code self-enforces five gates:
cp examples/CLAUDE.md .claude/CLAUDE.mdHooks read JSON from stdin per the Claude Code hooks protocol and block actions via exit code 2.
cp -r hooks/ ./hooks/
# Add to .claude/settings.json — see examples/settings.jsonSeven hooks across three lifecycle events:
| Hook | Event | Blocks |
|---|---|---|
spec-gate.sh |
PreToolUse (Edit/Write) | No spec file |
diagnosis-gate.sh |
PreToolUse (Edit) | No diagnosis |
test-gate.sh |
PreToolUse (Bash) | Tests not run before commit |
completion-gate.sh |
Stop | TODO/FIXME in last message |
task-quality-gate.sh |
TaskCompleted | Failing tests |
For semantic evaluation (LLM-evaluated, not string matching), use Claude Code's native agent or prompt hook types — see examples/semantic-hooks.json:
{
"type": "agent",
"prompt": "Read .diagnosis and evaluate whether it identifies a root cause, not just symptoms. Exit 2 if insufficient."
}This spawns a full agent with file access to verify your gate condition. No Python required.
pip install agent-gatesfrom agent_gates import Pipeline
pipeline = Pipeline([
("SpecGate", "spec", "_"),
("DiagnosisGate", "diagnosis", "_"),
("QCGate", "qc_report", "_"),
])
storage = {
"spec": "Add retry logic to API client",
"diagnosis": "Timeout errors — connection pool exhausted under concurrent load",
"qc_report": {"verdict": "PASS"},
}
pipeline.run(storage) # All gates pass| Gate | When | Blocks unless |
|---|---|---|
| Spec | Before coding | A specification exists and the task traces back to it |
| Test | Before accepting fix | Tests cover the specific behavior being changed |
| Diagnosis | Before bug fix | Root cause identified with sufficient detail (min 20 chars) |
| QC | After code written | Independent review verdict is PASS |
| Evidence | Before merge | Evidence beyond "tests pass" — edge cases verified |
The built-in gates are structural checks: they verify presence, format, and minimum substance. They catch "you skipped a step" — not "your reasoning is wrong."
For checks that require understanding context, use LLMGate with your own evaluation function:
from agent_gates import LLMGate
def eval_diagnosis(storage: dict) -> tuple[bool, str]:
import anthropic
client = anthropic.Anthropic()
diagnosis = storage.get("diagnosis", "")
resp = client.messages.create(
model="claude-sonnet-4-6",
max_tokens=200,
messages=[{"role": "user", "content":
f"Does this diagnosis identify a root cause (not just symptoms)? "
f"Answer PASS or FAIL with one sentence.\n\n{diagnosis}"
}],
)
text = resp.content[0].text
passed = text.strip().upper().startswith("PASS")
return (passed, text)
gate = LLMGate(eval_fn=eval_diagnosis)
passed, reason = gate.check(storage)This is where "semantic" becomes real. The LLM reads your project context — a semantic gate can tell you that a function's error handling contradicts the project's stated error philosophy. A linter can only tell you the function exists.
Structural (built-in): "Does a diagnosis exist with >20 chars?"
Semantic (LLMGate): "Does this diagnosis identify a root cause?"
Both are useful. Structural gates are free and fast. Semantic gates cost tokens but catch what structural gates can't.
Three patterns, ~150 lines of Python:
- Operator —
run(storage, input_key, output_key). Universal interface. - Registry —
@REGISTRY.register. Name-based discovery. - Pipeline — Chain operators with gate enforcement. Gates raise
GateErroron failure.
from agent_gates import GateOperator, REGISTRY
@REGISTRY.register
class CustomGate(GateOperator):
def check(self, storage: dict) -> tuple[bool, str]:
if meets_your_standards(storage):
return (True, "")
return (False, "What failed and why")Small samples, consistent direction:
- Diagnosis gate (A/B, Claude Sonnet, N=3): debugging accuracy 0/3 without → 2/3 with diagnosis-first enforcement. Consistent with Tang et al.'s theoretical result that external signal beats greedy iteration.
- QC gate (medical QA pipeline, not a coding context): win rate 68.6% (p=0.02), Bradley-Terry Δβ=1.063 (p=0.01). Dose-response: 1 gate → 56%, 2 → 75%, 3 → 86% win rate.
These are preliminary. The experiment we need: a controlled trial comparing gates-on vs gates-off on real codebases, measuring defect rate and velocity. If you run this on your codebase, open an issue — we want the data.
agent-gates (evaluation logic) Claude Code (enforcement engine)
├── SpecGate ├── PreToolUse hooks
├── DiagnosisGate ├── PostToolUse hooks
├── TestGate ←→ ├── Stop hooks
├── QCGate ├── TaskCompleted hooks
├── EvidenceGate ├── prompt/agent hook types
└── LLMGate └── 31 lifecycle events
agent-gates defines what to check. Claude Code hooks define when and how to enforce. Use them together or independently.
- Structural gate operators (5 built-in)
- LLMGate for semantic evaluation (Python)
- Claude Code hooks — stdin JSON protocol, 5 hooks across 3 lifecycle events
- Semantic hook examples — native
agent/prompthook types - Cost tracking per gate invocation
- RCT framework: gates-on vs gates-off measurement
- VS Code / Cursor extension
MIT