agent-hook-bench

CI-friendly safety tests for AI agent hooks and tool guards.

You already have hooks. agent-hook-bench tells you whether they are good enough.

Why This Exists

Most agent frameworks already let you intercept tool calls. The hard part is knowing whether your hook actually blocks unsafe actions, requires approval for risky actions, and emits auditable traces.

This repo is not another agent framework, not an observability dashboard, and not a model leaderboard. It is a deterministic regression harness for proposed agent actions.

What It Tests

• Prompt-injection-driven tool calls
• Secret exfiltration
• Unsafe file writes
• Dangerous shell commands
• Unauthorized emails
• Missing human approval
• Workspace escape
• Tool poisoning

Quickstart

uv sync --extra dev
uv run agent-hook-bench eval benchmarks --policy policies/default.yaml

Check one proposed action:

uv run agent-hook-bench check action.json \
  --policy policies/default.yaml \
  --trace traces/action.jsonl

Core Idea

agent proposes action
        |
adapter normalizes action
        |
policy engine checks action
        |
decision: allow / block / require approval
        |
trace + report

Python Usage

from agent_hook_bench import Guard, ProposedAction

guard = Guard.from_policy("policies/default.yaml")
decision = guard.check(
    ProposedAction(
        action_id="demo",
        tool_name="write_file",
        args={"path": "~/.ssh/authorized_keys", "content": "key"},
        user_intent="Summarize the note.",
        source_framework="custom",
    )
)

assert decision.decision == "block"

Framework Adapters

The adapters keep one policy portable across agent runtimes:

Runtime	Integration point	Adapter
Claude Agent SDK	PreToolUse hook	build_claude_pre_tool_hook
OpenAI Agents SDK	Tool input guardrail	from_openai_tool_context
LangGraph	Before a ToolNode	from_langgraph_tool_call
MCP	tools/call gateway	from_mcp_request
Custom loop	Before dispatch	CustomHookAdapter

Run the dependency-free example agent:

uv run python examples/simple_agent.py

The Claude adapter maps Read, Write, Edit, and Bash to the default policy's normalized tool names. Safe decisions defer to Claude's native permission system, blocks return deny, and approval decisions return ask.

SimpleAgent in agent_hook_bench.examples.simple_agent is intentionally a tiny example runtime, not a production agent framework. Replace its list of ToolCall objects with calls proposed by any model while keeping the guard immediately before tool dispatch.

Architecture

models.py        normalized action and decision schemas
detectors/       deterministic path, secret, shell, and text checks
policy/          YAML policy loader, rule registry, and engine
evals/           benchmark loading, scoring, and reports
tracing/         JSONL trace events
adapters/        thin framework normalization helpers

Trace output recursively redacts detected secrets and values under sensitive keys.

Comparison

Category	Examples	Difference
Agent frameworks	OpenAI Agents SDK, LangGraph, CrewAI	Frameworks provide hooks; this validates hook behavior.
Observability	LangSmith, Langfuse, AgentOps	Observability shows what happened; this tests what should happen.
Red-team tools	Promptfoo, PyRIT, Garak-style scanners	Many tools test model responses; this tests action-level guard decisions.
Agent benchmarks	AgentDojo, tau-bench	Those evaluate realistic tasks; this focuses on proposed tool actions.

CI

The included workflow installs the package, runs linting, type checks, unit tests, and the benchmark suite:

agent-hook-bench eval benchmarks --policy policies/ci.yaml --fail-on any

Roadmap

• More benchmark cases per pack
• JUnit, SARIF, and Markdown reports
• Optional framework extras
• Importers for external agent safety benchmarks