Skip to content

fix: calibrate agent precision benchmark gate - #4

Merged
hlinor-systems merged 1 commit into
mainfrom
agent/precision-benchmark
Aug 18, 2026
Merged

fix: calibrate agent precision benchmark gate#4
hlinor-systems merged 1 commit into
mainfrom
agent/precision-benchmark

Conversation

@hlinor-systems

Copy link
Copy Markdown
Collaborator

What changed

  • preserve dynamic model declaration signals without emitting code-like model values
  • reject primitive and dotted identifier values from agent names
  • ignore Dependabot configuration and GitHub Actions workflow metadata
  • add regression coverage for MCP policy scope, self-report idempotence, owner punctuation, workflow jobs, and permission locality
  • add a pinned six-repository benchmark with scheduled and manual GitHub Actions coverage
  • calibrate benchmark bounds from the latest pinned snapshot counts with approximately 40% slack

Why

The scanner had moved between under-detection and noisy inventory as declaration heuristics were tightened. This change separates candidate signals from reportable metadata, narrows non-agent configuration sources, and makes the external regression gate tolerant to small heuristic changes.

Validation

  • go test ./...
  • go vet ./...
  • go test -race ./internal/scan ./cmd/agentctl-benchmark
  • demo SARIF regeneration and fixture checks
  • staged pre-commit secret and agentctl gates
  • pinned external benchmark: 6/6 passing, zero source ACP-005 and zero code-expression metadata hits

@hlinor-systems
hlinor-systems marked this pull request as ready for review August 18, 2026 07:32
@hlinor-systems
hlinor-systems merged commit 6b688c9 into main Aug 18, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants