Skip to content

fix(pairing): add call-site verification to strategy-1 name matching (#14) - #16

Merged
jflowers merged 1 commit into
mainfrom
opsx/fix-strategy1-callsite-verification
Sep 2, 2026
Merged

fix(pairing): add call-site verification to strategy-1 name matching (#14)#16
jflowers merged 1 commit into
mainfrom
opsx/fix-strategy1-callsite-verification

Conversation

@jflowers

@jflowers jflowers commented Sep 2, 2026

Copy link
Copy Markdown
Contributor

Summary

Strategy-1 (name-convention) pairing in quality/pairing.py now verifies
call-site presence before emitting a pairing. When a test function's name
matches a target via prefix stripping (e.g., test_addadd), the
system walks the test function's AST body for direct calls to the matched
target. If no matching call is found, the pairing is suppressed entirely,
allowing the test to fall through to strategy-2 or strategy-3.

This eliminates false pairings where a test is named after a function it
never actually invokes, improving Gaze's test quality metrics accuracy.

Closes #14

How to Test

# Run the full test suite (574 tests, 95.78% coverage)
uv run pytest --cov=snake_eyes --cov-report=term-missing --cov-fail-under=85

# Run only the new call-site verification tests
uv run pytest tests/test_test_mapping_method.py -k "callsite" -v

# Run all CI gates
uv run ruff check src/ tests/
uv run ruff format --check src/ tests/
uv run mypy src/

How to Demo

  1. Create a fixture project with a source file containing def add(): ...
    and a test file containing def test_add(): assert True (no call to add)
  2. Run snake-eyes --stdio and send a test_mapping request
  3. Observe that test_add does NOT pair with add at confidence 90
    (previously it would have been a false positive)
  4. Add add() to the test body and re-run — now the pairing is emitted correctly

Key Files Changed

  • src/snake_eyes/quality/pairing.py — Added optional test_tree parameter
    to _name_match; call-site verification via _direct_call_names before
    emitting pairings (both exact-match and case-only paths)
  • tests/test_test_mapping_method.py — Updated 14 existing tests to include
    target calls; added 8 new tests covering call-site verification scenarios
  • openspec/changes/fix-strategy1-callsite-verification/ — OpenSpec change
    artifacts (proposal, design, spec with 8 scenarios, tasks with 15 items)

This PR was generated by /uf.finale (AI-assisted).

…14)

- Add optional test_tree parameter to _name_match for call-site verification
- Verify name-convention matches against direct calls in test body via
  _direct_call_names before emitting pairings
- Suppress false pairings where test name matches target but never calls it
- Update 14 existing tests to include target function calls in test bodies
- Add 8 new tests: direct/attribute/suppression/fallthrough/indirect/
  case-only/integration coverage
- Add OpenSpec change artifacts (proposal, design, spec, tasks)

Closes #14

Assisted-by: claude-opus-4
Generated with AI assistance (claude-opus-4)

@jflowers jflowers left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Council Verdict: APPROVE

Note: Could not post as APPROVE due to GitHub's self-review prohibition (author cannot approve own PR). Posted as COMMENT instead. Original verdict: APPROVE.

Reviewers: Adversary, Architect, Curator, Envoy, Guard, Herald, Scribe, SRE, Testing
Iterations: 1

Adversary (APPROVE)

No findings.

Architect (APPROVE)

2 LOW findings omitted.

Curator (APPROVE)

No findings.

Envoy (APPROVE)

3 LOW findings omitted.

Guard (APPROVE)

1 LOW finding omitted.

Herald (APPROVE)

3 LOW findings omitted.

Scribe (APPROVE)

5 LOW findings omitted.

SRE (APPROVE)

1 LOW finding omitted.

Testing (APPROVE)

3 LOW findings omitted.


This review was generated by /review-council (AI-assisted).

@jflowers
jflowers merged commit fbbd530 into main Sep 2, 2026
2 checks passed
@jflowers

jflowers commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

PR Cost Report: #16

Session: Change proposal artifact workflow

ID: ses_fa122f139ffeQ63laznpIoOdU5

Metric Value
Cost (parent only) $11.12
Input tokens 119
Output tokens 51,384
Cache read tokens 9,707,717
Cache write tokens 797,731

Timeline: 2026-09-01 17:25:40 — 2026-09-02 08:33:07

Child sessions: 16 — additional cost: $15.58
Session tree total: $26.71

Child session breakdown
Session Cost Output Tokens
Adversary spec review (@divisor-adversary subagent) $0.90 4,620
Architect spec review (@divisor-architect subagent) $1.03 5,255
Guard spec review (@divisor-guard subagent) $0.93 6,392
Testing spec review (@divisor-testing subagent) $1.20 5,364
SRE spec review (@divisor-sre subagent) $0.60 3,322
Spec re-review adversary (@divisor-adversary subagent) $0.56 3,958
Spec re-review architect (@divisor-architect subagent) $0.66 4,394
Spec re-review guard (@divisor-guard subagent) $0.35 3,441
Spec re-review testing (@divisor-testing subagent) $0.67 3,560
Spec re-review SRE (@divisor-sre subagent) $0.50 2,667
Implement Group 1 + Group 2 tasks (@cobalt-crush-dev subagent) $3.16 14,686
Code review - adversary (@divisor-adversary subagent) $0.90 2,975
Code review - architect (@divisor-architect subagent) $1.15 3,857
Code review - guard (@divisor-guard subagent) $0.93 3,394
Code review - testing (@divisor-testing subagent) $1.05 3,644
Code review - SRE (@divisor-sre subagent) $1.00 2,917

Session: Code and spec review council

ID: ses_f9de40430ffeXowOdFqgj2VNfp

Metric Value
Cost (parent only) $4.10
Input tokens 59
Output tokens 24,227
Cache read tokens 3,175,236
Cache write tokens 304,546

Timeline: 2026-09-02 08:33:16 — 2026-09-02 09:27:47

Child sessions: 10 — additional cost: $12.23
Session tree total: $16.33

Child session breakdown
Session Cost Output Tokens
Gaze full quality report (@gaze-reporter subagent) $2.27 10,549
Divisor Adversary review (@divisor-adversary subagent) $1.13 3,176
Divisor Architect review (@divisor-architect subagent) $1.24 4,450
Divisor Guard review (@divisor-guard subagent) $1.22 4,197
Divisor Testing review (@divisor-testing subagent) $1.17 4,061
Divisor SRE review (@divisor-sre subagent) $1.17 4,046
Divisor Curator review (@divisor-curator subagent) $1.05 3,223
Divisor Envoy review (@divisor-envoy subagent) $1.08 2,834
Divisor Herald review (@divisor-herald subagent) $0.96 2,453
Divisor Scribe review (@divisor-scribe subagent) $0.95 3,008

Grand Total: $43.04 across 2 session tree(s) (28 sessions)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: strategy-1 name-convention pairing should verify call-site presence

1 participant