feat(test-mapping): add test mapping pipeline and test_mapping method (#6) by jflowers · Pull Request #11 · zero-dot-force/snake-eyes

jflowers · 2026-09-01T13:00:45Z

Summary

Adds a Python test-to-assertion mapping capability (issue #6). A new
src/snake_eyes/quality/ package pairs test functions to the production
functions they exercise, classifies each assertion, and infers the
side-effect type each test targets — served over the Gaze analyzer
protocol via a new test_mapping JSON-RPC method. The initialize
capability flag test_mapping flips false → true. No CRAP/scoring math
(Gaze owns scoring); no new runtime dependency (astroid already shipped).

Pairing (quality/pairing.py, lifted+adapted from gaze-py): three
first-match-wins strategies — name convention (90 exact / 70 case-only),
direct call (80), astroid transitive call-graph BFS depth 5 (75) with
graceful degradation.
Assertion detection (quality/assertions.py, lifted+adapted): one row
per assertion; exhaustive pytest+unittest classification into
equality/comparison/identity/membership/error_check/generic.
Effect-type inference (quality/mapping.py, fresh): consumes detector
FunctionRecord.side_effects.
Pipeline (quality/pipeline.py, fresh): run_test_mapping() —
static-analysis-only, deterministic, targets restricted to source files.
Server: _test_mapping handler + DEFAULT_DISPATCH; protocol.py
capability flip.

How to Test

uv sync --locked
uv run ruff check src/ tests/
uv run ruff format --check src/ tests/
uv run mypy src/
uv run pytest --cov=snake_eyes --cov-report=term-missing --cov-fail-under=85

Expected: 557 passed, 95.75% coverage (per-file assertions 95 / mapping 100 /
pairing 93 / pipeline 91).

Stdio smoke:

printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"root_path":"tests/fixtures/sample_project"}}' | uv run snake-eyes --stdio

Expected: capabilities include "test_mapping": true; a test_mapping
request over tests/fixtures/sample_project returns 7 mapping rows.

How to Demo

Point test_mapping at tests/fixtures/sample_project and observe the 7
rows: name-strategy (test_add→add, 90), case-only (test_Add→add, 70),
direct-call (test_it_divides→divide, 80), error-check
(test_divide_error→divide, ErrorReturn), class-qualified unittest
method (TestCounter.test_inc→inc, ReceiverMutation), and a
multi-assertion test emitting one row per assertion.

Key Files Changed

src/snake_eyes/quality/ (new): __init__.py, pairing.py,
assertions.py, mapping.py, pipeline.py
src/snake_eyes/protocol.py, server.py: capability flip + handler/dispatch
tests/test_test_mapping_method.py (new), tests/fixtures/sample_project/**
(new), tests/conftest.py (fixture collection-ignore), + capability-assertion
updates in test_protocol.py / test_discover_method.py /
test_classify_signals_method.py / test_server.py
openspec/changes/test-mapping/: proposal, design, tasks, 5 capability specs
README.md, AGENTS.md, pyproject.toml (docs + slow marker)

Known Issues

Non-blocking LOW advisories accepted by the unanimous code-review council
(tracked for optional follow-up; none affect correctness):

LOW: two quality/pairing.py except sites are silent/defensive
(the nodes_of_class(Call) guard and the finally cache-clear) — could
add a stderr diagnostic for consistency.
LOW: _STRATEGY3_DEGRADE omits InferenceError/OSError (cosmetic
diagnostic-label parity; OSError intentionally excluded so
FileNotFoundError re-raises cleanly to -32602).
LOW: defined_names name-collision residual (an external callee whose
name collides with an in-project function can still trigger .infer()) —
identical accepted residual as analysis/inference.py, bounded by the
MemoryError degrade.
LOW: two tests carry or rows == [] escapes and a few strategy-3
smoke tests assert only graph is not None; optional BFS memoization per
test file.

This PR was generated by /uf.finale (AI-assisted).

… passed)

…method (#6)

jflowers

Council Verdict: APPROVE

Note: Could not post as APPROVE due to GitHub's self-review prohibition. Posted as COMMENT instead. Original verdict: APPROVE.

Reviewers: Adversary, Architect, Guard, Testing, SRE, Curator, Envoy, Herald, Scribe
Iterations: 1
Pre-flight: All CI gates PASS (95.75% coverage, 557 tests)

Adversary (APPROVE)

2 LOW findings omitted.

Architect (APPROVE)

2 LOW findings omitted.

Guard (APPROVE)

1 LOW finding omitted.

Testing (APPROVE)

3 LOW findings omitted.

SRE (APPROVE)

3 LOW findings omitted.

Curator (APPROVE)

1 LOW finding omitted. Blog milestone issue filed as #12.

Envoy (APPROVE)

No findings.

Herald (APPROVE)

2 LOW findings omitted.

Scribe (APPROVE)

[MEDIUM] _test_mapping handler lacks docstring (server.py:136)
[MEDIUM] _AssertionVisitor class docstring minimal (assertions.py:251)
[MEDIUM] _CallGraph BFS algorithm lacks inline docs (pairing.py:169)
4 LOW findings omitted.

Linked Issues

#6: snake-eyes: test mapping pipeline and test_mapping method (O1)

This review was generated by /review-council (AI-assisted).

jflowers · 2026-09-01T14:39:40Z

    return {"signals": signals}


+def _test_mapping(params: dict[str, Any] | None) -> dict[str, Any]:


[MEDIUM] Missing handler docstring (Scribe) -- All JSON-RPC handlers in this file lack docstrings (pre-existing pattern), but per TD-007 public-facing API handlers should document parameters, return values, and error conditions. Consider adding a one-liner, e.g.:

def _test_mapping(params: dict[str, Any] | None) -> dict[str, Any]: """Handle the ``test_mapping`` JSON-RPC method; wraps pipeline output in ``{"mappings": [...]}``."""

jflowers · 2026-09-01T14:39:40Z

+# ---------------------------------------------------------------------------
+
+
+class _AssertionVisitor:


[MEDIUM] Minimal class docstring (Scribe) -- _AssertionVisitor constructor parameters (rel_path, skip_call_ids, max_depth) are undocumented. Per TD-007, non-trivial class parameters should be documented to help maintainers understand that rel_path formats assertion_location, skip_call_ids prevents with-item double-counting, and max_depth bounds traversal depth.

jflowers · 2026-09-01T14:39:40Z

+
+
+@dataclass
+class _CallGraph:


[MEDIUM] BFS algorithm lacks inline documentation (Scribe) -- The _CallGraph.reachable_files BFS has no inline comments explaining traversal semantics: visited includes the start file at depth 0, and depth > depth_limit is the termination condition. The module-level docstring mentions "BFS with depth_limit=5" but the implementation details are undocumented.

jflowers · 2026-09-02T20:43:20Z

PR Cost Report: #11

Session: Proposing changes with artifacts

ID: ses_fa6136c9fffeFAhODaq02oZRNb

Metric	Value
Cost (parent only)	$186.82
Input tokens	494
Output tokens	300,114
Reasoning tokens	297,806
Cache read tokens	64,323,044
Cache write tokens	9,154,115

Timeline: 2026-08-31 18:24:31 — 2026-09-01 09:03:33

Child sessions: 47 — additional cost: $107.85
Session tree total: $294.67

Child session breakdown

Session	Cost	Output Tokens
Spec review: Adversary (@divisor-adversary subagent)	$2.08	7,854
Spec review: Architect (@divisor-architect subagent)	$2.08	7,814
Spec review: Guard (@divisor-guard subagent)	$2.11	7,706
Spec review: Tester (@divisor-testing subagent)	$1.88	8,636
Spec review: Operator (@divisor-sre subagent)	$0.97	4,769
Spec review: Curator (@divisor-curator subagent)	$1.85	6,298
Spec review iter2 - adversary (@divisor-adversary subagent)	$3.55	4,922
Spec review iter2 - architect (@divisor-architect subagent)	$1.77	4,686
Spec review iter2 - guard (@divisor-guard subagent)	$1.50	4,156
Spec review iter2 - testing (@divisor-testing subagent)	$1.54	1,612
Spec review iter2 - sre (@divisor-sre subagent)	$1.41	3,015
Spec review iter2 - curator (@divisor-curator subagent)	$1.78	3,794
Apply iter-3 spec fixes (@cobalt-crush-dev subagent)	$1.36	1,021
Spec review iter3 adversary (@divisor-adversary subagent)	$1.37	2,002
Spec review iter3 architect (@divisor-architect subagent)	$2.18	3,998
Spec review iter3 guard (@divisor-guard subagent)	$2.27	5,269
Spec review iter3 testing (@divisor-testing subagent)	$1.37	3,451
Spec review iter3 sre (@divisor-sre subagent)	$1.23	2,919
Spec review iter3 curator (@divisor-curator subagent)	$1.09	2,538
Confirm spec review: adversary (@divisor-adversary subagent)	$1.78	3,035
Confirm spec review: architect (@divisor-architect subagent)	$1.01	2,722
Confirm spec review: guard (@divisor-guard subagent)	$0.95	3,401
Confirm spec review: testing (@divisor-testing subagent)	$0.73	1,590
Confirm spec review: sre (@divisor-sre subagent)	$0.60	1,507
Confirm spec review: curator (@divisor-curator subagent)	$0.73	1,803
Implement test-mapping feature (@cobalt-crush-dev subagent)	$3.38	3,799
Implement test-mapping feature (@general subagent)	$21.25	72,656
Code review: adversary (@divisor-adversary subagent)	$2.03	6,080
Code review: architect (@divisor-architect subagent)	$7.00	8,371
Code review: guard (@divisor-guard subagent)	$1.74	4,962
Code review: testing (@divisor-testing subagent)	$5.27	6,943
Code review: sre (@divisor-sre subagent)	$1.98	5,795
Code review: curator (@divisor-curator subagent)	$1.56	5,095
Fix code-review findings (@general subagent)	$3.52	27,889
CR iter2: adversary (@divisor-adversary subagent)	$1.20	2,997
CR iter2: architect (@divisor-architect subagent)	$1.88	3,282
CR iter2: guard (@divisor-guard subagent)	$2.12	5,131
CR iter2: testing (@divisor-testing subagent)	$2.22	5,232
CR iter2: sre (@divisor-sre subagent)	$1.14	3,656
CR iter2: curator (@divisor-curator subagent)	$0.78	3,131
Apply CR iter-2 fixes (@general subagent)	$1.52	12,635
Code review iter3: adversary (@divisor-adversary subagent)	$2.15	4,914
Code review iter3: architect (@divisor-architect subagent)	$1.66	5,992
Code review iter3: guard (@divisor-guard subagent)	$2.17	6,982
Code review iter3: testing (@divisor-testing subagent)	$2.16	3,828
Code review iter3: sre (@divisor-sre subagent)	$0.99	4,022
Code review iter3: curator (@divisor-curator subagent)	$0.96	4,609

Total: $294.67 (48 sessions)

jflowers added 4 commits August 31, 2026 23:02

docs(test-mapping): add OpenSpec change artifacts for #6 (spec-review…

4492a1f

… passed)

feat(test-mapping): implement test mapping pipeline and test_mapping …

292de9e

…method (#6)

fix(test-mapping): address code-review findings (#6)

7bf46f3

docs(test-mapping): mark code-review passed

8647e0b

jflowers commented Sep 1, 2026

View reviewed changes

jflowers merged commit fb04a9c into main Sep 1, 2026
2 checks passed

jflowers mentioned this pull request Sep 1, 2026

snake-eyes: test mapping pipeline and test_mapping method (O1) #6

Closed

Navigation Menu

Uh oh!

feat(test-mapping): add test mapping pipeline and test_mapping method (#6) - #11

feat(test-mapping): add test mapping pipeline and test_mapping method (#6)#11
jflowers merged 4 commits into
mainfrom
opsx/test-mapping

jflowers commented Sep 1, 2026

Uh oh!

jflowers left a comment

Uh oh!

jflowers Sep 1, 2026

Uh oh!

jflowers Sep 1, 2026

Uh oh!

jflowers Sep 1, 2026

Uh oh!

Uh oh!

jflowers commented Sep 2, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

		return {"signals": signals}


		def _test_mapping(params: dict[str, Any] \| None) -> dict[str, Any]:

		# ---------------------------------------------------------------------------


		class _AssertionVisitor:

Uh oh!

Conversation

jflowers commented Sep 1, 2026

Summary

How to Test

How to Demo

Key Files Changed

Known Issues

Uh oh!

jflowers left a comment

Choose a reason for hiding this comment

Council Verdict: APPROVE

Adversary (APPROVE)

Architect (APPROVE)

Guard (APPROVE)

Testing (APPROVE)

SRE (APPROVE)

Curator (APPROVE)

Envoy (APPROVE)

Herald (APPROVE)

Scribe (APPROVE)

Linked Issues

Uh oh!

jflowers Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

jflowers Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

jflowers Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

jflowers commented Sep 2, 2026

PR Cost Report: #11

Session: Proposing changes with artifacts

Total: $294.67 (48 sessions)

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant