Skip to content

feat(classify-signals): classification signal extraction + classify_signals method (#5) - #10

Merged
jflowers merged 6 commits into
mainfrom
opsx/classify-signals
Aug 31, 2026
Merged

feat(classify-signals): classification signal extraction + classify_signals method (#5)#10
jflowers merged 6 commits into
mainfrom
opsx/classify-signals

Conversation

@jflowers

Copy link
Copy Markdown
Contributor

Summary

Implements the Gaze classify_signals analyzer capability (issue #5). Snake Eyes now emits raw classification signals that Gaze's Go core scores — it does not classify or score itself.

  • 5 signal extractors (src/snake_eyes/signals/), reconstructed from documented gaze-py behavior (upstream source unavailable), weights pinned as source-of-truth literals: interface (30), visibility (+10/-10), caller_count (0→none, 1-4→5, 5→15, 20→25), naming_convention (+15/-10), docstring (15).
  • Caller-count inference (analysis/inference.py): build_caller_index/CallerIndex/count_callers over astroid, callees matched by resolved defining file path (robust to src/ layouts).
  • Adapter fan-out (signals/adapter.py extract_signals) + JSON-RPC classify_signals server method; initialize capability flag flipped to true.
  • RAW signals only — no scoring, no contractual/incidental/ambiguous labels; engine.py deliberately NOT lifted (avoids scoring drift; Gaze owns scoring).
  • Analysis Safety (Constitution V): static-only (never imports analyzed code), 16 MiB byte-cap applied before astroid parses, isolated per-request manager, graceful degradation to 0 on any astroid failure.

How to Test

uv sync --locked
uv run ruff check src/ tests/
uv run ruff format --check src/ tests/
uv run mypy src/
uv run pytest --cov=snake_eyes --cov-report=term-missing --cov-fail-under=85

All green: 409 passed, 96.19% coverage, ruff + mypy clean.

How to Demo

printf '%s\n%s\n' \
  '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"root_path":"<proj>"}}' \
  '{"jsonrpc":"2.0","id":2,"method":"classify_signals","params":{"root_path":"<proj>"}}' \
  | uv run snake-eyes --stdio

initialize reports capabilities.classify_signals: true; classify_signals returns a signals[] array of {function, package, side_effect_type, source, weight, reasoning}.

Key Files Changed

38 files, +1975/-29.

  • New capabilitysrc/snake_eyes/signals/ (__init__, _routing, _types, adapter, interface, visibility, caller, naming, docstring); src/snake_eyes/analysis/inference.py.
  • Wiringsrc/snake_eyes/server.py (_classify_signals handler + dispatch), src/snake_eyes/protocol.py (capability flag).
  • Depspyproject.toml, uv.lock (astroid >=3.0,<4).
  • DocsREADME.md, AGENTS.md.
  • Tests — 11 new (test_signals_*, test_inference*, test_classify_signals_method) + 3 updated (test_discover_method, test_protocol, test_server).
  • Specopenspec/changes/classify-signals/ (proposal, design, tasks, 4 specs).

Known Issues

Accepted-LOW review-council findings (non-blocking; sweep at archive):

  • LOW: tasks.md §6 fixtures are realized as tmp_path-built trees inside the test modules rather than a committed tests/fixtures/signals/ directory.
  • LOW: provenance terminology ("lifted" vs "reconstructed") is inconsistent across file headers / NOTICE / README.
  • LOW: AGENTS.md project-structure tree has minor ASCII-glyph misalignment.
  • LOW: pytest emits a CoverageWarning (module-not-measured) from the -m snake_eyes determinism subprocess (cosmetic).
  • LOW: no dedicated two-trees-in-sequence caller-isolation regression test (isolation is implemented via MANAGER.clear_cache() and implicitly validated).

This PR was generated by /uf.finale (AI-assisted).

…classify_signals method (#5)

Add the signals/ package (interface, visibility, caller, naming, docstring
extractors + adapter fan-out) and analysis/inference.py (astroid caller-count
index), wire the classify_signals JSON-RPC method, and flip the capability
flag to true. Extractors reconstructed from documented gaze-py behavior
(gaze-py source unavailable); the engine is deliberately not lifted.

Section 6 fixture scenarios are realized as tmp_path-built trees within the
test modules rather than committed tests/fixtures/signals/ files.
…dedup, and adapter branch-coverage tests (code review)
Record the code-review gate marker in the OpenSpec change tasks after the review council's unanimous (9/9) approval.

Assisted-by: claude-opus
Generated with AI assistance (claude-opus)

@jflowers jflowers left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: Could not post as APPROVE (GitHub blocks self-approval / insufficient permissions). Posted as COMMENT instead. Original council verdict: APPROVE.

Council Verdict: APPROVE

Reviewers: The Adversary, The Architect, The Guard, The Tester, The Operator, The Curator, The Envoy, The Herald, The Scribe (9/9 discovered invoked, 0 absent)
Iterations: 1 (initial REQUEST CHANGES → APPROVE after fixes)

The Adversary (APPROVE)

  • Resolved: astroid 16 MiB cap-bypass (MEDIUM) via in-project name-gated inference + finally clear_cache; non-degrade exception → -32603 (LOW) via broadened per-call except.
  • LOW/advisory (non-blocking): method-name collision residual; stale docstring inference.py:26-27; dev-only ast-serialize/librt transitive dep (maintainer confirm).

The Architect (APPROVE — alignment 9/10)

  • Resolved HIGH provenance self-contradiction: "lifted" vs "reconstructed" now consistent per origin (detector/complexity lifted #4; signals/*+_routing reconstructed #5; adapter/inference original; engine not lifted).

The Guard (APPROVE)

  • Resolved HIGH provenance drift; LOW _routing mislabel. Report-only: verify reconstructed weights vs canonical gaze-py (escalate to CRITICAL if any diverge — none changed here). LOW wording residual (design.md:90).

The Tester (APPROVE)

  • Resolved MEDIUM degradation-branch coverage (S-6) + MEDIUM zero-signal protocol test (S-7); inference.py 90→94%. LOW residual: oversized-file test stubs the cap.

The Operator (APPROVE)

  • Resolved 3 MEDIUM: stderr diagnostics on all degrade paths; broadened per-call catch; inference scoping + finally-clear + honest docs. CI gates intact, astroid pinned. 2 LOW micro-opts.

The Curator (APPROVE)

  • Resolved: AGENTS ASCII tree; README naming "source"; _routing parenthetical. LOW cosmetic: signals/ ordering differs README vs AGENTS.

The Envoy (APPROVE)

  • Resolved 3 LOW: spelling (behaviour/recognised); provenance wording.

The Herald (APPROVE)

  • Resolved 2 LOW: provenance verbs; README _routing suffix.

The Scribe (APPROVE)

  • Resolved: provenance, ASCII connectors, naming "source", design.md claims verified. LOW deferred: proposal/design "New files" omit _routing.py/_types.py.

LOW findings: ~10 across personas (summarized above); none blocking.

Report-only items for maintainer:

  1. Verify the 5 reconstructed extractor weights vs canonical gaze-py source (interface 30; visibility ±10; caller 5/15/25; naming +15/-10; docstring 15). No weight changed/lowered on this branch; escalate to CRITICAL if any later diverges.
  2. Per-file copyright-header form on reconstructed code + NOTICE "lifted files" line (NOTICE out of this branch's diff; accurate for genuinely-lifted #4 files) — licensing/legal judgment.
  3. Confirm dev-only transitive deps ast-serialize/librt (pulled by mypy) + intended mypy version; runtime footprint unaffected.

Note: tasks.md checkboxes/review markers + proposal fixture list were intentionally left untouched per the author's direction (by-design /uf.unleash progress tracking).


This review was generated by /review-council (AI-assisted).

…ion (review council)

Change "lifted/verbatim" to "reconstructed from documented gaze-py behavior" for the signal extractors, which were reconstructed from the issue's documented behavior rather than copied. Fix AGENTS.md analysis/ tree connectors and README naming.py comment. Correct design.md on-disk-resolution and degradation claims to match inference.py.
…dings

Scope astroid inference to in-project defined names to prevent cap-bypassing transitive parsing, and clear the MANAGER cache in a finally block. Broaden the per-call-site catch so an uninferable call is omitted rather than failing the whole request. Emit stderr diagnostics on all degrade paths. Add degradation-branch tests (outer degrade, parametrized per-file exceptions, Uninferable skip, non-degrade exception) and protocol non-empty/source/weight assertions; fix minor docstring spelling.
@jflowers

Copy link
Copy Markdown
Contributor Author

Review Council — Formal Review (what I would have said as a formal review)

Posted as a PR comment because GitHub blocks a PR author from
submitting a formal APPROVE on their own PR (HTTP 422). The
council's formal verdict is APPROVE.

Council Verdict: APPROVE — 9/9 discovered divisor reviewers APPROVE
after 1 fix iteration. Mode: Code Review (OpenSpec). Scope: full
branch diff main...HEAD. Gaze quality analysis skipped at the
maintainer's request (not yet wired for Python).

The remediation is committed in this push:

  • 769c739 docs(classify-signals): reconcile provenance wording with implementation
  • 47119cf fix(classify-signals): harden caller inference per review council findings

Gates green after fixes: 416 tests pass, coverage 96.43% (gate 85%),
ruff check + ruff format --check + mypy --strict all pass. No
protected gate lowered.


Per-reviewer verdicts

Reviewer Verdict Resolution
Adversary (security/resilience) APPROVE MEDIUM astroid cap-bypass closed by scoping inference to in-project names + finally: clear_cache; LOW -32603-on-untrusted-input closed by broadened per-call catch.
Architect (structure/conventions) APPROVE (9/10) HIGH provenance self-contradiction fully reconciled — "lifted" vs "reconstructed" now consistent with each file's true origin.
Guard (intent/governance) APPROVE HIGH provenance drift resolved; _routing.py mislabel fixed. No gate weakened.
Tester (test quality) APPROVE Degradation branches now tested (outer degrade, parametrized per-file exceptions, Uninferable skip, non-degrade exception); protocol test no longer passes with zero signals. inference.py 90%→94%.
Operator/SRE (ops/deps) APPROVE All 3 MEDIUM resolved: stderr diagnostics on every degrade path, broadened per-call catch, inference short-circuit + honest docs. CI gates intact; astroid pinned.
Curator (docs pipeline) APPROVE AGENTS analysis/ tree connectors fixed; README naming.py "source" added; _routing.py comment harmonized.
Envoy (comms/PR) APPROVE British→American spelling fixed; provenance wording reconciled.
Herald (release notes) APPROVE Provenance verbs consistent; README _routing.py suffix aligned.
Scribe (technical docs) APPROVE Doc/spec provenance reconciled; design.md technical claims verified against inference.py.

What was fixed this iteration

  • C-1 — Provenance accuracy (was HIGH): the five signal extractors
    and _routing.py were reconstructed from gaze-py's documented
    behavior
    , not copied verbatim. Wording reconciled across
    README.md, AGENTS.md, proposal.md, design.md, and
    signal-extractors/spec.md. detector.py/complexity.py remain
    correctly described as "lifted" (issue snake-eyes: side effect detector, complexity, and coverage — required analysis methods #4); adapter.py/inference.py
    as original; engine.py as never brought over.
  • C-2 — Resource safety (was MEDIUM): astroid inference is now
    short-circuited to call sites whose unqualified name matches an
    in-project FunctionDef, so stdlib/third-party modules are not parsed
    during inference in the common case (the 16 MiB cap can no longer be
    bypassed by transitive parsing). The MANAGER cache is cleared in a
    finally. Docs corrected to describe the true, honest behavior.
  • C-4 — Resilience (was MEDIUM): the per-call-site catch is
    broadened so any astroid inference exception on untrusted input omits
    that one call instead of failing the whole request with -32603.
  • C-5 — Observability (was MEDIUM): all three degrade paths
    (whole-index, per-file, per-call) now emit a one-line stderr
    diagnostic, matching the documented troubleshooting workflow.
  • S-6 / S-7 — Test coverage (was MEDIUM): added degradation-branch
    tests and strengthened the protocol conformance test (asserts
    non-empty signals + source ∈ the five sources + integer weight).
  • LOWs: spelling (recognisedrecognized, behaviourbehavior),
    exact-count assertion, and a _routing unknown-type (ValueError)
    guard test.

No circular findings were encountered.


Report-only items for the maintainer (cannot be resolved by review)

  • (A) Weight verification. The five reconstructed extractor weights
    (interface 30; visibility ±10; caller 5/15/25; naming +15/−10;
    docstring 15) should be verified against canonical gaze-py source when
    it is available. None were changed or lowered on this branch. If any
    is later found to diverge, escalate to CRITICAL
    (these are Gaze
    gate values).
  • (B) Copyright-header form. The reconstructed files carry a gaze-py
    copyright line plus an Apache-2.0 §4(b) "reconstructed" notice; whether
    that attribution form is correct for documented-behavior
    reconstruction is a licensing judgment. The NOTICE "lifted files"
    line is out of this branch's diff and remains accurate for the
    genuinely-lifted snake-eyes: side effect detector, complexity, and coverage — required analysis methods #4 files. Headers left as-is.
  • (C) Dev-only transitive deps. Confirm the provenance of
    ast-serialize/librt (pulled in by mypy) and that any mypy
    version bump was intended. Runtime footprint is unaffected — astroid
    has zero runtime transitive dependencies.

Non-blocking LOW follow-ups (optional)

  • Stale wording in inference.py module docstring (MemoryError-during-
    infer now skips one call rather than emptying the index).
  • design.md risk bullet still reads "if a lifted weight ever seems
    wrong" (lone residual "lifted" applied to a weight value).
  • test_oversized_file_skipped_before_astroid stubs is_analyzable_file
    rather than exercising the real 16 MiB byte-cap.
  • README and AGENTS list the signals/ files in a different order, and
    proposal.md/design.md "New files" omit _routing.py/_types.py.

Note on scope

The tasks.md completion checkboxes (6.1–6.7), the
code-review: passed / spec-review: passed markers, and the
proposal.md fixture-file list were intentionally left untouched, per
the maintainer's ruling that these are by-design /uf.unleash progress
tracking (this review is the review step of that workflow).


This review was generated by /uf.review-council (AI-assisted).

@jflowers
jflowers merged commit e12fbb8 into main Aug 31, 2026
2 checks passed
@jflowers

jflowers commented Sep 2, 2026

Copy link
Copy Markdown
Contributor Author

PR Cost Report: #10

Session: Propose change with artifacts

ID: ses_fb09d735bffeC3RwlKbHDQg9Uz

Metric Value
Cost (parent only) $486.10
Input tokens 898
Output tokens 515,855
Reasoning tokens 304,959
Cache read tokens 138,773,492
Cache write tokens 30,190,330

Timeline: 2026-08-29 17:17:33 — 2026-08-31 15:51:13

Child sessions: 19 — additional cost: $48.06
Session tree total: $534.16

Child session breakdown
Session Cost Output Tokens
Spec review: adversary (@divisor-adversary subagent) $1.64 7,147
Spec review: architect (@divisor-architect subagent) $1.56 4,722
Spec review: guard (@divisor-guard subagent) $1.73 6,956
Spec review: testing (@divisor-testing subagent) $1.04 4,579
Spec review: sre (@divisor-sre subagent) $1.06 6,054
Spec review: curator (@divisor-curator subagent) $1.07 3,743
Spec review: scribe (@divisor-scribe subagent) $1.79 5,169
Spec review: envoy (@divisor-envoy subagent) $1.10 4,056
Spec review: herald (@divisor-herald subagent) $0.83 2,829
Re-run spec review council (@cobalt-crush-dev subagent) $4.54 9,641
Apply iteration-2 spec fixes (@cobalt-crush-dev subagent) $2.71 1,291
Spec review council iteration 2/3 (@cobalt-crush-dev subagent) $6.19 6,457
Spec review council iteration 3/3 (@cobalt-crush-dev subagent) $4.93 9,510
Spec review council final confirm (@cobalt-crush-dev subagent) $3.45 7,713
Implement classify-signals feature (@cobalt-crush-dev subagent) $4.91 4,044
Code review council (impl) (@cobalt-crush-dev subagent) $6.11 11,832
Code review council iter 2/3 (@cobalt-crush-dev subagent) $3.19 9,524
Investigate gaze+snake-eyes self-analysis (@explore subagent) $0.15 6,952
Investigate gaze analyzer protocol transport (@explore subagent) $0.05 1,469

Session: Skip Gaze Python review

ID: ses_fa69e517fffep8Lq96JY60alMb

Metric Value
Cost (parent only) $24.05
Input tokens 202
Output tokens 90,999
Reasoning tokens 113,841
Cache read tokens 14,137,777
Cache write tokens 1,600,535

Timeline: 2026-08-31 15:52:49 — 2026-08-31 18:15:15

Child sessions: 18 — additional cost: $27.85
Session tree total: $51.89

Child session breakdown
Session Cost Output Tokens
Adversary review classify-signals (@divisor-adversary subagent) $3.50 8,099
Architect review classify-signals (@divisor-architect subagent) $4.70 9,963
Guard review classify-signals (@divisor-guard subagent) $2.55 9,140
Testing review classify-signals (@divisor-testing subagent) $2.39 9,268
SRE review classify-signals (@divisor-sre subagent) $2.15 7,736
Curator review classify-signals (@divisor-curator subagent) $1.64 6,363
Envoy review classify-signals (@divisor-envoy subagent) $1.47 4,156
Herald review classify-signals (@divisor-herald subagent) $1.24 3,865
Scribe review classify-signals (@divisor-scribe subagent) $2.22 6,523
Re-review (verify): adversary (@divisor-adversary subagent) $0.73 2,787
Re-review (verify): architect (@divisor-architect subagent) $0.86 2,534
Re-review (verify): guard (@divisor-guard subagent) $0.57 1,837
Re-review (verify): testing (@divisor-testing subagent) $0.58 1,908
Re-review (verify): sre (@divisor-sre subagent) $0.83 2,371
Re-review (verify): curator (@divisor-curator subagent) $0.68 1,864
Re-review (verify): envoy (@divisor-envoy subagent) $0.46 1,039
Re-review (verify): herald (@divisor-herald subagent) $0.48 1,233
Re-review (verify): scribe (@divisor-scribe subagent) $0.81 1,637

Grand Total: $586.05 across 2 session tree(s) (39 sessions)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant