feat: graph-based targeted test selection for health checker (#1451)

[colehurwitz]

Summary

• Adds find_dependent_tests() to factory/graph.py — reverse-import BFS over the existing graph.json to compute which test files are transitively affected by changed source files
• Threads an optional test_paths parameter through PythonEvaluator → compute_hygiene_results() → run_eval() so pytest runs only the affected subset
• Adds factory eval --targeted CLI flag that computes changed files, resolves dependencies via the graph, and falls back to full suite when selection cannot be trusted
• Updates the health checker prompt to use --targeted by default
• Includes a grep-based inline import scan that catches test files importing changed modules inside function bodies (invisible to graphifyy's static graph)

Safety design

• find_dependent_tests() returns None (not empty set) on any ambiguity → callers always fall back to full suite
• Hard-coded full-suite triggers: conftest.py, __init__.py, pyproject.toml, CI files
• Stale graph → full suite; unknown files → full suite; fan-out >80% → full suite
• Sacred Rule 9 / check_qa_execution() is unaffected (checks event presence, not pytest scope)

Inline import coverage

Adversarial testing found 56% of test files use inline imports invisible to graphifyy's static graph. The grep-based scan (runs after the reverse-BFS) catches these by searching test files for the changed module's dotted import path. Validated on factory/runners/claude.py — previously missed test_ceo_message_events.py and test_verification.py, now found.

Files changed

• factory/graph.py — find_dependent_tests() with reverse BFS + grep-based inline import scan
• factory/eval/languages/python.py — test_paths param on run_tests_with_coverage() / run_tests()
• factory/eval/hygiene.py — thread test_paths through hygiene pipeline
• factory/eval/runner.py — thread test_paths through run_eval()
• factory/cli/eval_cmds.py — --targeted flag + _compute_targeted_test_paths()
• factory/cli/_parser_groups.py — argparse flag registration
• factory/agents/prompts/health_checker.md — one-line prompt update
• tests/test_targeted_tests.py — 39 tests covering BFS, CLI integration, grep scan

Test plan

• 39 tests pass (reverse BFS, multi-hop, staleness gates, exclusion triggers, fan-out threshold, test_paths threading, CLI integration, grep-based scan)
• Full test suite: 2391 passed (no regressions)
• Empirical validation on 3 real commits: targeted selection equivalent to full suite
• Grep scan verified: previously-missed inline importers now found

Closes #1451

🤖 Generated with Claude Code

https://claude.ai/code/session_01YYiei7JyDnjMrt2Dg6jkJu

[github-actions]

Sentrux Quality Report

Absolute

Scanning ....
[scan] git ls-files: 638 total, 624 kept, 14 dropped (ext:14, meta:0, big:0)
[build_project_map] 624 files, 109 unique dirs, 101 cache misses, 5.3ms
[resolve] 1216 resolved, 1612 unresolved (of 2828 total specs)
[resolve_imports] project_map 5.5ms, suffix_idx 1.3ms, suffix_resolve 20.1ms, total 26.9ms
[build_graphs] 624 files | maps 2.6ms, imports 27.1ms, calls+inherit 8.7ms, total 38.5ms | 1215 import, 8991 call, 2 inherit edges
sentrux check — 3 rules checked

Quality: 4201

✗ [Error] max_cc: 6 function(s) exceed max cyclomatic complexity of 30
    factory/cli/_ceo_helpers.py:_validate_ceo_flags (cc=43)
    factory/cli/_ceo_helpers.py:_execute_ceo (cc=43)
    factory/graph.py:find_dependent_tests (cc=41)
    examples/chess_evolve.py:main (cc=33)
    factory/cli/_task_builder.py:_build_ceo_task (cc=32)
    factory/outer_loop/mutations.py:validate_and_repair (cc=31)

✗ 1 violation(s) found

Diff (vs base branch)

Scanning ....
[scan] git ls-files: 638 total, 624 kept, 14 dropped (ext:14, meta:0, big:0)
[build_project_map] 624 files, 109 unique dirs, 101 cache misses, 5.2ms
[resolve] 1216 resolved, 1612 unresolved (of 2828 total specs)
[resolve_imports] project_map 5.4ms, suffix_idx 1.1ms, suffix_resolve 18.9ms, total 25.4ms
[build_graphs] 624 files | maps 1.9ms, imports 25.6ms, calls+inherit 6.1ms, total 33.5ms | 1215 import, 8991 call, 2 inherit edges
sentrux gate — structural regression check

Quality:      4203 -> 4201
Coupling:     0.80 → 0.80
Cycles:       4 → 4
God files:    2 → 2

Distance from Main Sequence: 0.35

✗ DEGRADED
  ✗ Complex functions increased: 68 → 69

[codecov]

Codecov Report

❌ Patch coverage is 92.56757% with 11 lines in your changes missing coverage. Please review.
✅ Project coverage is 82.36%. Comparing base (332af60) to head (506c9d0).
⚠️ Report is 2 commits behind head on main.

Files with missing lines	Patch %	Lines
factory/graph.py	88.77%	6 Missing and 5 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1454      +/-   ##
==========================================
+ Coverage   82.33%   82.36%   +0.03%     
==========================================
  Files         222      224       +2     
  Lines       24555    24904     +349     
  Branches     3925     4005      +80     
==========================================
+ Hits        20217    20512     +295     
- Misses       3405     3445      +40     
- Partials      933      947      +14     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[colehurwitz]

@ceo-review

[github-actions]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: QA: CLEAN — 601 tests pass, composite 0.9631, code review 7/7 PASS, adversarial 13/13 PASS, 0 issues

QA Analysis

Adversarial_Tester Agent Output

• timestamp: 2026-09-04T18:45:19Z
• exit_code: 0

---

---

Adversarial QA Verdict: PASS ✅

Evidence Summary

#	What I Tested	How	Result
Smoke	Project-wide smoke test	pytest tests/test_models.py tests/test_guards.py tests/test_runners.py	165 passed
1	Full targeted test suite	pytest tests/test_targeted_tests.py -v	35/35 passed
2	Import path	python -c 'from factory.graph import find_dependent_tests'	Clean import, correct signature
3	CLI flag	factory eval --help	--targeted present with correct help text
4	Functional: reverse BFS	Real graph.json on disk, multiple scenarios	Correct transitive + direct results
5	Edge: no graph.json	Called with empty temp dir	Returns None (graceful fallback)
6	Edge: corrupt graph.json	Invalid JSON content	Returns None with warning (no crash)
7	Edge: "links" key format	Alternative graph format	Works correctly
8-9	Backward compat	Without --targeted, without attr	_compute_targeted_test_paths never called
10	Call chain threading	inspect.signature on 5 functions	test_paths param present everywhere
11	Fan-out threshold	80% boundary vs 90%	Correct > behavior (80% passes, 90% triggers)
12	Real git repo	_compute_targeted_test_paths(Path('.'))	Graceful None in detached HEAD
13	Signature guard	hygiene.py introspection	Correctly omits test_paths for older evaluators

One minor observation (not a blocker): Root-level conftest.py without a directory prefix won't match **/conftest.py in fnmatch. This is a low-risk gap with multiple safety nets.

The feature is solid. Every failure path returns None → full suite fallback. No crashes, no regressions.

⚠ CEO IDENTITY RE-ANCHOR (Sacred Rule 8)
You are the Factory CEO. You orchestrate, delegate, and decide. You do NOT implement.
If you are about to write code, run tests, do research, or fix bugs — STOP and spawn the appropriate agent.
Re-read your Permitted/Forbidden Actions lists in the Identity section above.

---

Posted by Factory CEO

[colehurwitz]

@ceo-review

[colehurwitz]

@lambdabaa a small PR to lower the amount of testing for health checker based on what is affected by the PR. This step is extremely slow and times out a lot.

[github-actions]

✅ Factory Review: KEEP

Verdict: KEEP
Reason: QA: CLEAN — 601 tests pass, composite 0.963, all 7 code review categories PASS, all 8 adversarial acceptance criteria verified. No critical issues.

QA Analysis

Adversarial QA Report — PR #1454: Graph-Based Targeted Test Selection

Project type: Library (CLI + eval framework)
Date: 2026-09-04
Verdict: ✅ PASS

---

Smoke Test

Command: uv run pytest tests/test_models.py tests/test_guards.py tests/test_runners.py -x -q --tb=short -k 'not (BobAuth or preflight_error_unchanged)'
Result: 165 passed in 4.48s
Status: ✅ PASS

---

Test Plan (derived from PR scope)

#	Criterion	Status
A	All 39 unit tests pass	✅ VERIFIED
B	find_dependent_tests works with real temp graph	✅ VERIFIED
C	Edge cases: stale/conftest/unknown/fan-out	✅ VERIFIED
D	Grep-based inline import scan works	✅ VERIFIED
E	inspect.signature guard works	✅ VERIFIED
F	Adversarial inputs don't crash	✅ VERIFIED
G	--targeted CLI flag exists and parses	✅ VERIFIED
H	test_paths threads through full chain	✅ VERIFIED

---

Test A: All 39 unit tests pass

Command: uv run pytest tests/test_targeted_tests.py -v
Output: 39 passed in 0.29s — all tests green
Status: ✅ VERIFIED

---

Test B: find_dependent_tests with real temp graph

Command: Python script creating temp graph.json and calling find_dependent_tests directly
Evidence:

• Change src/a.py → found ['tests/test_a.py', 'tests/test_b.py'] (transitive BFS)
• Change src/b.py → found ['tests/test_b.py'] only (direct import)
• Change tests/test_a.py → found ['tests/test_a.py'] (self-inclusion)
  Status: ✅ VERIFIED

---

Test C: Edge cases

All tested via direct Python calls with mocked staleness:

Edge Case	Input	Expected	Actual	Status
Stale graph	is_graph_stale=True	None	None	✅
Unknown staleness	is_graph_stale=None	None	None	✅
conftest.py	['tests/conftest.py']	None	None	✅
Nested conftest	['tests/sub/conftest.py']	None	None	✅
__init__.py	['factory/__init__.py']	None	None	✅
Unknown .py file	['src/unknown.py']	None	None	✅
Fan-out 100%	10/10 tests depend	None	None	✅
Empty list	[]	None	None	✅
pyproject.toml	['pyproject.toml']	None	None	✅
CI workflow	['.github/workflows/ci.yml']	None	None	✅
Fan-out exactly 80%	8/10 tests depend	set(8)	set(8)	✅
Fan-out 90%	9/10 tests depend	None	None	✅

Status: ✅ VERIFIED

---

Test D: Grep-based inline import scan

Command: Python script creating test files with inline imports on disk
Evidence:

• Test file with from factory.runners.claude import run inside function body → grep found it
• Test file without matching import → grep correctly returned empty
• Top-level file (setup.py, parts < 2) → grep skipped correctly
  Status: ✅ VERIFIED

---

Test E: inspect.signature guard

Command: Python scripts testing both old-style and new-style evaluators
Evidence:

• PythonEvaluator.run_tests_with_coverage has test_paths param → ✅
• PythonEvaluator.run_tests has test_paths param → ✅
• _collect_test_and_coverage source contains inspect.signature → ✅
• Old evaluator without test_paths param → guard omits it, no crash → ✅
• New evaluator with test_paths param → guard passes it through → ✅
  Status: ✅ VERIFIED

---

Test F: Adversarial inputs

Test	Input	Expected	Actual	Status
Empty list	[]	None	None	✅
File with hyphen	src/my-module.py	set()	set()	✅
File with space	src/my module.py	set()	set()	✅
10-hop transitive chain	mod_0 ← ... ← mod_9 ← test_deep	{tests/test_deep.py}	{tests/test_deep.py}	✅
Corrupted JSON	{invalid json!!!}	None	None	✅
Empty nodes/edges	{'nodes':[], 'edges':[]}	None	None	✅
Non-python file	Makefile	None	None	✅
Mixed full-suite trigger	['src/a.py', 'tests/conftest.py']	None	None	✅
links key alias	graph with links not edges	set with results	set with results	✅

Status: ✅ VERIFIED

---

Test G: --targeted CLI flag

Command: parser.parse_args(['eval', '/tmp/fake', '--targeted'])
Evidence:

• With --targeted: args.targeted = True → ✅
• Without flag: args.targeted = False → ✅
• Flag registered in factory/cli/_parser_groups.py with action="store_true" → ✅
  Status: ✅ VERIFIED

---

Test H: test_paths threading through full chain

Command: inspect.signature() on each function in the chain
Evidence:

• cmd_eval → reads args.targeted, calls _compute_targeted_test_paths, passes to run_eval → ✅
• run_eval has test_paths param → ✅
• compute_hygiene_results has test_paths param → ✅
• _collect_test_and_coverage has test_paths param → ✅
• PythonEvaluator.run_tests_with_coverage appends paths to pytest command after -q flag → ✅
• PythonEvaluator.run_tests delegates to run_tests_with_coverage with test_paths → ✅
  Status: ✅ VERIFIED

---

Process Cleanup

No servers started. No tmux sessions created. No orphaned processes.

---

Adversarial Verdict: ✅ PASS

All 8 acceptance criteria verified with concrete evidence. The feature:

1. Correctly performs reverse BFS over import graphs to find affected tests
2. Correctly falls back to full suite on stale graphs, conftest changes, unknown files, and high fan-out
3. Catches inline imports via grep that graphify misses
4. Guards against evaluators that lack the test_paths parameter via inspect.signature
5. Threads test_paths through the full CLI → runner → hygiene → evaluator chain
6. Handles adversarial inputs (empty lists, special chars, deep chains, corrupted JSON) gracefully without crashes

---

Posted by Factory CEO