Skip to content

feat: graph-based targeted test selection for health checker (#1451) - #1455

Closed
colehurwitz wants to merge 1 commit into
akashgit:mainfrom
colehurwitz:factory/run-c378458d
Closed

colehurwitz wants to merge 1 commit into
akashgit:mainfrom
colehurwitz:factory/run-c378458d

Conversation

@colehurwitz

Copy link
Copy Markdown
Collaborator

Summary

  • Adds find_dependent_tests() to factory/graph.py — reverse-import BFS over the existing graph.json to compute which test files are transitively affected by changed source files
  • Threads an optional test_paths parameter through PythonEvaluatorcompute_hygiene_results()run_eval() so pytest runs only the affected subset
  • Adds factory eval --targeted CLI flag that computes changed files, resolves dependencies via the graph, and falls back to full suite when selection cannot be trusted
  • Updates the health checker prompt to use --targeted by default

Safety design

  • find_dependent_tests() returns None (not empty set) on any ambiguity → callers always fall back to full suite
  • Hard-coded full-suite triggers: conftest.py, __init__.py, pyproject.toml, CI files
  • Stale graph → full suite; unknown files → full suite; fan-out >80% → full suite
  • Sacred Rule 9 / check_qa_execution() is unaffected (checks event presence, not pytest scope)

Files changed

  • factory/graph.py — new find_dependent_tests() (122 lines)
  • factory/eval/languages/python.pytest_paths param on run_tests_with_coverage() / run_tests()
  • factory/eval/hygiene.py — thread test_paths through hygiene pipeline
  • factory/eval/runner.py — thread test_paths through run_eval()
  • factory/cli/eval_cmds.py--targeted flag + _compute_targeted_test_paths()
  • factory/cli/_parser_groups.py — argparse flag registration
  • factory/agents/prompts/health_checker.md — one-line prompt update
  • tests/test_targeted_tests.py — 21 tests covering all critical paths

Test plan

  • 21 new tests pass (reverse BFS, multi-hop, staleness gates, exclusion triggers, fan-out threshold, test_paths threading)
  • Full test suite: 2391 passed (no regressions)
  • Lint clean, type check clean (1 pre-existing error in unrelated file)
  • Adversarial testing: 35 edge cases verified with evidence
  • Code review: 6/7 categories PASS, 1 important (CLI integration test gap, acceptable for v1)

Closes #1451

🤖 Generated with Claude Code

https://claude.ai/code/session_01YYiei7JyDnjMrt2Dg6jkJu

…t#1451)

Add reverse-import BFS over graph.json to select only tests affected by
changed files, reducing health-check gate time from 8-10 min to 1-3 min
for focused changes. Falls back to full suite on any ambiguity (stale
graph, conftest changes, unknown files, >80% fan-out).

- factory/graph.py: find_dependent_tests() with conservative None fallback
- factory/eval/languages/python.py: optional test_paths parameter
- factory/eval/runner.py + cli: --targeted flag with structlog auditing
- factory/agents/prompts/health_checker.md: use factory eval --targeted
- 21 new tests covering BFS, fan-out, triggers, and path threading

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@colehurwitz

Copy link
Copy Markdown
Collaborator Author

Duplicate of #1454 — closing this one.

@colehurwitz colehurwitz closed this Sep 4, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Health checker: graph-based targeted test selection

1 participant