diff --git a/docs/superpowers/plans/2026-03-23-commit-semantic-domain-quality-gstack-plan.md b/docs/superpowers/plans/2026-03-23-commit-semantic-domain-quality-gstack-plan.md new file mode 100644 index 0000000..5d4466a --- /dev/null +++ b/docs/superpowers/plans/2026-03-23-commit-semantic-domain-quality-gstack-plan.md @@ -0,0 +1,406 @@ +# GStack Implementation Plan — Commit-Semantic Domain Quality Optimization + +Branch: `feature/commit-semantic-domain` +Spec: `docs/superpowers/specs/2026-03-23-commit-semantic-domain-quality-design.md` +Mode: **gstack plan** (not superpowers) + +## Goal + +Make `commit-semantic` domain output good enough for stable downstream use with an **LLM-only** semantic path. +This plan fixes three things together: +1. domain schema quality +2. classification quality +3. runtime mode/provenance clarity for LLM execution + +This plan does **not** preserve semantic fallback behavior. If LLM discover/classify is unavailable or invalid, the pipeline should fail fast rather than emit low-quality semantic output. + +--- + +## What already exists + +These are already real and should be reused, not rebuilt: + +- `skills/commit-semantic/run.py` + - full 5-stage runner exists + - real repo pipeline already runs end-to-end + - discover / ingest / aggregate / distill / export hooks already exist +- `src/commit_semantic/domain_utils.py` + - contains pure helpers already used by the pipeline + - best current home for new normalization/scoring logic +- `tests/e2e/test_commit_semantic.py` + - already covers real pipeline behavior, fallback behavior, and repo-style output checks +- `tests/test_commit_semantic_domain.py` + - already covers pure-function behavior and is the right place for more rule-focused tests +- repo-level baseline already exists + - current real baseline: `uncategorized_ratio = 0.1762` + +--- + +## NOT in scope + +Do **not** include any of the following in this implementation: + +- changing aggregate / distill scoring formulas +- changing `commit-extract` output schema +- integrating demand stage +- adding write-back / feedback correction loops +- building a new cache subsystem +- refactoring unrelated pipeline files just because they are large +- changing spec during implementation; if spec/code conflict appears, stop and ask + +--- + +## File responsibilities + +### Files to modify + +#### 1. `src/commit_semantic/domain_utils.py` +Add pure rule logic here: +- domain name normalization +- noise filtering +- duplicate / near-duplicate merge +- deterministic scoring weights +- ambiguity gate +- path-disable-after-multi-domain-failure behavior + +This file should own “what the rules are.” + +#### 2. `skills/commit-semantic/run.py` +Keep orchestration here: +- call pure helpers +- persist discover provenance in `domains.json` +- restore provenance on cache hit +- set runtime mode metadata +- export runtime mode fields +- enforce **LLM-only** discover/classify behavior +- fail fast when LLM execution is unavailable or invalid + +This file should own “when each rule path is used.” + +#### 3. `tests/test_commit_semantic_domain.py` +Add focused unit tests for: +- normalization behavior +- merge thresholds +- scoring weights +- ambiguity gate +- path scoring disabled after commit-level multi-domain failure + +#### 4. `tests/e2e/test_commit_semantic.py` +Add pipeline-level tests for: +- normalized `domains.json` +- cache hit restoring provenance +- summary mode fields +- multi-domain path failure behavior +- repo-level regression checks + +#### 5. `tests/test_export_dataclasses.py` +Only touch if needed for summary schema assertions. +Prefer keeping mode/provenance tests in E2E unless there is a clean export-only assertion. + +#### 6. `skills/commit-semantic/SKILL.md` +Update only after runtime mode/export behavior is actually implemented and verified. + +--- + +## 4 execution checkpoints + +This plan is intentionally split into 4 hard checkpoints. Do not start the next one until the current one is green. + +```text +Checkpoint 1: Schema normalization +Checkpoint 2: Deterministic classify upgrade +Checkpoint 3: Mode / provenance reporting +Checkpoint 4: LLM-first default switch +``` + +At the end of each checkpoint: +- targeted tests pass +- no broken existing tests in touched areas +- repo-level pipeline still runs + +--- + +## Checkpoint 1 — Schema normalization + +### Objective +Make `domains.json` structurally cleaner before trying to lower `uncategorized`. + +### Work + +#### Task 1.1 — add pure normalization helpers +Modify: +- `src/commit_semantic/domain_utils.py` + +Add pure helpers for: +- normalize domain name +- singular/plural merge (`test` + `tests` => `tests`) +- noise token rejection +- duplicate / near-duplicate merge +- minimum quality gate +- winner selection priority during merge + +#### Task 1.2 — wire normalization into discover save paths +Modify: +- `skills/commit-semantic/run.py` + +Apply the same normalization path to: +- local fallback discover +- `complete_discover()` LLM path + +Keep fingerprint logic unchanged. + +### Required tests + +Add unit tests in `tests/test_commit_semantic_domain.py` for: +- exact duplicate merge +- singular/plural merge +- keyword-overlap merge threshold +- path-overlap merge threshold +- noise filtering +- winner selection priority + +Add E2E fixture in `tests/e2e/test_commit_semantic.py` for a **combined normalization case**: +- duplicate `test/tests` +- noisy process tokens +- overlapping keyword/path domains +- expected cleaned `domains.json` + +### Exit criteria +- duplicate normalized domains = 0 in test fixture +- `test/tests` collapse correctly +- bad process tokens do not survive as top-level domains unless justified by spec rules + +--- + +## Checkpoint 2 — Deterministic classify upgrade + +### Objective +Lower bad fallback assignments and reduce `uncategorized` only where evidence is genuinely strong. + +### Work + +#### Task 2.1 — move scoring rules into pure functions +Modify: +- `src/commit_semantic/domain_utils.py` + +Implement deterministic scoring contract exactly as spec says: +- path-prefix = 5 +- theme token = 3 +- summary token = 2 +- section-name token = 2 +- domain-keyword = 1 +- repeated hits do not stack per signal type +- minimum assignment score = 4 +- ambiguous if `top1 - top2 < 2` + +#### Task 2.2 — enforce path-disable-after-commit-failure +Modify: +- `skills/commit-semantic/run.py` +- maybe `src/commit_semantic/domain_utils.py` if context flag belongs there + +Rule: +- if commit-level path convergence fails because files span multiple candidate domains, + unit-level fallback scoring must **not** use path-prefix evidence for that commit. + +#### Task 2.3 — keep single-domain fast path intact +Preserve existing behavior: +- if commit-level path assignment cleanly converges to one domain, keep commit-level fast path + +### Required tests + +Add unit tests in `tests/test_commit_semantic_domain.py` for: +- scoring weights +- non-stacking hits +- minimum score gate +- ambiguity gate + +Add E2E tests in `tests/e2e/test_commit_semantic.py` for: +- single-domain path fast path still works +- multi-domain commit falls back to unit scoring +- after multi-domain failure, path scoring is disabled +- ambiguous non-path signals remain `uncategorized` when no stronger signal exists + +### Exit criteria +- no regression in single-domain assignment +- multi-domain failure path no longer silently reuses whole-commit path evidence at unit level +- fallback assignments become more conservative, not more eager + +--- + +## Checkpoint 3 — Mode / provenance reporting + +### Objective +Make output truthful about how it was produced, especially across cache hits and degraded runs. + +### Work + +#### Task 3.1 — persist discover provenance in `domains.json` +Modify: +- `skills/commit-semantic/run.py` + +Persist alongside `_fingerprint` and `domains`: +- `discover_mode` +- `orchestration_mode_at_discover` + +#### Task 3.2 — restore provenance on cache hit +Modify: +- `skills/commit-semantic/run.py` + +On discover cache hit: +- restore persisted mode data into `HarnessState.metadata` +- do not silently treat cache hit as “current default mode” + +#### Task 3.3 — add exported mode fields +Modify: +- `skills/commit-semantic/run.py` +- `skills/commit-semantic/SKILL.md` + +Export in `summary.json`: +- `orchestration_mode` +- `discover_mode` +- `classify_mode` + +### Required tests + +Add E2E tests for: +- discover cache hit restores provenance correctly +- summary mode fields exist +- local fallback run marks fallback/degraded truthfully +- mixed-degraded scenario is represented correctly + +### Critical failure-path tests +These are hard requirements, not optional: +- discover cache hit should not erase provenance +- mode fields must reflect actual execution, not default assumptions + +### Exit criteria +- summary always tells the truth about execution mode +- cache hit path is no longer a silent provenance lie + +--- + +## Checkpoint 4 — LLM-only execution + +### Objective +Make discover/classify strictly LLM-only and fail fast on invalid or unavailable LLM execution. + +### Work + +#### Task 4.0 — implement fail-fast decision table as a first-class task +Modify: +- `skills/commit-semantic/run.py` +- tests in `tests/e2e/test_commit_semantic.py` + +Do **not** leave this implicit. Implement and test each failure branch directly: + +- discover: LLM output empty/invalid + - fail immediately +- discover: LLM unavailable + - fail immediately +- classify: any batch fails + - fail immediately +- classify: all batches must succeed before semantic output is considered valid +- local/no-orchestrator mode + - fail immediately instead of emitting fallback semantic output + +Required tests for Task 4.0: +- discover invalid/empty LLM output fails +- discover unavailable orchestration fails +- classify partial batch failure fails the stage +- classify total failure fails the stage +- no semantic output is emitted as a successful degraded fallback + +#### Task 4.1 — define actual default behavior in runner +Modify: +- `skills/commit-semantic/run.py` + +Desired behavior: +- default runtime is LLM-only +- no semantic fallback path remains for discover/classify +- exported mode fields describe LLM execution only when the run succeeds + +### Required tests + +Add tests for: +- default run requires LLM orchestration +- local run without orchestration fails cleanly +- success path exports truthful LLM mode fields only after valid completion + +### Exit criteria +- default behavior is LLM-only +- failure branches are explicitly implemented and tested +- no low-quality semantic fallback output is emitted + +--- + +## Verification plan + +### Targeted tests after each checkpoint +Run the smallest relevant set first. + +### Required full verification before calling the work done +Run all of these: + +```bash +pytest tests/test_commit_semantic_domain.py -q +pytest tests/e2e/test_commit_semantic.py -q +pytest tests/test_export_dataclasses.py -q +pytest tests/test_commit_extract_rewrite.py tests/e2e/test_pipeline_e2e.py tests/test_repo_structure.py -q +pytest tests -q +ruff check . +``` + +### Required real repo manual validation +Re-run the real worktree pipeline with real LLM orchestration and inspect: +- `data/commit-semantic/domains.json` +- `data/commit-semantic/domains-aggregated.jsonl` +- `data/commit-semantic/summary.json` + +Manual checklist: +- no duplicate domains like `test/tests` +- `uncategorized_ratio < 0.1762` +- top 5 domains no longer dominated by obvious process/noise buckets +- top 5 domains have stronger paths or stronger deduplicated keywords +- summary contains mode fields +- summary mode fields show successful LLM execution, not fallback/degraded semantics + +--- + +## Failure modes to watch while implementing + +1. **Cache lies about provenance** +- Risk: cache hit reports current mode instead of original discover mode +- Must be covered by tests before checkpoint 3 is complete + +2. **Multi-domain commits get false-confidence classification** +- Risk: path evidence leaks into ambiguous classification logic and produces overconfident wrong assignments +- Must be covered by tests before checkpoint 2 is complete + +3. **Normalization becomes too aggressive** +- Risk: legitimate distinct domains collapse into one bucket +- Mitigation: thresholded merge tests and explicit winner rules + +4. **Silent semantic degradation** +- Risk: pipeline emits seemingly-valid semantic output even though LLM execution failed or never happened +- Mitigation: LLM-only fail-fast behavior and stage-level failure tests + +--- + +## Minimal commit strategy + +Commit at the end of each checkpoint, not at the end of the whole effort. + +Suggested commit boundaries: +1. `feat: normalize commit semantic domains` +2. `feat: gate commit semantic fallback classification` +3. `feat: persist commit semantic runtime provenance` +4. `feat: prefer llm-first commit semantic execution` + +--- + +## Plan-specific notes + +- Do not edit the spec during implementation. If spec and code reality conflict, stop and ask. +- Prefer putting new rule logic in `domain_utils.py`, not `run.py`. +- Do not add new infra or generalized caches just because it seems cleaner. +- Keep diffs explicit and checkpointed. diff --git a/docs/superpowers/plans/2026-03-23-commit-semantic-domain-quality-implementation.md b/docs/superpowers/plans/2026-03-23-commit-semantic-domain-quality-implementation.md new file mode 100644 index 0000000..a1edfa0 --- /dev/null +++ b/docs/superpowers/plans/2026-03-23-commit-semantic-domain-quality-implementation.md @@ -0,0 +1,547 @@ +# Commit-Semantic Domain Quality Optimization Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Improve `commit-semantic` domain quality by cleaning domain schema, reducing incorrect/over-eager classification, reporting runtime provenance, and switching the default runtime to LLM-first semantics with explicit degraded fallback behavior. + +**Architecture:** Keep `skills/commit-semantic/run.py` as orchestration and move domain normalization / deterministic classification rules into pure functions in `src/commit_semantic/domain_utils.py`. Implement in four checkpoints: schema normalization, deterministic classification gating, mode/provenance reporting, then LLM-first default switching. Every checkpoint must be independently testable and keep the repo-level pipeline runnable. + +**Tech Stack:** Python 3.10+, pytest, JSONL artifacts, existing `HarnessState`, existing `commit-semantic` prompts and pipeline. + +--- + +## File Structure + +### Primary files to modify +- `src/commit_semantic/domain_utils.py` + - Add pure functions for domain normalization, duplicate merging, noise filtering, deterministic scoring, and ambiguity gating. +- `skills/commit-semantic/run.py` + - Keep orchestration only: call pure functions, persist provenance, export mode fields, and switch default runtime behavior. +- `tests/test_commit_semantic_domain.py` + - Add pure-function tests for normalization, merge thresholds, deterministic scoring, and path-disable-after-multi-domain-failure logic. +- `tests/e2e/test_commit_semantic.py` + - Add pipeline-level tests for cache/provenance restore, summary mode fields, local fallback behavior, and repo-style classification scenarios. +- `skills/commit-semantic/SKILL.md` + - Update runtime mode semantics and summary/output contract after code/tests are done. + +### Secondary files (only if needed) +- `docs/superpowers/specs/2026-03-23-commit-semantic-domain-quality-design.md` + - Only if implementation reveals wording mismatch requiring a tiny spec correction. + +### Files that should NOT be touched in this plan +- `skills/commit-extract/run.py` +- `skills/repo_structure/run.py` +- `src/demand/**` +- aggregate/distill scoring formulas outside the mode/schema/classify work + +--- + +## Execution Order + +1. **Checkpoint A — Domain schema normalization** +2. **Checkpoint B — Deterministic classify upgrade** +3. **Checkpoint C — Mode/provenance reporting** +4. **Checkpoint D — LLM-first default switch** + +Do not start the next checkpoint until the current checkpoint tests pass. + +--- + +### Task 1: Add domain normalization pure functions + +**Files:** +- Modify: `src/commit_semantic/domain_utils.py` +- Test: `tests/test_commit_semantic_domain.py` + +- [ ] **Step 1: Write failing normalization tests** + +Add tests for: +- singular/plural merge: `test` + `tests` => `tests` +- exact duplicate merge +- near-duplicate merge with keyword Jaccard overlap >= 0.6 +- near-duplicate merge with path-prefix overlap >= 0.5 +- noise token rejection (`add`, `update`, `fix`, `impl`, `phase`, `final`, `worktree`) +- winner selection priority: + 1. non-empty paths + 2. more keywords + 3. non-noise name + 4. lexical tie-break + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/test_commit_semantic_domain.py -q +``` +Expected: new normalization tests fail because functions do not exist yet. + +- [ ] **Step 3: Implement minimal pure functions** + +Add pure helpers in `src/commit_semantic/domain_utils.py`: +- `normalize_domain_name(name: str) -> str` +- `is_noise_domain_name(name: str) -> bool` +- `merge_domain_candidates(domains: list[dict]) -> list[dict]` +- `normalize_domains(domains: list[dict]) -> list[dict]` + +Rules must follow the spec exactly: +- preserve `domain` field name +- lowercase + dash-case +- singular/plural merge on exact normalized stems +- near-duplicate merge on keyword Jaccard or path overlap thresholds +- apply minimum quality gate + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/test_commit_semantic_domain.py -q +``` +Expected: all normalization tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add src/commit_semantic/domain_utils.py tests/test_commit_semantic_domain.py +git commit -m "feat: normalize commit semantic domains" +``` + +--- + +### Task 2: Wire normalization into discover paths + +**Files:** +- Modify: `skills/commit-semantic/run.py` +- Test: `tests/e2e/test_commit_semantic.py` +- Test: `tests/test_commit_semantic_domain.py` + +- [ ] **Step 1: Write failing discover integration tests** + +Add tests proving: +- `complete_discover()` normalizes LLM output before saving +- local fallback discover also normalizes before saving +- a combined fixture with `test/tests` + noise tokens + overlapping keywords produces a cleaned `domains.json` + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py tests/test_commit_semantic_domain.py -q +``` +Expected: new discover normalization tests fail. + +- [ ] **Step 3: Implement minimal orchestration changes** + +In `skills/commit-semantic/run.py`: +- import and call `normalize_domains()` in both: + - local fallback discover path + - `complete_discover()` +- do not change fingerprint behavior +- keep existing prompt preparation behavior intact + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py tests/test_commit_semantic_domain.py -q +``` +Expected: discover normalization tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add skills/commit-semantic/run.py tests/e2e/test_commit_semantic.py tests/test_commit_semantic_domain.py +git commit -m "feat: normalize discovered commit semantic domains" +``` + +--- + +### Task 3: Add deterministic scoring and ambiguity gate as pure functions + +**Files:** +- Modify: `src/commit_semantic/domain_utils.py` +- Test: `tests/test_commit_semantic_domain.py` + +- [ ] **Step 1: Write failing scoring tests** + +Add pure-function tests for: +- scoring weights: + - path-prefix = 5 + - theme token = 3 + - summary token = 2 + - section-name token = 2 + - domain-keyword = 1 +- repeated hits do not stack beyond one hit per signal type +- minimum score gate = 4 +- ambiguous if `top1 - top2 < 2` +- when commit-level multi-domain failure already occurred, unit-level scoring must ignore path-prefix signals + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/test_commit_semantic_domain.py -q +``` +Expected: new scoring tests fail. + +- [ ] **Step 3: Implement minimal pure helpers** + +Add helpers such as: +- `score_unit_for_domain(...)` +- `pick_domain_for_unit(...)` +- `classify_units_locally(...)` + +Make the API explicit enough that `run.py` just passes context and receives decisions. + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/test_commit_semantic_domain.py -q +``` +Expected: scoring tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add src/commit_semantic/domain_utils.py tests/test_commit_semantic_domain.py +git commit -m "feat: add deterministic domain classification scoring" +``` + +--- + +### Task 4: Replace local classify logic in runner with pure-function orchestration + +**Files:** +- Modify: `skills/commit-semantic/run.py` +- Test: `tests/e2e/test_commit_semantic.py` + +- [ ] **Step 1: Write failing ingest/classify integration tests** + +Add tests proving: +- multi-domain commit failure disables path-based scoring at unit fallback time +- strong single-domain path match still assigns at commit level +- ambiguous unit remains `uncategorized` if no LLM and no sufficient non-path score +- mixed/no-path unit uses non-path scoring only + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py -q +``` +Expected: new integration tests fail. + +- [ ] **Step 3: Implement minimal runner changes** + +In `skills/commit-semantic/run.py`: +- keep commit-level `assign_domain_by_path()` fast path +- when commit-level convergence fails, pass explicit context to pure functions so unit fallback does NOT use path scoring +- keep external orchestration metadata path intact +- keep `complete_classify()` behavior intact for external LLM responses + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py -q +``` +Expected: ingest/classify integration tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add skills/commit-semantic/run.py tests/e2e/test_commit_semantic.py +git commit -m "feat: gate commit semantic fallback classification" +``` + +--- + +### Task 5: Add runtime mode provenance persistence and cache restore + +**Files:** +- Modify: `skills/commit-semantic/run.py` +- Test: `tests/e2e/test_commit_semantic.py` +- Test: `tests/test_commit_semantic_domain.py` (if helper extraction is needed) + +- [ ] **Step 1: Write failing provenance tests** + +Add tests proving: +- `domains.json` persists discover provenance fields +- cache hit restores `discover_mode` into `HarnessState.metadata` +- export reports actual execution mode, not default assumptions +- local fallback and mixed-degraded cases are distinguishable in summary output + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py -q +``` +Expected: provenance/mode tests fail. + +- [ ] **Step 3: Implement minimal provenance changes** + +In `skills/commit-semantic/run.py`: +- persist discover provenance into `domains.json` +- restore provenance on cache hit +- set `orchestration_mode`, `discover_mode`, `classify_mode` in `HarnessState.metadata` +- preserve current behavior for `external_orchestration` + +Do not invent a generic state framework; keep changes local to this pipeline. + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py -q +``` +Expected: provenance tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add skills/commit-semantic/run.py tests/e2e/test_commit_semantic.py +git commit -m "feat: persist commit semantic runtime provenance" +``` + +--- + +### Task 6: Export mode fields and update summary contract + +**Files:** +- Modify: `skills/commit-semantic/run.py` +- Modify: `skills/commit-semantic/SKILL.md` +- Test: `tests/test_export_dataclasses.py` +- Test: `tests/e2e/test_commit_semantic.py` + +- [ ] **Step 1: Write failing export tests** + +Add/adjust tests asserting `summary.json` includes: +- `orchestration_mode` +- `discover_mode` +- `classify_mode` +- existing fields remain intact + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/test_export_dataclasses.py tests/e2e/test_commit_semantic.py -q +``` +Expected: export mode-field tests fail. + +- [ ] **Step 3: Implement minimal export change** + +In `skills/commit-semantic/run.py`: +- emit the three mode fields from `state.metadata` +- do not remove existing summary fields + +In `skills/commit-semantic/SKILL.md`: +- document the new mode fields and degraded-mode visibility + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/test_export_dataclasses.py tests/e2e/test_commit_semantic.py -q +``` +Expected: export tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add skills/commit-semantic/run.py skills/commit-semantic/SKILL.md tests/test_export_dataclasses.py tests/e2e/test_commit_semantic.py +git commit -m "feat: report commit semantic runtime modes" +``` + +--- + +### Task 7: Precompute normalization and scoring inputs + +**Files:** +- Modify: `src/commit_semantic/domain_utils.py` +- Test: `tests/test_commit_semantic_domain.py` + +- [ ] **Step 1: Write failing precompute-focused tests** + +Add tests that lock behavior while allowing internal optimization: +- normalized keywords are deduplicated once +- repeated scoring on the same domains/units reuses precomputed normalized structures +- classification behavior stays identical after precompute refactor + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/test_commit_semantic_domain.py -q +``` +Expected: new tests fail. + +- [ ] **Step 3: Implement minimal precompute layer** + +Add a small pure precompute step for: +- normalized domain keywords +- normalized domain names +- path-prefix structures if needed + +Do not add a separate cache subsystem. + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/test_commit_semantic_domain.py -q +``` +Expected: precompute tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add src/commit_semantic/domain_utils.py tests/test_commit_semantic_domain.py +git commit -m "refactor: precompute commit semantic scoring inputs" +``` + +--- + +### Task 8: Switch default runtime to LLM-first semantics + +**Files:** +- Modify: `skills/commit-semantic/run.py` +- Test: `tests/e2e/test_commit_semantic.py` + +- [ ] **Step 1: Write failing mode-default tests** + +Add tests proving: +- default path is now LLM-first semantics +- fallback remains available and explicitly marked degraded +- repo-style run without external orchestration still succeeds via fallback, but exported mode reflects that it degraded + +- [ ] **Step 2: Run tests to verify RED** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py -q +``` +Expected: mode-default tests fail. + +- [ ] **Step 3: Implement minimal default-switch behavior** + +Update `skills/commit-semantic/run.py` so that: +- design intent is LLM-first by default +- local execution still succeeds via fallback when orchestration is unavailable +- exported mode fields tell the truth about what happened + +Do not remove fallback. + +- [ ] **Step 4: Run tests to verify GREEN** + +Run: +```bash +pytest tests/e2e/test_commit_semantic.py -q +``` +Expected: mode-default tests pass. + +- [ ] **Step 5: Commit** + +```bash +git add skills/commit-semantic/run.py tests/e2e/test_commit_semantic.py +git commit -m "feat: prefer llm-first commit semantic execution" +``` + +--- + +### Task 9: Run full verification and real-repo regression + +**Files:** +- Modify: none unless failures require fixes in files above +- Test: `tests/test_commit_semantic_domain.py` +- Test: `tests/e2e/test_commit_semantic.py` +- Test: `tests/test_export_dataclasses.py` +- Test: any touched repo-structure tests if summary contract impacts them + +- [ ] **Step 1: Run targeted test suite** + +Run: +```bash +pytest tests/test_commit_semantic_domain.py tests/e2e/test_commit_semantic.py tests/test_export_dataclasses.py -q +``` +Expected: all pass. + +- [ ] **Step 2: Run broader regression suite** + +Run: +```bash +pytest tests/test_commit_extract_rewrite.py tests/test_repo_structure.py tests/e2e/test_pipeline_e2e.py -q +``` +Expected: all pass. + +- [ ] **Step 3: Run full test suite** + +Run: +```bash +pytest tests -q +``` +Expected: full suite green. + +- [ ] **Step 4: Run lint** + +Run: +```bash +ruff check . +``` +Expected: `All checks passed!` + +- [ ] **Step 5: Re-run real repo manual validation** + +Run: +```bash +python skills/commit-semantic/run.py run --force +``` +Then inspect: +- `data/commit-semantic/domains.json` +- `data/commit-semantic/domains-aggregated.jsonl` +- `data/commit-semantic/summary.json` + +Verify manually: +- no duplicate `test/tests` +- `uncategorized_ratio < 0.1762` +- top 5 domains have stronger paths/keywords +- mode fields are present and truthful + +- [ ] **Step 6: Commit** + +```bash +git add skills/commit-semantic/run.py src/commit_semantic/domain_utils.py skills/commit-semantic/SKILL.md tests/test_commit_semantic_domain.py tests/e2e/test_commit_semantic.py tests/test_export_dataclasses.py +git commit -m "feat: improve commit semantic domain quality" +``` + +--- + +## Test Plan Artifact + +Affected areas to verify during QA / manual validation: +- `commit-semantic` full local run on real repo data +- discover cache hit behavior +- mixed/no-path classification fallback behavior +- summary mode reporting +- repo-level top domain quality and `uncategorized_ratio` + +Critical paths: +- real repo `commit-extract -> commit-semantic` run +- local fallback discover/classify path +- future external orchestration compatibility through `complete_discover()` / `complete_classify()` + +--- + +## Plan Review Notes + +This plan intentionally avoids: +- changing aggregate/distill scoring formulas +- touching demand integration +- modifying commit-extract schema +- adding heavy new caching or infrastructure + +It assumes the current approved spec at: +- `docs/superpowers/specs/2026-03-23-commit-semantic-domain-quality-design.md` + +is the source of truth. diff --git a/docs/superpowers/specs/2026-03-23-commit-semantic-domain-quality-design.md b/docs/superpowers/specs/2026-03-23-commit-semantic-domain-quality-design.md new file mode 100644 index 0000000..21a210c --- /dev/null +++ b/docs/superpowers/specs/2026-03-23-commit-semantic-domain-quality-design.md @@ -0,0 +1,434 @@ +# Commit-Semantic Domain Quality Optimization Design + +Date: 2026-03-23 +Status: APPROVED +Scope: commit-semantic domain quality optimization after repo-level manual validation + +## Problem Statement + +Repo-level manual validation now succeeds end-to-end, but domain quality is still not good enough for stable downstream use. + +Current observed output on the real repository: +- `domain_count = 7` +- `uncategorized_ratio = 0.1762` +- top domains include `tests`, `test`, `skill`, `review`, `claude` +- `uncategorized` is still ranked #2 + +This means the execution chain is working, but the semantic layer is still too dependent on heuristic token buckets rather than stable repository domains. + +## Goals + +This iteration optimizes three things together: + +1. **Domain schema quality** + - remove duplicate or near-duplicate domains like `test` / `tests` + - improve domain naming so it reflects stable repo-level areas rather than commit-title noise + - produce cleaner `domains.json` for downstream use + +2. **Classification coverage** + - reduce `uncategorized_ratio` below the current repo-level result + - improve deterministic assignment without aggressively forcing weak matches + +3. **Runtime priority clarity** + - make LLM-based discover/classify the preferred path + - keep heuristic fallback as explicit degraded mode only + - expose mode information in outputs so manual review can distinguish high-quality vs fallback runs + +## Non-Goals + +This iteration does **not**: +- redesign aggregate or distill scoring formulas +- change commit-extract output schema +- integrate demand stage +- build a full human feedback / write-back correction loop + +## Recommended Approach + +Use a three-layer convergence strategy, but execute it in ordered sub-phases: + +1. **Phase A — Stabilize the domain schema first** +2. **Phase B — Use the better schema to reduce `uncategorized`** +3. **Phase C — Make LLM the primary strategy and fallback explicit** +4. **Phase D — Expose runtime mode in exported artifacts** + +Rationale: +- If domain definitions are noisy, lowering `uncategorized` just pushes more units into bad buckets. +- Schema cleanup increases the value of deterministic path/keyword matching. +- LLM-first policy should remain the target architecture, but fallback is still needed for local execution and CI. +- Runtime mode must be visible so fallback output is not mistaken for full semantic output. + +## Alternatives Considered + +### A. Three-layer convergence (recommended) +- Clean domain schema +- Improve deterministic classification quality and coverage +- Switch runtime priority to LLM-first, fallback-only + +**Pros** +- Solves all three user goals in the right order +- Keeps local runnability while improving final architecture +- Minimizes the risk of pushing more units into unstable domains + +**Cons** +- Broadest scope of the options +- Needs careful verification to avoid regressions + +### B. LLM-first immediately +- Prioritize true LLM discover/classify now +- Defer schema cleanup to later + +**Pros** +- Most aligned with the target architecture +- Avoids investing heavily in heuristics + +**Cons** +- Real output quality may still be unstable if post-processing is weak +- Local fallback remains under-specified +- Does not directly solve duplicate domain names + +### C. Heuristic-only quality pass +- Improve fallback discover/classify and stop there + +**Pros** +- Fastest path to local quality improvement +- Easy to verify in CI and local runs + +**Cons** +- Makes fallback the de facto primary architecture +- Drifts from the intended LLM-first design + +## Design + +## Canonical Domain Schema + +Normalized domains must preserve the current `domain` field name for compatibility with the existing pipeline. +The canonical normalized domain object is: + +```json +{ + "domain": "tests", + "description": "Repository test and verification surface", + "paths": ["tests/", "src/..."], + "keywords": ["tests", "pytest", "verification"] +} +``` + +Required fields: +- `domain: str` +- `description: str` +- `paths: list[str]` +- `keywords: list[str]` + +Merge output rules: +- merged domain keeps the winner `domain` string after normalization +- `description` prefers the first non-empty description from the highest-quality candidate +- `paths` are unioned and deduplicated, preserving shortest-prefix-first ordering +- `keywords` are unioned and deduplicated after lowercase normalization + +Winner selection priority during merge: +1. domain with non-empty paths +2. domain with more normalized keywords +3. domain with non-noise normalized name +4. stable lexical order as final tiebreak + +## Runtime State Contract + +The implementation must use `HarnessState.metadata` as the source of truth for runtime mode. + +Required metadata keys: +- `external_orchestration: bool` +- `orchestration_mode: "llm_preferred" | "local_fallback" | "mixed_degraded"` +- `discover_mode: "llm" | "fallback" | "cached_llm" | "cached_fallback"` +- `classify_mode: "llm" | "fallback" | "mixed" | "cached"` + +Persistence contract: +- `domains.json` must persist discover provenance alongside `_fingerprint` and `domains` +- required persisted fields in `domains.json`: + - `discover_mode` + - `orchestration_mode_at_discover` +- on discover cache hit, the runner must restore `discover_mode` from cached provenance into `HarnessState.metadata` +- if classify is skipped due to a future cache/resume optimization, the same rule applies: persisted provenance must be restored before export +- export must report actual execution provenance, not default assumptions + +Mode transitions: +- discover via LLM + classify via LLM => `orchestration_mode = "llm_preferred"` +- discover via fallback + classify via fallback => `orchestration_mode = "local_fallback"` +- any mixed combination, cache reuse of one mode plus execution of another mode, or any LLM stage that falls back after failure => `orchestration_mode = "mixed_degraded"` + +Failure / degradation decision table: + +| Stage | Failure case | Action | Exported mode | +|------|--------------|--------|---------------| +| discover | LLM output empty/invalid | retry once if orchestration exists, else fallback normalize+save | `mixed_degraded` if fallback used | +| discover | LLM unavailable | fallback normalize+save | `local_fallback` or `mixed_degraded` | +| classify | some batches fail | fallback-local classify failed batches, keep successful LLM batches | `mixed_degraded` | +| classify | all batches fail and no fallback match | leave unresolved units as `uncategorized` | `mixed_degraded` | +| classify | no LLM path available | deterministic/fallback classify only | `local_fallback` | + +Export contract: +`summary.json` must include: +- `orchestration_mode` +- `discover_mode` +- `classify_mode` + +The runner is responsible for setting these values before export. External orchestrators may set `external_orchestration=true`, but exported mode fields must still reflect actual execution, not intent. + +### 1. Discover redesign + +Discover will now have two conceptual layers: + +#### 1.1 Preferred path: LLM discover + +LLM discover remains responsible for producing the initial semantic domain list from: +- `units/all.jsonl` summary +- optional architecture document context + +Expected LLM output remains: +- `domain` +- `description` +- `paths` +- `keywords` + +But the post-processing contract becomes stricter: +- domain name must be normalized and singular/plural-safe +- paths must be meaningful for later commit-level assignment +- keywords must represent domain semantics, not just copied commit-title fragments + +#### 1.2 Required post-normalization (applies to both LLM and fallback) + +Every discovered domain list must go through normalization before being accepted. +The implementation contract is: + +```python +normalize_domains(domains: list[dict]) -> list[dict] +``` + +Call order: +1. parse or generate raw domains +2. normalize domains +3. validate normalized domains +4. save `domains.json` + +This normalization step must be applied in both paths: +- LLM path: `complete_discover()` +- fallback path: local discover writer + +Fingerprint semantics: +- input fingerprint remains based on input artifacts (`units/all.jsonl` and optional architecture doc) +- normalization does not change fingerprint inputs +- normalization only affects saved domain output quality + +Normalization rules: + +1. **Name normalization** + - lowercase + - dash-case + - singular/plural merge when normalized stems match exactly (`test` + `tests` => `tests`) + +2. **Noise filtering** + - reject or down-rank generic workflow verbs/nouns such as: + - `add`, `update`, `fix`, `impl`, `phase`, `final`, `worktree` + - reject pure process buckets unless they have strong repo support via paths or keyword overlap + +3. **Duplicate and near-duplicate merging** + - exact same normalized domain name => merge + - near-duplicate merge only when either: + - keyword Jaccard overlap >= 0.6, or + - path-prefix overlap >= 0.5 + - merged domain keeps deduplicated keyword union and path union + +4. **Minimum quality gate** + - reject a domain if all of the following are true: + - `paths == []` + - fewer than 3 deduplicated keywords + - normalized name is in the noise-token list + - LLM output may bypass this only when description explicitly references a stable repo area, boundary, or subsystem + +#### 1.3 Fallback discover role + +Fallback discover should no longer try to produce a broad set of token-bucket domains. +It should instead produce a **small, conservative domain skeleton** using stronger signals: +- stable path fragments +- repeated repo nouns +- high-signal theme tokens after noise filtering + +Fallback discover should prefer fewer, stronger domains over broad coverage. + +### 2. Classify redesign + +Classification should also separate preferred path from degraded path. + +Classification granularity must be explicit: +- **commit-level assignment** is used when reliable `file_paths` exist and all changed files converge on one domain through deterministic path matching +- **unit-level scoring** is used only for mixed commits, missing-path commits, or low-confidence deterministic cases + +Path-evidence rule after commit-level failure: +- if commit-level convergence fails because the commit spans multiple candidate domains, unit-level fallback scoring must **disable path-prefix scoring** for that commit +- in that case, unit-level fallback may use only: + - theme token match + - summary token match + - section-name token match + - domain-keyword match +- if those non-path signals are still ambiguous, the unit must go to LLM when available, otherwise remain `uncategorized` +- path-prefix scoring remains allowed only when commit-level path evidence already converged to a single domain or when a future implementation introduces per-unit path attribution + +#### 2.1 Preferred path: LLM classify + +LLM classify is preferred when: +- `is_mixed = true` +- file paths are absent +- path-based assignment spans multiple candidate domains +- deterministic scoring is low-confidence or tied + +This preserves the original design intent: semantic classification is LLM-first for ambiguous cases. + +#### 2.2 Deterministic classify upgrade + +Deterministic assignment remains important for speed and local execution, but must become more structured. + +Scoring priority: +1. path-prefix match +2. theme token match +3. summary token match +4. section-name token match +5. domain-keyword match + +Scoring weights: +- path-prefix match = 5 +- theme token match = 3 +- summary token match = 2 +- section-name token match = 2 +- domain-keyword match = 1 + +Scoring rules: +- repeated token hits do not stack beyond one hit per signal type +- path-prefix match is evaluated once per candidate domain +- keyword overlap is deduplicated after lowercase normalization +- minimum deterministic assignment score = 4 +- ambiguous if `top1 - top2 < 2` +- if score < 4 => do not assign deterministically +- if ambiguous => send to LLM when available, otherwise leave as `uncategorized` + +This reduces accidental assignment while still lowering `uncategorized` where evidence is strong. + +### 3. Runtime priority and degradation policy + +Three runtime modes should be explicit: + +#### 3.1 `llm_preferred` +Default mode. +- discover prefers LLM +- classify prefers LLM for ambiguous work +- fallback is used only when LLM path is unavailable or fails + +#### 3.2 `local_fallback` +Explicit local mode. +- discover/classify use deterministic fallback only +- intended for local testing, CI, or offline execution + +#### 3.3 `mixed_degraded` +Partial degradation mode. +- some stages used LLM, others used fallback +- used when one stage succeeds semantically and another falls back + +### 4. Output visibility + +`summary.json` should include explicit runtime mode markers so human review can evaluate quality in context. + +Recommended additions: +- `orchestration_mode` +- `discover_mode` +- `classify_mode` + +This prevents fallback results from being mistaken for fully semantic LLM-backed output. + +## Validation Strategy + +### Automated validation + +1. **Domain normalization tests** + - singular/plural merge (`test` + `tests`) + - duplicate keyword/path merge + - noise token rejection + +2. **Classification confidence tests** + - strong path match assigns deterministically + - weak ambiguous match remains unresolved until LLM/fallback decision + - `uncategorized` decreases only when evidence is sufficient + +3. **Mode reporting tests** + - summary reports correct discover/classify mode + - fallback runs are clearly marked degraded + +4. **Repo-level regression test** + - end-to-end local run should no longer produce duplicate domains like `test/tests` + - `uncategorized_ratio` should improve from the current baseline + +### Required test matrix + +The implementation plan must cover at least these fixture cases: +- LLM discover output containing duplicate domains like `test/tests` +- fallback discover output containing noisy token buckets +- path-based single-domain commit assignment +- path-based multi-domain commit ambiguity +- mixed/no-path unit classification tie +- git/path failure with degraded mode reporting preserved in `summary.json` +- summary schema assertions for: + - `orchestration_mode` + - `discover_mode` + - `classify_mode` + +### Manual validation + +Run the real repository again and review: +- top 5 domains should look like stable repo areas, not token buckets +- `uncategorized` should not rank near the top if domain quality improved +- duplicate domains should disappear + +## Success Criteria + +Implementation planning must treat the current repo-level run used in this session as the baseline snapshot. +Baseline source: +- worktree: `/Users/yan./git/3p/sematic-harness/.worktrees/commit-semantic-domain` +- summary artifact: `data/commit-semantic/summary.json` +- observed baseline: `uncategorized_ratio = 0.1762` + +Success criteria are mode-specific: + +### Deterministic / CI gate (`local_fallback`) +1. No duplicate or near-duplicate top-level domains such as `test/tests` +2. Duplicate normalized domain names after post-processing = 0 +3. Top 5 domains contain no banned noise tokens as their final normalized names +4. `summary.json` includes `orchestration_mode`, `discover_mode`, and `classify_mode` +5. Top 5 repo-level domains are more stable than the current baseline, meaning each top domain has at least one of: + - non-empty path prefixes, or + - at least 3 normalized keywords after deduplication + +### Repo-level quality gate (`llm_preferred` or `mixed_degraded`) +6. `uncategorized_ratio` is lower than the baseline `0.1762` when re-run against the same baseline worktree snapshot +7. The repo-level result is reviewed manually, not treated as a hard deterministic CI gate, unless model/input conditions are explicitly frozen in a later iteration + +## Risks + +1. **Over-normalization risk** + - aggressive merge rules may collapse legitimately distinct domains + +2. **False-confidence risk** + - lowering `uncategorized` too aggressively can hide uncertainty by forcing bad assignments + +3. **Fallback drift risk** + - if fallback becomes too feature-rich, it may replace the intended LLM-first architecture + +Mitigation: +- keep normalization rules narrow and test-driven +- require confidence thresholds before deterministic assignment +- keep fallback explicitly marked as degraded mode + +## Implementation Notes + +Planned implementation should likely touch: +- `skills/commit-semantic/run.py` +- optionally `src/commit_semantic/domain_utils.py` if normalization logic needs a pure-function home +- targeted tests in `tests/e2e/test_commit_semantic.py` and/or related domain tests + +No downstream demand changes are included in this spec. diff --git a/prompts/classify_units.md b/prompts/classify_units.md new file mode 100644 index 0000000..3ec2c60 --- /dev/null +++ b/prompts/classify_units.md @@ -0,0 +1,13 @@ +Given the following domain list and commit semantic units, classify each unit into the most appropriate domain. + +Domain list: +{domains_json} + +Units to classify: +{units_json} + +Requirements: +1. For each unit output: {"id": "", "domain": ""} +2. domain must be a value from the domain list, or "uncategorized" +3. Judge by semantic content (theme, summary, operation type), not just keywords +4. Output a JSON array only, no explanation diff --git a/prompts/discover_domains.md b/prompts/discover_domains.md new file mode 100644 index 0000000..865d1e1 --- /dev/null +++ b/prompts/discover_domains.md @@ -0,0 +1,13 @@ +Given the following semantic units from a codebase's git history, cluster them into core domains. + +Units summary: +{units_summary} + +Architecture document (if available): +{architecture_content} + +Requirements: +1. Each domain must have: domain (short identifier), description (one sentence), paths (associated directory prefixes), keywords (associated keywords) +2. Target 5-15 domains, maximum 20. If fewer than 5 natural domains exist, output the actual count. If more than 20, merge similar domains until under 20. +3. Cluster based on semantic content (themes, operations, summaries), not just directory structure +4. Output a JSON array only, no explanation diff --git a/skills/commit-extract/run.py b/skills/commit-extract/run.py index 2c3669f..f94cbdc 100644 --- a/skills/commit-extract/run.py +++ b/skills/commit-extract/run.py @@ -14,6 +14,7 @@ import argparse import logging +import os import re import subprocess import sys @@ -31,6 +32,7 @@ OUTPUT_BASE = Path("data/commit-extract") TMP_DIR = OUTPUT_BASE / "tmp" +USE_TASK_AGENTS_ENV = "COMMIT_EXTRACT_USE_TASK_AGENTS" # Adaptive batching constants WEIGHT_BUDGET = 3000 @@ -318,12 +320,156 @@ def _run_collect(self, state: HarnessState) -> bool: manifest_path = str(TMP_DIR / "manifest.json") save_json(manifest, manifest_path) print(f"\n Manifest written to {manifest_path}") - print(f" Workers should write to {TMP_DIR}/batch_NNNN.jsonl") - print(f" After all workers complete, run merge to consolidate.") + + if self._use_task_agents(): + print(f" Task-agent orchestration enabled via {USE_TASK_AGENTS_ENV}=1") + print(f" Workers should write to {TMP_DIR}/batch_NNNN.jsonl") + print(" After all workers complete, run merge to consolidate.") + else: + print(" Running local worker fallback...") + processed = self._run_local_workers(manifest) + merged = merge_tmp_files(OUTPUT_BASE, TMP_DIR) + print(f" Local fallback wrote {processed} records") + print(f" Merged {merged} new records into monthly JSONL") self.add_artifact(state, str(OUTPUT_BASE)) return True + def _use_task_agents(self) -> bool: + """Return True when external task-agent orchestration is explicitly enabled.""" + return os.environ.get(USE_TASK_AGENTS_ENV, "").lower() in ("1", "true", "yes") + + def _run_local_workers(self, manifest: dict) -> int: + """Process manifest batches locally with deterministic git-derived extraction.""" + total = 0 + for batch in manifest.get("batches", []): + output_path = batch.get("output_path") + if not output_path: + continue + + records = [] + for sha in batch.get("shas", []): + record = self._extract_commit_record(sha) + if record is not None: + records.append(record) + + if records: + append_jsonl(records, output_path) + total += len(records) + + return total + + def _extract_commit_record(self, sha: str) -> dict | None: + """Build a schema-valid commit-extract record from git metadata.""" + try: + meta_result = subprocess.run( + [ + "git", "-C", self.repo_path, + "show", "--no-patch", + "--format=%an%x00%aI%x00%B", + sha, + ], + capture_output=True, + text=True, + check=True, + ) + stat_result = subprocess.run( + ["git", "-C", self.repo_path, "show", "--stat", "--summary", "--format=", sha], + capture_output=True, + text=True, + check=True, + ) + except subprocess.CalledProcessError as e: + logger.warning("Failed to extract commit %s: %s", sha, e) + return None + + parts = meta_result.stdout.split("\x00", 2) + author = parts[0].strip() if len(parts) > 0 else "" + date = parts[1].strip() if len(parts) > 1 else "" + message = parts[2].strip() if len(parts) > 2 else "" + summary = next((line.strip() for line in message.splitlines() if line.strip()), "") + body_lines = [line.strip() for line in message.splitlines()[1:] if line.strip()] + + weight = parse_stat(stat_result.stdout) + summary_lower = summary.lower() + op = self._classify_op(summary_lower) + theme = self._derive_theme(summary) + section_name = self._derive_section_name(summary) + item_summary = body_lines[0] if body_lines else (summary or f"Update in {theme}") + + rules_invariants = [] + for line in body_lines[1:]: + if any(keyword in line.lower() for keyword in ("must", "should", "ensure", "always", "never")): + rules_invariants.append({ + "kind": "rule", + "statement": line, + "enforced_by_commit": False, + }) + + if not rules_invariants: + for line in body_lines: + if any(keyword in line.lower() for keyword in ("must", "should", "ensure", "always", "never")): + rules_invariants.append({ + "kind": "rule", + "statement": line, + "enforced_by_commit": False, + }) + + return { + "sha": sha, + "author": author, + "date": date, + "is_large_aggregate": weight >= WEIGHT_BUDGET, + "is_mixed": False, + "sections": [{ + "name": section_name, + "theme": theme, + "importance": "primary", + "items": [{ + "op": op, + "summary": item_summary, + }], + }], + "rules_invariants": rules_invariants, + } + + def _classify_op(self, summary_lower: str) -> str: + """Map commit summary text to the existing commit-extract op taxonomy.""" + if any(token in summary_lower for token in ("bugfix", "fix", "hotfix")): + return "bugfix" + if "refactor" in summary_lower: + return "refactor" + if any(token in summary_lower for token in ("test", "spec")): + return "test" + if any(token in summary_lower for token in ("config", "ci", "build", "infra")): + return "config" + if any(token in summary_lower for token in ("feat", "feature", "add", "implement")): + return "feat" + return "other" + + def _derive_theme(self, summary: str) -> str: + """Derive a stable-ish theme slug from commit summary text.""" + text = summary.strip() + if not text: + return "misc" + if ":" in text: + text = text.split(":", 1)[1].strip() or text + slug = re.sub(r"[^a-z0-9]+", "-", text.lower()).strip("-") + return slug or "misc" + + def _derive_section_name(self, summary: str) -> str: + """Create a readable section name from the commit summary.""" + text = summary.strip() + if not text: + return "General changes" + if ":" in text: + prefix, rest = text.split(":", 1) + label = rest.strip() or prefix.strip() + else: + label = text + label = label[:1].upper() + label[1:] + return label + def handle_merge(self) -> int: """Merge tmp files after workers complete.""" if not TMP_DIR.exists(): diff --git a/skills/commit-semantic/SKILL.md b/skills/commit-semantic/SKILL.md index 1adfcc5..57a4cc8 100644 --- a/skills/commit-semantic/SKILL.md +++ b/skills/commit-semantic/SKILL.md @@ -1,6 +1,6 @@ --- name: commit-semantic -description: Analyze commit patterns from structured JSONL (4-stage pipeline) +description: Analyze commit patterns from structured JSONL (5-stage pipeline) entrypoint: skills.commit-semantic.run.run_commit_semantic triggers: - commit-semantic @@ -10,7 +10,7 @@ triggers: # Commit Semantic -4-stage pipeline consuming commit-extract JSONL output: ingest → aggregate → distill → export. +5-stage pipeline consuming commit-extract JSONL output: discover → ingest → aggregate → distill → export. ## Prerequisites @@ -18,61 +18,69 @@ Requires `data/commit-extract/*.jsonl` files produced by `/commit-extract run`. ## Pipeline Stages -### 1. ingest +### 1. discover -Expand sections into semantic units + collect rules_invariants. +Bottom-up domain discovery from semantic units, cached by `domains.json` fingerprint. -- Each section's items become individual units with `sha`, `date`, `author`, `theme`, `importance`, `op`, `summary` -- Commit-level `is_large_aggregate` and `is_mixed` flags carried to each unit -- `rules_invariants` collected separately -- Skips invalid JSON lines with warning +- Builds domains from unit-level semantic signals +- Reuses cached `domains.json` when fingerprint matches current inputs +- First run may bootstrap by running ingest first to create units +- Output: `data/commit-semantic/domains.json` + +### 2. ingest + +Expand sections into semantic units, collect invariants, and assign domains. + +- Each section item becomes a unit with commit metadata, semantic fields, and domain assignment when `domains.json` exists +- Collects invariants separately into `invariants.jsonl` +- Mixed or no-path commits may require LLM classification - Output: `data/commit-semantic/units/all.jsonl`, `data/commit-semantic/invariants.jsonl` -### 2. aggregate +### 3. aggregate -Group units by theme, compute statistics. +Group units by domain and compute domain-level statistics. -- Primary key: `theme` (cross-commit semantic theme) -- Same theme from different `section_name` values merged -- Statistics: `op` distribution, `importance` ratio (primary/secondary) -- Threshold: theme must appear in >= 3 distinct commits -- Output: `data/commit-semantic/patterns.jsonl` +- Primary key: `domain`, not theme +- Preserves `sub_themes` within each domain +- `uncategorized` remains an independent domain bucket +- Output: `data/commit-semantic/domains-aggregated.jsonl` -### 3. distill +### 4. distill -Extract canonical demands from patterns, scored and ranked. +Extract canonical demands from aggregated domains, score them, and rank them. -- Score: `distinct_commits × importance_weight` where `primary=2, secondary=1` -- Tie-break: `distinct_commits` desc → `theme` alpha -- Invariants appearing in >= 3 commits get extra weight +- Uses multi-dimensional scoring with invariant SHA association and caps +- Emits score breakdown fields for downstream review +- Produces ranked canonical demands per domain cluster - Output: `data/commit-semantic/canonical-demands.jsonl` -### 4. export +### 5. export -Generate summary statistics. +Generate summary statistics for the domain-based pipeline. -- Total units, patterns, op distribution, bugfix ratio -- Top patterns by score -- Date range +- `summary.json` includes `top_domains`, `domain_count`, `uncategorized_ratio`, `file_paths_available` +- Also includes `op_distribution`, `invariant_count`, `date_range`, and `bugfix_ratio` +- Also reports runtime provenance via `orchestration_mode`, `discover_mode`, and `classify_mode` - Output: `data/commit-semantic/summary.json` ## Output Schema ``` data/commit-semantic/ - units/all.jsonl # Expanded semantic units - invariants.jsonl # Rules and invariants - patterns.jsonl # Aggregated patterns (threshold >= 3) + domains.json # Discovered domains + fingerprint cache + domains-aggregated.jsonl # Aggregated domain statistics canonical-demands.jsonl # Scored and ranked demands - summary.json # Summary statistics + summary.json # Domain summary statistics + units/all.jsonl # Expanded semantic units + invariants.jsonl # Collected invariants ``` ## Usage ```bash -/commit-semantic run # Full pipeline (4 stages) -/commit-semantic run --stage ingest # Run specific stage -/commit-semantic step # Run next stage only -/commit-semantic resume # Continue from breakpoint -/commit-semantic reset # Clear state, keep artifacts +/commit-semantic run # Full pipeline (5 stages) +/commit-semantic run --stage discover # Run specific stage +/commit-semantic step # Run next stage only +/commit-semantic resume # Continue from breakpoint +/commit-semantic reset # Clear state, keep artifacts ``` diff --git a/skills/commit-semantic/run.py b/skills/commit-semantic/run.py index fc623b2..42a0097 100644 --- a/skills/commit-semantic/run.py +++ b/skills/commit-semantic/run.py @@ -1,11 +1,12 @@ #!/usr/bin/env python3 """commit-semantic skill implementation. -4 阶段消费 commit-extract JSONL 输出： - 1. ingest - 展开 sections 为 semantic units + 收集 rules_invariants - 2. aggregate - 按 theme 聚合，统计 op 分布 + importance 分布 - 3. distill - 提取 canonical demands，评分排序 - 4. export - 汇总统计，生成 summary.json +5 阶段管道，从 commit-extract JSONL 构建领域知识： + 0. discover - 从 units 语义内容聚类出领域（自底向上，首次运行时） + 1. ingest - 展开 sections 为 semantic units + 按领域归入 + 2. aggregate - 按领域聚合，统计 op 分布 + importance 分布 + 3. distill - 多维评分排序 + 4. export - 汇总统计，生成 summary.json Input: data/commit-extract/*.jsonl Output: data/commit-semantic/ @@ -14,32 +15,74 @@ from __future__ import annotations import argparse +import json import logging +import shutil import sys -from collections import defaultdict -from datetime import datetime +from collections import Counter, defaultdict from pathlib import Path +import re sys.path.insert(0, str(Path(__file__).parent.parent.parent)) -from src.harness_state import HarnessState, save_state -from src.skill_runner import SkillRunner, run_skill +from src.harness_state import HarnessState, load_state, save_state +from src.host_executor import HostExecutor +from src.skill_runner import SkillRunner from src.io_utils import load_jsonl, save_jsonl, save_json +from src.commit_semantic.domain_utils import ( + build_sha_file_map, + assign_domain_by_path, + classify_unit_locally, + normalize_domains, + parse_llm_domains, + parse_llm_classifications, + compute_fingerprint, + fingerprint_matches, + build_units_summary, +) logger = logging.getLogger(__name__) EXTRACT_OUTPUT = Path("data/commit-extract") SEMANTIC_OUTPUT = Path("data/commit-semantic") +ARCH_CANDIDATES = [ + Path("docs/superpowers/ARCHITECTURE.md"), + Path("docs/ARCHITECTURE.md"), + Path("ARCHITECTURE.md"), +] +PROMPT_DIR = Path(__file__).resolve().parents[2] / "prompts" +LLM_CLASSIFY_BATCH = 50 +LEGACY_EXPORT_PATHS = [ + Path("patterns"), + Path("canonical-demands.yaml"), + Path("functional"), + Path("non-functional"), +] + + +def _domains_file(): + return SEMANTIC_OUTPUT / "domains.json" + + +def _units_file(): + return SEMANTIC_OUTPUT / "units" / "all.jsonl" + + +def _invariants_file(): + return SEMANTIC_OUTPUT / "invariants.jsonl" class CommitSemanticRunner(SkillRunner): - """Runner for commit-semantic pipeline (4 stages).""" + """Runner for commit-semantic pipeline (5 stages).""" - STAGES = ["ingest", "aggregate", "distill", "export"] + STAGES = ["discover", "ingest", "aggregate", "distill", "export"] PIPELINE = "commit-semantic" + def __init__(self, executor: HostExecutor | None = None) -> None: + super().__init__() + self.executor = executor + def _check_prerequisites(self) -> tuple[bool, str]: - """Check if commit-extract JSONL output exists.""" if not EXTRACT_OUTPUT.exists(): return False, "commit-extract output not found" jsonl_files = list(EXTRACT_OUTPUT.glob("*.jsonl")) @@ -54,9 +97,231 @@ def _require_prerequisites(self) -> bool: return False return True + def _find_arch_file(self) -> Path | None: + for p in ARCH_CANDIDATES: + if p.exists(): + return p + return None + + def _tokenize_text(self, value: str) -> list[str]: + return re.findall(r"[a-z0-9_/-]+", (value or "").lower()) + + def _build_local_domains(self, units: list[dict]) -> list[dict]: + token_counts: Counter[str] = Counter() + path_counts: dict[str, Counter[str]] = defaultdict(Counter) + + for unit in units: + for field in ("section_name", "theme", "summary"): + for token in self._tokenize_text(unit.get(field, "")): + if len(token) >= 4 and token not in {"with", "from", "into", "flow", "local"}: + token_counts[token] += 1 + for path in unit.get("file_paths", []): + parts = [part for part in path.split("/") if part] + for prefix_len in range(1, min(len(parts), 3) + 1): + prefix = "/".join(parts[:prefix_len]) + "/" + path_counts[prefix][path] += 1 + + domains: list[dict] = [] + used_names: set[str] = set() + for token, count in token_counts.most_common(8): + if count < 1 or token in used_names: + continue + keywords = [candidate for candidate, _ in token_counts.most_common() if token in candidate or candidate in token][:5] + matching_prefixes = [ + prefix for prefix, counter in path_counts.items() + if any(token in fp.lower() for fp in counter) + ] + domains.append({ + "domain": token.replace("_", "-").replace("/", "-"), + "description": f"Local heuristic domain for {token}", + "paths": sorted(matching_prefixes)[:5], + "keywords": keywords or [token], + }) + used_names.add(token) + if len(domains) >= 6: + break + + if not domains: + domains.append({ + "domain": "core", + "description": "Fallback domain inferred locally", + "paths": [], + "keywords": ["core"], + }) + return domains + + + def _assign_domains_locally(self, units: list[dict], domains: list[dict], *, allow_path_scoring: bool = True) -> int: + assigned = 0 + for unit in units: + if unit.get("domain") and unit["domain"] != "uncategorized": + continue + best_domain = classify_unit_locally( + unit, + domains, + allow_path_scoring=allow_path_scoring and not unit.get("path_scoring_disabled", False), + ) + if best_domain: + unit["domain"] = best_domain + assigned += 1 + return assigned + + def _use_local_fallback(self, state: HarnessState) -> bool: + return self.executor is None and not state.metadata.get("external_orchestration", False) + + def _build_discover_context(self, units_summary: str, arch_content: str) -> dict[str, str]: + return { + "units_summary": units_summary, + "architecture_content": arch_content or "(none)", + } + + def _execute_discover(self, state: HarnessState, prompt: str, context: dict[str, str]) -> bool: + if self.executor is None: + print(" ! Discover orchestration unavailable") + return False + try: + response = self.executor( + prompt, + context, + artifact_name="domains", + sampling_mode="auto", + ) + except Exception as exc: + print(f" ! Discover orchestration failed: {exc}") + return False + return self.complete_discover(response, state) + + def _build_classify_batches(self, needs_llm: list[dict], domains: list[dict]) -> list[dict[str, str]]: + prompt_template = (PROMPT_DIR / "classify_units.md").read_text(encoding="utf-8") + batches: list[dict[str, str]] = [] + for start in range(0, len(needs_llm), LLM_CLASSIFY_BATCH): + batch_units = needs_llm[start:start + LLM_CLASSIFY_BATCH] + context = { + "domains_json": json.dumps(domains, ensure_ascii=False, indent=2), + "units_json": json.dumps( + [ + { + "id": str(i), + "section_name": unit.get("section_name", ""), + "theme": unit.get("theme", ""), + "summary": unit.get("summary", ""), + "op": unit.get("op", ""), + } + for i, unit in enumerate(batch_units, start=start) + ], + ensure_ascii=False, + indent=2, + ), + } + prompt = prompt_template.replace("{domains_json}", context["domains_json"]) + prompt = prompt.replace("{units_json}", context["units_json"]) + batches.append({"prompt": prompt, "context": context}) + return batches + + def _execute_classify_batches( + self, + state: HarnessState, + needs_llm: list[dict], + domains: list[dict], + units: list[dict], + ) -> bool: + if self.executor is None: + print(" ! Classify orchestration unavailable") + return False + + responses: list[str] = [] + for batch in self._build_classify_batches(needs_llm, domains): + try: + response = self.executor( + batch["prompt"], + batch["context"], + artifact_name="classify-units", + sampling_mode="auto", + ) + except Exception as exc: + print(f" ! LLM classification batch failed: {exc}") + return False + responses.append(response) + + return self._apply_classify_responses(responses, state, units=units) + + def _set_discover_mode(self, state: HarnessState, mode: str) -> None: + state.metadata["discover_mode"] = mode + + def _set_classify_mode(self, state: HarnessState, mode: str) -> None: + state.metadata["classify_mode"] = mode + + def _refresh_orchestration_mode(self, state: HarnessState) -> None: + discover_mode = state.metadata.get("discover_mode") + classify_mode = state.metadata.get("classify_mode") + + if not discover_mode and not classify_mode: + return + + discover_family = ( + "fallback" if discover_mode in {"fallback", "cached_fallback"} + else "llm" if discover_mode in {"llm", "cached_llm"} + else None + ) + classify_family = ( + "fallback" if classify_mode == "fallback" + else "llm" if classify_mode in {"llm", "cached"} + else "mixed" if classify_mode == "mixed" + else None + ) + + if discover_family == "fallback" and classify_family in {None, "fallback", "llm"}: + state.metadata["orchestration_mode"] = "local_fallback" + return + if discover_family == "llm" and classify_family in {None, "llm"}: + state.metadata["orchestration_mode"] = "llm_preferred" + return + state.metadata["orchestration_mode"] = "mixed_degraded" + + def _persist_domains(self, *, fingerprint: dict, domains: list[dict], discover_mode: str, orchestration_mode_at_discover: str) -> None: + save_json( + { + "_fingerprint": fingerprint, + "discover_mode": discover_mode, + "orchestration_mode_at_discover": orchestration_mode_at_discover, + "domains": domains, + }, + str(_domains_file()), + ) + + def _write_local_domains(self, units: list[dict], fingerprint: dict, state: HarnessState) -> list[dict]: + domains = normalize_domains(self._build_local_domains(units)) + self._set_discover_mode(state, "fallback") + self._refresh_orchestration_mode(state) + self._persist_domains( + fingerprint=fingerprint, + domains=domains, + discover_mode="fallback", + orchestration_mode_at_discover=state.metadata.get("orchestration_mode", "local_fallback"), + ) + return domains + + def _check_state_compat(self, state: HarnessState) -> HarnessState: + """Detect old 4-stage state and reset if incompatible.""" + completed = state.metadata.get("completed_stages", []) + if completed and "discover" not in self.STAGES[:1]: + return state + # If state has completed stages but none match new STAGES[0], + # it's from the old 4-stage pipeline + if completed and all(s in ["ingest", "aggregate", "distill", "export"] for s in completed): + if "discover" not in completed: + logger.warning( + "Detected old 4-stage state (completed: %s). " + "Resetting for new 5-stage pipeline.", completed + ) + return self.init_state() + return state + + def run_stage(self, stage: str, state: HarnessState) -> bool: print(f"\n[{self.PIPELINE}] Running stage: {stage}") dispatch = { + "discover": self._run_discover, "ingest": self._run_ingest, "aggregate": self._run_aggregate, "distill": self._run_distill, @@ -68,16 +333,272 @@ def run_stage(self, stage: str, state: HarnessState) -> bool: return True # ------------------------------------------------------------------- - # Stage 1: ingest + # Stage 0: discover + # ------------------------------------------------------------------- + + def _run_discover(self, state: HarnessState) -> bool: + """Cluster units into domains (bottom-up). Only runs on init or fingerprint change.""" + print(" -> Running domain discovery") + SEMANTIC_OUTPUT.mkdir(parents=True, exist_ok=True) + + # If units don't exist yet, we need ingest first (first-run bootstrap) + if not _units_file().exists(): + print(" Units not found — running ingest first (no domain assignment)") + if not self._run_ingest_raw(state): + return False + + units = load_jsonl(str(_units_file()), skip_errors=True) + if not units: + print(" ! No units to cluster") + return True + + arch_file = self._find_arch_file() + current_fp = compute_fingerprint(_units_file(), arch_file) + + # Check cache + force = state.metadata.get("force", False) + if _domains_file().exists() and not force: + try: + with open(_domains_file()) as f: + cached = json.load(f) + if fingerprint_matches(cached, current_fp): + n = len(cached.get("domains", [])) + cached_mode = cached.get("discover_mode") + if cached_mode == "llm": + self._set_discover_mode(state, "cached_llm") + if "orchestration_mode_at_discover" in cached: + state.metadata["orchestration_mode_at_discover"] = cached["orchestration_mode_at_discover"] + self._refresh_orchestration_mode(state) + print(f" Cache hit ({n} domains, fingerprint matches). Skipping discovery.") + return True + print(" Non-LLM discovery cache incompatible — re-running discovery") + else: + print(" Fingerprint changed — re-running discovery") + except (json.JSONDecodeError, OSError): + print(" Invalid cache — re-running discovery") + + # Build prompt input + units_summary = build_units_summary(units) + arch_content = "" + if arch_file and arch_file.exists(): + arch_content = arch_file.read_text(encoding="utf-8")[:3000] + + prompt_template = (PROMPT_DIR / "discover_domains.md").read_text(encoding="utf-8") + context = self._build_discover_context(units_summary, arch_content) + prompt = prompt_template.replace("{units_summary}", context["units_summary"]) + prompt = prompt.replace("{architecture_content}", context["architecture_content"]) + + state.metadata["discover_prompt"] = prompt + state.metadata["discover_fingerprint"] = current_fp + print(f" Prepared discovery prompt ({len(units)} units, {len(units_summary)} chars)") + if self.executor is not None: + return self._execute_discover(state, prompt, context) + if self._use_local_fallback(state): + print(" ! Discover orchestration unavailable") + return False + print(" [ORCHESTRATOR] Send discover_prompt to LLM, then call complete_discover()") + return True + + def complete_discover(self, llm_response: str, state: HarnessState) -> bool: + """Called by orchestrator after LLM returns domain list.""" + domains = normalize_domains(parse_llm_domains(llm_response)) + if not domains: + print(" ! LLM returned no valid domains") + return False + + if len(domains) > 20: + logger.warning("LLM returned %d domains (>20), truncating to 20", len(domains)) + domains = domains[:20] + + fp = state.metadata.get("discover_fingerprint", {}) + self._set_discover_mode(state, "llm") + self._refresh_orchestration_mode(state) + self._persist_domains( + fingerprint=fp, + domains=domains, + discover_mode="llm", + orchestration_mode_at_discover=state.metadata.get("orchestration_mode", "llm_preferred"), + ) + print(f" Discovered {len(domains)} domains → {_domains_file()}") + return True + + # ------------------------------------------------------------------- + # Stage 1: ingest (raw — no domain assignment) + # ------------------------------------------------------------------- + + def _run_ingest_raw(self, state: HarnessState) -> bool: + """Expand sections into units WITHOUT domain assignment. Used for bootstrap.""" + print(" -> Ingesting commit-extract JSONL (raw, no domain assignment)") + units_dir = SEMANTIC_OUTPUT / "units" + units_dir.mkdir(parents=True, exist_ok=True) + + all_units, all_invariants = self._expand_records() + + save_jsonl(all_units, str(_units_file())) + save_jsonl(all_invariants, str(_invariants_file())) + print(f" Raw ingest: {len(all_units)} units, {len(all_invariants)} invariants") + return True + + # ------------------------------------------------------------------- + # Stage 1: ingest (with domain assignment) # ------------------------------------------------------------------- def _run_ingest(self, state: HarnessState) -> bool: - """Read JSONL → expand sections into semantic units + collect invariants.""" + """Expand sections into units + assign domains.""" print(" -> Ingesting commit-extract JSONL") - units_dir = SEMANTIC_OUTPUT / "units" units_dir.mkdir(parents=True, exist_ok=True) + all_units, all_invariants = self._expand_records() + + # Domain assignment (only if domains.json exists) + domains = self._load_domains() + file_paths_available = True + if domains and "discover_mode" not in state.metadata: + cached_discover_mode = self._load_domains_data().get("discover_mode") + if cached_discover_mode == "llm": + self._set_discover_mode(state, "cached_llm") + elif cached_discover_mode == "fallback": + self._set_discover_mode(state, "cached_fallback") + + if domains: + # Build SHA → file_paths map + shas = list(set(u["sha"] for u in all_units if u.get("sha"))) + repo_path = str(Path.cwd()) + sha_file_map, git_ok = build_sha_file_map(repo_path, shas) + file_paths_available = git_ok + + # Group units by commit SHA + by_sha: dict[str, list[dict]] = defaultdict(list) + for u in all_units: + by_sha[u.get("sha", "")].append(u) + + # Assign domains at commit level + needs_llm: list[dict] = [] + for sha, units in by_sha.items(): + file_paths = sha_file_map.get(sha, []) + is_mixed = any(u.get("is_mixed", False) for u in units) + + # Attach file_paths to each unit + for u in units: + u["file_paths"] = file_paths + + if not file_paths or is_mixed: + # Mixed commit or no paths → needs LLM + needs_llm.extend(units) + else: + domain = assign_domain_by_path(file_paths, domains) + if domain: + for u in units: + u["domain"] = domain + else: + # Paths span multiple domains → needs LLM + for u in units: + u["path_scoring_disabled"] = True + needs_llm.extend(units) + + # Store units needing LLM classification for orchestrator + if needs_llm: + state.metadata["needs_llm_classify"] = len(needs_llm) + state.metadata["classify_units"] = [ + {"id": str(i), "section_name": u.get("section_name", ""), + "theme": u.get("theme", ""), "summary": u.get("summary", ""), + "op": u.get("op", "")} + for i, u in enumerate(needs_llm) + ] + state.metadata["classify_unit_indices"] = [ + all_units.index(u) for u in needs_llm + ] + if self.executor is not None: + if not self._execute_classify_batches(state, needs_llm, domains, all_units): + return False + elif self._use_local_fallback(state): + print(" ! Classify orchestration unavailable") + return False + else: + self._set_classify_mode(state, "llm") + print(f" {len(needs_llm)} units need LLM classification") + print(" [ORCHESTRATOR] Send classify batches to LLM, then call complete_classify()") + else: + self._set_classify_mode(state, "cached") + state.metadata["needs_llm_classify"] = 0 + + # Mark uncategorized for units without domain + for u in all_units: + if "domain" not in u: + u["domain"] = "uncategorized" + else: + print(" No domains.json — skipping domain assignment") + + self._refresh_orchestration_mode(state) + state.metadata["file_paths_available"] = file_paths_available + + save_jsonl(all_units, str(_units_file())) + save_jsonl(all_invariants, str(_invariants_file())) + + categorized = sum(1 for u in all_units if u.get("domain", "uncategorized") != "uncategorized") + total = len(all_units) or 1 + print(f" Ingested {len(all_units)} units, {len(all_invariants)} invariants") + if domains: + print(f" Domain assignment: {categorized}/{total} ({categorized/total:.0%}) categorized") + self.add_artifact(state, str(units_dir)) + return True + + def _apply_classify_responses( + self, + llm_responses: list[str], + state: HarnessState, + *, + units: list[dict] | None = None, + ) -> bool: + indices = state.metadata.get("classify_unit_indices", []) + current_units = units if units is not None else load_jsonl(str(_units_file()), skip_errors=True) + + classified = 0 + classified_ids: set[int] = set() + staged_units = [dict(unit) for unit in current_units] + for response in llm_responses: + mapping = parse_llm_classifications(response) + if not mapping: + try: + parsed = json.loads(response.strip()) + except json.JSONDecodeError: + parsed = None + if isinstance(parsed, dict): + mapping = {str(k): v for k, v in parsed.items() if isinstance(v, str)} + if not mapping: + print(" ! LLM classification batch returned invalid output") + return False + for id_str, domain in mapping.items(): + idx = int(id_str) + if 0 <= idx < len(indices) and indices[idx] < len(staged_units): + staged_units[indices[idx]]["domain"] = domain + classified += 1 + classified_ids.add(idx) + + unresolved_count = sum( + 1 for idx, unit_idx in enumerate(indices) + if idx not in classified_ids and unit_idx < len(staged_units) + ) + if unresolved_count: + print(f" ! LLM classification incomplete: {unresolved_count} units unresolved") + return False + + if units is None: + save_jsonl(staged_units, str(_units_file())) + else: + units[:] = staged_units + self._set_classify_mode(state, "llm") + self._refresh_orchestration_mode(state) + print(f" LLM classified {classified} units") + return True + + def complete_classify(self, llm_responses: list[str], state: HarnessState) -> bool: + """Called by orchestrator after LLM classification batches return.""" + return self._apply_classify_responses(llm_responses, state) + + def _expand_records(self) -> tuple[list[dict], list[dict]]: + """Read JSONL and expand sections into units + collect invariants.""" all_units: list[dict] = [] all_invariants: list[dict] = [] @@ -90,7 +611,6 @@ def _run_ingest(self, state: HarnessState) -> bool: is_mixed = record.get("is_mixed", False) sections = record.get("sections", []) - # Expand each section's items into units for section in sections: section_name = section.get("name", "") theme = section.get("theme", "") @@ -110,7 +630,6 @@ def _run_ingest(self, state: HarnessState) -> bool: "is_mixed": is_mixed, }) - # Collect rules_invariants for inv in record.get("rules_invariants", []): all_invariants.append({ "sha": sha, @@ -120,67 +639,103 @@ def _run_ingest(self, state: HarnessState) -> bool: "enforced_by_commit": inv.get("enforced_by_commit", False), }) - save_jsonl(all_units, str(units_dir / "all.jsonl")) - save_jsonl(all_invariants, str(SEMANTIC_OUTPUT / "invariants.jsonl")) - - print(f" Ingested {len(all_units)} units, {len(all_invariants)} invariants") - self.add_artifact(state, str(units_dir)) - return True + return all_units, all_invariants + + def _load_domains_data(self) -> dict: + """Load raw domains.json payload. Returns empty dict if not available.""" + if not _domains_file().exists(): + return {} + try: + with open(_domains_file()) as f: + return json.load(f) + except (json.JSONDecodeError, OSError): + return {} + + def _load_domains(self) -> list[dict]: + """Load domains from cache. Returns empty list if not available.""" + return self._load_domains_data().get("domains", []) + + def _remove_legacy_export_artifacts(self) -> list[str]: + """Remove legacy export artifacts from previous commit-semantic output.""" + removed: list[str] = [] + for relative_path in LEGACY_EXPORT_PATHS: + legacy_path = SEMANTIC_OUTPUT / relative_path + if not legacy_path.exists(): + continue + if legacy_path.is_dir(): + shutil.rmtree(legacy_path) + else: + legacy_path.unlink() + removed.append(str(relative_path)) + return removed # ------------------------------------------------------------------- # Stage 2: aggregate # ------------------------------------------------------------------- def _run_aggregate(self, state: HarnessState) -> bool: - """Group units by theme, compute op distribution + importance ratio.""" - print(" -> Aggregating by theme") + """Group units by domain, compute statistics.""" + print(" -> Aggregating by domain") - units_file = SEMANTIC_OUTPUT / "units" / "all.jsonl" - if not units_file.exists(): + if not _units_file().exists(): print(" ! No units to aggregate") return True - units = load_jsonl(str(units_file)) + units = load_jsonl(str(_units_file())) - # Group by theme - by_theme: dict[str, list[dict]] = defaultdict(list) + # Group by domain + by_domain: dict[str, list[dict]] = defaultdict(list) for unit in units: - theme = unit.get("theme", "unknown") - by_theme[theme].append(unit) + domain = unit.get("domain", "uncategorized") + by_domain[domain].append(unit) - patterns: list[dict] = [] - for theme, theme_units in sorted(by_theme.items()): - distinct_commits = len(set(u["sha"] for u in theme_units)) - - # Threshold: >= 3 distinct commits - if distinct_commits < 3: - continue + aggregated: list[dict] = [] + for domain, domain_units in sorted(by_domain.items()): + distinct_commits = len(set(u["sha"] for u in domain_units)) # Op distribution op_dist: dict[str, int] = defaultdict(int) importance_counts = {"primary": 0, "secondary": 0} + + # Date range + min_date = "" + max_date = "" + + # Sub-themes + by_theme: dict[str, int] = defaultdict(int) summaries: list[str] = [] - for u in theme_units: + for u in domain_units: op_dist[u.get("op", "other")] += 1 imp = u.get("importance", "secondary") if imp in importance_counts: importance_counts[imp] += 1 + d = u.get("date", "") + if d: + if not min_date or d < min_date: + min_date = d + if not max_date or d > max_date: + max_date = d + theme = u.get("theme", "unknown") + by_theme[theme] += 1 if u.get("summary") and len(summaries) < 3: summaries.append(u["summary"]) - patterns.append({ - "theme": theme, - "count": len(theme_units), + aggregated.append({ + "domain": domain, + "is_uncategorized": domain == "uncategorized", + "count": len(domain_units), "distinct_commits": distinct_commits, "op_distribution": dict(op_dist), "importance_ratio": importance_counts, + "date_range": {"from": min_date, "to": max_date} if min_date else {}, + "sub_themes": dict(sorted(by_theme.items(), key=lambda x: -x[1])[:10]), "representative_summaries": summaries, }) - save_jsonl(patterns, str(SEMANTIC_OUTPUT / "patterns.jsonl")) - print(f" Found {len(patterns)} patterns (threshold >= 3 distinct commits)") - self.add_artifact(state, str(SEMANTIC_OUTPUT / "patterns.jsonl")) + save_jsonl(aggregated, str(SEMANTIC_OUTPUT / "domains-aggregated.jsonl")) + print(f" Aggregated {len(aggregated)} domains") + self.add_artifact(state, str(SEMANTIC_OUTPUT / "domains-aggregated.jsonl")) return True # ------------------------------------------------------------------- @@ -188,44 +743,95 @@ def _run_aggregate(self, state: HarnessState) -> bool: # ------------------------------------------------------------------- def _run_distill(self, state: HarnessState) -> bool: - """Extract canonical demands from patterns, scored and ranked.""" + """Score and rank domains with multi-dimensional formula.""" print(" -> Distilling canonical demands") - patterns_file = SEMANTIC_OUTPUT / "patterns.jsonl" - if not patterns_file.exists(): - print(" ! No patterns to distill") + agg_file = SEMANTIC_OUTPUT / "domains-aggregated.jsonl" + if not agg_file.exists(): + print(" ! No aggregated domains to distill") return True - patterns = load_jsonl(str(patterns_file)) + aggregated = load_jsonl(str(agg_file)) + + # Load invariants for SHA-based association + invariants = load_jsonl(str(_invariants_file())) if _invariants_file().exists() else [] + units = load_jsonl(str(_units_file())) if _units_file().exists() else [] + + # Build domain → set of SHAs + domain_shas: dict[str, set[str]] = defaultdict(set) + for u in units: + domain = u.get("domain", "uncategorized") + sha = u.get("sha", "") + if sha: + domain_shas[domain].add(sha) + + # Build invariant SHA set + inv_by_sha: dict[str, list[dict]] = defaultdict(list) + for inv in invariants: + sha = inv.get("sha", "") + if sha: + inv_by_sha[sha].append(inv) + + from datetime import datetime, timedelta + now = datetime.now() + cutoff_90d = (now - timedelta(days=90)).strftime("%Y-%m-%d") - # Score each pattern demands: list[dict] = [] - for pattern in patterns: - distinct = pattern.get("distinct_commits", 0) - imp_ratio = pattern.get("importance_ratio", {}) + for entry in aggregated: + domain = entry["domain"] + distinct = entry.get("distinct_commits", 0) + imp_ratio = entry.get("importance_ratio", {}) primary = imp_ratio.get("primary", 0) secondary = imp_ratio.get("secondary", 0) total_imp = primary + secondary - if total_imp > 0: - importance_weight = (primary * 2 + secondary * 1) / total_imp - else: - importance_weight = 1.0 - - score = distinct * importance_weight + importance_weight = (primary * 2 + secondary * 1) / total_imp if total_imp > 0 else 1.0 + + base_score = distinct * importance_weight + + # Diversity bonus + op_dist = entry.get("op_distribution", {}) + total_ops = sum(op_dist.values()) or 1 + unique_ops = len(op_dist) + diversity_bonus = unique_ops / total_ops + + # Invariant bonus (SHA association, cap=5) + shas = domain_shas.get(domain, set()) + domain_invariants: set[str] = set() + for sha in shas: + for inv in inv_by_sha.get(sha, []): + domain_invariants.add(inv.get("statement", "")) + invariant_bonus = min(len(domain_invariants), 5) + + # Recency weight + recent = 0 + for u in units: + if u.get("domain") == domain and u.get("date", "") >= cutoff_90d: + recent += 1 + total_domain = entry.get("count", 1) or 1 + recency_weight = recent / total_domain + + final_score = ( + base_score + * (1 + diversity_bonus) + * (1 + invariant_bonus * 0.3) + * (1 + recency_weight * 0.2) + ) demands.append({ - "theme": pattern["theme"], - "score": round(score, 2), + "domain": domain, + "is_uncategorized": entry.get("is_uncategorized", False), + "final_score": round(final_score, 2), + "base_score": round(base_score, 2), + "diversity_bonus": round(diversity_bonus, 4), + "invariant_bonus": invariant_bonus, + "recency_weight": round(recency_weight, 4), "distinct_commits": distinct, - "op_distribution": pattern.get("op_distribution", {}), "importance_weight": round(importance_weight, 2), - "representative_summaries": pattern.get("representative_summaries", []), + "op_distribution": op_dist, + "representative_summaries": entry.get("representative_summaries", []), }) - # Sort: score desc → distinct_commits desc → theme alpha - demands.sort(key=lambda d: (-d["score"], -d["distinct_commits"], d["theme"])) - - # Add rank + demands.sort(key=lambda d: (-d["final_score"], -d["distinct_commits"], d["domain"])) for i, d in enumerate(demands, 1): d["rank"] = i @@ -242,20 +848,19 @@ def _run_export(self, state: HarnessState) -> bool: """Generate summary statistics.""" print(" -> Generating export summary") - # Load units for stats - units_file = SEMANTIC_OUTPUT / "units" / "all.jsonl" - units = load_jsonl(str(units_file)) if units_file.exists() else [] + units = load_jsonl(str(_units_file())) if _units_file().exists() else [] + invariants = load_jsonl(str(_invariants_file())) if _invariants_file().exists() else [] - patterns_file = SEMANTIC_OUTPUT / "patterns.jsonl" - patterns = load_jsonl(str(patterns_file)) if patterns_file.exists() else [] + agg_file = SEMANTIC_OUTPUT / "domains-aggregated.jsonl" + aggregated = load_jsonl(str(agg_file)) if agg_file.exists() else [] - invariants_file = SEMANTIC_OUTPUT / "invariants.jsonl" - invariants = load_jsonl(str(invariants_file)) if invariants_file.exists() else [] + demands_file = SEMANTIC_OUTPUT / "canonical-demands.jsonl" + demands = load_jsonl(str(demands_file)) if demands_file.exists() else [] - # Op distribution across all units + # Op distribution op_dist: dict[str, int] = defaultdict(int) - min_date: str = "" - max_date: str = "" + min_date = "" + max_date = "" for u in units: op_dist[u.get("op", "other")] += 1 d = u.get("date", "") @@ -269,32 +874,40 @@ def _run_export(self, state: HarnessState) -> bool: total = len(units) or 1 bugfix_ratio = round(bugfix_count / total, 4) - # Top patterns by score - demands_file = SEMANTIC_OUTPUT / "canonical-demands.jsonl" - demands = load_jsonl(str(demands_file)) if demands_file.exists() else [] - top_patterns = [ - {"theme": d["theme"], "score": d["score"], "distinct_commits": d["distinct_commits"]} + # Uncategorized ratio + uncategorized = sum(1 for u in units if u.get("domain", "uncategorized") == "uncategorized") + uncategorized_ratio = round(uncategorized / total, 4) + + # Top domains + top_domains = [ + {"domain": d["domain"], "final_score": d["final_score"], + "distinct_commits": d["distinct_commits"]} for d in demands[:10] ] - # Date range - date_range = {} - if min_date: - date_range = {"from": min_date, "to": max_date} - + removed_legacy_paths = self._remove_legacy_export_artifacts() summary = { "total_units": len(units), - "total_patterns": len(patterns), + "domain_count": len(aggregated), + "uncategorized_ratio": uncategorized_ratio, "op_distribution": dict(op_dist), - "top_patterns": top_patterns, + "top_domains": top_domains, "bugfix_ratio": bugfix_ratio, "invariant_count": len(invariants), - "date_range": date_range, + "date_range": {"from": min_date, "to": max_date} if min_date else {}, + "file_paths_available": state.metadata.get("file_paths_available", True), + "orchestration_mode": state.metadata.get("orchestration_mode", "local_fallback"), + "discover_mode": state.metadata.get("discover_mode", "fallback"), + "classify_mode": state.metadata.get("classify_mode", "cached"), } + if removed_legacy_paths: + summary["removed_legacy_paths"] = removed_legacy_paths save_json(summary, str(SEMANTIC_OUTPUT / "summary.json")) - print(f" Exported: {len(units)} units, {len(patterns)} patterns, " - f"bugfix ratio {bugfix_ratio:.1%}") + print(f" Exported: {len(units)} units, {len(aggregated)} domains, " + f"uncategorized {uncategorized_ratio:.1%}, bugfix {bugfix_ratio:.1%}") + if removed_legacy_paths: + print(f" Removed legacy artifacts: {', '.join(removed_legacy_paths)}") self.add_artifact(state, str(SEMANTIC_OUTPUT / "summary.json")) return True @@ -316,6 +929,7 @@ def handle_run(self, remaining: list[str] | None = None) -> int: argv = remaining or [] parser = argparse.ArgumentParser() parser.add_argument("--stage", help="Run a specific stage") + parser.add_argument("--force", action="store_true", help="Force re-discovery") args = parser.parse_args(argv) if args.stage: @@ -326,11 +940,26 @@ def handle_run(self, remaining: list[str] | None = None) -> int: if not self._require_prerequisites(): return 1 state = self.init_state() + if args.force: + state.metadata["force"] = True save_state(self.PIPELINE, state) success = self.run_stage(args.stage, state) return 0 if success else 1 - return super().handle_run() + if not self._require_prerequisites(): + return 1 + + # Check state compatibility + old_state = load_state(self.PIPELINE) + if not self.is_fresh(old_state): + old_state = self._check_state_compat(old_state) + save_state(self.PIPELINE, old_state) + + state = self.init_state() + if args.force: + state.metadata["force"] = True + save_state(self.PIPELINE, state) + return self.handle_resume() def run_commit_semantic() -> None: diff --git a/skills/repo_structure/preflight.py b/skills/repo_structure/preflight.py index 7bd2e7e..d7ae3f3 100644 --- a/skills/repo_structure/preflight.py +++ b/skills/repo_structure/preflight.py @@ -14,7 +14,6 @@ import subprocess from dataclasses import dataclass, field from pathlib import Path -from typing import Literal @dataclass @@ -139,13 +138,13 @@ def check(repo_root: Path | str = ".") -> PreflightResult: result.ok = False result.invalid.append(PreflightIssue( "EMPTY_ARTIFACT", str(fpath.relative_to(root)), - f"gsd file is empty", + "gsd file is empty", producer="gsd::map-codebase")) except FileNotFoundError: result.ok = False result.missing.append(PreflightIssue( "MISSING_INPUT", str(fpath.relative_to(root)), - f"gsd file not found", + "gsd file not found", producer="gsd::map-codebase", suggestion="Run gsd map-codebase first")) diff --git a/skills/repo_structure/run.py b/skills/repo_structure/run.py index 1b5f112..16e53a5 100644 --- a/skills/repo_structure/run.py +++ b/skills/repo_structure/run.py @@ -20,18 +20,15 @@ import sys import uuid from collections import defaultdict -from concurrent.futures import ThreadPoolExecutor -from datetime import datetime, date +from datetime import date, datetime from pathlib import Path from typing import Any -import yaml - sys.path.insert(0, str(Path(__file__).parent.parent.parent)) -from src.io_utils import save_yaml, load_yaml -from src.skill_runner import SkillRunner, run_skill from src.harness_state import HarnessState +from src.io_utils import load_jsonl, load_yaml, save_yaml +from src.skill_runner import SkillRunner from .preflight import check as preflight_check, REQUIRED_GSD_FILES @@ -238,22 +235,19 @@ def _run_hotspot(self, state: HarnessState) -> bool: print(f" ERROR: commit-extract output not found at {commit_extract_dir}") return False - monthly_files = sorted(commit_extract_dir.glob("????-??.yaml")) + monthly_files = sorted(commit_extract_dir.glob("????-??.jsonl")) print(f" Found {len(monthly_files)} monthly commit files") - patterns_dir = Path("data/commit-semantic/patterns") - patterns: list = [] - if patterns_dir.exists(): - for pf in patterns_dir.glob("*.yaml"): - try: - data = load_yaml(str(pf)) - if "patterns" in data: - patterns.extend(data["patterns"]) - except Exception as e: - print(f" WARNING: could not load {pf}: {e}") + aggregated_domains_path = Path("data/commit-semantic/domains-aggregated.jsonl") + aggregated_domains: list[dict[str, Any]] = [] + if aggregated_domains_path.exists(): + try: + aggregated_domains = load_jsonl(str(aggregated_domains_path), skip_errors=True) + except Exception as e: + print(f" WARNING: could not load {aggregated_domains_path}: {e}") head = self._get_repo_head() - hotspots = self._aggregate_hotspots(monthly_files, patterns, head) + hotspots = self._aggregate_hotspots(monthly_files, aggregated_domains, head) version = self._next_version("hotspot_map") maps_dir = OUTPUT_BASE / "maps" @@ -265,7 +259,8 @@ def _run_hotspot(self, state: HarnessState) -> bool: "repo_snapshot_commit": head, "generated_at": datetime.now().isoformat(), "monthly_files": [str(f) for f in monthly_files], - "total_patterns": len(patterns), + "aggregated_domains_path": str(aggregated_domains_path), + "total_domains": len(aggregated_domains), }, "facts": hotspots, }, str(out_path)) @@ -274,20 +269,24 @@ def _run_hotspot(self, state: HarnessState) -> bool: return True def _aggregate_hotspots( - self, monthly_files: list[Path], patterns: list, head: str + self, monthly_files: list[Path], aggregated_domains: list[dict[str, Any]], head: str ) -> list[dict[str, Any]]: - """Aggregate commit-extract data and commit-semantic patterns into hotspot facts.""" + """Aggregate commit-extract JSONL and commit-semantic domain output into hotspot facts.""" module_commit_count: dict[str, int] = defaultdict(int) module_files: dict[str, set[str]] = defaultdict(set) for mf in monthly_files: try: - data = load_yaml(str(mf)) - for commit in data.get("commits", []): - for f in commit.get("files", []): - module = str(f).split("/")[0] if "/" in str(f) else "root" + commits = load_jsonl(str(mf), skip_errors=True) + for commit in commits: + file_list = commit.get("file_paths") or commit.get("files") or [] + if not isinstance(file_list, list): + continue + for file_path in file_list: + file_path = str(file_path) + module = file_path.split("/")[0] if "/" in file_path else "root" module_commit_count[module] += 1 - module_files[module].add(str(f)) + module_files[module].add(file_path) except Exception as e: print(f" WARNING: skipped malformed file {mf}: {e}") @@ -308,7 +307,7 @@ def _aggregate_hotspots( "source": "hotspot", "evidence": [{ "source_type": "hotspot", - "file_path": "data/commit-extract/*.yaml", + "file_path": "data/commit-extract/*.jsonl", "locator_type": "file_path", "locator": module, "stable_ref": f"module:{module}", @@ -319,25 +318,29 @@ def _aggregate_hotspots( "files": sorted(module_files[module]), }) - for pattern in patterns[:10]: - pid = pattern.get("pattern_id", "unknown") + for domain in aggregated_domains[:10]: + domain_name = domain.get("domain") or domain.get("domain_id") or "unknown" + commit_count = domain.get("commit_count") or len(domain.get("commit_shas", []) or []) + file_list = domain.get("file_paths") or domain.get("files") or [] hotspots.append({ "fact_id": str(uuid.uuid4()), "fact_type": "hotspot_signal", - "domain": "semantic_pattern", - "statement": f"Recurring pattern: {pattern.get('description', pid)}", + "domain": "semantic_domain", + "statement": f"Domain '{domain_name}' is a semantic hotspot across {commit_count} commits", "confidence": "confirmed", "status": "active", "repo_snapshot_commit": head, "source": "hotspot", "evidence": [{ "source_type": "hotspot", - "file_path": "data/commit-semantic/patterns/", + "file_path": "data/commit-semantic/domains-aggregated.jsonl", "locator_type": "section_ref", - "locator": pid, - "stable_ref": f"pattern:{pid}", - "rationale": "From commit-semantic pattern extraction", + "locator": str(domain_name), + "stable_ref": f"domain:{domain_name}", + "rationale": "From commit-semantic aggregated domain output", }], + "commit_count": commit_count, + "files": sorted(str(path) for path in file_list) if isinstance(file_list, list) else [], }) return hotspots @@ -348,7 +351,7 @@ def _run_extract(self, state: HarnessState) -> bool: manifest_path = OUTPUT_BASE / "sample" / "manifest.yaml" if not manifest_path.exists(): print(f" ERROR: sample manifest not found at {manifest_path}") - print(f" Run 'repo-structure --stage sample' first") + print(" Run 'repo-structure --stage sample' first") return False manifest = load_yaml(str(manifest_path)) @@ -943,7 +946,10 @@ def _next_version(self, artifact_name: str) -> str: """Get next version number for an artifact.""" maps_dir = OUTPUT_BASE / "maps" existing = sorted(maps_dir.glob(f"{artifact_name}.v*.yaml")) - return "v0" if not existing else f"v{int(existing[-1].stem.split(".")[-1][1:]) + 1}" + if not existing: + return "v0" + latest_version = existing[-1].stem.split(".")[-1] + return f"v{int(latest_version[1:]) + 1}" # ------------------------------------------------------------------------- # Override run to inject preflight diff --git a/src/commit_semantic/domain_utils.py b/src/commit_semantic/domain_utils.py new file mode 100644 index 0000000..f0cc4d7 --- /dev/null +++ b/src/commit_semantic/domain_utils.py @@ -0,0 +1,421 @@ +"""Domain utilities for commit-semantic pipeline. + +Pure functions for domain discovery and assignment. +LLM orchestration stays in run.py; this module is fully testable without mocks. +""" + +from __future__ import annotations + +import hashlib +import json +import logging +import re +import subprocess +from pathlib import Path + +logger = logging.getLogger(__name__) + +NOISE_DOMAIN_NAMES = { + "misc", + "other", + "others", + "general", + "generic", + "unknown", + "uncategorized", +} + + +def _normalize_name(value: str) -> str: + return (value or "").strip().lower() + + +def _singular_plural_forms(name: str) -> set[str]: + normalized = _normalize_name(name) + forms = {normalized} + if normalized.endswith("s") and len(normalized) > 1: + forms.add(normalized[:-1]) + elif normalized: + forms.add(f"{normalized}s") + return forms + + +def _unique_sorted_strings(values: list[str]) -> list[str]: + return sorted({value for value in values if value}) + + +def _overlap_ratio(left: set[str], right: set[str]) -> float: + if not left or not right: + return 0.0 + return len(left & right) / min(len(left), len(right)) + + +def _is_noise_domain(domain: dict) -> bool: + return _normalize_name(domain.get("domain", "")) in NOISE_DOMAIN_NAMES + + +def choose_domain_winner(left: dict, right: dict) -> dict: + """Choose the stronger canonical domain between two merge candidates.""" + left_name = _normalize_name(left.get("domain", "")) + right_name = _normalize_name(right.get("domain", "")) + + if left_name != right_name and left_name in _singular_plural_forms(right_name): + if right_name.endswith("s") and not left_name.endswith("s"): + return right + if left_name.endswith("s") and not right_name.endswith("s"): + return left + + left_score = ( + len(_unique_sorted_strings(left.get("paths", []))), + len(_unique_sorted_strings(left.get("keywords", []))), + len((left.get("description", "") or "").strip()), + ) + right_score = ( + len(_unique_sorted_strings(right.get("paths", []))), + len(_unique_sorted_strings(right.get("keywords", []))), + len((right.get("description", "") or "").strip()), + ) + return right if right_score > left_score else left + + +def should_merge_domains(left: dict, right: dict) -> bool: + """Return True when two discovered domains represent the same cluster.""" + left_name = _normalize_name(left.get("domain", "")) + right_name = _normalize_name(right.get("domain", "")) + if not left_name or not right_name: + return False + if left_name == right_name: + return True + if left_name in _singular_plural_forms(right_name): + return True + + left_keywords = {_normalize_name(value) for value in left.get("keywords", []) if _normalize_name(value)} + right_keywords = {_normalize_name(value) for value in right.get("keywords", []) if _normalize_name(value)} + if len(left_keywords & right_keywords) >= 2 and _overlap_ratio(left_keywords, right_keywords) >= 0.5: + return True + + left_paths = {value for value in left.get("paths", []) if value} + right_paths = {value for value in right.get("paths", []) if value} + if left_paths and right_paths and _overlap_ratio(left_paths, right_paths) >= 0.6: + return True + + return False + + +def _merge_domain_pair(left: dict, right: dict) -> dict: + winner = choose_domain_winner(left, right) + loser = right if winner is left else left + return { + "domain": winner.get("domain", ""), + "description": winner.get("description", "") or loser.get("description", ""), + "paths": _unique_sorted_strings(list(winner.get("paths", [])) + list(loser.get("paths", []))), + "keywords": _unique_sorted_strings(list(winner.get("keywords", [])) + list(loser.get("keywords", []))), + } + + +def normalize_domains(domains: list[dict]) -> list[dict]: + """Filter noise and merge overlapping discovered domains into canonical entries.""" + normalized: list[dict] = [] + for domain in domains: + candidate = { + "domain": domain.get("domain", ""), + "description": domain.get("description", ""), + "paths": _unique_sorted_strings(list(domain.get("paths", []))), + "keywords": _unique_sorted_strings(list(domain.get("keywords", []))), + } + if not candidate["domain"] or _is_noise_domain(candidate): + continue + + merged = False + for index, existing in enumerate(normalized): + if should_merge_domains(existing, candidate): + normalized[index] = _merge_domain_pair(existing, candidate) + merged = True + break + if not merged: + normalized.append(candidate) + + return normalized + + +def build_sha_file_map(repo_path: str, shas: list[str]) -> tuple[dict[str, list[str]], bool]: + """Batch-fetch SHA → file_paths mapping. Single git call. + + Returns (sha_map, success). If git fails, returns empty lists for all SHAs + and success=False so caller can record degradation in summary. + + Critical: --no-walk prevents git from walking full history per SHA. + """ + if not shas: + return {}, True + + result = subprocess.run( + ["git", "log", "--name-only", "--format=%H", "--stdin", "--no-walk"], + input="\n".join(shas), + capture_output=True, text=True, cwd=repo_path, + ) + if result.returncode != 0: + logger.warning("git log failed (rc=%d): %s", result.returncode, result.stderr[:200]) + return {sha: [] for sha in shas}, False + + sha_map: dict[str, list[str]] = {} + current_sha = None + for line in result.stdout.splitlines(): + line = line.strip() + if not line: + continue + if len(line) == 40 and all(c in "0123456789abcdef" for c in line): + current_sha = line + sha_map[current_sha] = [] + elif current_sha: + sha_map[current_sha].append(line) + return sha_map, True + + +def assign_domain_by_path(file_paths: list[str], domains: list[dict]) -> str | None: + """Path-prefix matching at commit level. Longest prefix wins. + + Returns domain name if ALL file_paths resolve to the same domain (single-domain commit). + Returns None if paths span multiple domains or no match (needs LLM classification). + """ + if not file_paths or not domains: + return None + + matched_domains: set[str] = set() + for fp in file_paths: + best_domain = None + best_prefix_len = 0 + for domain in domains: + for path_prefix in domain.get("paths", []): + if fp.startswith(path_prefix) and len(path_prefix) > best_prefix_len: + best_prefix_len = len(path_prefix) + best_domain = domain["domain"] + if best_domain: + matched_domains.add(best_domain) + + if len(matched_domains) == 1: + return matched_domains.pop() + return None + + +def parse_llm_domains(raw: str) -> list[dict]: + """Parse LLM output for domain discovery. Expects JSON array. + + Returns parsed domains list, or empty list on parse failure. + Each domain must have at least 'domain' and 'description' keys. + """ + raw = raw.strip() + # Strip markdown code fences if present + if raw.startswith("```"): + lines = raw.splitlines() + lines = [line for line in lines if not line.strip().startswith("```")] + raw = "\n".join(lines).strip() + + try: + data = json.loads(raw) + except json.JSONDecodeError: + logger.warning("Failed to parse LLM domains output as JSON") + return [] + + if not isinstance(data, list): + logger.warning("LLM domains output is not a list") + return [] + + valid = [] + for item in data: + if isinstance(item, dict) and "domain" in item: + valid.append({ + "domain": item["domain"], + "description": item.get("description", ""), + "paths": item.get("paths", []), + "keywords": item.get("keywords", []), + }) + return valid + + +def parse_llm_classifications(raw: str) -> dict[str, str]: + """Parse LLM output for unit classification. Expects JSON array of {id, domain}. + + Returns {unit_id: domain} mapping. Unparseable entries are skipped. + """ + raw = raw.strip() + if raw.startswith("```"): + lines = raw.splitlines() + lines = [line for line in lines if not line.strip().startswith("```")] + raw = "\n".join(lines).strip() + + try: + data = json.loads(raw) + except json.JSONDecodeError: + logger.warning("Failed to parse LLM classification output as JSON") + return {} + + if not isinstance(data, list): + return {} + + result: dict[str, str] = {} + for item in data: + if isinstance(item, dict) and "id" in item and "domain" in item: + result[str(item["id"])] = item["domain"] + return result + + +TOKEN_PATTERN = re.compile(r"[a-z0-9_/-]+") +PATH_PREFIX_WEIGHT = 5 +THEME_TOKEN_WEIGHT = 3 +SUMMARY_TOKEN_WEIGHT = 2 +SECTION_NAME_TOKEN_WEIGHT = 2 +DOMAIN_KEYWORD_WEIGHT = 1 +MIN_ASSIGNMENT_SCORE = 4 +AMBIGUITY_DELTA = 2 + + +def _tokenize_signal(value: str) -> set[str]: + normalized = (value or "").lower().replace("-", " ").replace("_", " ").replace("/", " ") + return {token for token in TOKEN_PATTERN.findall(normalized) if token} + + +def _split_keywords(values: list[str]) -> set[str]: + tokens: set[str] = set() + for value in values: + tokens.update(_tokenize_signal(value)) + return tokens + + +def _domain_tokens(domain: dict) -> set[str]: + domain_name = str(domain.get("domain", "") or "").replace("-", " ").replace("_", " ") + return _tokenize_signal(domain_name) + + +def score_unit_for_domain(unit: dict, domain: dict, *, allow_path_scoring: bool = True) -> int: + """Deterministically score one unit against one domain.""" + score = 0 + + unit_paths = [path for path in unit.get("file_paths", []) if path] + if allow_path_scoring and unit_paths: + if any( + path.startswith(path_prefix) + for path_prefix in domain.get("paths", []) + if path_prefix + for path in unit_paths + ): + score += PATH_PREFIX_WEIGHT + + domain_tokens = _domain_tokens(domain) + keyword_tokens = _split_keywords(list(domain.get("keywords", []))) + signal_tokens = domain_tokens | keyword_tokens + + theme_tokens = _tokenize_signal(unit.get("theme", "")) + if signal_tokens & theme_tokens: + score += THEME_TOKEN_WEIGHT + + summary_tokens = _tokenize_signal(unit.get("summary", "")) + if signal_tokens & summary_tokens: + score += SUMMARY_TOKEN_WEIGHT + + section_tokens = _tokenize_signal(unit.get("section_name", "")) + if signal_tokens & section_tokens: + score += SECTION_NAME_TOKEN_WEIGHT + + if keyword_tokens & (theme_tokens | summary_tokens | section_tokens): + score += DOMAIN_KEYWORD_WEIGHT + + return score + + +def classify_unit_locally(unit: dict, domains: list[dict], *, allow_path_scoring: bool = True) -> str | None: + """Return a deterministic domain only when evidence is strong and unambiguous.""" + scored: list[tuple[int, str]] = [] + for domain in domains: + score = score_unit_for_domain(unit, domain, allow_path_scoring=allow_path_scoring) + scored.append((score, domain.get("domain", ""))) + + if not scored: + return None + + scored.sort(key=lambda item: (-item[0], item[1])) + top_score, top_domain = scored[0] + if top_score < MIN_ASSIGNMENT_SCORE: + return None + + second_score = scored[1][0] if len(scored) > 1 else 0 + if top_score - second_score < AMBIGUITY_DELTA: + return None + + return top_domain or None + + +def compute_fingerprint(units_file: Path, arch_file: Path | None = None) -> dict: + """Compute content fingerprint for cache invalidation. + + Returns dict with units_hash, arch_hash (or null), for embedding in domains.json. + """ + units_hash = "" + if units_file.exists(): + h = hashlib.sha256() + with open(units_file, "rb") as f: + for chunk in iter(lambda: f.read(8192), b""): + h.update(chunk) + units_hash = h.hexdigest()[:16] + + arch_hash = None + if arch_file and arch_file.exists(): + h = hashlib.sha256() + with open(arch_file, "rb") as f: + for chunk in iter(lambda: f.read(8192), b""): + h.update(chunk) + arch_hash = h.hexdigest()[:16] + + return {"units_hash": units_hash, "arch_hash": arch_hash} + + +def fingerprint_matches(domains_data: dict, current_fp: dict) -> bool: + """Check if cached domains.json fingerprint matches current inputs.""" + cached_fp = domains_data.get("_fingerprint", {}) + return ( + cached_fp.get("units_hash") == current_fp.get("units_hash") + and cached_fp.get("arch_hash") == current_fp.get("arch_hash") + ) + + +def build_units_summary(units: list[dict], max_themes: int = 30, max_summaries: int = 5) -> str: + """Build a compact summary of units for the discover LLM prompt. + + Includes theme distribution, op distribution, and representative summaries. + """ + from collections import Counter + + theme_counts = Counter(u.get("theme", "unknown") for u in units) + op_counts = Counter(u.get("op", "other") for u in units) + + top_themes = theme_counts.most_common(max_themes) + summaries = [] + seen_themes: set[str] = set() + for u in units: + theme = u.get("theme", "") + summary = u.get("summary", "") + if summary and theme not in seen_themes and len(summaries) < max_summaries: + summaries.append(f"[{theme}] {summary}") + seen_themes.add(theme) + + lines = [ + f"Total units: {len(units)}", + f"Distinct themes: {len(theme_counts)}", + "", + "Theme distribution (top {}):" .format(min(max_themes, len(top_themes))), + ] + for theme, count in top_themes: + lines.append(f" {theme}: {count}") + + lines.append("") + lines.append("Op distribution:") + for op, count in op_counts.most_common(): + lines.append(f" {op}: {count}") + + if summaries: + lines.append("") + lines.append("Representative summaries:") + for s in summaries: + lines.append(f" - {s}") + + return "\n".join(lines) diff --git a/tests/e2e/test_commit_extract.py b/tests/e2e/test_commit_extract.py index fd77a9e..6d64239 100644 --- a/tests/e2e/test_commit_extract.py +++ b/tests/e2e/test_commit_extract.py @@ -76,6 +76,58 @@ def test_collect_produces_manifest(self): assert len(manifest["batches"]) == 1 assert len(manifest["batches"][0]["shas"]) == 1 + def test_collect_local_fallback_writes_monthly_jsonl(self): + """Full local collect path writes schema-valid monthly JSONL output.""" + mod = load_commit_extract_module() + + with tempfile.TemporaryDirectory() as tmpdir: + repo_path = Path(tmpdir) / "test_repo" + repo_path.mkdir() + subprocess.run(["git", "init"], cwd=repo_path, capture_output=True, check=True) + subprocess.run(["git", "config", "user.email", "t@t.com"], cwd=repo_path, capture_output=True, check=True) + subprocess.run(["git", "config", "user.name", "Test User"], cwd=repo_path, capture_output=True, check=True) + + (repo_path / "f.txt").write_text("hello\n") + subprocess.run(["git", "add", "."], cwd=repo_path, capture_output=True, check=True) + subprocess.run( + ["git", "commit", "-m", "feat: add login\n\nEnsure auth stays enabled."], + cwd=repo_path, + capture_output=True, + check=True, + ) + + runner = mod.CommitExtractRunner() + runner.repo_path = str(repo_path) + + saved_base = mod.OUTPUT_BASE + saved_tmp = mod.TMP_DIR + mod.OUTPUT_BASE = Path(tmpdir) / "output" + mod.TMP_DIR = mod.OUTPUT_BASE / "tmp" + + from src.harness_state import HarnessState + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + result = runner._run_collect(state) + + month_files = list((Path(tmpdir) / "output").glob("*.jsonl")) + + mod.OUTPUT_BASE = saved_base + mod.TMP_DIR = saved_tmp + + assert result is True + assert len(month_files) == 1 + + records = [json.loads(line) for line in month_files[0].read_text().splitlines() if line.strip()] + assert len(records) == 1 + record = records[0] + assert record["sha"] + assert record["author"] == "Test User" + assert "T" in record["date"] + assert record["sections"] + assert record["sections"][0]["items"][0]["op"] == "feat" + assert record["rules_invariants"] == [ + {"kind": "rule", "statement": "Ensure auth stays enabled.", "enforced_by_commit": False} + ] + class TestCommitExtractParseStat: """Tests for git show --stat parsing.""" diff --git a/tests/e2e/test_commit_semantic.py b/tests/e2e/test_commit_semantic.py index a0e2aca..8766c7a 100644 --- a/tests/e2e/test_commit_semantic.py +++ b/tests/e2e/test_commit_semantic.py @@ -1,11 +1,14 @@ -"""E2E tests for commit-semantic skill (4-stage JSONL pipeline). +"""E2E tests for commit-semantic skill (5-stage domain pipeline). -Tests: ingest → aggregate → distill → export consuming JSONL from commit-extract. +Tests: discover → ingest → aggregate → distill → export consuming JSONL from commit-extract. +Note: discover stage requires LLM, so most tests skip it and run ingest→export directly. +Without domains.json, all units are "uncategorized" — this is expected behavior. """ from __future__ import annotations import json +import subprocess import sys from pathlib import Path @@ -13,7 +16,66 @@ sys.path.insert(0, str(Path(__file__).parent.parent.parent)) -from src.io_utils import save_jsonl, load_jsonl, load_json +from src.io_utils import save_jsonl, load_jsonl, load_json, save_json + + +class FakeHostExecutor: + """Deterministic host executor for commit-semantic tests.""" + + def __init__(self) -> None: + self.calls: list[dict] = [] + + def __call__( + self, + prompt_text: str, + context: dict[str, str], + *, + artifact_name: str, + sampling_mode: str = "auto", + ) -> str: + self.calls.append( + { + "prompt_text": prompt_text, + "context": context, + "artifact_name": artifact_name, + "sampling_mode": sampling_mode, + } + ) + + if artifact_name == "domains": + return json.dumps( + [ + { + "domain": "auth", + "description": "Authentication and session flows", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "demand", + "description": "Issue and demand mapping", + "paths": ["src/demand/"], + "keywords": ["demand", "issue", "requirement", "card"], + }, + ] + ) + + if artifact_name == "classify-units": + units = json.loads(context["units_json"]) + classifications = [] + for unit in units: + text = " ".join( + [ + unit.get("section_name", ""), + unit.get("theme", ""), + unit.get("summary", ""), + ] + ).lower() + domain = "demand" if "demand" in text or "issue" in text else "auth" + classifications.append({"id": unit["id"], "domain": domain}) + return json.dumps(classifications) + + raise AssertionError(f"Unexpected artifact_name: {artifact_name}") def load_commit_semantic_module(): @@ -82,6 +144,10 @@ def sample_jsonl_records(): ] +# Skip discover in tests (requires LLM). Run ingest→aggregate→distill→export. +NON_LLM_STAGES = ["ingest", "aggregate", "distill", "export"] + + def _setup_and_run(tmp_path, sample_jsonl_records, stages=None): """Helper: write fixture, run stages, return (mod, semantic_dir).""" mod = load_commit_semantic_module() @@ -98,12 +164,22 @@ def _setup_and_run(tmp_path, sample_jsonl_records, stages=None): runner = mod.CommitSemanticRunner() state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) - for stage in (stages or runner.STAGES): + for stage in (stages or NON_LLM_STAGES): runner.run_stage(stage, state) return mod, semantic_dir +def _write_semantic_inputs(semantic_dir, *, units=None, invariants=None, aggregated=None): + """Write semantic stage fixtures directly for aggregate/distill tests.""" + units_dir = semantic_dir / "units" + units_dir.mkdir(parents=True, exist_ok=True) + save_jsonl(units or [], str(units_dir / "all.jsonl")) + save_jsonl(invariants or [], str(semantic_dir / "invariants.jsonl")) + if aggregated is not None: + save_jsonl(aggregated, str(semantic_dir / "domains-aggregated.jsonl")) + + class TestCommitSemanticSkill: """Basic skill structure.""" @@ -111,7 +187,7 @@ def test_skill_exists(self): mod = load_commit_semantic_module() runner = mod.CommitSemanticRunner() assert runner.PIPELINE == "commit-semantic" - assert runner.STAGES == ["ingest", "aggregate", "distill", "export"] + assert runner.STAGES == ["discover", "ingest", "aggregate", "distill", "export"] class TestCommitSemanticPrerequisites: @@ -194,78 +270,273 @@ def test_zero_sections_commit(self, tmp_path): class TestCommitSemanticAggregate: - """Stage 2: aggregate.""" + """Stage 2: aggregate by domain with audit fields.""" - def test_theme_threshold(self, tmp_path, sample_jsonl_records): - """Only themes with >= 3 distinct commits become patterns.""" + def test_uncategorized_without_domains(self, tmp_path, sample_jsonl_records): + """Without domains.json, all units aggregate into uncategorized.""" _, semantic_dir = _setup_and_run(tmp_path, sample_jsonl_records, ["ingest", "aggregate"]) - patterns = load_jsonl(str(semantic_dir / "patterns.jsonl")) + aggregated = load_jsonl(str(semantic_dir / "domains-aggregated.jsonl")) + assert len(aggregated) == 1 + assert aggregated[0]["domain"] == "uncategorized" + assert aggregated[0]["is_uncategorized"] is True + assert aggregated[0]["count"] == 6 - themes = [p["theme"] for p in patterns] - assert "auth-flow" in themes # 4 distinct commits - assert "config-mgmt" not in themes # 1 commit - assert "retry-logic" not in themes # 1 commit + def test_groups_by_domain_and_preserves_sub_themes(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + _write_semantic_inputs( + semantic_dir, + units=[ + { + "sha": "a1", + "date": "2026-03-01T10:00:00", + "domain": "auth", + "theme": "login", + "importance": "primary", + "op": "feat", + "summary": "Add password login", + }, + { + "sha": "a2", + "date": "2026-03-02T10:00:00", + "domain": "auth", + "theme": "login", + "importance": "secondary", + "op": "bugfix", + "summary": "Fix login redirect", + }, + { + "sha": "a3", + "date": "2026-03-03T10:00:00", + "domain": "auth", + "theme": "session", + "importance": "primary", + "op": "refactor", + "summary": "Simplify session middleware", + }, + { + "sha": "u1", + "date": "2026-03-04T10:00:00", + "domain": "uncategorized", + "theme": "misc-cleanup", + "importance": "secondary", + "op": "chore", + "summary": "Clean up misc paths", + }, + ], + ) - def test_op_distribution(self, tmp_path, sample_jsonl_records): - _, semantic_dir = _setup_and_run(tmp_path, sample_jsonl_records, ["ingest", "aggregate"]) - patterns = load_jsonl(str(semantic_dir / "patterns.jsonl")) - auth = [p for p in patterns if p["theme"] == "auth-flow"][0] - assert auth["op_distribution"]["feat"] == 3 - assert auth["op_distribution"]["refactor"] == 1 + mod.SEMANTIC_OUTPUT = semantic_dir + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) - def test_importance_ratio(self, tmp_path, sample_jsonl_records): - _, semantic_dir = _setup_and_run(tmp_path, sample_jsonl_records, ["ingest", "aggregate"]) - patterns = load_jsonl(str(semantic_dir / "patterns.jsonl")) - auth = [p for p in patterns if p["theme"] == "auth-flow"][0] - assert auth["importance_ratio"]["primary"] == 3 - assert auth["importance_ratio"]["secondary"] == 1 + runner._run_aggregate(state) - def test_theme_below_threshold(self, tmp_path): - """Theme with exactly 2 commits should NOT be a pattern.""" - records = [ - {"sha": "a", "date": "2026-03-01", "sections": [ - {"name": "X", "theme": "two-commit-theme", "importance": "primary", - "items": [{"op": "feat", "summary": "s1"}]} - ], "rules_invariants": []}, - {"sha": "b", "date": "2026-03-02", "sections": [ - {"name": "X", "theme": "two-commit-theme", "importance": "primary", - "items": [{"op": "feat", "summary": "s2"}]} - ], "rules_invariants": []}, - ] - _, semantic_dir = _setup_and_run(tmp_path, records, ["ingest", "aggregate"]) - patterns = load_jsonl(str(semantic_dir / "patterns.jsonl")) - assert len(patterns) == 0 + aggregated = load_jsonl(str(semantic_dir / "domains-aggregated.jsonl")) + assert [entry["domain"] for entry in aggregated] == ["auth", "uncategorized"] + + auth = aggregated[0] + assert auth["is_uncategorized"] is False + assert auth["distinct_commits"] == 3 + assert auth["op_distribution"] == {"feat": 1, "bugfix": 1, "refactor": 1} + assert auth["importance_ratio"] == {"primary": 2, "secondary": 1} + assert auth["sub_themes"] == {"login": 2, "session": 1} + + uncat = aggregated[1] + assert uncat["is_uncategorized"] is True + assert uncat["sub_themes"] == {"misc-cleanup": 1} class TestCommitSemanticDistill: - """Stage 3: distill.""" + """Stage 3: distill with domain scoring and ranking.""" - def test_scoring_formula(self, tmp_path, sample_jsonl_records): + def test_scoring_produces_audit_breakdown(self, tmp_path, sample_jsonl_records): _, semantic_dir = _setup_and_run(tmp_path, sample_jsonl_records, ["ingest", "aggregate", "distill"]) demands = load_jsonl(str(semantic_dir / "canonical-demands.jsonl")) assert len(demands) >= 1 - d = demands[0] - # auth-flow: 4 distinct, importance_weight = (3*2+1*1)/4 = 1.75, score = 7.0 - assert d["theme"] == "auth-flow" - assert d["score"] == 7.0 - assert d["rank"] == 1 - - def test_tiebreak_order(self, tmp_path): - """Same score → distinct_commits desc → theme alpha.""" + demand = demands[0] + assert demand["rank"] == 1 + assert set(demand.keys()) >= { + "domain", + "is_uncategorized", + "final_score", + "base_score", + "diversity_bonus", + "invariant_bonus", + "recency_weight", + "distinct_commits", + "importance_weight", + "op_distribution", + "representative_summaries", + } + + def test_invariant_bonus_uses_same_sha_association_and_caps_at_five(self, tmp_path): mod = load_commit_semantic_module() semantic_dir = tmp_path / "data" / "commit-semantic" - semantic_dir.mkdir(parents=True) + aggregated = [ + { + "domain": "auth", + "is_uncategorized": False, + "count": 2, + "distinct_commits": 2, + "op_distribution": {"feat": 1, "bugfix": 1}, + "importance_ratio": {"primary": 1, "secondary": 1}, + "date_range": {"from": "2026-03-01T00:00:00", "to": "2026-03-02T00:00:00"}, + "sub_themes": {"login": 2}, + "representative_summaries": ["Add login", "Fix login"], + } + ] + units = [ + {"sha": "sha-1", "date": "2026-03-01T00:00:00", "domain": "auth"}, + {"sha": "sha-2", "date": "2026-03-02T00:00:00", "domain": "auth"}, + ] + invariants = [ + {"sha": "sha-1", "statement": "inv-1"}, + {"sha": "sha-1", "statement": "inv-2"}, + {"sha": "sha-1", "statement": "inv-3"}, + {"sha": "sha-2", "statement": "inv-4"}, + {"sha": "sha-2", "statement": "inv-5"}, + {"sha": "sha-2", "statement": "inv-6"}, + {"sha": "other-sha", "statement": "ignored"}, + {"sha": "sha-2", "statement": "inv-6"}, + ] + _write_semantic_inputs(semantic_dir, units=units, invariants=invariants, aggregated=aggregated) + + mod.SEMANTIC_OUTPUT = semantic_dir + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + runner._run_distill(state) - patterns = [ - {"theme": "zebra", "count": 6, "distinct_commits": 3, - "op_distribution": {"feat": 6}, "importance_ratio": {"primary": 3, "secondary": 3}, - "representative_summaries": []}, - {"theme": "alpha", "count": 6, "distinct_commits": 3, - "op_distribution": {"feat": 6}, "importance_ratio": {"primary": 3, "secondary": 3}, - "representative_summaries": []}, + demands = load_jsonl(str(semantic_dir / "canonical-demands.jsonl")) + assert len(demands) == 1 + assert demands[0]["domain"] == "auth" + assert demands[0]["invariant_bonus"] == 5 + + def test_ranking_uses_score_then_distinct_commits_then_domain(self, tmp_path): + """Same score -> distinct_commits desc -> domain alpha.""" + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + aggregated = [ + { + "domain": "zebra", + "is_uncategorized": False, + "count": 6, + "distinct_commits": 3, + "op_distribution": {"feat": 6}, + "importance_ratio": {"primary": 3, "secondary": 3}, + "date_range": {}, + "sub_themes": {}, + "representative_summaries": [], + }, + { + "domain": "alpha", + "is_uncategorized": False, + "count": 6, + "distinct_commits": 3, + "op_distribution": {"feat": 6}, + "importance_ratio": {"primary": 3, "secondary": 3}, + "date_range": {}, + "sub_themes": {}, + "representative_summaries": [], + }, + { + "domain": "bravo", + "is_uncategorized": False, + "count": 6, + "distinct_commits": 4, + "op_distribution": {"feat": 6}, + "importance_ratio": {"primary": 2, "secondary": 4}, + "date_range": {}, + "sub_themes": {}, + "representative_summaries": [], + }, ] - save_jsonl(patterns, str(semantic_dir / "patterns.jsonl")) - save_jsonl([], str(semantic_dir / "invariants.jsonl")) + _write_semantic_inputs(semantic_dir, units=[], invariants=[], aggregated=aggregated) + + mod.SEMANTIC_OUTPUT = semantic_dir + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + runner._run_distill(state) + + demands = load_jsonl(str(semantic_dir / "canonical-demands.jsonl")) + assert [d["domain"] for d in demands] == ["bravo", "alpha", "zebra"] + assert [d["rank"] for d in demands] == [1, 2, 3] + + def test_guardrail_top_three_vs_bottom_three_gap(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + aggregated = [ + { + "domain": "auth", + "is_uncategorized": False, + "count": 10, + "distinct_commits": 5, + "op_distribution": {"feat": 4, "bugfix": 3, "refactor": 3}, + "importance_ratio": {"primary": 8, "secondary": 2}, + "date_range": {}, + "sub_themes": {"login": 6, "session": 4}, + "representative_summaries": [], + }, + { + "domain": "billing", + "is_uncategorized": False, + "count": 9, + "distinct_commits": 4, + "op_distribution": {"feat": 3, "bugfix": 3, "config": 3}, + "importance_ratio": {"primary": 6, "secondary": 3}, + "date_range": {}, + "sub_themes": {"invoice": 5, "refund": 4}, + "representative_summaries": [], + }, + { + "domain": "search", + "is_uncategorized": False, + "count": 8, + "distinct_commits": 4, + "op_distribution": {"feat": 4, "refactor": 4}, + "importance_ratio": {"primary": 5, "secondary": 3}, + "date_range": {}, + "sub_themes": {"query": 8}, + "representative_summaries": [], + }, + { + "domain": "docs", + "is_uncategorized": False, + "count": 6, + "distinct_commits": 2, + "op_distribution": {"docs": 6}, + "importance_ratio": {"primary": 0, "secondary": 6}, + "date_range": {}, + "sub_themes": {"guides": 6}, + "representative_summaries": [], + }, + { + "domain": "tooling", + "is_uncategorized": False, + "count": 5, + "distinct_commits": 1, + "op_distribution": {"chore": 5}, + "importance_ratio": {"primary": 0, "secondary": 5}, + "date_range": {}, + "sub_themes": {"ci": 5}, + "representative_summaries": [], + }, + { + "domain": "misc", + "is_uncategorized": True, + "count": 4, + "distinct_commits": 1, + "op_distribution": {"chore": 4}, + "importance_ratio": {"primary": 0, "secondary": 4}, + "date_range": {}, + "sub_themes": {"misc": 4}, + "representative_summaries": [], + }, + ] + _write_semantic_inputs(semantic_dir, units=[], invariants=[], aggregated=aggregated) mod.SEMANTIC_OUTPUT = semantic_dir from src.harness_state import HarnessState @@ -274,8 +545,9 @@ def test_tiebreak_order(self, tmp_path): runner._run_distill(state) demands = load_jsonl(str(semantic_dir / "canonical-demands.jsonl")) - assert demands[0]["theme"] == "alpha" - assert demands[1]["theme"] == "zebra" + top_three = [d["final_score"] for d in demands[:3]] + bottom_three = [d["final_score"] for d in demands[-3:]] + assert min(top_three) >= max(bottom_three) * 2 class TestCommitSemanticExport: @@ -286,11 +558,13 @@ def test_summary_json(self, tmp_path, sample_jsonl_records): summary = load_json(str(semantic_dir / "summary.json")) assert summary["total_units"] == 6 - assert summary["total_patterns"] >= 1 + assert summary["domain_count"] >= 1 assert 0 <= summary["bugfix_ratio"] <= 1 assert summary["invariant_count"] == 2 assert summary["date_range"]["from"] == "2026-03-01T10:00:00" assert summary["date_range"]["to"] == "2026-03-10T14:00:00" + assert "uncategorized_ratio" in summary + assert "file_paths_available" in summary def test_op_distribution_in_summary(self, tmp_path, sample_jsonl_records): _, semantic_dir = _setup_and_run(tmp_path, sample_jsonl_records) @@ -298,19 +572,929 @@ def test_op_distribution_in_summary(self, tmp_path, sample_jsonl_records): assert "feat" in summary["op_distribution"] assert summary["op_distribution"]["feat"] == 3 + def test_top_domains_in_summary(self, tmp_path, sample_jsonl_records): + _, semantic_dir = _setup_and_run(tmp_path, sample_jsonl_records) + summary = load_json(str(semantic_dir / "summary.json")) + assert "top_domains" in summary + assert len(summary["top_domains"]) >= 1 + + +class TestCommitSemanticDiscoverNormalization: + """Discover stage normalization behavior.""" + + def test_complete_discover_normalizes_combined_domains_json(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + semantic_dir.mkdir(parents=True) + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState( + stage="discover", + metadata={ + "discover_fingerprint": {"units_hash": "abc123", "arch_hash": None}, + "completed_stages": [], + "artifacts_written": [], + "status": "ok", + }, + ) + + llm_response = json.dumps([ + { + "domain": "test", + "description": "Single test utilities", + "paths": ["tests/unit/", "tests/e2e/"], + "keywords": ["pytest", "fixture"], + }, + { + "domain": "tests", + "description": "Test infrastructure and end-to-end flows", + "paths": ["tests/e2e/", "tests/integration/"], + "keywords": ["pytest", "integration", "fixture"], + }, + { + "domain": "quality", + "description": "Quality checks and validation", + "paths": ["tests/e2e/", "tests/integration/"], + "keywords": ["pytest", "integration", "validation"], + }, + { + "domain": "misc", + "description": "Catch-all bucket", + "paths": [], + "keywords": ["misc"], + }, + ]) + + assert runner.complete_discover(llm_response, state) is True + + domains = load_json(str(semantic_dir / "domains.json")) + assert domains["_fingerprint"] == {"units_hash": "abc123", "arch_hash": None} + assert domains["discover_mode"] == "llm" + assert domains["orchestration_mode_at_discover"] == "llm_preferred" + assert domains["domains"] == [ + { + "domain": "tests", + "description": "Test infrastructure and end-to-end flows", + "paths": ["tests/e2e/", "tests/integration/", "tests/unit/"], + "keywords": ["fixture", "integration", "pytest", "validation"], + } + ] + + def test_discover_cache_hit_restores_provenance(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + units_dir = semantic_dir / "units" + units_dir.mkdir(parents=True) + mod.SEMANTIC_OUTPUT = semantic_dir + + units = [ + { + "sha": "a1", + "date": "2026-03-01T10:00:00", + "section_name": "Auth", + "theme": "login", + "summary": "Add login flow", + "op": "feat", + } + ] + save_jsonl(units, str(units_dir / "all.jsonl")) + fingerprint = mod.compute_fingerprint(units_dir / "all.jsonl", None) + save_json( + { + "_fingerprint": fingerprint, + "discover_mode": "llm", + "orchestration_mode_at_discover": "llm_preferred", + "domains": [ + { + "domain": "auth", + "description": "Authentication", + "paths": ["src/auth/"], + "keywords": ["auth", "login"], + } + ], + }, + str(semantic_dir / "domains.json"), + ) + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + runner._find_arch_file = lambda: None + state = HarnessState(stage="discover", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + + assert runner._run_discover(state) is True + assert state.metadata["discover_mode"] == "cached_llm" + assert state.metadata["orchestration_mode_at_discover"] == "llm_preferred" + + def test_complete_discover_invalid_or_empty_output_fails_without_fallback(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + units_dir = semantic_dir / "units" + units_dir.mkdir(parents=True) + mod.SEMANTIC_OUTPUT = semantic_dir + + units = [ + { + "sha": "a1", + "date": "2026-03-01T10:00:00", + "section_name": "Auth", + "theme": "login", + "summary": "Add login flow", + "op": "feat", + } + ] + save_jsonl(units, str(units_dir / "all.jsonl")) + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState( + stage="discover", + metadata={ + "completed_stages": [], + "artifacts_written": [], + "status": "ok", + "discover_fingerprint": mod.compute_fingerprint(units_dir / "all.jsonl", None), + "external_orchestration": True, + }, + ) + + assert runner.complete_discover("not json", state) is False + assert runner.complete_discover("[]", state) is False + assert not (semantic_dir / "domains.json").exists() + assert "discover_mode" not in state.metadata + + def test_run_discover_executes_host_executor_and_persists_domains(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + units_dir = semantic_dir / "units" + units_dir.mkdir(parents=True) + mod.SEMANTIC_OUTPUT = semantic_dir + + save_jsonl( + [ + { + "sha": "a1", + "date": "2026-03-01T10:00:00", + "section_name": "Auth rollout", + "theme": "auth-session", + "summary": "Add login session refresh token flow", + "op": "feat", + } + ], + str(units_dir / "all.jsonl"), + ) + + from src.harness_state import HarnessState + executor = FakeHostExecutor() + runner = mod.CommitSemanticRunner(executor=executor) + runner._find_arch_file = lambda: None + state = HarnessState(stage="discover", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + + assert runner._run_discover(state) is True + domains_payload = load_json(str(semantic_dir / "domains.json")) + + assert len(executor.calls) == 1 + assert executor.calls[0]["artifact_name"] == "domains" + assert "units_json" not in executor.calls[0]["context"] + assert domains_payload["discover_mode"] == "llm" + assert [domain["domain"] for domain in domains_payload["domains"]] == ["auth", "demand"] + assert state.metadata["discover_mode"] == "llm" + + def test_run_ingest_executes_host_executor_for_classify_batches(self, tmp_path): + mod = load_commit_semantic_module() + extract_dir = tmp_path / "data" / "commit-extract" + semantic_dir = tmp_path / "data" / "commit-semantic" + extract_dir.mkdir(parents=True) + semantic_dir.mkdir(parents=True) + + save_jsonl( + [ + { + "sha": "a" * 40, + "author": "yan.", + "date": "2026-03-01T10:00:00", + "is_large_aggregate": False, + "is_mixed": True, + "sections": [ + { + "name": "Auth rollout", + "theme": "auth-session", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Add login session refresh token flow"}, + {"op": "feat", "summary": "Map issue inputs into demand cards"}, + ], + } + ], + "rules_invariants": [], + } + ], + str(extract_dir / "2026-03.jsonl"), + ) + save_json( + { + "discover_mode": "llm", + "orchestration_mode_at_discover": "llm_preferred", + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "demand", + "description": "Issue and demand mapping", + "paths": ["src/demand/"], + "keywords": ["demand", "issue", "requirement", "card"], + }, + ], + }, + str(semantic_dir / "domains.json"), + ) + + mod.EXTRACT_OUTPUT = extract_dir + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + executor = FakeHostExecutor() + runner = mod.CommitSemanticRunner(executor=executor) + state = HarnessState( + stage="ingest", + metadata={ + "completed_stages": [], + "artifacts_written": [], + "status": "ok", + "external_orchestration": True, + }, + ) + + assert runner._run_ingest(state) is True + units = load_jsonl(str(semantic_dir / "units" / "all.jsonl")) + + classify_calls = [call for call in executor.calls if call["artifact_name"] == "classify-units"] + assert len(classify_calls) == 1 + assert len(json.loads(classify_calls[0]["context"]["units_json"])) == 2 + assert [unit["domain"] for unit in units] == ["auth", "demand"] + assert state.metadata["classify_mode"] == "llm" + assert state.metadata["needs_llm_classify"] == 2 + class TestCommitSemanticFullPipeline: - """Full pipeline integration.""" + """Full pipeline integration (without discover).""" def test_all_stages_produce_output(self, tmp_path, sample_jsonl_records): _, semantic_dir = _setup_and_run(tmp_path, sample_jsonl_records) assert (semantic_dir / "units" / "all.jsonl").exists() assert (semantic_dir / "invariants.jsonl").exists() - assert (semantic_dir / "patterns.jsonl").exists() + assert (semantic_dir / "domains-aggregated.jsonl").exists() assert (semantic_dir / "canonical-demands.jsonl").exists() assert (semantic_dir / "summary.json").exists() + def test_local_run_without_orchestration_fails_cleanly_instead_of_succeeding_via_fallback(self, tmp_path): + extract_dir = tmp_path / "data" / "commit-extract" + extract_dir.mkdir(parents=True) + + records = [ + { + "sha": "a" * 40, + "author": "yan.", + "date": "2026-03-01T10:00:00", + "is_large_aggregate": False, + "is_mixed": False, + "sections": [ + { + "name": "Commit semantic pipeline", + "theme": "domain-classification", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Add local domain classification fallback"} + ], + } + ], + "rules_invariants": [], + } + ] + save_jsonl(records, str(extract_dir / "2026-03.jsonl")) + + result = subprocess.run( + [sys.executable, str(Path(__file__).parent.parent.parent / "skills/commit-semantic/run.py"), "run"], + capture_output=True, + text=True, + cwd=tmp_path, + ) + assert result.returncode == 1 + assert not (tmp_path / "data" / "commit-semantic" / "summary.json").exists() + assert "discover orchestration unavailable" in (result.stderr + result.stdout).lower() + + def test_ingest_needing_classify_with_no_orchestration_fails(self, tmp_path): + mod = load_commit_semantic_module() + extract_dir = tmp_path / "data" / "commit-extract" + semantic_dir = tmp_path / "data" / "commit-semantic" + extract_dir.mkdir(parents=True) + semantic_dir.mkdir(parents=True) + + save_jsonl( + [ + { + "sha": "c" * 40, + "author": "yan.", + "date": "2026-03-03T10:00:00", + "is_large_aggregate": False, + "is_mixed": True, + "sections": [ + { + "name": "Auth rollout", + "theme": "auth-session", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Add login session refresh token flow"} + ], + } + ], + "rules_invariants": [], + } + ], + str(extract_dir / "2026-03.jsonl"), + ) + save_json( + { + "discover_mode": "cached_llm", + "orchestration_mode_at_discover": "llm_preferred", + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": [], + "keywords": ["auth", "login", "session", "token"], + } + ] + }, + str(semantic_dir / "domains.json"), + ) + + mod.EXTRACT_OUTPUT = extract_dir + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + + ok = runner._run_ingest(state) + assert ok is False + assert not (semantic_dir / "units" / "all.jsonl").exists() + assert state.metadata["needs_llm_classify"] == 1 + + + def test_summary_includes_truthful_mode_fields(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + semantic_dir.mkdir(parents=True) + (semantic_dir / "units").mkdir(parents=True) + mod.SEMANTIC_OUTPUT = semantic_dir + + save_jsonl( + [ + { + "sha": "a", + "date": "2026-03-01T10:00:00", + "op": "feat", + "summary": "add login", + "domain": "auth", + } + ], + str(semantic_dir / "units" / "all.jsonl"), + ) + save_jsonl([], str(semantic_dir / "domains-aggregated.jsonl")) + save_jsonl([], str(semantic_dir / "canonical-demands.jsonl")) + save_jsonl([], str(semantic_dir / "invariants.jsonl")) + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState( + metadata={ + "completed_stages": [], + "artifacts_written": [], + "status": "ok", + "orchestration_mode": "mixed_degraded", + "discover_mode": "cached_llm", + "classify_mode": "fallback", + "file_paths_available": False, + } + ) + + assert runner._run_export(state) is True + summary = load_json(str(semantic_dir / "summary.json")) + assert summary["orchestration_mode"] == "mixed_degraded" + assert summary["discover_mode"] == "cached_llm" + assert summary["classify_mode"] == "fallback" + assert summary["file_paths_available"] is False + + def test_complete_classify_partial_failure_fails_overall_instead_of_degrading(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + (semantic_dir / "units").mkdir(parents=True) + mod.SEMANTIC_OUTPUT = semantic_dir + + save_jsonl( + [ + { + "sha": "a", + "date": "2026-03-01T10:00:00", + "section_name": "Auth rollout", + "theme": "auth-session", + "summary": "Add login session refresh token flow", + }, + { + "sha": "b", + "date": "2026-03-02T10:00:00", + "section_name": "Demand cards", + "theme": "issue-mapping", + "summary": "Map issue inputs into demand cards", + }, + ], + str(semantic_dir / "units" / "all.jsonl"), + ) + save_json( + { + "discover_mode": "llm", + "orchestration_mode_at_discover": "llm_preferred", + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": [], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "demand", + "description": "Issue and demand mapping", + "paths": [], + "keywords": ["issue", "demand", "cards"], + }, + ], + }, + str(semantic_dir / "domains.json"), + ) + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState( + metadata={ + "completed_stages": [], + "artifacts_written": [], + "status": "ok", + "external_orchestration": True, + "discover_mode": "cached_llm", + "classify_unit_indices": [0, 1], + "classify_units": [ + {"id": "0", "section_name": "Auth rollout", "theme": "auth-session", "summary": "Add login session refresh token flow", "op": "feat"}, + {"id": "1", "section_name": "Demand cards", "theme": "issue-mapping", "summary": "Map issue inputs into demand cards", "op": "feat"}, + ], + } + ) + + assert runner.complete_classify([json.dumps({"0": "auth"}), "not json"], state) is False + units = load_jsonl(str(semantic_dir / "units" / "all.jsonl")) + assert all("domain" not in unit for unit in units) + assert "classify_mode" not in state.metadata + + def test_classify_total_failure_fails_instead_of_leaving_uncategorized(self, tmp_path): + mod = load_commit_semantic_module() + semantic_dir = tmp_path / "data" / "commit-semantic" + (semantic_dir / "units").mkdir(parents=True) + mod.SEMANTIC_OUTPUT = semantic_dir + + save_jsonl( + [ + { + "sha": "a", + "date": "2026-03-01T10:00:00", + "section_name": "Infra cleanup", + "theme": "maintenance", + "summary": "Refactor helper wiring", + } + ], + str(semantic_dir / "units" / "all.jsonl"), + ) + save_json( + { + "discover_mode": "llm", + "orchestration_mode_at_discover": "llm_preferred", + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": [], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "demand", + "description": "Issue and demand mapping", + "paths": [], + "keywords": ["issue", "demand", "requirement"], + }, + ], + }, + str(semantic_dir / "domains.json"), + ) + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState( + metadata={ + "completed_stages": [], + "artifacts_written": [], + "status": "ok", + "external_orchestration": True, + "discover_mode": "cached_llm", + "classify_unit_indices": [0], + "classify_units": [ + {"id": "0", "section_name": "Infra cleanup", "theme": "maintenance", "summary": "Refactor helper wiring", "op": "refactor"}, + ], + } + ) + + assert runner.complete_classify(["not json"], state) is False + units = load_jsonl(str(semantic_dir / "units" / "all.jsonl")) + assert all("domain" not in unit for unit in units) + assert "classify_mode" not in state.metadata + + def test_default_run_prefers_llm_semantics_when_orchestration_available(self, tmp_path): + mod = load_commit_semantic_module() + extract_dir = tmp_path / "data" / "commit-extract" + semantic_dir = tmp_path / "data" / "commit-semantic" + extract_dir.mkdir(parents=True) + semantic_dir.mkdir(parents=True) + + save_jsonl( + [ + { + "sha": "h" * 40, + "author": "yan.", + "date": "2026-03-03T10:00:00", + "is_large_aggregate": False, + "is_mixed": True, + "sections": [ + { + "name": "Auth rollout", + "theme": "auth-session", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Add login session refresh token flow"} + ], + } + ], + "rules_invariants": [], + } + ], + str(extract_dir / "2026-03.jsonl"), + ) + save_json( + { + "discover_mode": "llm", + "orchestration_mode_at_discover": "llm_preferred", + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": [], + "keywords": ["auth", "login", "session", "token"], + } + ], + }, + str(semantic_dir / "domains.json"), + ) + + mod.EXTRACT_OUTPUT = extract_dir + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState( + stage="init", + metadata={ + "completed_stages": [], + "artifacts_written": [], + "status": "ok", + "external_orchestration": True, + }, + ) + + assert runner._run_ingest(state) is True + units = load_jsonl(str(semantic_dir / "units" / "all.jsonl")) + assert units[0]["domain"] == "uncategorized" + assert state.metadata["discover_mode"] == "cached_llm" + assert state.metadata["classify_mode"] == "llm" + assert state.metadata["needs_llm_classify"] == 1 + assert state.metadata["orchestration_mode"] == "llm_preferred" + + + def test_local_run_without_orchestration_does_not_emit_degraded_summary(self, tmp_path): + extract_dir = tmp_path / "data" / "commit-extract" + extract_dir.mkdir(parents=True) + + records = [ + { + "sha": "a" * 40, + "author": "yan.", + "date": "2026-03-01T10:00:00", + "is_large_aggregate": False, + "is_mixed": False, + "sections": [ + { + "name": "Commit semantic pipeline", + "theme": "domain-classification", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Add local domain classification fallback"} + ], + } + ], + "rules_invariants": [], + } + ] + save_jsonl(records, str(extract_dir / "2026-03.jsonl")) + + result = subprocess.run( + [sys.executable, str(Path(__file__).parent.parent.parent / "skills/commit-semantic/run.py"), "run"], + capture_output=True, + text=True, + cwd=tmp_path, + ) + assert result.returncode == 1, result.stderr + "\n" + result.stdout + + semantic_dir = tmp_path / "data" / "commit-semantic" + assert not (semantic_dir / "domains.json").exists() + assert not (semantic_dir / "summary.json").exists() + + + def test_local_classify_keeps_single_domain_path_fast_path(self, tmp_path): + mod = load_commit_semantic_module() + extract_dir = tmp_path / "data" / "commit-extract" + semantic_dir = tmp_path / "data" / "commit-semantic" + extract_dir.mkdir(parents=True) + semantic_dir.mkdir(parents=True) + + save_jsonl( + [ + { + "sha": "d" * 40, + "author": "yan.", + "date": "2026-03-03T10:00:00", + "is_large_aggregate": False, + "is_mixed": False, + "sections": [ + { + "name": "Unrelated section", + "theme": "maintenance", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Refactor helper wiring"} + ], + } + ], + "rules_invariants": [], + } + ], + str(extract_dir / "2026-03.jsonl"), + ) + save_json( + { + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "session", "token"], + } + ] + }, + str(semantic_dir / "domains.json"), + ) + + mod.EXTRACT_OUTPUT = extract_dir + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + + original_build_sha_file_map = mod.build_sha_file_map + mod.build_sha_file_map = lambda repo_path, shas: ({"d" * 40: ["src/auth/login.py"]}, True) + try: + ok = runner._run_ingest(state) + finally: + mod.build_sha_file_map = original_build_sha_file_map + + assert ok is True + units = load_jsonl(str(semantic_dir / "units" / "all.jsonl")) + assert units[0]["domain"] == "auth" + assert state.metadata["needs_llm_classify"] == 0 + + def test_multi_domain_commit_requires_orchestration_instead_of_unit_scoring(self, tmp_path): + mod = load_commit_semantic_module() + extract_dir = tmp_path / "data" / "commit-extract" + semantic_dir = tmp_path / "data" / "commit-semantic" + extract_dir.mkdir(parents=True) + semantic_dir.mkdir(parents=True) + + save_jsonl( + [ + { + "sha": "e" * 40, + "author": "yan.", + "date": "2026-03-03T10:00:00", + "is_large_aggregate": False, + "is_mixed": False, + "sections": [ + { + "name": "Auth rollout", + "theme": "auth-session", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Add login session refresh token flow"} + ], + } + ], + "rules_invariants": [], + } + ], + str(extract_dir / "2026-03.jsonl"), + ) + save_json( + { + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "demand", + "description": "Issue and demand mapping", + "paths": ["src/demand/"], + "keywords": ["demand", "issue", "requirement"], + }, + ] + }, + str(semantic_dir / "domains.json"), + ) + + mod.EXTRACT_OUTPUT = extract_dir + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + + original_build_sha_file_map = mod.build_sha_file_map + mod.build_sha_file_map = lambda repo_path, shas: ({"e" * 40: ["src/auth/login.py", "src/demand/card.py"]}, True) + try: + ok = runner._run_ingest(state) + finally: + mod.build_sha_file_map = original_build_sha_file_map + + assert ok is False + assert state.metadata["needs_llm_classify"] == 1 + assert not (semantic_dir / "units" / "all.jsonl").exists() + + def test_after_multi_domain_failure_requires_orchestration_instead_of_degrading(self, tmp_path): + mod = load_commit_semantic_module() + extract_dir = tmp_path / "data" / "commit-extract" + semantic_dir = tmp_path / "data" / "commit-semantic" + extract_dir.mkdir(parents=True) + semantic_dir.mkdir(parents=True) + + save_jsonl( + [ + { + "sha": "f" * 40, + "author": "yan.", + "date": "2026-03-03T10:00:00", + "is_large_aggregate": False, + "is_mixed": False, + "sections": [ + { + "name": "Infra cleanup", + "theme": "maintenance", + "importance": "primary", + "items": [ + {"op": "refactor", "summary": "Refactor helper wiring"} + ], + } + ], + "rules_invariants": [], + } + ], + str(extract_dir / "2026-03.jsonl"), + ) + save_json( + { + "discover_mode": "cached_llm", + "orchestration_mode_at_discover": "llm_preferred", + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "demand", + "description": "Issue and demand mapping", + "paths": ["src/demand/"], + "keywords": ["demand", "issue", "requirement"], + }, + ] + }, + str(semantic_dir / "domains.json"), + ) + + mod.EXTRACT_OUTPUT = extract_dir + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + + original_build_sha_file_map = mod.build_sha_file_map + mod.build_sha_file_map = lambda repo_path, shas: ({"f" * 40: ["src/auth/login.py", "src/demand/card.py"]}, True) + try: + ok = runner._run_ingest(state) + finally: + mod.build_sha_file_map = original_build_sha_file_map + + assert ok is False + assert state.metadata["needs_llm_classify"] == 1 + assert not (semantic_dir / "units" / "all.jsonl").exists() + + def test_ambiguous_non_path_signals_require_orchestration(self, tmp_path): + mod = load_commit_semantic_module() + extract_dir = tmp_path / "data" / "commit-extract" + semantic_dir = tmp_path / "data" / "commit-semantic" + extract_dir.mkdir(parents=True) + semantic_dir.mkdir(parents=True) + + save_jsonl( + [ + { + "sha": "g" * 40, + "author": "yan.", + "date": "2026-03-03T10:00:00", + "is_large_aggregate": True, + "is_mixed": True, + "sections": [ + { + "name": "Update", + "theme": "auth issue", + "importance": "primary", + "items": [ + {"op": "feat", "summary": "Refine handling"} + ], + } + ], + "rules_invariants": [], + } + ], + str(extract_dir / "2026-03.jsonl"), + ) + save_json( + { + "domains": [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "issue-triage", + "description": "Issue processing", + "paths": [], + "keywords": ["issue", "routing", "auth"], + }, + ] + }, + str(semantic_dir / "domains.json"), + ) + + mod.EXTRACT_OUTPUT = extract_dir + mod.SEMANTIC_OUTPUT = semantic_dir + + from src.harness_state import HarnessState + runner = mod.CommitSemanticRunner() + state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) + + ok = runner._run_ingest(state) + + assert ok is False + assert state.metadata["needs_llm_classify"] == 1 + assert not (semantic_dir / "units" / "all.jsonl").exists() + if __name__ == "__main__": pytest.main([__file__, "-v"]) diff --git a/tests/e2e/test_pipeline_e2e.py b/tests/e2e/test_pipeline_e2e.py index b855b75..30f6ae9 100644 --- a/tests/e2e/test_pipeline_e2e.py +++ b/tests/e2e/test_pipeline_e2e.py @@ -19,7 +19,7 @@ class TestFullPipelineE2E: """Test full pipeline with temp git repo.""" def test_commit_extract_produces_manifest(self, temp_git_repo: Path, tmp_path: Path): - """commit-extract produces batch manifest for workers.""" + """commit-extract run completes locally and writes monthly JSONL output.""" result = subprocess.run( [sys.executable, str(repo_root / "skills/commit-extract/run.py"), "run", "--repo", str(temp_git_repo)], @@ -29,12 +29,19 @@ def test_commit_extract_produces_manifest(self, temp_git_repo: Path, tmp_path: P extract_dir = tmp_path / "data" / "commit-extract" manifest = extract_dir / "tmp" / "manifest.json" + monthly_files = sorted(extract_dir.glob("????-??.jsonl")) + assert manifest.exists(), f"manifest.json not found at {manifest}" + assert monthly_files, f"monthly JSONL not found in {extract_dir}" data = json.loads(manifest.read_text()) assert data["total_shas"] >= 1 assert len(data["batches"]) >= 1 + records = [json.loads(line) for line in monthly_files[0].read_text().splitlines() if line.strip()] + assert records, "monthly JSONL should contain at least one record" + assert {"sha", "author", "date", "sections", "rules_invariants"}.issubset(records[0]) + def test_commit_semantic_ingest_stage(self, temp_git_repo: Path, tmp_path: Path): """commit-semantic ingest reads JSONL and produces units.""" # Create fixture JSONL (simulating worker output) @@ -91,6 +98,6 @@ def test_pipeline_produces_correct_output_structure(self, temp_git_repo: Path, t semantic_dir = tmp_path / "data" / "commit-semantic" assert (semantic_dir / "units" / "all.jsonl").exists() assert (semantic_dir / "invariants.jsonl").exists() - assert (semantic_dir / "patterns.jsonl").exists() + assert (semantic_dir / "domains-aggregated.jsonl").exists() assert (semantic_dir / "canonical-demands.jsonl").exists() assert (semantic_dir / "summary.json").exists() diff --git a/tests/test_commit_extract_rewrite.py b/tests/test_commit_extract_rewrite.py index 1096741..0f4b0c3 100644 --- a/tests/test_commit_extract_rewrite.py +++ b/tests/test_commit_extract_rewrite.py @@ -352,20 +352,12 @@ def test_theme_grouping_and_threshold(self, tmp_path_clean, sample_commit_record runner._run_ingest(state) runner._run_aggregate(state) - patterns = load_jsonl(str(semantic_dir / "patterns.jsonl")) + aggregated = load_jsonl(str(semantic_dir / "domains-aggregated.jsonl")) - # auth-flow: 4 distinct commits → pattern - auth = [p for p in patterns if p["theme"] == "auth-flow"] - assert len(auth) == 1 - assert auth[0]["distinct_commits"] == 4 - - # config-mgmt: 1 commit → NOT a pattern - config = [p for p in patterns if p["theme"] == "config-mgmt"] - assert len(config) == 0 - - # retry-logic: 1 commit → NOT a pattern - retry = [p for p in patterns if p["theme"] == "retry-logic"] - assert len(retry) == 0 + # Without domains.json, all units are uncategorized → 1 domain + assert len(aggregated) == 1 + assert aggregated[0]["domain"] == "uncategorized" + assert aggregated[0]["count"] == 6 # 2+1+2+1 items def test_op_distribution(self, tmp_path_clean, sample_commit_records, monkeypatch): mod = _load_semantic_module() @@ -385,10 +377,10 @@ def test_op_distribution(self, tmp_path_clean, sample_commit_records, monkeypatc runner._run_ingest(state) runner._run_aggregate(state) - patterns = load_jsonl(str(semantic_dir / "patterns.jsonl")) - auth = [p for p in patterns if p["theme"] == "auth-flow"][0] - assert auth["op_distribution"]["feat"] == 3 - assert auth["op_distribution"]["refactor"] == 1 + aggregated = load_jsonl(str(semantic_dir / "domains-aggregated.jsonl")) + uncat = aggregated[0] + assert uncat["op_distribution"]["feat"] == 3 + assert uncat["op_distribution"]["refactor"] == 1 # --------------------------------------------------------------------------- @@ -418,27 +410,29 @@ def test_scoring_and_ranking(self, tmp_path_clean, sample_commit_records, monkey demands = load_jsonl(str(semantic_dir / "canonical-demands.jsonl")) assert len(demands) >= 1 assert demands[0]["rank"] == 1 - assert demands[0]["theme"] == "auth-flow" - # score = 4 distinct * importance_weight (3 primary + 1 secondary → (3*2+1*1)/4 = 1.75) - assert demands[0]["score"] == 7.0 + assert demands[0]["domain"] == "uncategorized" + assert "final_score" in demands[0] + assert "base_score" in demands[0] def test_tiebreak(self, tmp_path_clean, monkeypatch): - """When scores are equal, sort by distinct_commits desc then theme alpha.""" + """When scores are equal, sort by distinct_commits desc then domain alpha.""" mod = _load_semantic_module() semantic_dir = tmp_path_clean / "data" / "commit-semantic" - semantic_dir.mkdir(parents=True) + units_dir = semantic_dir / "units" + units_dir.mkdir(parents=True) - # Two patterns with same score - patterns = [ - {"theme": "zebra", "count": 6, "distinct_commits": 3, + # Two domains with same stats + aggregated = [ + {"domain": "zebra", "is_uncategorized": False, "count": 6, "distinct_commits": 3, "op_distribution": {"feat": 6}, "importance_ratio": {"primary": 3, "secondary": 3}, - "representative_summaries": []}, - {"theme": "alpha", "count": 6, "distinct_commits": 3, + "date_range": {}, "sub_themes": {}, "representative_summaries": []}, + {"domain": "alpha", "is_uncategorized": False, "count": 6, "distinct_commits": 3, "op_distribution": {"feat": 6}, "importance_ratio": {"primary": 3, "secondary": 3}, - "representative_summaries": []}, + "date_range": {}, "sub_themes": {}, "representative_summaries": []}, ] - save_jsonl(patterns, str(semantic_dir / "patterns.jsonl")) + save_jsonl(aggregated, str(semantic_dir / "domains-aggregated.jsonl")) save_jsonl([], str(semantic_dir / "invariants.jsonl")) + save_jsonl([], str(units_dir / "all.jsonl")) monkeypatch.setattr(mod, "SEMANTIC_OUTPUT", semantic_dir) @@ -451,8 +445,8 @@ def test_tiebreak(self, tmp_path_clean, monkeypatch): demands = load_jsonl(str(semantic_dir / "canonical-demands.jsonl")) assert len(demands) == 2 # Same score, same distinct_commits → alpha order - assert demands[0]["theme"] == "alpha" - assert demands[1]["theme"] == "zebra" + assert demands[0]["domain"] == "alpha" + assert demands[1]["domain"] == "zebra" # --------------------------------------------------------------------------- @@ -475,14 +469,14 @@ def test_summary_output(self, tmp_path_clean, sample_commit_records, monkeypatch from src.harness_state import HarnessState state = HarnessState(stage="init", metadata={"completed_stages": [], "artifacts_written": [], "status": "ok"}) - for stage in runner.STAGES: + for stage in ["ingest", "aggregate", "distill", "export"]: runner.run_stage(stage, state) from src.io_utils import load_json summary = load_json(str(semantic_dir / "summary.json")) assert summary["total_units"] == 6 - assert summary["total_patterns"] >= 1 + assert summary["domain_count"] >= 1 assert 0 <= summary["bugfix_ratio"] <= 1 assert "from" in summary["date_range"] assert "to" in summary["date_range"] diff --git a/tests/test_commit_semantic_domain.py b/tests/test_commit_semantic_domain.py new file mode 100644 index 0000000..21708dd --- /dev/null +++ b/tests/test_commit_semantic_domain.py @@ -0,0 +1,548 @@ +"""Tests for commit-semantic domain utilities and pipeline stages.""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path + +import pytest + +# Ensure src is importable +import sys +sys.path.insert(0, str(Path(__file__).parent.parent)) + +from src.commit_semantic.domain_utils import ( + assign_domain_by_path, + build_sha_file_map, + build_units_summary, + choose_domain_winner, + classify_unit_locally, + compute_fingerprint, + fingerprint_matches, + normalize_domains, + parse_llm_classifications, + parse_llm_domains, + score_unit_for_domain, + should_merge_domains, +) + + +# --------------------------------------------------------------- +# T7-T11: build_sha_file_map +# --------------------------------------------------------------- + +class TestBuildShaFileMap: + """Tests for build_sha_file_map.""" + + def test_normal_commit(self, tmp_path): + """T7: Normal commit returns correct file list.""" + # Use the actual repo + repo = str(Path(__file__).parent.parent) + result = subprocess.run( + ["git", "log", "--format=%H", "-1"], + capture_output=True, text=True, cwd=repo, + ) + if result.returncode != 0: + pytest.skip("Not in a git repo") + sha = result.stdout.strip() + sha_map, ok = build_sha_file_map(repo, [sha]) + assert ok is True + assert sha in sha_map + assert isinstance(sha_map[sha], list) + + def test_empty_shas(self): + """Empty SHA list returns empty map.""" + sha_map, ok = build_sha_file_map(".", []) + assert sha_map == {} + assert ok is True + + def test_missing_sha(self): + """T10: Non-existent SHA — git may fail or return empty, either is acceptable.""" + repo = str(Path(__file__).parent.parent) + sha_map, ok = build_sha_file_map(repo, ["0" * 40]) + # git log --no-walk with invalid SHA returns rc=128, which is a git failure + # Either ok=True with empty map, or ok=False with fallback — both acceptable + if ok: + assert "0" * 40 not in sha_map or sha_map["0" * 40] == [] + else: + assert sha_map == {"0" * 40: []} + + def test_git_failure(self, tmp_path): + """T11: git failure returns empty lists + success=False.""" + sha_map, ok = build_sha_file_map(str(tmp_path), ["abc123"]) + assert ok is False + assert sha_map == {"abc123": []} + + +# --------------------------------------------------------------- +# T12-T14: assign_domain_by_path +# --------------------------------------------------------------- + +SAMPLE_DOMAINS = [ + {"domain": "semantic", "paths": ["src/semantic/"], "keywords": ["semantic"]}, + {"domain": "commit", "paths": ["src/commit_semantic/", "skills/commit-"], "keywords": ["commit"]}, + {"domain": "demand", "paths": ["src/demand/"], "keywords": ["demand"]}, +] + + +class TestAssignDomainByPath: + """Tests for assign_domain_by_path.""" + + def test_single_domain(self): + """T12: All files in same domain → returns that domain.""" + result = assign_domain_by_path( + ["src/semantic/signals.py", "src/semantic/candidates.py"], + SAMPLE_DOMAINS, + ) + assert result == "semantic" + + def test_mixed_domains(self): + """T13: Files span multiple domains → returns None.""" + result = assign_domain_by_path( + ["src/semantic/signals.py", "src/demand/pipeline.py"], + SAMPLE_DOMAINS, + ) + assert result is None + + def test_no_match(self): + """Files match no domain → returns None.""" + result = assign_domain_by_path( + ["README.md", "setup.py"], + SAMPLE_DOMAINS, + ) + assert result is None + + def test_empty_paths(self): + """Empty file paths → returns None.""" + assert assign_domain_by_path([], SAMPLE_DOMAINS) is None + + def test_empty_domains(self): + """Empty domain list → returns None.""" + assert assign_domain_by_path(["src/foo.py"], []) is None + + def test_longest_prefix_wins(self): + """Longest prefix match wins over shorter.""" + domains = [ + {"domain": "broad", "paths": ["src/"], "keywords": []}, + {"domain": "specific", "paths": ["src/semantic/"], "keywords": []}, + ] + result = assign_domain_by_path(["src/semantic/foo.py"], domains) + assert result == "specific" + + +# --------------------------------------------------------------- +# T1-T6: discover (parse_llm_domains, fingerprint) +# --------------------------------------------------------------- + +class TestDeterministicClassification: + """Tests for deterministic unit scoring and classification.""" + + DOMAINS = [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "session", "token"], + }, + { + "domain": "demand", + "description": "Issue and demand mapping", + "paths": ["src/demand/"], + "keywords": ["demand", "issue", "requirement"], + }, + ] + + def test_scoring_weights(self): + unit = { + "section_name": "Auth token refresh", + "theme": "auth-session", + "summary": "Add login session refresh token flow", + "file_paths": ["src/auth/login.py"], + } + + score = score_unit_for_domain(unit, self.DOMAINS[0]) + + assert score == 13 + + def test_repeated_hits_do_not_stack_per_signal_type(self): + unit = { + "section_name": "Auth auth auth", + "theme": "auth auth auth", + "summary": "auth auth login session token", + "file_paths": ["src/auth/login.py", "src/auth/session.py"], + } + + score = score_unit_for_domain(unit, self.DOMAINS[0]) + + assert score == 13 + + def test_minimum_score_gate(self): + unit = { + "section_name": "Minor cleanup", + "theme": "maintenance", + "summary": "Adjust token naming", + "file_paths": [], + } + + assert classify_unit_locally(unit, self.DOMAINS) is None + + def test_ambiguity_gate(self): + unit = { + "section_name": "Update", + "theme": "auth issue", + "summary": "Refine handling", + "file_paths": [], + } + domains = [ + self.DOMAINS[0], + { + "domain": "issue-triage", + "description": "Issue processing", + "paths": [], + "keywords": ["issue", "routing", "auth"], + }, + ] + + assert classify_unit_locally(unit, domains) is None + + def test_disable_path_scoring_after_multi_domain_failure(self): + unit = { + "section_name": "Auth rollout", + "theme": "maintenance", + "summary": "Refactor helpers", + "file_paths": ["src/auth/login.py"], + } + + assert classify_unit_locally(unit, self.DOMAINS, allow_path_scoring=False) is None + + +class TestParseLlmDomains: + """Tests for parse_llm_domains.""" + + def test_valid_json(self): + """T1: Valid JSON array of domains.""" + raw = json.dumps([ + {"domain": "core", "description": "Core module", "paths": ["src/"], "keywords": ["core"]}, + {"domain": "test", "description": "Tests", "paths": ["tests/"], "keywords": ["test"]}, + ]) + result = parse_llm_domains(raw) + assert len(result) == 2 + assert result[0]["domain"] == "core" + + def test_with_code_fences(self): + """Handles markdown code fences.""" + raw = '```json\n[{"domain": "x", "description": "y"}]\n```' + result = parse_llm_domains(raw) + assert len(result) == 1 + + def test_invalid_json(self): + """T5: Invalid JSON returns empty list.""" + assert parse_llm_domains("not json") == [] + + def test_not_a_list(self): + """Non-list JSON returns empty list.""" + assert parse_llm_domains('{"domain": "x"}') == [] + + def test_missing_domain_key(self): + """Items without 'domain' key are skipped.""" + raw = json.dumps([{"description": "no domain key"}, {"domain": "ok", "description": "has it"}]) + result = parse_llm_domains(raw) + assert len(result) == 1 + assert result[0]["domain"] == "ok" + + def test_schema_validation(self): + """T2: Output has required fields.""" + raw = json.dumps([{"domain": "x", "description": "y", "paths": ["a/"], "keywords": ["k"]}]) + result = parse_llm_domains(raw) + assert "domain" in result[0] + assert "description" in result[0] + assert "paths" in result[0] + assert "keywords" in result[0] + + +class TestFingerprint: + """Tests for compute_fingerprint and fingerprint_matches.""" + + def test_fingerprint_computation(self, tmp_path): + """Fingerprint changes when file content changes.""" + f = tmp_path / "units.jsonl" + f.write_text('{"a": 1}\n') + fp1 = compute_fingerprint(f) + + f.write_text('{"a": 1}\n{"b": 2}\n') + fp2 = compute_fingerprint(f) + + assert fp1["units_hash"] != fp2["units_hash"] + + def test_fingerprint_matches_true(self, tmp_path): + """T3: Fingerprint matches when content unchanged.""" + f = tmp_path / "units.jsonl" + f.write_text('{"a": 1}\n') + fp = compute_fingerprint(f) + data = {"_fingerprint": fp, "domains": []} + assert fingerprint_matches(data, fp) is True + + def test_fingerprint_matches_false(self, tmp_path): + """T4: Fingerprint doesn't match when content changed.""" + f = tmp_path / "units.jsonl" + f.write_text('{"a": 1}\n') + fp_old = compute_fingerprint(f) + data = {"_fingerprint": fp_old, "domains": []} + + f.write_text('{"a": 1}\n{"b": 2}\n') + fp_new = compute_fingerprint(f) + assert fingerprint_matches(data, fp_new) is False + + def test_arch_file_included(self, tmp_path): + """Architecture file hash is included when present.""" + units = tmp_path / "units.jsonl" + units.write_text("{}\n") + arch = tmp_path / "ARCH.md" + arch.write_text("# Architecture\n") + + fp_with = compute_fingerprint(units, arch) + fp_without = compute_fingerprint(units) + + assert fp_with["arch_hash"] is not None + assert fp_without["arch_hash"] is None + + +# --------------------------------------------------------------- +# T14: parse_llm_classifications +# --------------------------------------------------------------- + +class TestParseLlmClassifications: + """Tests for parse_llm_classifications.""" + + def test_valid_response(self): + """T14 partial: Valid classification response.""" + raw = json.dumps([ + {"id": "0", "domain": "semantic"}, + {"id": "1", "domain": "commit"}, + ]) + result = parse_llm_classifications(raw) + assert result == {"0": "semantic", "1": "commit"} + + def test_invalid_json(self): + """LLM failure returns empty dict.""" + assert parse_llm_classifications("broken") == {} + + def test_with_code_fences(self): + """Handles markdown fences.""" + raw = '```json\n[{"id": "0", "domain": "x"}]\n```' + result = parse_llm_classifications(raw) + assert result == {"0": "x"} + + +# --------------------------------------------------------------- +# build_units_summary +# --------------------------------------------------------------- + +class TestBuildUnitsSummary: + """Tests for build_units_summary.""" + + def test_basic_summary(self): + """Produces readable summary with theme and op distribution.""" + units = [ + {"theme": "auth", "op": "feature", "summary": "Add login"}, + {"theme": "auth", "op": "bugfix", "summary": "Fix token"}, + {"theme": "test", "op": "feature", "summary": "Add test"}, + ] + result = build_units_summary(units) + assert "Total units: 3" in result + assert "auth: 2" in result + assert "feature: 2" in result + + def test_empty_units(self): + """Empty units list produces valid summary.""" + result = build_units_summary([]) + assert "Total units: 0" in result + + +class TestDomainNormalization: + """Tests for discover-stage domain normalization helpers.""" + + def test_singular_plural_merge_prefers_plural_tests(self): + domains = [ + { + "domain": "test", + "description": "Single test utilities", + "paths": ["tests/unit/"], + "keywords": ["pytest", "fixture"], + }, + { + "domain": "tests", + "description": "Test infrastructure", + "paths": ["tests/e2e/"], + "keywords": ["pytest", "integration"], + }, + ] + + assert normalize_domains(domains) == [ + { + "domain": "tests", + "description": "Test infrastructure", + "paths": ["tests/e2e/", "tests/unit/"], + "keywords": ["fixture", "integration", "pytest"], + } + ] + + def test_exact_duplicate_merge(self): + domains = [ + { + "domain": "auth", + "description": "Authentication", + "paths": ["src/auth/"], + "keywords": ["login", "token"], + }, + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/", "src/session/"], + "keywords": ["login", "token", "session"], + }, + ] + + assert normalize_domains(domains) == [ + { + "domain": "auth", + "description": "Authentication and sessions", + "paths": ["src/auth/", "src/session/"], + "keywords": ["login", "session", "token"], + } + ] + + def test_keyword_overlap_merge_threshold(self): + left = { + "domain": "auth", + "description": "Authentication flows", + "paths": ["src/auth/"], + "keywords": ["auth", "login", "token"], + } + right = { + "domain": "authentication", + "description": "Authentication and sessions", + "paths": ["src/session/"], + "keywords": ["auth", "login", "session"], + } + other = { + "domain": "billing", + "description": "Billing", + "paths": ["src/billing/"], + "keywords": ["invoice", "payment", "refund"], + } + + assert should_merge_domains(left, right) is True + assert should_merge_domains(left, other) is False + + def test_path_overlap_merge_threshold(self): + left = { + "domain": "demand", + "description": "Demand cards", + "paths": ["src/demand/", "tests/demand/"], + "keywords": ["demand", "issue"], + } + right = { + "domain": "demands", + "description": "Demand processing", + "paths": ["src/demand/", "tests/demand/", "docs/demand/"], + "keywords": ["requirements"], + } + other = { + "domain": "commit", + "description": "Commit pipeline", + "paths": ["src/commit_semantic/"], + "keywords": ["commit"], + } + + assert should_merge_domains(left, right) is True + assert should_merge_domains(left, other) is False + + def test_noise_filtering(self): + domains = [ + {"domain": "misc", "description": "noise", "paths": [], "keywords": ["misc"]}, + {"domain": "other", "description": "noise", "paths": [], "keywords": ["other"]}, + {"domain": "general", "description": "noise", "paths": [], "keywords": ["general"]}, + {"domain": "auth", "description": "Authentication", "paths": ["src/auth/"], "keywords": ["login"]}, + ] + + assert normalize_domains(domains) == [ + { + "domain": "auth", + "description": "Authentication", + "paths": ["src/auth/"], + "keywords": ["login"], + } + ] + + def test_winner_selection_priority(self): + singular = { + "domain": "test", + "description": "Short description", + "paths": ["tests/unit/"], + "keywords": ["pytest"], + } + plural = { + "domain": "tests", + "description": "Longer and more specific test infrastructure domain", + "paths": ["tests/unit/", "tests/e2e/"], + "keywords": ["pytest", "fixture", "integration"], + } + + assert choose_domain_winner(singular, plural)["domain"] == "tests" + + +# --------------------------------------------------------------- +# T26: state compatibility +# --------------------------------------------------------------- + +class TestStateCompat: + """Tests for state compatibility detection.""" + + def _make_runner_and_state(self, completed: list[str]): + """Create a CommitSemanticRunner and HarnessState with given completed stages.""" + from src.harness_state import HarnessState + sys.path.insert(0, str(Path(__file__).parent.parent / "skills" / "commit-semantic")) + # Import directly from the module file + import importlib.util + spec = importlib.util.spec_from_file_location( + "commit_semantic_run", + Path(__file__).parent.parent / "skills" / "commit-semantic" / "run.py", + ) + mod = importlib.util.module_from_spec(spec) + spec.loader.exec_module(mod) + + runner = mod.CommitSemanticRunner() + state = HarnessState( + stage="test", + metadata={ + "completed_stages": completed, + "artifacts_written": [], + "status": "ok", + }, + ) + return runner, state + + def test_old_4stage_detected(self): + """T26: Old 4-stage completed_stages triggers reset.""" + runner, state = self._make_runner_and_state( + completed=["ingest", "aggregate", "distill", "export"] + ) + new_state = runner._check_state_compat(state) + assert new_state.metadata.get("completed_stages") == [] + + def test_new_5stage_preserved(self): + """New 5-stage state is preserved.""" + runner, state = self._make_runner_and_state( + completed=["discover", "ingest"] + ) + new_state = runner._check_state_compat(state) + assert new_state.metadata.get("completed_stages") == ["discover", "ingest"] + + def test_empty_state_preserved(self): + """Fresh state is preserved.""" + runner, state = self._make_runner_and_state(completed=[]) + new_state = runner._check_state_compat(state) + assert new_state.metadata.get("completed_stages") == [] + diff --git a/tests/test_export_dataclasses.py b/tests/test_export_dataclasses.py index a2c493a..7aa20da 100644 --- a/tests/test_export_dataclasses.py +++ b/tests/test_export_dataclasses.py @@ -173,7 +173,7 @@ def setup(self, tmp_path): def _write_fixtures(self, units, patterns=None, demands=None, invariants=None): from src.io_utils import save_jsonl save_jsonl(units, str(self.semantic_dir / "units" / "all.jsonl")) - save_jsonl(patterns or [], str(self.semantic_dir / "patterns.jsonl")) + save_jsonl(patterns or [], str(self.semantic_dir / "domains-aggregated.jsonl")) save_jsonl(demands or [], str(self.semantic_dir / "canonical-demands.jsonl")) save_jsonl(invariants or [], str(self.semantic_dir / "invariants.jsonl")) @@ -237,19 +237,62 @@ def test_summary_json_is_json_serializable(self): assert "total_units" in summary assert "bugfix_ratio" in summary - def test_top_patterns_in_summary(self): + def test_top_domains_in_summary(self): self._write_fixtures( - units=[{"sha": "a", "date": "2026-03-01", "op": "feat", "summary": "s1"}], + units=[{"sha": "a", "date": "2026-03-01", "op": "feat", "summary": "s1", "domain": "auth"}], demands=[ - {"theme": "auth", "score": 10.0, "distinct_commits": 5, "rank": 1}, - {"theme": "api", "score": 8.0, "distinct_commits": 4, "rank": 2}, + {"domain": "auth", "final_score": 10.0, "distinct_commits": 5, "rank": 1}, + {"domain": "api", "final_score": 8.0, "distinct_commits": 4, "rank": 2}, ], ) runner = CommitSemanticRunner() runner._run_export(HarnessState()) from src.io_utils import load_json summary = load_json(str(self.semantic_dir / "summary.json")) - top = summary["top_patterns"] + top = summary["top_domains"] assert len(top) == 2 - assert top[0]["theme"] == "auth" - assert top[0]["score"] == 10.0 + assert top[0]["domain"] == "auth" + assert top[0]["final_score"] == 10.0 + + def test_export_removes_legacy_artifacts_and_keeps_new_outputs(self): + self._write_fixtures([ + {"sha": "a", "date": "2026-03-01", "op": "feat", "summary": "s1", "domain": "auth"}, + ]) + (self.semantic_dir / "patterns").mkdir() + (self.semantic_dir / "patterns" / "legacy.json").write_text("{}", encoding="utf-8") + (self.semantic_dir / "canonical-demands.yaml").write_text("legacy: true\n", encoding="utf-8") + (self.semantic_dir / "functional").mkdir() + (self.semantic_dir / "functional" / "legacy.md").write_text("legacy", encoding="utf-8") + (self.semantic_dir / "non-functional").mkdir() + (self.semantic_dir / "non-functional" / "legacy.md").write_text("legacy", encoding="utf-8") + + runner = CommitSemanticRunner() + runner._run_export(HarnessState()) + + assert (self.semantic_dir / "summary.json").exists() + assert (self.semantic_dir / "canonical-demands.jsonl").exists() + assert not (self.semantic_dir / "patterns").exists() + assert not (self.semantic_dir / "canonical-demands.yaml").exists() + assert not (self.semantic_dir / "functional").exists() + assert not (self.semantic_dir / "non-functional").exists() + + from src.io_utils import load_json + summary = load_json(str(self.semantic_dir / "summary.json")) + assert summary["removed_legacy_paths"] == [ + "patterns", + "canonical-demands.yaml", + "functional", + "non-functional", + ] + + def test_export_is_idempotent_when_no_legacy_artifacts_exist(self): + self._write_fixtures([ + {"sha": "a", "date": "2026-03-01", "op": "feat", "summary": "s1", "domain": "auth"}, + ]) + + runner = CommitSemanticRunner() + runner._run_export(HarnessState()) + + from src.io_utils import load_json + summary = load_json(str(self.semantic_dir / "summary.json")) + assert "removed_legacy_paths" not in summary diff --git a/tests/test_repo_structure.py b/tests/test_repo_structure.py index 6b148cb..950a18b 100644 --- a/tests/test_repo_structure.py +++ b/tests/test_repo_structure.py @@ -1,6 +1,7 @@ """tests/test_repo_structure.py — repo-structure skill tests.""" from __future__ import annotations +import json from pathlib import Path import subprocess import sys @@ -155,13 +156,14 @@ def test_hotspot_consumes_commit_semantic(self, tmp_path, monkeypatch): monkeypatch.chdir(tmp_path) (tmp_path / ".git").mkdir() (tmp_path / "data/commit-extract").mkdir(parents=True) - (tmp_path / "data/commit-semantic/patterns").mkdir(parents=True) + (tmp_path / "data/commit-semantic").mkdir(parents=True) - yaml.dump({"metadata": {"month": "2025-01"}, "commits": [ - {"commit_id": "abc", "files": ["src/hermes/registry.py", "src/hermes/registry.py"]} - ]}, (tmp_path / "data/commit-extract/2025-01.yaml").open("w")) - yaml.dump({"patterns": [{"pattern_id": "p1", "description": "Test pattern"}]}, - (tmp_path / "data/commit-semantic/patterns/canonical.yaml").open("w")) + (tmp_path / "data/commit-extract/2025-01.jsonl").write_text( + json.dumps({"sha": "abc", "file_paths": ["src/hermes/registry.py", "src/hermes/registry.py"]}) + "\n" + ) + (tmp_path / "data/commit-semantic/domains-aggregated.jsonl").write_text( + json.dumps({"domain": "registry", "commit_count": 2, "file_paths": ["src/hermes/registry.py"]}) + "\n" + ) from src.harness_state import HarnessState from skills.repo_structure.run import RepoStructureRunner @@ -319,7 +321,7 @@ def test_full_pipeline_produces_baseline(self, tmp_path, monkeypatch): # Set up all required inputs (tmp_path / "data/commit-extract").mkdir(parents=True) - (tmp_path / "data/commit-semantic/patterns").mkdir(parents=True) + (tmp_path / "data/commit-semantic").mkdir(parents=True) gsd_dir = tmp_path / ".planning/codebase" gsd_dir.mkdir(parents=True) @@ -328,13 +330,14 @@ def test_full_pipeline_produces_baseline(self, tmp_path, monkeypatch): (gsd_dir / fname).write_text(f"## Section\nTest content for {fname}.\n") # commit-extract artifact - yaml.dump({"metadata": {"month": "2025-01"}, "commits": [ - {"commit_id": "abc", "files": ["src/hermes/registry.py", "src/hermes/registry.py"]} - ]}, (tmp_path / "data/commit-extract/2025-01.yaml").open("w")) + (tmp_path / "data/commit-extract/2025-01.jsonl").write_text( + json.dumps({"sha": "abc", "file_paths": ["src/hermes/registry.py", "src/hermes/registry.py"]}) + "\n" + ) - # commit-semantic patterns - yaml.dump({"patterns": [{"pattern_id": "p1", "description": "Test pattern"}]}, - (tmp_path / "data/commit-semantic/patterns/canonical.yaml").open("w")) + # commit-semantic aggregated domains + (tmp_path / "data/commit-semantic/domains-aggregated.jsonl").write_text( + json.dumps({"domain": "registry", "commit_count": 2, "file_paths": ["src/hermes/registry.py"]}) + "\n" + ) from src.harness_state import HarnessState from skills.repo_structure.run import RepoStructureRunner