diff --git a/CONFORMANCE_TESTING_IMPROVEMENT_PLAN.md b/CONFORMANCE_TESTING_IMPROVEMENT_PLAN.md new file mode 100644 index 00000000..8eb0e0bd --- /dev/null +++ b/CONFORMANCE_TESTING_IMPROVEMENT_PLAN.md @@ -0,0 +1,552 @@ +# Conformance testing improvement — implementation plan + +**Status:** Iteration 1 implemented and in review — client PR +[codeplain#252](https://github.com/Codeplain-ai/codeplain/pull/252), backend PR +[plain2code_rest_api#122](https://github.com/Codeplain-ai/plain2code_rest_api/pull/122). +**Iteration 2 implemented (2026-07-19)** on `feature/conformance-single-run-suite` in both +repos (stacked on the Iteration 1 branches), all chunks verified e2e against a local backend: + +- 2a (`5a8d4d8`): loop-capable golang + cypress runners (degenerate case preserves old behavior). +- 2b.1 (`4aaa6e4`): failure attribution helpers + tests. +- 2b.2 (`36d09ce`): whole-suite execution flip. Verified: side-by-side per-FRID vs whole-module + verdicts agree; happy path 2-FRID render = 2 invocations (was 3, grew quadratically); + forced cross-FRID conflict → failure attributed to the earlier FRID, fix loop scoped to its + subfolder, backend classified conflicting requirements, render failed with a clear report. +- 2b.3 (`e435de3`): dead FRID-walk machinery removed (~100 lines). +- 2c.1 (backend `40bbf75`): optional module-root-relative output paths behind a request flag. +- 2c.2 (`03c9b0d`): client stores at module root; shared root files included in context; two-tier + guard (other-suite paths rejected; shared root files insertion-only via difflib) with one retry. +- 2c.3 (client `68ee4a1`, backend `747dfb9`): per-FRID summarization dropped. Evidence: its only + consumer was the plan-stage dedup section; A/B renders of a 3-FRID project with and without + summaries produced identical dedup quality (no cross-FRID duplication either way). The + `folder_name`/`functional_requirement` map stays (attribution + fix routing depend on it); + the backend endpoint stays for older clients. Saves one LLM call per functionality. + +Note from 2c e2e: the render prompt permits shared root helpers, but on small projects the LLM +keeps suites self-contained — sharing is opportunity-based, which is the intended contract. + +Feature branch (both repos): `feature/improve-conformance-testing` + +- Client worktree: `codeplain/.claude/worktrees/feature+improve-conformance-testing` +- Backend worktree: `plain2code_rest_api/.claude/worktrees/feature+improve-conformance-testing` + +## Local environment (verified 2026-07-18) + +- Both branches are synced with their `origin/main` and fully green locally. +- **Client (codeplain):** run tests and quality gates with the `plain2code_client` conda env + (`/opt/miniconda3/envs/plain2code_client/bin/python`). Baseline: 293 tests pass; black, isort, + flake8, mypy all clean. (Running with the miniconda *base* env produces spurious failures — + wrong tool versions; don't use it.) +- **Backend (plain2code_rest_api):** run tests with the `plain2code_server` conda env. Baseline: + 225 tests pass. Prerequisites: local Postgres container running + (`docker-compose -f docker-compose.dev.yml up -d`) and the gitignored `src/.env` present in + the worktree (copied from the main checkout's `src/.env`; `load_dotenv()` resolves it relative + to `src/config.py`). + +## Background and goal + +Today the renderer creates a **separate conformance test suite per FRID** +(`conformance_tests///`), and when generating tests for a new FRID the +backend's plan stage (`DeviseConformanceTestsPlanTemplate`) only sees lossy **text summaries** +of previous FRIDs' tests (`conformance_tests_json`). The implementation stage sees no prior +tests at all and outputs new files only, into a fresh folder. + +Consequences, especially when specs build incrementally on one another: + +- Duplicated test coverage across suites (each new suite re-tests ground already covered). +- Duplicated setup code (each suite re-implements the fixtures earlier suites already built). +- Inconsistent conventions between suites (naming, structure, helpers drift). + +**Long-term direction (see "Roadmap after this iteration"):** one test suite per module that is +run as a whole (one script invocation per module) with shared setup, where rendering a new FRID +only **adds** tests — existing tests are never modified at render time. **This iteration** takes +the first, deliberately small step: keep everything as-is (per-FRID folders, new-files-only +output, folder naming, regression walk, fix loop) and only **send the existing conformance-test +files to the backend** so the render prompts can (a) avoid duplicating existing coverage and +(b) match existing conventions. + +## Decisions already made + +- Existing test files go to **both** prompt stages (plan + implementation). +- Send the **current module's own suites only** (required modules' copies stay covered by the + `conformance_tests_json` summaries). +- `conformance_tests_json` summaries **stay in the prompt unchanged** (minimal prompt surgery). +- The new API field is **optional** → fully backward compatible with older clients. + +## Delivery model + +Implement one step at a time; **STOP after each step for user review** of code and results +before starting the next. Every step leaves both repos fully working: all tests and quality +gates green, old behavior preserved. Backend changes land **before** the client starts sending +the new field, so at no point does the client send something the server doesn't accept. + +**Every step cleans up after itself.** Whatever a step makes obsolete (dead code paths, unused +state-machine wiring, scaffolding introduced by an earlier step) is removed in that same step — +there is no deferred "cleanup iteration" at the end. + +| Step | Repo | What ships | Why the system still fully works after it | +|------|------|-----------|-------------------------------------------| +| 1 | codeplain | Fetch helper + unit tests (not yet called by the render flow) | Pure addition; render behavior unchanged | +| 2 | rest_api | API plumbing: optional request field threaded to the handler (prompts untouched) | Field accepted and logged but unused; old clients unaffected | +| 3 | rest_api | Prompt changes: inject the files into both templates (empty-safe) | No field sent → empty section → prompts render exactly as today | +| 4 | codeplain | Wire the helper into the render action + API client; feature live | End of feature; verified end-to-end against a local API | + +--- + +## Step 1 — client fetch helper (`codeplain`) + +**Goal:** a tested, unused building block that collects all existing conformance-test files of a +module. + +### Changes + +`render_machine/conformance_tests.py` — add to the `ConformanceTests` class: + +``` +fetch_all_existing_conformance_test_files(module_name) -> dict[str, str] +``` + +- Iterate folders from the existing `fetch_existing_conformance_test_folder_names(module_name)` + (already excludes hidden `.` copies of required-module tests — matches the + "current module only" decision). Sort folder names for deterministic output. +- Per folder, reuse `file_utils.list_all_text_files` + `file_utils.get_existing_files_content` + (same primitives the existing `fetch_existing_conformance_test_files` uses, + `conformance_tests.py:147-166`). +- Key files as `/` so the LLM can tell which suite a file + belongs to. +- `conformance_tests.json` (the definition file) is naturally excluded — it lives at the module + folder root and only subfolders are walked; cover this with a test. +- Returns `{}` when no prior suites exist (first FRID). + +`tests/test_conformance_tests.py` — new test file covering: empty/missing module folder, +multi-folder collection incl. nested subdirectories, hidden-folder exclusion, definition-file +exclusion, binary-file skipping. + +### Done-check (all must be green before review) + +- `pytest tests/ -v` (new tests pass; no new failures vs. baseline) +- `black --check`, `isort --check-only`, `flake8`, `mypy . --check-untyped-defs` + (no new complaints vs. baseline) + +**→ STOP for user review.** + +--- + +## Step 2 — backend plumbing (`plain2code_rest_api`) + +**Goal:** the API accepts the new optional field end-to-end without using it yet. + +### Changes + +`src/app.py`: + +- `render_conformance_tests_model` (line ~348): add + `"existing_conformance_tests_files": fields.Nested(file_content_model, required=False, + description="Content of the module's existing conformance test files, keyed by + /")`. +- Route handler (line ~850): + `existing_conformance_tests_files = data.get("existing_conformance_tests_files", {})`, + passed through to `codeplain_instance.render_conformance_tests(...)`. + +`src/codeplain.py` — `render_conformance_tests` (line ~1034): + +- Accept the new parameter (default `{}`, `None`-safe). +- Log the received file count. +- **Do not use it in prompts yet.** + +### Done-check + +- Backend test posting to `/render_conformance_tests` **without** the field (backward compat) + and **with** it — both accepted. +- Backend test suite green (no new failures vs. baseline). + +**→ STOP for user review.** + +--- + +## Step 3 — backend prompt changes (`plain2code_rest_api`) + +**Goal:** both LLM stages see the existing test files; behavior identical when none are sent. + +### Changes + +`src/codeplain.py` — `render_conformance_tests`: + +- Prefix incoming keys with the existing `CONFORMANCE_TESTS_FOLDER_MARKER` (line 67) the same + way the fix flow does (line ~1296): `{f"{CONFORMANCE_TESTS_FOLDER_MARKER}/{key}": value ...}` + and merge into `files_content` so the LLM chain can view file contents. Do **not** add them to + `applicable_files_content` — they are read-only context, not implementation files. +- Build the prompt section with the existing `_get_conformance_tests_files_prompt_section` + (line 217) — reuse, don't reimplement. +- Add the rendered section to the input dicts of **both** LLM calls (plan stage at ~line 1114, + implementation stage at ~line 1183) under a new key, e.g. + `"previous_conformance_tests_files"`. Empty string when nothing was sent (first FRID or old + client) so templates degrade gracefully. + +`src/prompt_templates/conformance_test_template.py`: + +- New template block, e.g. `PREVIOUS_CONFORMANCE_TESTS_FILES_TEXT`: these are the actual source + files of :ConformanceTests: implemented for previous functionalities; they are **read-only + reference** — do not modify or re-emit them; use them to (a) avoid duplicating test coverage + that already exists and (b) follow the same structure, naming conventions, fixtures and + configuration patterns. (Mirror the phrasing of the acceptance-test template, + `acceptance_test_template.py:33-43`, which already implements the "extend, don't duplicate" + stance.) +- `DeviseConformanceTestsPlanTemplate` — extend Task 2 (the dedup task, ~line 175): a planned + test is also removed if its expectations are **already implemented in the previous + conformance-test files** (real code, not just summaries). Keep the existing summary-based + criteria intact. +- `ConformanceTestsImplementationTemplate` — include the new section in + `get_template_text_list()`; instruct that new tests must follow the conventions visible in the + previous test files. Output format stays `LLM_SOURCE_OUTPUT_NEW_FILE` — unchanged. +- Both templates render the section conditionally (empty input → empty/omitted section). + +### Done-check + +- Backend tests green, covering both the with-field and without-field request paths. +- Prompt output with empty input is byte-identical to today's (no accidental drift for old + clients). + +**→ STOP for user review.** + +--- + +## Step 4 — client wiring (`codeplain`) + +**Goal:** the feature goes live; the client sends existing tests on every conformance render. + +### Changes + +`render_machine/actions/render_conformance_tests.py` — in `_render_conformance_tests` (the +full-conformance-render path only, **not** the acceptance path): + +- Call the Step 1 helper for `render_context.module_name`. +- Print the files via `console.print_files` (matches the existing "Files sent as input..." + pattern). +- Pass them as a new argument to `render_context.codeplain_api.render_conformance_tests(...)`. + +`codeplain_REST_api.py` (lines ~306-339): + +- Add parameter `existing_conformance_tests_files` to `render_conformance_tests`; include it in + the JSON payload as `"existing_conformance_tests_files"`. + +### Done-check + +- Client tests + quality gates green. +- **End-to-end verification** (manual, per CLAUDE.md cross-repo workflow): + 1. Start local Postgres + API (`python src/app.py` in the backend worktree, port 5000). + 2. Render a multi-FRID example (e.g. `examples/example_hello_world_python`) with + `--api http://127.0.0.1:5000`. + 3. Confirm: FRID 1 renders with an empty section; FRID ≥ 2 requests contain + `existing_conformance_tests_files`; the plan-stage output shows dedup decisions that + reference actual previous tests; rendered suites still pass their conformance runs. + +**→ feature complete, final user review.** + +--- + +## Roadmap after this iteration + +### Iteration 2 — single-run suite with shared setup (two-tier immutability) + +Decided design: + +- **Keep the per-FRID subfolders inside the module suite** (minimal structural change). A + failing test's path still names its FRID, so attribution stays free — no ownership index + needed. Storage, folder naming, and `conformance_tests_json` stay as-is. + - Scope note: the depth-1 "subfolder directly under the module folder" layout holds for all + three shipped runners (python, golang, cypress) and is what Iteration 2 implements. For + deep-layout ecosystems it generalizes later — see *Deep-layout ecosystems* below. +- **Whole-suite always — the test-script contract never changes.** The scripts' contract stays + "run the tests found under `$2`"; the client simply passes the module suite folder + (`conformance_tests//`) instead of a per-FRID subfolder. No script in this repo — and, + critically, no user project's custom script referenced from `config.yaml` — needs editing. + Consequence: there is no standalone-first run of the current FRID's fresh tests; every + conformance invocation runs the whole module suite. The client classifies failures + ("my new test is broken" vs. "new code broke an old test") from the failing tests' subfolder + paths. Trade-off accepted: fix-loop cycles re-run the full suite (wall-clock cost on modules + with many FRIDs); revisit a two-path script contract later only if that hurts in practice. +- **Two-tier immutability at render time** (refined from an earlier stricter "append-only" + rule, which broke on legitimately evolving shared files — build manifests, shared helpers, + model factories): + - **Other FRIDs' test files are strictly immutable at render time** — the load-bearing + guarantee: another functionality's expectations are never weakened or rewritten. Only the + fix loop may edit them, same as today. + - **Shared setup files (suite root: manifests, helpers, factories) are extendable at render + time, insertion-only** — mechanically enforced: every existing line must survive in order; + the new version may only insert lines (new factory/trait, new dependency line, new import). + Deletions or edits of existing lines → reject. One diff check covers `pom.xml`, + `requirements.txt`, and `factories.rb` uniformly. + - Backstops: the whole-suite run immediately verifies extensions against old tests (an + insertion that changes existing helper behavior fails fast and the fix loop repairs it), + and the prompt instructs "add new entries; do not alter the behavior of existing helpers". + Genuine modifications (e.g. bumping a pinned dependency version) remain fix-loop-only. + +**Empirical grounding (verified 2026-07-18 on the Iteration 1 e2e project):** invoking the +unchanged `run_conformance_tests_python.sh` with the module suite folder as `$2` discovered and +ran both FRIDs' suites in one invocation (`Ran 2 tests ... OK`). This works because (a) the +Python template's `***test reqs***` already mandates `__init__.py` in test subfolders +(`python-console-app-template.plain:16`), making each per-FRID subfolder a distinct package, and +(b) `generate_folder_name_from_functional_requirement` already guarantees unique subfolder +names. The whole-suite premise is proven for Python; golang/cypress need the 2a audit. + +Delivered in three self-contained chunks, each committed+pushed with the system fully working. + +(A shadow-run phase was considered and dropped: shadow phases earn their keep when a change +ships to a fleet you observe, but codeplain renders happen on machines we don't — the only +shadow data would come from our own renders, which 2b's side-by-side verification provides +without building and then deleting renderer machinery.) + +--- + +#### Chunk 2a — discovery-safe suites (loop-capable runners; no behavior change) + +##### Audit findings (2026-07-19) + +- **python — works as-is.** `unittest discover` recurses; per-FRID subfolders are packages + (`__init__.py` pinned in the template). Proven empirically on the Iteration 1 e2e project. +- **golang — script change required.** The runner does no discovery at all: it executes one + hardcoded file, `go run "$2/conformance_tests.go"` (`run_conformance_tests_golang.sh:79`). + The single-main-file convention is pinned in the golang template + (`golang-console-app-template.plain:21-23`) and stays. With `$2` = module folder there is no + root `conformance_tests.go` → immediate failure. +- **cypress — script change required.** Each suite is a standalone Cypress project + (`cypress.config.ts`, `package.json`, `cypress/e2e/...` — confirmed in + `examples/example_hello_world_react/harness_tests/hello_world_display/`). The runner copies + `$2/*` to a scratch dir and runs `npx cypress run` there — with `$2` = module folder there is + no config at the copied root → hard fail. The runner also builds and starts the React app on + every invocation — today paid once per FRID, so the flip amortizes it to once per module: + the biggest single speedup of Iteration 2. +- **Templates need no changes**; suite structures stay as they are. + +##### Resolution: subfolder-loop pattern, shipped in 2a + +Updated golang/cypress runners implement "run all tests under `$2`" as an internal loop, with a +**degenerate-case check** that keeps today's behavior byte-identical: if `$2` itself looks like +a single suite (root `conformance_tests.go` / root `cypress.config.*`), run it directly exactly +as today. Because the client is untouched in 2a (still passes per-FRID folders → always the +degenerate path), 2a ships with zero behavior change; 2b then only flips `$2`. + +##### Step 2a.1 — golang loop runner (`run_conformance_tests_golang.sh` + `.ps1`) + +- Keep: arg validation, `/tmp/go_` staging of `$1`, `go get` in the build folder, exit + codes. +- Degenerate case: `$2/conformance_tests.go` exists → current behavior verbatim (including the + optional `go get` in `$2` when it has a `go.mod`). +- Loop case: iterate sorted immediate subfolders of `$2` that contain `conformance_tests.go`; + for each: optional `go get` in the subfolder (mirrors today's per-suite behavior), then + `go run "/conformance_tests.go"` from the build dir; print a + `=== conformance suite: ===` header before each suite's output (feeds 2b's + attribution). +- Aggregation: run **all** suites (don't stop at first failure — 2b needs the full implicated + set); exit with the first failing suite's exit code; zero suites found → exit 1 (the + "no tests discovered" convention). +- Verify: synthetic two-suite fixture from the golang example's harness artifacts + a + single-suite degenerate check; golang example still renders green (client unchanged). + +##### Step 2a.2 — cypress loop runner (`run_conformance_tests_cypress.sh` + `.ps1`) + +- Keep: port-3000 cleanup, `$1` staging, `npm install` + `npm run build` + app start — + **once per invocation** (Step 1 of the script is untouched). +- Degenerate case: `$2` has a root `cypress.config.*` → current behavior verbatim. +- Loop case: iterate sorted immediate subfolders of `$2` that contain `cypress.config.*`; + for each: stage into the scratch dir (wipe between suites, preserving `node_modules` / + `package-lock.json` as today), `npm install cypress` (cheap after first — offline cache), + `npx cypress run`; same `=== conformance suite: ===` headers. +- Aggregation: same as golang. +- Verify: two-suite fixture built from the react example's harness suite (duplicated with a + second spec) + degenerate check; react example still renders green. + +##### Verification matrix (2026-07-19, chunk implemented) + +| Case | golang `.sh` (real go 1.25) | cypress `.sh` (stubbed npm/npx) | +|---|---|---| +| Degenerate single suite, pass | exit 0 ✓ | exit 0 ✓ | +| Degenerate single suite, fail | output printed, exit 1 ✓ | exit 1 ✓ | +| Module folder: pass+fail suites | both run with `=== conformance suite: ===` headers, failure output under its header, exit 1 ✓ | same ✓ | +| Hidden `.module` subfolder | skipped ✓ | skipped ✓ | +| Empty module folder | "No conformance test suites discovered", exit 1 ✓ | same ✓ | +| Missing `$2` | exit 69 ✓ | exit 69 ✓ | + +`.ps1` variants mirrored by careful review; not executable on this machine (no pwsh) — same +verification status as the repo's other PowerShell scripts. Python runner untouched. +Latent quirk noted (pre-existing, unchanged): the cypress script's +`npm install | grep -Ev ` under `pipefail` fails if npm's entire output matches the +filter; real npm always emits a surviving line. + +A full example render was not repeated: the client is unchanged and still invokes the scripts +with per-FRID folders — exactly the degenerate case verified above. + +#### Chunk 2b — flip execution to the single run (client only; the core chunk) + +Grounding fact that keeps this chunk small: the fix loop, memory creation, and conflict +detection all key off `conformance_tests_running_context.current_testing_frid` and derive the +suite folder from it (`fix_conformance_test.py:48-147`, `memory_management.py:35`). So the flip +reduces to: run whole suites, and on failure **set `current_testing_frid` via attribution** +before the existing machinery takes over. + +Also note: once `TESTING_CURRENT_FRID` runs the whole own-module suite, it already covers every +prior FRID — the own-module regression walk doesn't just shrink, it **disappears**; regression +reduces to running each required module's copied suite. + +##### Step 2b.1 — attribution + evidence helpers (pure addition, nothing calls them yet) + +New module `render_machine/failure_attribution.py`: + +- `attribute_failures(output, conformance_tests_json) -> list[str]`: FRIDs whose + `folder_name` basename appears in the output, ordered by spec order (json insertion order). +- `extract_frid_failure_evidence(output, folder_basename) -> str`: best-effort per-FRID slice + (Python unittest blocks are `======`-delimited); returns the full output when slicing fails. +- `format_other_frids_note(implicated_frids, current_frid) -> str`: the one-line "tests of + functionalities X also failed in this run; handled separately" summary. +- `detect_layout_failure(output) -> bool`: **conservative** migration guard — fires only when + zero tests ran AND an import-style signature is present ("Start directory is not importable", + discovery-time `ModuleNotFoundError`). A legit test failure must never trip it. + +Unit tests for all four (single/multiple/none implicated; slice + fallback; guard +true/false cases). Commit+push — behavior unchanged. + +##### Step 2b.2 — the flip (one behavioral commit) + +- `RunConformanceTests.execute` (`run_conformance_tests.py:19`): own module → + `$2 = get_module_conformance_tests_folder(module_name)`; required module → the module-level + copy root (`conformance_tests//./`, today's + `get_source_conformance_test_folder_name` logic at module granularity — new small helper in + `conformance_tests.py`). +- Phase orchestration (`render_context.py`): `TESTING_CURRENT_FRID` = one own-module suite run; + the regression phase iterates required modules only. Acceptance-test phases unchanged (their + re-runs are now whole-suite runs). "Code changed while fixing" simplifies from + "restart the FRID walk" to "re-run the affected module suites". +- Post-failure routing (in `RunConformanceTests.execute`, before returning `FAILED_OUTCOME`): + `attribute_failures` → set `ctx.current_testing_frid` (and module) to the earliest implicated + FRID → build the failure payload from `extract_frid_failure_evidence` + `format_other_frids_note` + instead of the raw output. Everything downstream — fix payload files, + `is_previous_conformance_tests_issue`, conflict detection, memory keying — works unchanged. + Order matters: attribution runs **before** `create_conformance_tests_memory`. +- Migration guard: `detect_layout_failure` → dispatch a render error ("regenerate conformance + tests for this module; if using a custom conformance script, it must run all tests under + `$2` recursively") instead of entering the fix loop. +- Budget: `ctx.fix_attempts` already lives on the per-implemented-FRID running context → the + global cap holds with no change. +- Verify: full client suite + gates; **side-by-side verification** — throwaway script runs the + per-FRID walk and the whole-suite invocation on the same rendered example and diffs verdicts; + e2e render against the local backend; hand-break an earlier FRID's behavior in the build and + confirm the fix loop targets that FRID's subfolder with filtered evidence. Commit+push. + +##### Step 2b.3 — cleanup (same chunk, separate commit for reviewability) + +Remove now-dead machinery + its tests: FRID-iteration branches of +`get_first/next_conformance_tests_running_context`, `_has_reached_implementation_frid`, +`_start_regression_phase`'s FRID bookkeeping, `code_changed_during_regression` restart logic, +and the `MOVE_TO_NEXT_CONFORMANCE_TEST` transitions in `state_machine_config.py:338-399` that +implement the walk (module iteration and acceptance-phase transitions stay). `requires`-chain +copying and `conformance_tests_json` bookkeeping stay. Full suite + gates green; one more e2e +render. Commit+push. + +#### Chunk 2c — shared setup at the suite root (backend prompts + client storage) + +Depends on 2b. **Scope: flat layouts (python) only** — golang/cypress suites stay standalone +per-suite projects under the loop runners; consolidating them to a shared root belongs to the +deep-layout forward path below. + +##### Step 2c.1 — backend: output-path contract + prompts + +- Conformance render output paths become **module-suite-root-relative**. The prompt still + receives the FRID subfolder name and instructs: test files go under + `/`; shared helpers/fixtures may be placed at the suite root; reuse existing + root helpers instead of re-implementing setup; other functionalities' test files must never + be re-emitted or modified; shared setup files at the suite root may be extended by emitting + the full new version, adding lines only, without altering the behavior of existing helpers. + Rewrite `CONFORMANCE_TESTS_FOLDER_NAME_HINT` (`conformance_test_folder_names.py`) + accordingly; extend the Iteration 1 previous-files section with the reuse instruction; same + treatment for `AcceptanceTestsImplementationTemplate`. Output format stays + `LLM_SOURCE_OUTPUT_NEW_FILE`. +- No new required API field: the client keeps sending `conformance_tests_folder_name` (the FRID + subfolder); the module root is its parent by construction. +- Template tests extended. Backend suite green. Commit+push (backward compatible: old clients + keep old-style paths because the hint text is driven by the same field they already send — + verify this explicitly in tests). + +##### Step 2c.2 — client: storage, context, and the two-tier guard + +- Store render/acceptance response files relative to the **module folder** + (`render_conformance_tests.py:135` and the acceptance path) instead of the FRID subfolder. +- `fetch_all_existing_conformance_test_files` (`conformance_tests.py`): **include root-level + files** (shared helpers) — today it walks subfolders only; keep excluding + `conformance_tests.json`. The render-context exclusion of the current FRID's subfolder stays. +- **Two-tier guard** on stored render responses: + - a response file whose path matches an existing file in another FRID's subfolder → reject + (retry the render call once with the violation named; then fail the render); + - a response file whose path matches an existing suite-root shared file → insertion-only + diff check (every existing line survives, in order — `difflib` opcodes contain no + `delete`/`replace`); violation → same reject-retry-fail path; + - new paths → store normally. +- `conformance_tests_json` `folder_name` bookkeeping unchanged. Client suite + gates green; + e2e: render a multi-FRID python example, confirm FRID ≥ 2 imports a root helper instead of + duplicating setup, whole suite green. Commit+push. + +##### Step 2c.3 — summaries decision (evaluation, then possibly a removal commit) + +With real files in prompts (Iteration 1) and one-run suites (2b), evaluate whether the +per-FRID `summarize_finished_conformance_tests` LLM call still earns its cost. The +`folder_name` map in `conformance_tests_json` **must stay** (attribution and fix routing depend +on it) — the question is only the `test_summary` content and its LLM call. Criteria: render the +examples with summaries suppressed from the plan prompt and compare dedup quality; if no +degradation, drop the call (client + backend + prompt cleanup in both repos). Record the +decision and evidence here either way. + +#### Deep-layout ecosystems (Java/Maven and similar) — deferred until Iteration 2 works + +Decision (2026-07-19): a Java example will be added to `examples/` **after** the first +implementation (chunks 2a-2c) works properly. At that point: audit the example's actual suite +structure (extend 2a's findings), add a Java loop runner as step 2a.3 (degenerate case: root +`pom.xml`; loop case: `mvn test` per subfolder — 2b then covers Java with no further change), +and turn this section into a concrete chunk 2d (single Maven project per module). Recorded here +so Iteration 2's decisions don't paint us into a corner. Theme: **the suite root owns language-specific +structure; a FRID owns only its namespace.** + +- **Target shape (Java example):** one Maven project per module — + `conformance_tests//pom.xml` at the suite root (shared setup per chunk 2c), shared + fixtures under `src/test/java//support/`, and one package per FRID + (`src/test/java///`). One `mvn test` runs everything. +- **Folder-name generation is reinterpreted, not removed:** + `generate_folder_name_from_functional_requirement` keeps producing a unique, + identifier-safe **namespace slug** per FRID (current outputs are already valid Python and + Java package names). What changes: the client stops composing the full path from the slug; + the prompt instead says "this functionality's tests live in a namespace/folder named + ``, placed where the suite's layout requires." Depth-1 for Python/golang/cypress + (byte-identical to Iteration 2 behavior); a package subtree under `src/test/java/` for Java. +- **Recorded path, not assumed path:** when storing response files, the client locates the + directory matching the slug (`**//`) and records the actual path in + `conformance_tests_json` alongside the slug. Fix-loop payloads, regeneration deletes, and + attribution all work from the recorded path. Slug uniqueness is checked against the known + slugs in `conformance_tests_json` rather than a filesystem listing. +- **Attribution is unchanged in mechanism:** the slug appears in fully qualified test names + (`com.example.conformance..FooTest`) and matches against failure output the same + way a depth-1 folder name does. +- **Build manifests are covered by the two-tier rule:** a shared `pom.xml` / `package.json` / + `go.mod` at the suite root is a shared setup file — extendable at render time under the + insertion-only diff check (see the design bullets above), so a later FRID can add a + dependency without a fix-loop round-trip. Version bumps and other in-place edits stay + fix-loop-only. + +**Migration note:** projects rendered before the flip may have suites that collide in a joint +run. The intended answer for old projects is regenerating conformance tests on the next full +render (surfaced by 2b's migration guard), not compatibility machinery. +Custom user scripts: 2b deepens `$2` (module folder instead of one suite subfolder). Scripts +that genuinely "run all tests under `$2`" recursively are unaffected; scripts that hard-assumed +the old layout fail fast on their first post-flip render — 2b's migration guard should +recognize this shape too and say "your conformance script must run all tests under `$2` +recursively" instead of burning fix attempts. Document the semantic change in the release +notes. + +**Logistics (decided 2026-07-19):** Iteration 2 is developed on +`feature/conformance-single-run-suite` in both repos, stacked on the Iteration 1 branches +(`feature/improve-conformance-testing`, PRs #252 / #122), and goes into separate PRs targeting +those branches (retarget to `main` once the Iteration 1 PRs merge). + +## Out of scope (removed from the roadmap) + +Render-time **modification** of existing tests — and everything it would require (declared +supersession, test→FRID ownership index, fix-diff constraints) — is **completely out of scope +for now**. It may be reconsidered only after Iteration 2 has proven itself in practice. diff --git a/codeplain_REST_api.py b/codeplain_REST_api.py index 7a86ec59..18d7599d 100644 --- a/codeplain_REST_api.py +++ b/codeplain_REST_api.py @@ -335,6 +335,7 @@ def render_conformance_tests( "conformance_tests_json": conformance_tests_json, "all_acceptance_tests": all_acceptance_tests, "existing_conformance_tests_files": existing_conformance_tests_files, + "conformance_tests_paths_relative_to_module_root": True, } response = self.post_request(endpoint_url, headers, payload, run_state) @@ -510,26 +511,3 @@ def fail_functional_requirement(self, frid, module_name: str, run_state: RunStat } return self.post_request(endpoint_url, headers, payload, run_state) - - def summarize_finished_conformance_tests( - self, - frid, - plain_source_tree, - linked_resources, - conformance_test_files_content, - module_name: str, - required_modules, - run_state: RunState, - ): - endpoint_url = f"{self.api_url}/summarize_finished_conformance_tests" - headers = {"X-API-Key": self.api_key, "Content-Type": "application/json"} - payload = { - "frid": frid, - "plain_source_tree": plain_source_tree, - "linked_resources": linked_resources, - "conformance_test_files_content": conformance_test_files_content, - "module_name": module_name, - "required_modules": required_modules, - } - - return self.post_request(endpoint_url, headers, payload, run_state) diff --git a/render_machine/actions/render_conformance_tests.py b/render_machine/actions/render_conformance_tests.py index 95ffe32a..2b753c4a 100644 --- a/render_machine/actions/render_conformance_tests.py +++ b/render_machine/actions/render_conformance_tests.py @@ -8,11 +8,12 @@ from render_machine.actions.base_action import BaseAction from render_machine.implementation_code_helpers import ImplementationCodeHelpers from render_machine.render_context import RenderContext -from render_machine.render_types import AcceptanceTestPhase, TestExecutionPhase +from render_machine.render_types import AcceptanceTestPhase, RenderError, TestExecutionPhase class RenderConformanceTests(BaseAction): SUCCESSFUL_OUTCOME = "conformance_test_rendered" + RESPONSE_VALIDATION_FAILED_OUTCOME = "conformance_test_response_validation_failed" def execute(self, render_context: RenderContext, _previous_action_payload: Any | None): if self._should_render_conformance_tests(render_context): @@ -123,33 +124,64 @@ def _render_conformance_tests(self, render_context: RenderContext): if not file_name.startswith(current_subfolder_prefix) } - response_files, implementation_plan_summary = render_context.codeplain_api.render_conformance_tests( - render_context.frid_context.frid, - render_context.conformance_tests_running_context.current_testing_frid, - render_context.plain_source_tree, - render_context.frid_context.linked_resources, - existing_files_content, - memory_files_content, - render_context.module_name, - render_context.get_required_modules_functionalities(), - conformance_tests_folder_name, - render_context.conformance_tests_running_context.get_conformance_tests_json( - render_context.conformance_tests_running_context.current_testing_module_name - ), - all_acceptance_tests, - existing_conformance_tests_files, - run_state=render_context.run_state, + current_subfolder_name = os.path.basename(conformance_tests_folder_name) + module_conformance_tests_folder = render_context.conformance_tests.get_module_conformance_tests_folder( + render_context.module_name ) + for attempt in range(2): + response_files, implementation_plan_summary = render_context.codeplain_api.render_conformance_tests( + render_context.frid_context.frid, + render_context.conformance_tests_running_context.current_testing_frid, + render_context.plain_source_tree, + render_context.frid_context.linked_resources, + existing_files_content, + memory_files_content, + render_context.module_name, + render_context.get_required_modules_functionalities(), + conformance_tests_folder_name, + render_context.conformance_tests_running_context.get_conformance_tests_json( + render_context.conformance_tests_running_context.current_testing_module_name + ), + all_acceptance_tests, + existing_conformance_tests_files, + run_state=render_context.run_state, + ) + + violations = render_context.conformance_tests.find_response_file_violations( + render_context.module_name, + current_subfolder_name, + response_files, + ) + + if not violations: + break + + console.warning( + "Generated conformance test files violate the suite layout rules:\n " + + "\n ".join(violations) + + ("\nRetrying the generation." if attempt == 0 else "") + ) + else: + return ( + self.RESPONSE_VALIDATION_FAILED_OUTCOME, + RenderError.encode( + message="Generated conformance test files repeatedly violated the suite layout rules " + "(files outside the functionality's subfolder or invalid changes to shared setup files).", + error_type="CONFORMANCE_TESTS_VALIDATION_ERROR", + violations="\n".join(violations), + ).to_payload(), + ) + render_context.conformance_tests_running_context.current_testing_frid_high_level_implementation_plan = ( implementation_plan_summary ) - file_utils.store_response_files(conformance_tests_folder_name, response_files, []) + file_utils.store_response_files(module_conformance_tests_folder, response_files, []) console.print_files( "Conformance test files generated:", - conformance_tests_folder_name, + module_conformance_tests_folder, response_files, style=console.OUTPUT_STYLE, ) diff --git a/render_machine/actions/run_conformance_tests.py b/render_machine/actions/run_conformance_tests.py index 3c7bb1f0..4d344056 100644 --- a/render_machine/actions/run_conformance_tests.py +++ b/render_machine/actions/run_conformance_tests.py @@ -4,11 +4,20 @@ import render_machine.render_utils as render_utils from plain2code_console import console from render_machine.actions.base_action import BaseAction +from render_machine.failure_attribution import detect_layout_failure from render_machine.render_context import RenderContext from render_machine.render_types import RenderError UNRECOVERABLE_ERROR_EXIT_CODES = [69] +LAYOUT_FAILURE_MESSAGE = ( + "Conformance test suites of this module could not be discovered or run together. " + "This usually means the conformance tests were generated by an older version of the renderer - " + "delete the module's conformance tests folder so they get regenerated on the next render. " + "If the project uses a custom conformance tests script, make sure it runs all tests found " + "under the folder it receives as its second argument, including tests in subfolders." +) + class RunConformanceTests(BaseAction): @@ -19,26 +28,16 @@ class RunConformanceTests(BaseAction): def execute(self, render_context: RenderContext, _previous_action_payload: Any | None): conformance_tests_script = os.path.normpath(render_context.conformance_tests_script) - if render_context.module_name == render_context.conformance_tests_running_context.current_testing_module_name: - conformance_tests_folder_name = ( - render_context.conformance_tests_running_context.get_current_conformance_test_folder_name() - ) - else: - [conformance_tests_folder_name, _] = ( - render_context.conformance_tests.get_source_conformance_test_folder_name( - render_context.module_name, - render_context.required_modules, - render_context.conformance_tests_running_context.current_testing_module_name, - render_context.conformance_tests_running_context.get_current_conformance_test_folder_name(), - ) - ) + conformance_tests_folder_name = render_context.conformance_tests.get_module_suite_run_folder( + render_context.module_name, + render_context.required_modules, + render_context.conformance_tests_running_context.current_testing_module_name, + ) console.info( f"Running conformance tests script {conformance_tests_script} " - + f"for {conformance_tests_folder_name} (" - + f"functionality {render_context.conformance_tests_running_context.current_testing_frid} " - + f"in module {render_context.conformance_tests_running_context.current_testing_module_name}" - + ")." + + f"for the test suite {conformance_tests_folder_name} " + + f"of module {render_context.conformance_tests_running_context.current_testing_module_name}." ) exit_code, conformance_tests_issue, conformance_tests_temp_log_file_path = render_utils.execute_script( conformance_tests_script, @@ -54,21 +53,23 @@ def execute(self, render_context: RenderContext, _previous_action_payload: Any | ) render_context.script_execution_history.should_update_script_outputs = True - render_context.memory_manager.create_conformance_tests_memory( - render_context, exit_code, conformance_tests_issue - ) - if exit_code == 0: + render_context.memory_manager.create_conformance_tests_memory( + render_context, exit_code, conformance_tests_issue + ) + # A passing whole-suite run of the module being rendered covers the FRID being + # implemented, so its unresolved memory entries can be cleared. if ( render_context.conformance_tests_running_context.current_testing_module_name == render_context.module_name - and render_context.conformance_tests_running_context.current_testing_frid - == render_context.frid_context.frid ): render_context.memory_manager.delete_unresolved_memory_files() return self.SUCCESSFUL_OUTCOME, None if exit_code in UNRECOVERABLE_ERROR_EXIT_CODES: + render_context.memory_manager.create_conformance_tests_memory( + render_context, exit_code, conformance_tests_issue + ) console.error(conformance_tests_issue) return ( self.UNRECOVERABLE_ERROR_OUTCOME, @@ -80,4 +81,23 @@ def execute(self, render_context: RenderContext, _previous_action_payload: Any | ).to_payload(), ) - return self.FAILED_OUTCOME, {"previous_conformance_tests_issue": conformance_tests_issue} + if detect_layout_failure(conformance_tests_issue): + console.error(conformance_tests_issue) + return ( + self.UNRECOVERABLE_ERROR_OUTCOME, + RenderError.encode( + message=LAYOUT_FAILURE_MESSAGE, + error_type="ENVIRONMENT_ERROR", + script=conformance_tests_script, + issue=conformance_tests_issue, + ).to_payload(), + ) + + # Attribute the failure to a FRID (re-pointing the running context for the fix loop) + # before creating memory, so the memory entry is keyed to the implicated FRID. + conformance_tests_evidence = render_context.route_conformance_failure_to_frid(conformance_tests_issue) + render_context.memory_manager.create_conformance_tests_memory( + render_context, exit_code, conformance_tests_issue + ) + + return self.FAILED_OUTCOME, {"previous_conformance_tests_issue": conformance_tests_evidence} diff --git a/render_machine/actions/summarize_conformance_tests.py b/render_machine/actions/summarize_conformance_tests.py deleted file mode 100644 index 14e4d719..00000000 --- a/render_machine/actions/summarize_conformance_tests.py +++ /dev/null @@ -1,35 +0,0 @@ -from typing import Any - -from plain2code_console import console -from render_machine.actions.base_action import BaseAction -from render_machine.render_context import RenderContext - - -class SummarizeConformanceTests(BaseAction): - SUCCESSFUL_OUTCOME = "conformance_tests_summarized" - - def execute(self, render_context: RenderContext, _previous_action_payload: Any | None): - console.info(f"Summarizing conformance tests for functionality {render_context.frid_context.frid}.") - - _, existing_conformance_test_files_content = ( - render_context.conformance_tests.fetch_existing_conformance_test_files( - render_context.module_name, - render_context.required_modules, - render_context.conformance_tests_running_context.current_testing_module_name, - render_context.conformance_tests_running_context.get_current_conformance_test_folder_name(), - ) - ) - - summary = render_context.codeplain_api.summarize_finished_conformance_tests( - frid=render_context.frid_context.frid, - plain_source_tree=render_context.plain_source_tree, - linked_resources=render_context.frid_context.linked_resources, - conformance_test_files_content=existing_conformance_test_files_content, - module_name=render_context.module_name, - required_modules=render_context.get_required_modules_functionalities(), - run_state=render_context.run_state, - ) - - render_context.conformance_tests_running_context.set_conformance_tests_summary(summary) - - return self.SUCCESSFUL_OUTCOME, None diff --git a/render_machine/conformance_tests.py b/render_machine/conformance_tests.py index 6bce88c5..de0ed56a 100644 --- a/render_machine/conformance_tests.py +++ b/render_machine/conformance_tests.py @@ -1,3 +1,4 @@ +import difflib import json import os @@ -9,6 +10,18 @@ CONFORMANCE_TESTS_DEFINITION_FILE_NAME = "conformance_tests.json" +def _is_insertion_only_change(existing_content: str, new_content: str) -> bool: + """Check that new_content only inserts lines into existing_content. + + Every existing line must survive, in order - the diff opcodes may contain only + "equal" and "insert" operations. + """ + matcher = difflib.SequenceMatcher( + None, existing_content.splitlines(keepends=True), new_content.splitlines(keepends=True), autojunk=False + ) + return all(opcode in ("equal", "insert") for opcode, *_ in matcher.get_opcodes()) + + class ConformanceTests: """Manages the state of conformance tests.""" @@ -144,14 +157,43 @@ def store_conformance_tests_files( style=console.OUTPUT_STYLE, ) + def get_module_suite_run_folder( + self, + module_name: str, + required_modules: list[PlainModule], + current_testing_module_name: str, + ) -> str: + """Resolve the folder to pass to the conformance test script for a whole-module run. + + For the module being rendered this is its own conformance tests folder. For a + required module it is the most specific existing copy of that module's tests + (mirroring get_source_conformance_test_folder_name at module granularity), falling + back to the required module's own folder when no copy exists yet. + """ + if current_testing_module_name == module_name: + return self.get_module_conformance_tests_folder(module_name) + + modules_list = [module_name] + [m.module_name for m in reversed(required_modules)] + + for copy_from_module in modules_list: + if copy_from_module == current_testing_module_name: + break + + candidate = self.get_module_conformance_tests_folder(copy_from_module + "/." + current_testing_module_name) + if os.path.exists(candidate): + return candidate + + return self.get_module_conformance_tests_folder(current_testing_module_name) + def fetch_all_existing_conformance_test_files(self, module_name: str) -> dict[str, str]: """Fetch the content of all existing conformance test files of the module. Files are collected from every conformance test subfolder of the module (one subfolder per functional requirement) and keyed as "/" so each - file's suite remains identifiable. Hidden subfolders (copies of required modules' tests) - and the conformance tests definition file (stored at the module folder root) are not - included. Returns an empty dict when the module has no conformance tests yet. + file's suite remains identifiable, plus any shared setup files at the module folder + root (keyed by their bare file name). Hidden subfolders (copies of required modules' + tests) and the conformance tests definition file are not included. Returns an empty + dict when the module has no conformance tests yet. """ all_files_content: dict[str, str] = {} module_folder = self.get_module_conformance_tests_folder(module_name) @@ -162,8 +204,66 @@ def fetch_all_existing_conformance_test_files(self, module_name: str) -> dict[st for file_name, content in files_content.items(): all_files_content[os.path.join(folder_name, file_name)] = content + if os.path.isdir(module_folder): + root_file_names = [ + entry.name + for entry in os.scandir(module_folder) + if entry.is_file() and entry.name != self.conformance_tests_definition_file_name + ] + root_files_content = file_utils.get_existing_files_content(module_folder, sorted(root_file_names)) + all_files_content.update(root_files_content) + return all_files_content + def find_response_file_violations( + self, + module_name: str, + current_subfolder_name: str, + response_files: dict[str, str], + ) -> list[str]: + """Check a conformance tests render response against the two-tier immutability rule. + + Paths are relative to the module's conformance tests folder. Allowed: any file under + the current functionality's subfolder, new files anywhere outside other + functionalities' subfolders, and extensions of existing shared setup files at the + suite root that only insert lines. Violations: files in other functionalities' + subfolders (or required-module copies), the conformance tests definition file, and + root-file changes that delete or modify existing lines. + """ + violations = [] + module_folder = self.get_module_conformance_tests_folder(module_name) + known_suite_folders = set(self.fetch_existing_conformance_test_folder_names(module_name)) + + for file_name, content in response_files.items(): + path_parts = file_name.replace(os.sep, "/").split("/") + top_level_name = path_parts[0] + + if top_level_name == current_subfolder_name: + continue + + if len(path_parts) > 1: + if top_level_name.startswith("."): + violations.append(f"{file_name}: files of required modules' test copies must not be changed") + elif top_level_name in known_suite_folders: + violations.append(f"{file_name}: belongs to another functionality's test suite") + continue + + if file_name == self.conformance_tests_definition_file_name: + violations.append(f"{file_name}: the conformance tests definition file must not be changed") + continue + + existing_file_path = os.path.join(module_folder, file_name) + if os.path.exists(existing_file_path): + with open(existing_file_path, "r") as f: + existing_content = f.read() + if content is None or not _is_insertion_only_change(existing_content, content): + violations.append( + f"{file_name}: shared setup files may only be extended by adding lines - " + "existing lines must not be changed or removed" + ) + + return violations + def fetch_existing_conformance_test_files( self, module_name: str, diff --git a/render_machine/failure_attribution.py b/render_machine/failure_attribution.py new file mode 100644 index 00000000..a24e7b08 --- /dev/null +++ b/render_machine/failure_attribution.py @@ -0,0 +1,114 @@ +"""Helpers for attributing whole-suite conformance test failures to functionalities. + +When the conformance test script runs a module's whole test suite, a failure can involve +tests belonging to several functionalities (FRIDs). These helpers map the failure output +back to the implicated FRIDs, extract the failure evidence belonging to one FRID, and +detect layout-level failures (e.g. suites from projects generated before whole-suite +execution that cannot be discovered together). +""" + +import os +import re + +# Failure block delimiter used by Python's unittest output. +_UNITTEST_BLOCK_DELIMITER = "=" * 70 + +# A horizontal rule inside/after unittest failure blocks and before the run summary. +_UNITTEST_SUMMARY_RULE = "-" * 70 + +_TESTS_RAN_PATTERN = re.compile(r"Ran [1-9]\d* tests?") + +# Signatures of failures caused by the suite layout rather than by failing tests. Kept +# deliberately narrow: a legitimate test failure must never match. They are only +# consulted when the output shows that no test ran at all. +_LAYOUT_FAILURE_SIGNATURES = ( + # unittest discovery cannot import the start directory (missing package structure) + "Start directory is not importable", + # unittest discovery found two suites clashing on the same module name + "module incorrectly imported from", + # the python conformance script discovered no tests + "No unittests discovered", + # the golang/cypress conformance scripts found no suites to run + "No conformance test suites discovered", +) + + +def attribute_failures(output: str, conformance_tests_json: dict) -> list[str]: + """Return the FRIDs whose conformance test suite appears in the failure output. + + Matches each suite's folder basename against the output text. This is + language-agnostic: test identifiers and paths in runner output contain the suite + folder name (as a Python package, a path segment, or a suite header printed by the + conformance script). The result follows the order of conformance_tests_json entries, + which is spec order. + """ + implicated_frids = [] + for frid, entry in conformance_tests_json.items(): + folder_basename = os.path.basename(entry.get("folder_name", "")) + if folder_basename and folder_basename in output: + implicated_frids.append(frid) + + return implicated_frids + + +def extract_frid_failure_evidence(output: str, folder_basename: str) -> str: + """Extract the failure blocks belonging to one suite from the run output. + + Best-effort: understands Python unittest's "="-delimited failure blocks and keeps the + run summary at the end. Returns the full output unchanged when the format doesn't + cooperate (no delimiters, or no block mentions the suite) so no evidence is ever lost. + """ + parts = output.split(_UNITTEST_BLOCK_DELIMITER) + if len(parts) < 2: + return output + + blocks = parts[1:] + + # The run summary ("Ran N tests..." / "FAILED (failures=N)") trails the last block + # after a second horizontal rule. Split it off so it can be kept unconditionally. + run_summary = "" + last_block = blocks[-1] + first_rule_position = last_block.find(_UNITTEST_SUMMARY_RULE) + last_rule_position = last_block.rfind(_UNITTEST_SUMMARY_RULE) + if last_rule_position != -1 and last_rule_position != first_rule_position: + blocks[-1] = last_block[:last_rule_position] + run_summary = last_block[last_rule_position:] + + matching_blocks = [block for block in blocks if folder_basename in block] + if not matching_blocks: + return output + + return "".join(_UNITTEST_BLOCK_DELIMITER + block for block in matching_blocks) + run_summary + + +def format_other_frids_note(implicated_frids: list[str], current_frid: str) -> str: + """Summarize other implicated FRIDs for the fix prompt without exposing their traces. + + The fix call for one FRID must not carry raw failure details of tests whose files are + not in its context. This note preserves the diagnostic signal (several functionalities + failing at once points at the implementation code) while keeping the fix scoped. + """ + other_frids = [frid for frid in implicated_frids if frid != current_frid] + if not other_frids: + return "" + + return ( + "\nNote: conformance tests of the following other functionalities also failed in this run: " + + ", ".join(other_frids) + + ". They are being handled separately - do not attempt to fix them or reference their files." + + " Several functionalities failing at once usually indicates the root cause is in the" + + " implementation code rather than in the conformance tests." + ) + + +def detect_layout_failure(output: str) -> bool: + """Detect a failure caused by the suite layout rather than by failing tests. + + Conservative by design: returns True only when no test ran at all AND the output + carries a known layout-failure signature. A legitimate test failure (which always + reports at least one test run) never matches. + """ + if _TESTS_RAN_PATTERN.search(output): + return False + + return any(signature in output for signature in _LAYOUT_FAILURE_SIGNATURES) diff --git a/render_machine/render_context.py b/render_machine/render_context.py index b415aca1..08c9bb49 100644 --- a/render_machine/render_context.py +++ b/render_machine/render_context.py @@ -1,3 +1,4 @@ +import os.path import threading from copy import deepcopy from typing import Callable, Optional @@ -11,7 +12,7 @@ from plain2code_events import RenderContextSnapshot from plain2code_state import RunState from plain_modules import PlainModule -from render_machine import triggers +from render_machine import failure_attribution, triggers from render_machine.conformance_tests import CONFORMANCE_TESTS_DEFINITION_FILE_NAME, ConformanceTests from render_machine.render_types import ( AcceptanceTestPhase, @@ -182,78 +183,6 @@ def start_unittests_processing(self): self.unit_tests_running_context = UnitTestsRunningContext(fix_attempts=0) self.run_state.increment_unittest_batch_id() - def _get_first_frid_conformance_test_running_context(self, module: PlainModule | None): - conformance_tests_running_context = self.conformance_tests_running_context - - if module is None: - conformance_tests_running_context.current_testing_module_name = self.module_name - if not conformance_tests_running_context.conformance_tests_json_has_module_populated( - conformance_tests_running_context.current_testing_module_name - ): - conformance_tests_running_context.set_conformance_tests_json( - conformance_tests_running_context.current_testing_module_name, - {}, - ) - else: - conformance_tests_running_context.current_testing_module_name = module.module_name - conformance_tests_running_context.set_conformance_tests_json( - conformance_tests_running_context.current_testing_module_name, - self.conformance_tests.get_conformance_tests_json( - conformance_tests_running_context.current_testing_module_name - ), - ) - - if module is None: - conformance_tests_running_context.current_testing_frid = plain_spec.get_first_frid(self.plain_source_tree) - else: - conformance_tests_running_context.current_testing_frid = next( - iter( - conformance_tests_running_context.get_conformance_tests_json( - conformance_tests_running_context.current_testing_module_name - ) - ) - ) - - return conformance_tests_running_context - - def get_first_conformance_tests_running_context(self): - if self.required_modules is None or len(self.required_modules) == 0: - return self._get_first_frid_conformance_test_running_context(None) - else: - return self._get_first_frid_conformance_test_running_context(self.required_modules[0]) - - def get_next_conformance_tests_running_context(self): - conformance_tests_running_context = self.conformance_tests_running_context - if conformance_tests_running_context.current_testing_module_name == self.module_name: - conformance_tests_running_context.current_testing_frid = plain_spec.get_next_frid( - self.plain_source_tree, - self.conformance_tests_running_context.current_testing_frid, - ) - else: - all_frids = list( - conformance_tests_running_context.get_conformance_tests_json( - conformance_tests_running_context.current_testing_module_name - ).keys() - ) - current_index = all_frids.index(conformance_tests_running_context.current_testing_frid) - if current_index + 1 < len(all_frids): - conformance_tests_running_context.current_testing_frid = all_frids[current_index + 1] - else: - next_module_index = -1 - for i, required_module in enumerate(self.required_modules): - if required_module.module_name == conformance_tests_running_context.current_testing_module_name: - next_module_index = i + 1 - break - - if next_module_index < len(self.required_modules): - conformance_tests_running_context = self._get_first_frid_conformance_test_running_context( - self.required_modules[next_module_index] - ) - else: - conformance_tests_running_context = self._get_first_frid_conformance_test_running_context(None) - - return conformance_tests_running_context - def finish_unittests_processing(self): existing_files = file_utils.list_all_text_files(self.build_folder) @@ -313,15 +242,6 @@ def finish_conformance_tests_processing(self): # ========== Helper Methods for Conformance Test Execution ========== - def _should_run_current_frid_tests(self) -> bool: - """Check if we should run/continue testing the current FRID.""" - ctx = self.conformance_tests_running_context - return ( - ctx.execution_phase == TestExecutionPhase.TESTING_CURRENT_FRID - and ctx.current_testing_module_name == self.module_name - and ctx.current_testing_frid == ctx.frid_being_implemented - ) - def _has_more_acceptance_test_phases(self) -> bool: """Check if there are more acceptance test phases to run.""" ctx = self.conformance_tests_running_context @@ -343,25 +263,79 @@ def _start_regression_phase(self): ctx.code_changed_during_regression = False ctx.execution_phase = TestExecutionPhase.RUNNING_REGRESSION - ctx.current_testing_frid = None # Will be set by get_first_conformance_tests_running_context + ctx.regression_module_index = None # Will be advanced by _get_next_regression_module + + def _module_has_conformance_tests(self, module_name: str) -> bool: + return len(self.conformance_tests.get_conformance_tests_json(module_name)) > 0 + + def _get_next_regression_module(self) -> Optional[str]: + """Advance to the next module whose whole suite should run during regression. + + The regression sequence is every required module (in requires order). When code + changed while fixing a conformance test, the module being rendered is appended so + its suite gets re-verified against the changed code. + """ + ctx = self.conformance_tests_running_context + + module_names = [module.module_name for module in (self.required_modules or [])] + if ctx.code_changed_during_regression: + module_names = module_names + [self.module_name] + + next_index = 0 if ctx.regression_module_index is None else ctx.regression_module_index + 1 + while next_index < len(module_names): + module_name = module_names[next_index] + if module_name == self.module_name or self._module_has_conformance_tests(module_name): + ctx.regression_module_index = next_index + return module_name + next_index += 1 - def _get_next_test_to_run(self): - """Determine which test to run next based on current phase.""" + return None + + def _switch_to_module_suite(self, module_name: str): + """Point the running context at a module so its whole suite is the next run.""" ctx = self.conformance_tests_running_context - if ctx.current_testing_frid is None: - return self.get_first_conformance_tests_running_context() + if module_name == self.module_name: + ctx.current_testing_module_name = self.module_name + ctx.current_testing_frid = ctx.frid_being_implemented else: - return self.get_next_conformance_tests_running_context() + if not ctx.conformance_tests_json_has_module_populated(module_name): + ctx.set_conformance_tests_json( + module_name, self.conformance_tests.get_conformance_tests_json(module_name) + ) + ctx.current_testing_module_name = module_name + ctx.current_testing_frid = next(iter(ctx.get_conformance_tests_json(module_name))) - def _has_reached_implementation_frid(self) -> bool: - """Check if regression has reached the FRID being implemented.""" + self._setup_test_specifications() + + def route_conformance_failure_to_frid(self, conformance_tests_issue: str) -> str: + """Attribute a failed whole-suite run to a functionality and scope the evidence to it. + + Points the running context at the earliest implicated FRID (the fix loop, memory + creation, and conflict detection all key off current_testing_frid) and returns the + failure evidence for that FRID: its own failure blocks plus a summary note about + other implicated FRIDs. Returns the issue unchanged when no FRID can be identified. + """ ctx = self.conformance_tests_running_context - return ( - ctx.execution_phase == TestExecutionPhase.RUNNING_REGRESSION - and ctx.current_testing_module_name == self.module_name - and (ctx.current_testing_frid is None or ctx.current_testing_frid == ctx.frid_being_implemented) - ) + module_json = ctx.get_conformance_tests_json(ctx.current_testing_module_name) + + implicated_frids = failure_attribution.attribute_failures(conformance_tests_issue, module_json) + if not implicated_frids: + return conformance_tests_issue + + target_frid = implicated_frids[0] + if target_frid != ctx.current_testing_frid: + console.info( + f"Conformance test failure attributed to functionality {target_frid} " + f"in module {ctx.current_testing_module_name}." + ) + ctx.current_testing_frid = target_frid + self._setup_test_specifications() + + folder_basename = os.path.basename(module_json[target_frid]["folder_name"]) + evidence = failure_attribution.extract_frid_failure_evidence(conformance_tests_issue, folder_basename) + + return evidence + failure_attribution.format_other_frids_note(implicated_frids, target_frid) def _setup_test_specifications(self): """Load specifications for the current test.""" @@ -469,32 +443,23 @@ def _handle_current_frid_testing(self): raise RuntimeError(f"Unexpected acceptance test phase: {ctx.acceptance_test_phase}") def _handle_regression_testing(self): - """Handle regression testing of all earlier FRIDs.""" - - # Get next test to run - self.conformance_tests_running_context = self._get_next_test_to_run() + """Handle regression testing: one whole-suite run per module. - # Get reference to the updated context + The module's own suite already ran in full during the current-FRID phase, so + regression only needs the required modules' suites (plus a re-run of the own + module's suite when code changed while fixing a conformance test). + """ ctx = self.conformance_tests_running_context - # Set up specs and run test - self._setup_test_specifications() + next_module = self._get_next_regression_module() - if ctx.current_conformance_tests_exist(): - # Check if this is the implementation FRID (last test to run) - if self._has_reached_implementation_frid(): - # Reached implementation FRID - only re-run it if code changed during regression - if ctx.code_changed_during_regression: - # Code changed - run the implementation FRID again to verify no regression - # After it passes, mark as completed on next iteration - ctx.execution_phase = TestExecutionPhase.COMPLETED - else: - # No code changes - skip re-running implementation FRID, mark as completed immediately - ctx.execution_phase = TestExecutionPhase.COMPLETED - self.machine.dispatch(triggers.MARK_ALL_CONFORMANCE_TESTS_PASSED) - return + if next_module is None: + ctx.execution_phase = TestExecutionPhase.COMPLETED + self.machine.dispatch(triggers.MARK_ALL_CONFORMANCE_TESTS_PASSED) + return - self.machine.dispatch(triggers.MARK_CONFORMANCE_TESTS_READY) + self._switch_to_module_suite(next_module) + self.machine.dispatch(triggers.MARK_CONFORMANCE_TESTS_READY) # ========== Main Conformance Test Orchestration ========== @@ -534,7 +499,17 @@ def start_conformance_tests_for_frid(self): return # ========== STEP 3: Handle Current FRID Testing ========== - if self._should_run_current_frid_tests(): + if ctx.execution_phase == TestExecutionPhase.TESTING_CURRENT_FRID: + # A failed whole-suite run may have re-pointed current_testing_frid at the + # implicated FRID for the fix loop; restore the FRID being implemented before + # continuing the current-FRID phases. + if ( + ctx.current_testing_module_name != self.module_name + or ctx.current_testing_frid != ctx.frid_being_implemented + ): + ctx.current_testing_module_name = self.module_name + ctx.current_testing_frid = ctx.frid_being_implemented + self._setup_test_specifications() self._handle_current_frid_testing() return diff --git a/render_machine/render_types.py b/render_machine/render_types.py index 44cda7a7..33bde3e8 100644 --- a/render_machine/render_types.py +++ b/render_machine/render_types.py @@ -83,6 +83,8 @@ def __init__( self.execution_phase: TestExecutionPhase = TestExecutionPhase.TESTING_CURRENT_FRID self.acceptance_test_phase: AcceptanceTestPhase = AcceptanceTestPhase.NOT_STARTED self.acceptance_tests_completed: int = 0 + # Index into the regression module sequence; None means regression has not started. + self.regression_module_index: Optional[int] = None self.frid_being_implemented: Optional[str] = frid_being_implemented self.test_that_triggered_code_change: Optional[tuple[str, str]] = None self.code_changed_during_regression: bool = False @@ -134,11 +136,6 @@ def get_current_acceptance_test(self) -> Optional[str]: return None return acceptance_tests[self.acceptance_tests_completed - 1] - def set_conformance_tests_summary(self, summary: list[dict]): - self.get_conformance_tests_json(self.current_testing_module_name)[self.current_testing_frid][ - "test_summary" - ] = summary - @dataclass class ScriptExecutionHistory: diff --git a/render_machine/state_machine_config.py b/render_machine/state_machine_config.py index 365e1d3a..de8833e3 100644 --- a/render_machine/state_machine_config.py +++ b/render_machine/state_machine_config.py @@ -24,7 +24,6 @@ from render_machine.actions.render_functional_requirement import RenderFunctionalRequirement from render_machine.actions.run_conformance_tests import RunConformanceTests from render_machine.actions.run_unit_tests import RunUnitTests -from render_machine.actions.summarize_conformance_tests import SummarizeConformanceTests from render_machine.render_context import RenderContext from render_machine.states import States @@ -52,7 +51,6 @@ def get_action_map(self) -> Dict[str, Any]: f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TEST_GENERATED.value}": PrepareTestingEnvironment(), f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TEST_ENV_PREPARED.value}": RunConformanceTests(), f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TEST_FAILED.value}": FixConformanceTest(), - f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.POSTPROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TESTS_READY_FOR_SUMMARY.value}": SummarizeConformanceTests(), f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.POSTPROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TESTS_READY_FOR_COMMIT.value}": CommitConformanceTestsChanges( git_utils.CONFORMANCE_TESTS_PASSED_COMMIT_MESSAGE, git_utils.FUNCTIONAL_REQUIREMENT_FINISHED_COMMIT_MESSAGE, @@ -85,6 +83,7 @@ def get_action_result_triggers_map(self) -> Dict[str, str]: FinishFunctionalRequirement.SUCCESSFUL_OUTCOME: triggers.PROCEED_FRID_PROCESSING, CreateDist.SUCCESSFUL_OUTCOME: triggers.FINISH_RENDER, RenderConformanceTests.SUCCESSFUL_OUTCOME: triggers.MARK_CONFORMANCE_TESTS_READY, + RenderConformanceTests.RESPONSE_VALIDATION_FAILED_OUTCOME: triggers.HANDLE_ERROR, PrepareTestingEnvironment.SUCCESSFUL_OUTCOME: triggers.MARK_TESTING_ENVIRONMENT_PREPARED, PrepareTestingEnvironment.FAILED_OUTCOME: triggers.HANDLE_ERROR, RunConformanceTests.SUCCESSFUL_OUTCOME: triggers.MOVE_TO_NEXT_CONFORMANCE_TEST, @@ -96,7 +95,6 @@ def get_action_result_triggers_map(self) -> Dict[str, str]: FixConformanceTest.REGENERATE_CONFORMANCE_TESTS_OUTCOME: triggers.MARK_REGENERATION_OF_CONFORMANCE_TESTS, CommitConformanceTestsChanges.SUCCESSFUL_OUTCOME_IMPLEMENTATION_UPDATED: triggers.MARK_NEXT_CONFORMANCE_TESTS_POSTPROCESSING_STEP, CommitConformanceTestsChanges.SUCCESSFUL_OUTCOME_IMPLEMENTATION_NOT_UPDATED: triggers.PROCEED_FRID_PROCESSING, - SummarizeConformanceTests.SUCCESSFUL_OUTCOME: triggers.MARK_NEXT_CONFORMANCE_TESTS_POSTPROCESSING_STEP, AnalyzeSpecificationAmbiguity.SUCCESSFUL_OUTCOME: triggers.PROCEED_FRID_PROCESSING, } @@ -120,9 +118,8 @@ def get_processing_unit_tests_states( def get_postprocessing_conformance_tests_states(self) -> Dict[str, Any]: return { "name": States.POSTPROCESSING_CONFORMANCE_TESTS.value, - "initial": States.CONFORMANCE_TESTS_READY_FOR_SUMMARY.value, + "initial": States.CONFORMANCE_TESTS_READY_FOR_COMMIT.value, "children": [ - States.CONFORMANCE_TESTS_READY_FOR_SUMMARY.value, States.CONFORMANCE_TESTS_READY_FOR_COMMIT.value, States.CONFORMANCE_TESTS_READY_FOR_AMBIGUITY_ANALYSIS.value, ], @@ -278,11 +275,6 @@ def get_transitions(self, render_context: RenderContext) -> List[Dict[str, Any]] "trigger": triggers.MARK_ALL_CONFORMANCE_TESTS_PASSED, "dest": f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.POSTPROCESSING_CONFORMANCE_TESTS.value}", }, - { - "source": f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.POSTPROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TESTS_READY_FOR_SUMMARY.value}", - "trigger": triggers.MARK_NEXT_CONFORMANCE_TESTS_POSTPROCESSING_STEP, - "dest": f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.POSTPROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TESTS_READY_FOR_COMMIT.value}", - }, { "source": f"{States.IMPLEMENTING_FRID.value}_{States.PROCESSING_CONFORMANCE_TESTS.value}_{States.POSTPROCESSING_CONFORMANCE_TESTS.value}_{States.CONFORMANCE_TESTS_READY_FOR_COMMIT.value}", "trigger": triggers.MARK_NEXT_CONFORMANCE_TESTS_POSTPROCESSING_STEP, diff --git a/render_machine/states.py b/render_machine/states.py index 86713853..cfd32ebd 100644 --- a/render_machine/states.py +++ b/render_machine/states.py @@ -44,7 +44,6 @@ class States(Enum): # Postprocessing conformance tests states POSTPROCESSING_CONFORMANCE_TESTS = "postprocessingConformanceTests" - CONFORMANCE_TESTS_READY_FOR_SUMMARY = "conformanceTestsReadyForSummary" CONFORMANCE_TESTS_READY_FOR_COMMIT = "conformanceTestsReadyForCommit" CONFORMANCE_TESTS_READY_FOR_AMBIGUITY_ANALYSIS = "conformanceTestsReadyForAmbiguityAnalysis" diff --git a/test_scripts/run_conformance_tests_cypress.ps1 b/test_scripts/run_conformance_tests_cypress.ps1 index 2eedd252..f63a6373 100755 --- a/test_scripts/run_conformance_tests_cypress.ps1 +++ b/test_scripts/run_conformance_tests_cypress.ps1 @@ -256,80 +256,143 @@ try { # Move back to the original directory Set-Location $current_dir + if (-not (Test-Path $ConformanceTestsFolder)) { + Write-Host "Error: Conformance tests folder '$ConformanceTestsFolder' does not exist." + exit $UNRECOVERABLE_ERROR_EXIT_CODE + } + # Define the path to the conformance tests subfolder $script:NODE_CONFORMANCE_TESTS_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "node_$(Split-Path $ConformanceTestsFolder -Leaf)" - if ($env:VERBOSE -eq "1") { - Write-Host "Preparing conformance tests Node subfolder: $script:NODE_CONFORMANCE_TESTS_SUBFOLDER" - } - - # Check if the conformance tests node subfolder exists - if (Test-Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER) { - # Delete all files and folders except "node_modules", "plain_modules", and "package-lock.json" - Get-ChildItem -Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | - Where-Object { - $_.Name -ne "node_modules" -and - $_.Name -ne "plain_modules" -and - $_.Name -ne "package-lock.json" - } | Remove-Item -Recurse -Force + # Stage a single conformance test suite into the scratch subfolder and run + # it. Returns the cypress run exit code. + function Invoke-CypressSuite { + param([string]$SuiteFolder) if ($env:VERBOSE -eq "1") { - Write-Host "Cleanup completed, keeping 'node_modules' and 'package-lock.json'." + Write-Host "Preparing conformance tests Node subfolder: $script:NODE_CONFORMANCE_TESTS_SUBFOLDER" } - } else { - if ($env:VERBOSE -eq "1") { - Write-Host "Subfolder does not exist. Creating it..." + + # Check if the conformance tests node subfolder exists + if (Test-Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER) { + # Delete all files and folders except "node_modules", "plain_modules", and "package-lock.json" + Get-ChildItem -Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | + Where-Object { + $_.Name -ne "node_modules" -and + $_.Name -ne "plain_modules" -and + $_.Name -ne "package-lock.json" + } | Remove-Item -Recurse -Force + + if ($env:VERBOSE -eq "1") { + Write-Host "Cleanup completed, keeping 'node_modules' and 'package-lock.json'." + } + } else { + if ($env:VERBOSE -eq "1") { + Write-Host "Subfolder does not exist. Creating it..." + } + + New-Item -ItemType Directory -Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | Out-Null } - New-Item -ItemType Directory -Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Force | Out-Null - } + Copy-Item -Path "$SuiteFolder/*" -Destination $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Recurse -Force - Copy-Item -Path "$ConformanceTestsFolder/*" -Destination $script:NODE_CONFORMANCE_TESTS_SUBFOLDER -Recurse -Force + # Move to the subfolder with Cypress tests + if (-not (Test-Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER)) { + Write-Host "Error: conformance tests Node folder '$script:NODE_CONFORMANCE_TESTS_SUBFOLDER' does not exist." + exit $UNRECOVERABLE_ERROR_EXIT_CODE + } - # Move to the subfolder with Cypress tests - if (-not (Test-Path $script:NODE_CONFORMANCE_TESTS_SUBFOLDER)) { - Write-Host "Error: conformance tests Node folder '$script:NODE_CONFORMANCE_TESTS_SUBFOLDER' does not exist." - exit $UNRECOVERABLE_ERROR_EXIT_CODE - } + Push-Location $script:NODE_CONFORMANCE_TESTS_SUBFOLDER - Push-Location $script:NODE_CONFORMANCE_TESTS_SUBFOLDER + try { + # Temporarily allow stderr output without throwing (npm may write warnings to stderr) + # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting + $script:ErrorActionPreference = 'Continue' + $npmInstallOutput = npm install cypress --save-dev --prefer-offline --no-audit --no-fund --loglevel error 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String + $script:ErrorActionPreference = 'Stop' + $npmInstallOutput -split "`n" | Where-Object { $_ -notmatch $NPM_INSTALL_OUTPUT_FILTER } | ForEach-Object { + if ($_.Trim()) { Write-Host $_ } + } - # Temporarily allow stderr output without throwing (npm may write warnings to stderr) - # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting - $ErrorActionPreference = 'Continue' - $npmInstallOutput = npm install cypress --save-dev --prefer-offline --no-audit --no-fund --loglevel error 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String - $ErrorActionPreference = 'Stop' - $npmInstallOutput -split "`n" | Where-Object { $_ -notmatch $NPM_INSTALL_OUTPUT_FILTER } | ForEach-Object { - if ($_.Trim()) { Write-Host $_ } - } + if ($env:VERBOSE -eq "1") { + Write-Host "Running Cypress conformance tests..." + } - if ($env:VERBOSE -eq "1") { - Write-Host "Running Cypress conformance tests..." + $script:ErrorActionPreference = 'Continue' + $cypress_info_output = npx cypress info 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String + $script:ErrorActionPreference = 'Stop' + $CYPRESS_BROWSER_FLAG = "" + if ($cypress_info_output -match "(?i)chrome") { + $CYPRESS_BROWSER_FLAG = "--browser=chrome" + } + Write-Host "CYPRESS_BROWSER_FLAG: $(if ($CYPRESS_BROWSER_FLAG) { $CYPRESS_BROWSER_FLAG } else { 'none' })" + + $env:BROWSERSLIST_IGNORE_OLD_DATA = "1" + if ($CYPRESS_BROWSER_FLAG) { + npx cypress run $CYPRESS_BROWSER_FLAG --config video=false 2>$null + } else { + npx cypress run --config video=false 2>$null + } + return $LASTEXITCODE + } finally { + Pop-Location + } } - $ErrorActionPreference = 'Continue' - $cypress_info_output = npx cypress info 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String - $ErrorActionPreference = 'Stop' - $CYPRESS_BROWSER_FLAG = "" - if ($cypress_info_output -match "(?i)chrome") { - $CYPRESS_BROWSER_FLAG = "--browser=chrome" + function Test-HasCypressConfig { + param([string]$Folder) + return [bool](Get-ChildItem -Path $Folder -Filter "cypress.config.*" -File -ErrorAction SilentlyContinue) } - Write-Host "CYPRESS_BROWSER_FLAG: $(if ($CYPRESS_BROWSER_FLAG) { $CYPRESS_BROWSER_FLAG } else { 'none' })" - $env:BROWSERSLIST_IGNORE_OLD_DATA = "1" - if ($CYPRESS_BROWSER_FLAG) { - npx cypress run $CYPRESS_BROWSER_FLAG --config video=false 2>$null - } else { - npx cypress run --config video=false 2>$null + if (Test-HasCypressConfig $ConformanceTestsFolder) { + # Single conformance test suite ($ConformanceTestsFolder is the suite folder itself). + $cypress_run_result = Invoke-CypressSuite $ConformanceTestsFolder + + if ($cypress_run_result -ne 0) { + if ($env:VERBOSE -eq "1") { + Write-Host "Error: Cypress conformance tests have failed." + } + exit 1 + } + + exit 0 } - $cypress_run_result = $LASTEXITCODE - if ($cypress_run_result -ne 0) { - if ($env:VERBOSE -eq "1") { - Write-Host "Error: Cypress conformance tests have failed." + # $ConformanceTestsFolder is a folder of conformance test suites: run every + # non-hidden subfolder that contains a cypress config file. + $suites_run = 0 + $aggregated_exit_code = 0 + + $suite_folders = Get-ChildItem -Path $ConformanceTestsFolder -Directory | + Where-Object { -not $_.Name.StartsWith(".") } | + Sort-Object Name + + foreach ($suite in $suite_folders) { + if (-not (Test-HasCypressConfig $suite.FullName)) { + continue + } + + Write-Host "=== conformance suite: $($suite.Name) ===" + + $cypress_run_result = Invoke-CypressSuite $suite.FullName + $suites_run += 1 + + # Keep running the remaining suites so the full set of failures is + # reported, but exit non-zero if any suite failed. + if ($cypress_run_result -ne 0) { + if ($env:VERBOSE -eq "1") { + Write-Host "Error: Cypress conformance tests have failed." + } + $aggregated_exit_code = 1 } + } + + if ($suites_run -eq 0) { + Write-Host "`nError: No conformance test suites discovered." exit 1 } + + exit $aggregated_exit_code } finally { Cleanup } diff --git a/test_scripts/run_conformance_tests_cypress.sh b/test_scripts/run_conformance_tests_cypress.sh index f3e41109..72252a97 100755 --- a/test_scripts/run_conformance_tests_cypress.sh +++ b/test_scripts/run_conformance_tests_cypress.sh @@ -209,51 +209,122 @@ printf "### Step 2: Running Cypress conformance tests $2...\n" # Move back to the original directory cd $current_dir -# Define the path to the conformance tests subfolder -NODE_CONFORMANCE_TESTS_SUBFOLDER="/tmp/node_$(basename "$2")" +# Resolve the conformance tests folder to an absolute path +CONFORMANCE_TESTS_FOLDER=$(cd "$2" 2>/dev/null && pwd) -if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then - printf "Preparing conformance tests Node subfolder: $NODE_CONFORMANCE_TESTS_SUBFOLDER\n" +if [ -z "$CONFORMANCE_TESTS_FOLDER" ]; then + printf "Error: Conformance tests folder '$2' does not exist.\n" + exit $UNRECOVERABLE_ERROR_EXIT_CODE fi -# Check if the conformance tests node subfolder exists -if [ -d "$NODE_CONFORMANCE_TESTS_SUBFOLDER" ]; then - # Find and delete all files and folders except "node_modules", "plain_modules", and "package-lock.json" - find "$NODE_CONFORMANCE_TESTS_SUBFOLDER" -mindepth 1 ! -path "$NODE_CONFORMANCE_TESTS_SUBFOLDER/node_modules*" ! -path "$NODE_CONFORMANCE_TESTS_SUBFOLDER/plain_modules*" ! -name "package-lock.json" -exec rm -rf {} + +# Define the path to the conformance tests subfolder +NODE_CONFORMANCE_TESTS_SUBFOLDER="/tmp/node_$(basename "$2")" + +# Stage a single conformance test suite into the scratch subfolder and run it. +# Returns the cypress run exit code (or exits the script on environment errors). +stage_and_run_suite() { + suite_folder="$1" if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then - printf "Cleanup completed, keeping 'node_modules' and 'package-lock.json'.\n" + printf "Preparing conformance tests Node subfolder: $NODE_CONFORMANCE_TESTS_SUBFOLDER\n" fi -else + + # Check if the conformance tests node subfolder exists + if [ -d "$NODE_CONFORMANCE_TESTS_SUBFOLDER" ]; then + # Find and delete all files and folders except "node_modules", "plain_modules", and "package-lock.json" + find "$NODE_CONFORMANCE_TESTS_SUBFOLDER" -mindepth 1 ! -path "$NODE_CONFORMANCE_TESTS_SUBFOLDER/node_modules*" ! -path "$NODE_CONFORMANCE_TESTS_SUBFOLDER/plain_modules*" ! -name "package-lock.json" -exec rm -rf {} + + + if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then + printf "Cleanup completed, keeping 'node_modules' and 'package-lock.json'.\n" + fi + else + if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then + printf "Subfolder does not exist. Creating it...\n" + fi + + mkdir -p $NODE_CONFORMANCE_TESTS_SUBFOLDER + fi + + cp -R "$suite_folder"/* $NODE_CONFORMANCE_TESTS_SUBFOLDER + + # Move to the subfolder with Cypress tests + cd "$NODE_CONFORMANCE_TESTS_SUBFOLDER" 2>/dev/null + + if [ $? -ne 0 ]; then + printf "Error: conformance tests Node folder '$NODE_CONFORMANCE_TESTS_SUBFOLDER' does not exist.\n" + exit $UNRECOVERABLE_ERROR_EXIT_CODE + fi + + npm install cypress --save-dev --prefer-offline --no-audit --no-fund --loglevel error | grep -Ev "$NPM_INSTALL_OUTPUT_FILTER" + if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then - printf "Subfolder does not exist. Creating it...\n" + printf "Running Cypress conformance tests...\n" fi - mkdir -p $NODE_CONFORMANCE_TESTS_SUBFOLDER -fi + BROWSERSLIST_IGNORE_OLD_DATA=1 npx cypress run --browser=chrome --config video=false 2>/dev/null + cypress_run_result=$? -cp -R $2/* $NODE_CONFORMANCE_TESTS_SUBFOLDER + # Move back to the original directory before the next suite + cd $current_dir -# Move to the subfolder with Cypress tests -cd "$NODE_CONFORMANCE_TESTS_SUBFOLDER" 2>/dev/null + return $cypress_run_result +} -if [ $? -ne 0 ]; then - printf "Error: conformance tests Node folder '$NODE_CONFORMANCE_TESTS_SUBFOLDER' does not exist.\n" - exit $UNRECOVERABLE_ERROR_EXIT_CODE -fi +has_cypress_config() { + ls "$1"/cypress.config.* >/dev/null 2>&1 +} -npm install cypress --save-dev --prefer-offline --no-audit --no-fund --loglevel error | grep -Ev "$NPM_INSTALL_OUTPUT_FILTER" +if has_cypress_config "$CONFORMANCE_TESTS_FOLDER"; then + # Single conformance test suite ("$2" is the suite folder itself). + stage_and_run_suite "$CONFORMANCE_TESTS_FOLDER" + cypress_run_result=$? -if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then - printf "Running Cypress conformance tests...\n" + if [ $cypress_run_result -ne 0 ]; then + if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then + printf "Error: Cypress conformance tests have failed.\n" + fi + exit 1 + fi + + exit 0 fi -BROWSERSLIST_IGNORE_OLD_DATA=1 npx cypress run --browser=chrome --config video=false 2>/dev/null -cypress_run_result=$? +# "$2" is a folder of conformance test suites: run every non-hidden subfolder +# that contains a cypress config file. +suites_run=0 +aggregated_exit_code=0 -if [ $cypress_run_result -ne 0 ]; then - if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then - printf "Error: Cypress conformance tests have failed.\n" +for suite_folder in "$CONFORMANCE_TESTS_FOLDER"/*/; do + suite_name=$(basename "$suite_folder") + + case "$suite_name" in + .*) continue ;; + esac + + if ! has_cypress_config "${suite_folder%/}"; then + continue fi + + printf "=== conformance suite: %s ===\n" "$suite_name" + + stage_and_run_suite "${suite_folder%/}" + cypress_run_result=$? + + suites_run=$((suites_run + 1)) + + # Keep running the remaining suites so the full set of failures is reported, + # but exit non-zero if any suite failed. + if [ $cypress_run_result -ne 0 ]; then + if [ "${VERBOSE:-}" -eq 1 ] 2>/dev/null; then + printf "Error: Cypress conformance tests have failed.\n" + fi + aggregated_exit_code=1 + fi +done + +if [ $suites_run -eq 0 ]; then + printf "\nError: No conformance test suites discovered.\n" exit 1 -fi \ No newline at end of file +fi + +exit $aggregated_exit_code diff --git a/test_scripts/run_conformance_tests_golang.ps1 b/test_scripts/run_conformance_tests_golang.ps1 index c3a3d35f..c7f69a97 100755 --- a/test_scripts/run_conformance_tests_golang.ps1 +++ b/test_scripts/run_conformance_tests_golang.ps1 @@ -29,6 +29,11 @@ if (-not [System.IO.Path]::IsPathRooted($ConformanceTestsFolder)) { $ConformanceTestsFolder = Join-Path $current_dir $ConformanceTestsFolder } +if (-not (Test-Path $ConformanceTestsFolder)) { + Write-Host "Error: Conformance tests folder '$ConformanceTestsFolder' does not exist." + exit $UNRECOVERABLE_ERROR_EXIT_CODE +} + $GO_BUILD_SUBFOLDER = Join-Path ([System.IO.Path]::GetTempPath()) "go_$(Split-Path $BuildFolder -Leaf)" if ($env:VERBOSE -eq "1") { @@ -59,48 +64,85 @@ if (-not (Test-Path $GO_BUILD_SUBFOLDER)) { exit $UNRECOVERABLE_ERROR_EXIT_CODE } +# Run a single conformance test suite. Expects the current working directory +# to be the build subfolder. Returns the suite's exit code. +function Invoke-ConformanceSuite { + param([string]$SuiteFolder) + + if (Test-Path (Join-Path $SuiteFolder "go.mod")) { + Write-Host "Running go get in conformance test directory..." + Push-Location $SuiteFolder + try { + go get + } finally { + Pop-Location + } + } else { + Write-Host "No go.mod found in conformance test directory, skipping go get" + } + + # Temporarily allow stderr output without throwing (Go may write to stderr) + # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting + $script:ErrorActionPreference = 'Continue' + $output = go run (Join-Path $SuiteFolder "conformance_tests.go") 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String + $suite_exit_code = $LASTEXITCODE + $script:ErrorActionPreference = 'Stop' + + # If there was an error, print the output + if ($suite_exit_code -ne 0) { + Write-Host $output + } + + return $suite_exit_code +} + Push-Location $GO_BUILD_SUBFOLDER try { Write-Host "Runinng go get in the build folder..." go get - # Move to conformance tests folder - Set-Location $ConformanceTestsFolder - if ($LASTEXITCODE -and $LASTEXITCODE -ne 0) { - Write-Host "Error: Conformance tests folder '$ConformanceTestsFolder' does not exist." - exit $UNRECOVERABLE_ERROR_EXIT_CODE - } + # Execute Go lang conformance tests + Write-Host "Running Golang conformance tests...`n" - Write-Host "Checking for go.mod in conformance test directory..." - if (Test-Path "go.mod") { - Write-Host "Running go get in conformance test directory..." - go get - } else { - Write-Host "No go.mod found in conformance test directory, skipping go get" + if (Test-Path (Join-Path $ConformanceTestsFolder "conformance_tests.go")) { + # Single conformance test suite ($ConformanceTestsFolder is the suite folder itself). + $exit_code = Invoke-ConformanceSuite $ConformanceTestsFolder + exit $exit_code } - # Move back to build directory - Set-Location $GO_BUILD_SUBFOLDER + # $ConformanceTestsFolder is a folder of conformance test suites: run every + # non-hidden subfolder that contains a conformance_tests.go file. + $suites_run = 0 + $aggregated_exit_code = 0 - # Execute Go lang conformance tests - Write-Host "Running Golang conformance tests...`n" + $suite_folders = Get-ChildItem -Path $ConformanceTestsFolder -Directory | + Where-Object { -not $_.Name.StartsWith(".") } | + Sort-Object Name - # Temporarily allow stderr output without throwing (Go may write to stderr) - # ForEach-Object converts ErrorRecord objects (from stderr) to plain strings to avoid verbose error formatting - $ErrorActionPreference = 'Continue' - $output = go run (Join-Path $ConformanceTestsFolder "conformance_tests.go") 2>&1 | ForEach-Object { if ($_ -is [System.Management.Automation.ErrorRecord]) { $_.Exception.Message } else { $_ } } | Out-String - $exit_code = $LASTEXITCODE - $ErrorActionPreference = 'Stop' + foreach ($suite in $suite_folders) { + if (-not (Test-Path (Join-Path $suite.FullName "conformance_tests.go"))) { + continue + } - # If there was an error, print the output and exit with the error code - if ($exit_code -ne 0) { - Write-Host $output - exit $exit_code + Write-Host "=== conformance suite: $($suite.Name) ===" + + $suite_exit_code = Invoke-ConformanceSuite $suite.FullName + $suites_run += 1 + + # Keep running the remaining suites so the full set of failures is + # reported, but exit with the first failing suite's exit code. + if ($suite_exit_code -ne 0 -and $aggregated_exit_code -eq 0) { + $aggregated_exit_code = $suite_exit_code + } + } + + if ($suites_run -eq 0) { + Write-Host "`nError: No conformance test suites discovered." + exit 1 } - # Exit with the exit code of the test command - exit $exit_code + exit $aggregated_exit_code } finally { Pop-Location if (Test-Path $GO_BUILD_SUBFOLDER) { diff --git a/test_scripts/run_conformance_tests_golang.sh b/test_scripts/run_conformance_tests_golang.sh index 14789621..12233a0a 100755 --- a/test_scripts/run_conformance_tests_golang.sh +++ b/test_scripts/run_conformance_tests_golang.sh @@ -16,6 +16,15 @@ if [ -z "$2" ]; then exit $UNRECOVERABLE_ERROR_EXIT_CODE fi +# Resolve the conformance tests folder to an absolute path so it can be used +# from the build subfolder (where we'll cd to next). +CONFORMANCE_TESTS_FOLDER=$(cd "$2" 2>/dev/null && pwd) + +if [ -z "$CONFORMANCE_TESTS_FOLDER" ]; then + printf "Error: Conformance tests folder '$2' does not exist.\n" + exit $UNRECOVERABLE_ERROR_EXIT_CODE +fi + GO_BUILD_SUBFOLDER="/tmp/go_$(basename "$1")" trap 'rm -rf "$GO_BUILD_SUBFOLDER"' EXIT @@ -53,35 +62,70 @@ fi echo "Runinng go get in the build folder..." go get -cd "$2" 2>/dev/null +# Run a single conformance test suite located in $1. Expects the current +# working directory to be the build subfolder. Returns the suite's exit code. +run_suite() { + suite_folder="$1" -if [ $? -ne 0 ]; then - printf "Error: Conformance tests folder '$2' does not exist.\n" - exit $UNRECOVERABLE_ERROR_EXIT_CODE -fi - -echo "Checking for go.mod in conformance test directory..." -if [ -f "go.mod" ]; then + if [ -f "$suite_folder/go.mod" ]; then echo "Running go get in conformance test directory..." - go get -else + (cd "$suite_folder" && go get) + else echo "No go.mod found in conformance test directory, skipping go get" -fi + fi -# Move back to build directory -cd "$GO_BUILD_SUBFOLDER" 2>/dev/null + output=$(go run "$suite_folder/conformance_tests.go" 2>&1) + suite_exit_code=$? + + # If there was an error, print the output + if [ $suite_exit_code -ne 0 ]; then + echo "$output" + fi + + return $suite_exit_code +} -# Execute Go lang conformance tests printf "Running Golang conformance tests...\n\n" -output=$(go run "$2/conformance_tests.go" 2>&1) -exit_code=$? +if [ -f "$CONFORMANCE_TESTS_FOLDER/conformance_tests.go" ]; then + # Single conformance test suite ("$2" is the suite folder itself). + run_suite "$CONFORMANCE_TESTS_FOLDER" + exit $? +fi + +# "$2" is a folder of conformance test suites: run every non-hidden subfolder +# that contains a conformance_tests.go file. +suites_run=0 +aggregated_exit_code=0 -# If there was an error, print the output and exit with the error code -if [ $exit_code -ne 0 ]; then - echo "$output" - exit $exit_code +for suite_folder in "$CONFORMANCE_TESTS_FOLDER"/*/; do + suite_name=$(basename "$suite_folder") + + case "$suite_name" in + .*) continue ;; + esac + + if [ ! -f "$suite_folder/conformance_tests.go" ]; then + continue + fi + + printf "=== conformance suite: %s ===\n" "$suite_name" + + run_suite "${suite_folder%/}" + suite_exit_code=$? + + suites_run=$((suites_run + 1)) + + # Keep running the remaining suites so the full set of failures is reported, + # but exit with the first failing suite's exit code. + if [ $suite_exit_code -ne 0 ] && [ $aggregated_exit_code -eq 0 ]; then + aggregated_exit_code=$suite_exit_code + fi +done + +if [ $suites_run -eq 0 ]; then + printf "\nError: No conformance test suites discovered.\n" + exit 1 fi -# Echo the original exit code of the unittest command -exit $exit_code \ No newline at end of file +exit $aggregated_exit_code diff --git a/tests/test_conformance_tests.py b/tests/test_conformance_tests.py index 69d75222..8a912c3c 100644 --- a/tests/test_conformance_tests.py +++ b/tests/test_conformance_tests.py @@ -76,3 +76,107 @@ def test_fetch_all_existing_conformance_test_files_skips_binary_files(conformanc files_content = conformance_tests.fetch_all_existing_conformance_test_files(MODULE_NAME) assert files_content == {os.path.join("some_functionality", "test_some.py"): "some test"} + + +class _FakeModule: + def __init__(self, module_name): + self.module_name = module_name + + +def test_get_module_suite_run_folder_own_module(conformance_tests, conformance_tests_dir): + folder = conformance_tests.get_module_suite_run_folder(MODULE_NAME, [], MODULE_NAME) + + assert folder == os.path.join(conformance_tests_dir, MODULE_NAME) + + +def test_get_module_suite_run_folder_required_module_with_copy(conformance_tests, conformance_tests_dir): + copy_folder = os.path.join(conformance_tests_dir, MODULE_NAME, ".required_module") + _write_file(copy_folder, os.path.join("some_frid", "test_x.py"), "copied test") + + folder = conformance_tests.get_module_suite_run_folder( + MODULE_NAME, [_FakeModule("required_module")], "required_module" + ) + + assert folder == copy_folder + + +def test_get_module_suite_run_folder_required_module_without_copy(conformance_tests, conformance_tests_dir): + folder = conformance_tests.get_module_suite_run_folder( + MODULE_NAME, [_FakeModule("required_module")], "required_module" + ) + + assert folder == os.path.join(conformance_tests_dir, "required_module") + + +def test_fetch_all_existing_conformance_test_files_includes_shared_root_files(conformance_tests, conformance_tests_dir): + module_folder = os.path.join(conformance_tests_dir, MODULE_NAME) + _write_file(module_folder, os.path.join("some_functionality", "test_some.py"), "some test") + _write_file(module_folder, "shared_helpers.py", "shared helper") + _write_file(module_folder, CONFORMANCE_TESTS_DEFINITION_FILE_NAME, json.dumps({})) + + files_content = conformance_tests.fetch_all_existing_conformance_test_files(MODULE_NAME) + + assert files_content == { + os.path.join("some_functionality", "test_some.py"): "some test", + "shared_helpers.py": "shared helper", + } + + +def test_find_response_file_violations_allows_own_subfolder_and_new_root_files( + conformance_tests, conformance_tests_dir +): + module_folder = os.path.join(conformance_tests_dir, MODULE_NAME) + _write_file(module_folder, os.path.join("earlier_suite", "test_earlier.py"), "earlier") + + violations = conformance_tests.find_response_file_violations( + MODULE_NAME, + "current_suite", + { + os.path.join("current_suite", "test_new.py"): "new test", + "shared_helpers.py": "brand new helper", + os.path.join("support", "util.py"): "new shared dir file", + }, + ) + + assert violations == [] + + +def test_find_response_file_violations_rejects_other_suites_and_hidden_copies(conformance_tests, conformance_tests_dir): + module_folder = os.path.join(conformance_tests_dir, MODULE_NAME) + _write_file(module_folder, os.path.join("earlier_suite", "test_earlier.py"), "earlier") + + violations = conformance_tests.find_response_file_violations( + MODULE_NAME, + "current_suite", + { + os.path.join("earlier_suite", "test_earlier.py"): "rewritten", + os.path.join(".required_module", "suite", "test_x.py"): "copied", + CONFORMANCE_TESTS_DEFINITION_FILE_NAME: "{}", + }, + ) + + assert len(violations) == 3 + + +def test_find_response_file_violations_shared_root_file_insertion_only(conformance_tests, conformance_tests_dir): + module_folder = os.path.join(conformance_tests_dir, MODULE_NAME) + _write_file(module_folder, "shared_helpers.py", "line one\nline two\n") + + extended = "line one\nnew line in between\nline two\nnew line at end\n" + assert ( + conformance_tests.find_response_file_violations(MODULE_NAME, "current_suite", {"shared_helpers.py": extended}) + == [] + ) + + modified = "line one CHANGED\nline two\n" + violations = conformance_tests.find_response_file_violations( + MODULE_NAME, "current_suite", {"shared_helpers.py": modified} + ) + assert len(violations) == 1 + assert "adding lines" in violations[0] + + truncated = "line one\n" + violations = conformance_tests.find_response_file_violations( + MODULE_NAME, "current_suite", {"shared_helpers.py": truncated} + ) + assert len(violations) == 1 diff --git a/tests/test_failure_attribution.py b/tests/test_failure_attribution.py new file mode 100644 index 00000000..69533098 --- /dev/null +++ b/tests/test_failure_attribution.py @@ -0,0 +1,178 @@ +from render_machine.failure_attribution import ( + attribute_failures, + detect_layout_failure, + extract_frid_failure_evidence, + format_other_frids_note, +) + +DELIMITER = "=" * 70 +RULE = "-" * 70 + +CONFORMANCE_TESTS_JSON = { + "1": {"folder_name": "conformance_tests/greeter/hello_world_display_conformance_tests"}, + "2": {"folder_name": "conformance_tests/greeter/shout_hello_world_conformance_tests"}, + "3": {"folder_name": "conformance_tests/greeter/quiet_mode_conformance_tests"}, +} + +TWO_SUITE_FAILURE_OUTPUT = f"""FF +{DELIMITER} +FAIL: test_greeting_output (hello_world_display_conformance_tests.test_conformance.TestGreeter.test_greeting_output) +{RULE} +Traceback (most recent call last): + File "conformance_tests/greeter/hello_world_display_conformance_tests/test_conformance.py", line 19 +AssertionError: greeting missing + +{DELIMITER} +FAIL: test_shout_output (shout_hello_world_conformance_tests.test_conformance.TestShout.test_shout_output) +{RULE} +Traceback (most recent call last): + File "conformance_tests/greeter/shout_hello_world_conformance_tests/test_conformance.py", line 19 +AssertionError: shout missing + +{RULE} +Ran 5 tests in 0.001s + +FAILED (failures=2) +""" + + +def test_attribute_failures_multiple_implicated_in_spec_order(): + assert attribute_failures(TWO_SUITE_FAILURE_OUTPUT, CONFORMANCE_TESTS_JSON) == ["1", "2"] + + +def test_attribute_failures_single_implicated(): + output = "FAIL: test_x (quiet_mode_conformance_tests.test_conformance.TestQuiet.test_x)" + assert attribute_failures(output, CONFORMANCE_TESTS_JSON) == ["3"] + + +def test_attribute_failures_none_implicated(): + assert attribute_failures("something unrelated failed", CONFORMANCE_TESTS_JSON) == [] + + +def test_attribute_failures_handles_missing_folder_name(): + assert attribute_failures("anything", {"1": {}}) == [] + + +def test_extract_evidence_keeps_only_matching_blocks_and_summary(): + evidence = extract_frid_failure_evidence(TWO_SUITE_FAILURE_OUTPUT, "hello_world_display_conformance_tests") + + assert "test_greeting_output" in evidence + assert "greeting missing" in evidence + assert "test_shout_output" not in evidence + assert "shout missing" not in evidence + assert "Ran 5 tests" in evidence + assert "FAILED (failures=2)" in evidence + + +def test_extract_evidence_returns_full_output_without_delimiters(): + output = "some completely different runner format: suite failed" + assert extract_frid_failure_evidence(output, "any_suite") == output + + +def test_extract_evidence_returns_full_output_when_no_block_matches(): + evidence = extract_frid_failure_evidence(TWO_SUITE_FAILURE_OUTPUT, "quiet_mode_conformance_tests") + assert evidence == TWO_SUITE_FAILURE_OUTPUT + + +def test_extract_evidence_single_block_output(): + output = f"""F +{DELIMITER} +FAIL: test_only (quiet_mode_conformance_tests.test_conformance.TestQuiet.test_only) +{RULE} +Traceback (most recent call last): +AssertionError: quiet broken + +{RULE} +Ran 1 test in 0.001s + +FAILED (failures=1) +""" + evidence = extract_frid_failure_evidence(output, "quiet_mode_conformance_tests") + + assert "quiet broken" in evidence + assert "Ran 1 test" in evidence + + +def test_format_other_frids_note_lists_only_other_frids(): + note = format_other_frids_note(["1", "2"], current_frid="1") + + assert "2" in note + assert "handled separately" in note + assert "implementation code" in note + + +def test_format_other_frids_note_empty_when_only_current(): + assert format_other_frids_note(["1"], current_frid="1") == "" + assert format_other_frids_note([], current_frid="1") == "" + + +def test_detect_layout_failure_on_unimportable_start_directory(): + output = "ImportError: Start directory is not importable: 'conformance_tests/greeter'" + assert detect_layout_failure(output) is True + + +def test_detect_layout_failure_on_module_name_collision(): + output = ( + "ImportError: 'test_conformance' module incorrectly imported from 'suite_a'. " + "Expected 'suite_b'. Is this module globally installed?" + ) + assert detect_layout_failure(output) is True + + +def test_detect_layout_failure_on_no_suites_discovered(): + assert detect_layout_failure("Error: No conformance test suites discovered.") is True + assert detect_layout_failure("Error: No unittests discovered.") is True + + +def test_detect_layout_failure_not_triggered_by_test_failures(): + assert detect_layout_failure(TWO_SUITE_FAILURE_OUTPUT) is False + + +def test_detect_layout_failure_not_triggered_when_tests_ran_despite_signature(): + output = "Ran 3 tests in 0.1s\nAssertionError: Start directory is not importable was printed by the app" + assert detect_layout_failure(output) is False + + +class _FakeRunningContext: + def __init__(self, module_name, frid, conformance_tests_json): + self.current_testing_module_name = module_name + self.current_testing_frid = frid + self._json = conformance_tests_json + + def get_conformance_tests_json(self, module_name): + return self._json + + +def _make_fake_render_context(running_context): + from types import SimpleNamespace + + return SimpleNamespace( + conformance_tests_running_context=running_context, + _setup_test_specifications=lambda: None, + ) + + +def test_route_conformance_failure_repoints_context_and_scopes_evidence(): + from render_machine.render_context import RenderContext + + running_context = _FakeRunningContext("greeter", "2", CONFORMANCE_TESTS_JSON) + fake_render_context = _make_fake_render_context(running_context) + + evidence = RenderContext.route_conformance_failure_to_frid(fake_render_context, TWO_SUITE_FAILURE_OUTPUT) + + assert running_context.current_testing_frid == "1" + assert "greeting missing" in evidence + assert "shout missing" not in evidence + assert "also failed in this run: 2" in evidence + + +def test_route_conformance_failure_keeps_context_when_nothing_matches(): + from render_machine.render_context import RenderContext + + running_context = _FakeRunningContext("greeter", "2", CONFORMANCE_TESTS_JSON) + fake_render_context = _make_fake_render_context(running_context) + + evidence = RenderContext.route_conformance_failure_to_frid(fake_render_context, "unattributable failure") + + assert running_context.current_testing_frid == "2" + assert evidence == "unattributable failure" diff --git a/tui/state_handlers.py b/tui/state_handlers.py index 7f74c137..b4552d75 100644 --- a/tui/state_handlers.py +++ b/tui/state_handlers.py @@ -240,13 +240,6 @@ def handle(self, segments: list[str], snapshot: RenderContextSnapshot, previous_ update_progress_item_substates( self.tui, TUIComponents.FRID_PROGRESS_CONFORMANCE_TEST.value, [Substate(fixing_text)] ) - else: - if segments[3] == States.CONFORMANCE_TESTS_READY_FOR_SUMMARY.value: - update_progress_item_substates( - self.tui, - TUIComponents.FRID_PROGRESS_CONFORMANCE_TEST.value, - [Substate("Summarizing conformance tests")], - ) class ScriptOutputsHandler(StateHandler):