Skip to content

[DRAFT] Send existing conformance tests as context when rendering new conformance tests - #252

Draft
pedjaradenkovic wants to merge 2 commits into
mainfrom
feature/improve-conformance-testing
Draft

[DRAFT] Send existing conformance tests as context when rendering new conformance tests#252
pedjaradenkovic wants to merge 2 commits into
mainfrom
feature/improve-conformance-testing

Conversation

@pedjaradenkovic

@pedjaradenkovic pedjaradenkovic commented Jul 18, 2026

Copy link
Copy Markdown
Contributor

Summary

First iteration of the conformance-testing improvement plan: when rendering conformance tests for a new functionality (FRID), the client now collects the module's existing conformance-test files and sends them to the API as context, so the generated tests don't duplicate coverage that already exists and follow the conventions of previous suites.

Pairs with Codeplain-ai/plain2code_rest_api#122 — the API field is optional, so this client change requires the backend to be deployed first, while the backend change is backward compatible with older clients.

Changes

  • New ConformanceTests.fetch_all_existing_conformance_test_files(module_name): collects all text files across the module's per-FRID conformance test subfolders, keyed as <frid_subfolder>/<relative path>; excludes hidden folders (required-module copies) and conformance_tests.json.
  • RenderConformanceTests._render_conformance_tests fetches these files (excluding the current FRID's own subfolder, so re-renders don't dedup against the tests being replaced), prints them, and passes them to the API.
  • codeplain_REST_api.render_conformance_tests sends them as the new existing_conformance_tests_files payload field.

Execution model is intentionally unchanged: per-FRID folders, per-FRID test runs, regression walk, and fix loop all stay as-is (see CONFORMANCE_TESTING_IMPROVEMENT_PLAN.md for the roadmap).

Testing

  • 5 new unit tests for the fetch helper; full suite 298 passed; black/isort/flake8/mypy clean.
  • E2E against a locally running backend from the paired branch: rendered a fresh 2-FRID project; FRID 2's request contained FRID 1's test files, and its plan produced only tests for the new behavior (no duplicated happy-path test); render exited 0 with all conformance runs passing.

@pedjaradenkovic pedjaradenkovic self-assigned this Jul 18, 2026
@pedjaradenkovic
pedjaradenkovic force-pushed the feature/improve-conformance-testing branch from f3af381 to b074d00 Compare July 18, 2026 18:06
@pedjaradenkovic pedjaradenkovic changed the title Send existing conformance tests as context when rendering new conformance tests [DRAFT] Send existing conformance tests as context when rendering new conformance tests Jul 19, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant