[DRAFT] Send existing conformance tests as context when rendering new conformance tests - #252
Draft
pedjaradenkovic wants to merge 2 commits into
Draft
[DRAFT] Send existing conformance tests as context when rendering new conformance tests#252pedjaradenkovic wants to merge 2 commits into
pedjaradenkovic wants to merge 2 commits into
Conversation
pedjaradenkovic
force-pushed
the
feature/improve-conformance-testing
branch
from
July 18, 2026 18:06
f3af381 to
b074d00
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First iteration of the conformance-testing improvement plan: when rendering conformance tests for a new functionality (FRID), the client now collects the module's existing conformance-test files and sends them to the API as context, so the generated tests don't duplicate coverage that already exists and follow the conventions of previous suites.
Pairs with Codeplain-ai/plain2code_rest_api#122 — the API field is optional, so this client change requires the backend to be deployed first, while the backend change is backward compatible with older clients.
Changes
ConformanceTests.fetch_all_existing_conformance_test_files(module_name): collects all text files across the module's per-FRID conformance test subfolders, keyed as<frid_subfolder>/<relative path>; excludes hidden folders (required-module copies) andconformance_tests.json.RenderConformanceTests._render_conformance_testsfetches these files (excluding the current FRID's own subfolder, so re-renders don't dedup against the tests being replaced), prints them, and passes them to the API.codeplain_REST_api.render_conformance_testssends them as the newexisting_conformance_tests_filespayload field.Execution model is intentionally unchanged: per-FRID folders, per-FRID test runs, regression walk, and fix loop all stay as-is (see CONFORMANCE_TESTING_IMPROVEMENT_PLAN.md for the roadmap).
Testing