What happened
On PR #115, the code agent created dashboard/test_csv_operator_version.py with 657 lines — 85% of the PR's 775 total additions. The actual production code changes were approximately 90 lines across gcsweb.py and server.py. The human's PR #114 made equivalent production changes and verified all 56 existing tests passed without adding any new test file. The module already had existing test coverage in test_gcsweb.py, test_build_health.py, and test_backfill_versions.py. The agent followed AGENTS.md rule #7 ('If no tests exist for the changed module, create a test file') but the module did have existing tests. The 7:1 test-to-production-code ratio made the PR harder to review and contributed to the human preferring their own cleaner implementation.
What could go better
The agent should scale test additions proportionally to the production code change and should check whether existing test files already cover the changed module before creating a new test file. 657 lines of tests for 90 lines of production code is excessive. The existing test files (test_gcsweb.py, test_build_health.py) already provided a framework for adding targeted tests. The human's closing comment acknowledged 'The test file from this PR will be adapted for the merged implementation in a follow-up,' suggesting some test coverage was warranted but the agent's volume was excessive. Confidence is medium — test quantity is somewhat subjective, but the 7:1 ratio and the creation of a new file when existing test files covered the module suggest over-testing.
Proposed change
Add guidance to the code-implementation skill (skills/code-implementation/SKILL.md in fullsend-ai/agents) in the testing step: 'Scale test additions proportionally to the production code change. For changes under 100 lines of production code, aim for test code at most 2-3x the production change size. Focus on critical paths: one happy-path test per new function, one edge-case test per error handling branch, and one regression test per bug fix. Before creating a new test file, check if an existing test file covers the changed module — prefer adding tests to an existing file. Do not write exhaustive combinatorial tests or duplicate coverage already provided by existing tests.'
Validation criteria
On next 5 code agent PRs with production changes under 100 lines, test additions should not exceed 3x the production code change size. New test files should only be created when no existing test file covers the changed module. Measure by comparing test-line additions to production-line additions in code agent PRs.
Generated by retro agent from redhat-community-ai-tools/ci-failure-tracker#115
What happened
On PR #115, the code agent created dashboard/test_csv_operator_version.py with 657 lines — 85% of the PR's 775 total additions. The actual production code changes were approximately 90 lines across gcsweb.py and server.py. The human's PR #114 made equivalent production changes and verified all 56 existing tests passed without adding any new test file. The module already had existing test coverage in test_gcsweb.py, test_build_health.py, and test_backfill_versions.py. The agent followed AGENTS.md rule #7 ('If no tests exist for the changed module, create a test file') but the module did have existing tests. The 7:1 test-to-production-code ratio made the PR harder to review and contributed to the human preferring their own cleaner implementation.
What could go better
The agent should scale test additions proportionally to the production code change and should check whether existing test files already cover the changed module before creating a new test file. 657 lines of tests for 90 lines of production code is excessive. The existing test files (test_gcsweb.py, test_build_health.py) already provided a framework for adding targeted tests. The human's closing comment acknowledged 'The test file from this PR will be adapted for the merged implementation in a follow-up,' suggesting some test coverage was warranted but the agent's volume was excessive. Confidence is medium — test quantity is somewhat subjective, but the 7:1 ratio and the creation of a new file when existing test files covered the module suggest over-testing.
Proposed change
Add guidance to the code-implementation skill (skills/code-implementation/SKILL.md in fullsend-ai/agents) in the testing step: 'Scale test additions proportionally to the production code change. For changes under 100 lines of production code, aim for test code at most 2-3x the production change size. Focus on critical paths: one happy-path test per new function, one edge-case test per error handling branch, and one regression test per bug fix. Before creating a new test file, check if an existing test file covers the changed module — prefer adding tests to an existing file. Do not write exhaustive combinatorial tests or duplicate coverage already provided by existing tests.'
Validation criteria
On next 5 code agent PRs with production changes under 100 lines, test additions should not exceed 3x the production code change size. New test files should only be created when no existing test file covers the changed module. Measure by comparing test-line additions to production-line additions in code agent PRs.
Generated by retro agent from redhat-community-ai-tools/ci-failure-tracker#115