Code agent should verify downstream consumers when deleting test fixtures or data directories

[fullsend-ai-retro]

What happened

On PR #205, the code agent removed eval/review/cases/001-clean-approve/ per issue #202. The deletion was correct but incomplete: (1) the agent didn't add a .gitkeep to preserve the empty cases/ directory, requiring a human /fs-fix intervention; (2) the agent didn't check that run-functional.sh would fail on an empty cases directory, requiring a human direct commit to add an early-exit guard. The code-implementation skill's step 7 ('Assess risk — will this change affect other callers?') and step 9 ('Verify API contracts per code path — if the fix removes, empties, or changes...') already cover this class of issue, but the agent did not apply them to a directory deletion.

What could go better

The code agent treated 'remove this directory' as a pure deletion without analyzing what consumes the directory. The run-functional.sh script iterates over case directories and passes them to workspace.py — removing all cases creates an empty input that the pipeline didn't handle. High confidence: this is directly observable from the PR timeline (two human interventions needed). The existing skill guidance is adequate but vague for deletion tasks — it talks about 'API contracts' and 'callers' in terms that map better to code interfaces than to directory-level filesystem contracts consumed by shell scripts.

Proposed change

Add a deletion-specific checklist item to skills/code-implementation/SKILL.md in the planning phase (step 8 area), something like: 'When removing files or directories, verify: (a) no scripts, workflows, or configs reference the deleted path — search for the path string in shell scripts, YAML configs, and Makefiles; (b) if removing the last item from a directory, check whether the directory itself needs to be preserved (.gitkeep) or whether consuming scripts handle the empty case; (c) run the specific test/lint commands that exercise the deleted content to confirm they handle the removal gracefully.' This makes the existing step 7 guidance concrete for deletion tasks.

Validation criteria

The next 3 code agent runs that involve file or directory deletion should demonstrate in their reasoning trace that they checked for scripts and configs referencing the deleted paths. Measure by reviewing agent transcripts for explicit search commands (grep/Grep) targeting the deleted paths in shell scripts and YAML files.

---

Generated by retro agent from #205

[fullsend-ai-triage]

🤖 Finished Triage · ✅ Success · Started 3:34 PM UTC · Completed 3:38 PM UTC
Commit: 12bd957 · View workflow run →

[fullsend-ai-triage]

Triage Summary

Severity: Medium · Category: Feature (skill enhancement)

Problem

The code-implementation skill's planning phase (steps 7–9 in SKILL.md) covers risk assessment and API contract verification using language oriented toward code interfaces. When the code agent performs file or directory deletions, it does not translate these checks into filesystem-specific actions — searching for scripts that reference deleted paths, checking whether consuming pipelines handle empty directories, or preserving directories with .gitkeep.

This gap directly caused the incident on PR #205, where deleting the last eval case directory required two human interventions because run-functional.sh wasn't checked.

Root Cause

Steps 7 and 9 in SKILL.md already cover this class of issue conceptually, but the language ("callers," "API contracts") maps to function signatures and API parameters — not to shell scripts iterating over directories or YAML configs referencing file paths.

Recommended Fix

Add a deletion-specific sub-checklist to skills/code-implementation/SKILL.md in the step 8 area (after the existing item 9 about API contracts):

• (a) Search for scripts, workflows, and configs that reference the deleted path (grep in .sh, .yml, .yaml, Makefile)
• (b) If removing the last item from a directory, check whether the directory needs preservation (.gitkeep) or whether consumers handle the empty case
• (c) Run the specific test/lint commands that exercise the deleted content to confirm graceful handling

Proposed Test Case

Validation: Review the next 3 code agent runs involving file/directory
deletion. Each should demonstrate in its reasoning trace that it:
1. Searched for scripts and configs referencing deleted paths
   (explicit grep/Grep commands targeting .sh, .yml, .yaml, Makefile)
2. Checked whether empty directories need .gitkeep preservation
3. Ran relevant tests to confirm the pipeline handles the removal

---

Labels: This is a retro-filed enhancement to the code-implementation skill, authored by a bot. Applying 'code-agent' (affected component) and 'enhancement' (new capability in skill guidance).

[fullsend-ai-prioritize]

RICE Priority Score: 3.6

Score breakdown

Dimension	Score	Reasoning
Reach	1.5	This affects all users of the code agent whenever a task involves file or directory deletion. Deletion is a common operation across code agent runs (test fixtures, migrations, refactors), so multiple team members and workflows are affected. Scored 1.5 for multiple internal users/customers impacted, though only during deletion-specific tasks.
Impact	1.5	When the agent fails to check downstream consumers of deleted paths, it produces broken PRs requiring human intervention — as demonstrated by PR #205 needing two separate human fixes. This is a significant pain point that erodes trust in agent autonomy. Scored 1.5 because it causes meaningful workflow disruption but isn't blocking core functionality outright.
Confidence	0.8	High confidence at 0.8. The issue is extremely well-documented with a concrete incident (PR #205), a clear root cause (skill guidance uses code-interface language that doesn't translate to filesystem operations), a specific proposed fix location (SKILL.md step 8 area), and measurable validation criteria. The only gap is whether the proposed checklist wording will fully prevent recurrence.
Effort	0.5	This is a small, well-scoped documentation change to a single file (skills/code-implementation/SKILL.md). The proposed checklist text is essentially already drafted in the issue body. No code changes, no architectural work — just adding 3 sub-items to an existing planning phase. Scored 0.5 for a simple, clearly defined change.

Formula: (1.5 x 1.5 x 0.8) / 0.5 = 3.6