What happened
In PR #1041, the code agent deleted 9 forge-specific policy files under policies/github/, policies/gitlab/, and policies/jira/ to consolidate shared filesystem_policy, landlock, and process blocks into policies/base.yaml. Each deleted file also contained a network_policies section with forge-specific network access rules (GitLab API endpoints, Jira Cloud API endpoints, access-level distinctions like read-only vs read-write). The code agent verified that the shared blocks were already present in base.yaml but did not verify that the network capabilities had replacement providers. No GitLab or Jira profiles existed at the time of deletion. The human caught this within 15 minutes, and the review bot's first complete review (delayed ~80 minutes by a cancelled first run) independently identified the fail-open. Two fix iterations were needed to create the missing profiles/providers and correct the fix direction, contributing to a total of 5 fix iterations over 5 days.
What could go better
The code agent understood the migration goal (consolidate shared config, delegate network rules to profiles) and correctly verified the shared blocks existed in base.yaml. But it treated the deleted files as single-purpose (shared policy containers) when they were actually multi-purpose (shared policy + forge-specific network rules). It should have enumerated all functional sections in each file being deleted and verified each had a replacement. Specifically, it should have checked whether GitLab and Jira profiles/providers existed before deleting the policy files that provided those network rules. This is distinct from issue #1070 (grep for references to deprecated keys): #1070 catches stale references, while this gap is about verifying that the capabilities themselves are preserved. Confidence: high — the code agent's log confirms it analyzed the policy files in detail, identified the network_policies sections, and correctly noted they should come from profiles, but never verified those profiles existed.
Proposed change
Add a migration-completeness check to the code-implementation skill (skills/code-implementation/SKILL.md). When the code agent deletes or consolidates config files that serve multiple functional purposes, it should:
- Enumerate all functional sections in each file being removed (not just the sections targeted for consolidation).
- For each section, verify that an equivalent capability is provided by an existing file — either via inheritance/fallback (e.g.,
base.yaml covers filesystem_policy) or via a separate mechanism (e.g., a profile covers network_policies).
- If any section lacks a replacement, either create the replacement in the same PR or flag it as an explicit gap in the PR description with a justification for why it can be deferred.
This complements #1070's grep-for-references approach by adding a capability-preservation check. Suggested wording for the skill (in the migration/refactoring guidance section):
When deleting config files, enumerate every functional section in each file. For each section, verify its functionality is preserved — either through inheritance from a base file, delegation to another mechanism (profiles, providers), or explicit replacement. Do not assume a section is covered; confirm the replacement file exists and provides equivalent capability.
Validation criteria
On the next 3 PRs where the code agent deletes or consolidates multi-section config files in this repo, the agent should (a) list all functional sections in deleted files in its PR description or commit message, and (b) verify each section has a replacement before committing. Success: zero critical review findings about missing replacement capabilities. Observable by reviewing PR descriptions and review agent output.
Generated by retro agent from #1041
What happened
In PR #1041, the code agent deleted 9 forge-specific policy files under
policies/github/,policies/gitlab/, andpolicies/jira/to consolidate sharedfilesystem_policy,landlock, andprocessblocks intopolicies/base.yaml. Each deleted file also contained anetwork_policiessection with forge-specific network access rules (GitLab API endpoints, Jira Cloud API endpoints, access-level distinctions like read-only vs read-write). The code agent verified that the shared blocks were already present inbase.yamlbut did not verify that the network capabilities had replacement providers. No GitLab or Jira profiles existed at the time of deletion. The human caught this within 15 minutes, and the review bot's first complete review (delayed ~80 minutes by a cancelled first run) independently identified the fail-open. Two fix iterations were needed to create the missing profiles/providers and correct the fix direction, contributing to a total of 5 fix iterations over 5 days.What could go better
The code agent understood the migration goal (consolidate shared config, delegate network rules to profiles) and correctly verified the shared blocks existed in
base.yaml. But it treated the deleted files as single-purpose (shared policy containers) when they were actually multi-purpose (shared policy + forge-specific network rules). It should have enumerated all functional sections in each file being deleted and verified each had a replacement. Specifically, it should have checked whether GitLab and Jira profiles/providers existed before deleting the policy files that provided those network rules. This is distinct from issue #1070 (grep for references to deprecated keys): #1070 catches stale references, while this gap is about verifying that the capabilities themselves are preserved. Confidence: high — the code agent's log confirms it analyzed the policy files in detail, identified thenetwork_policiessections, and correctly noted they should come from profiles, but never verified those profiles existed.Proposed change
Add a migration-completeness check to the code-implementation skill (
skills/code-implementation/SKILL.md). When the code agent deletes or consolidates config files that serve multiple functional purposes, it should:base.yamlcoversfilesystem_policy) or via a separate mechanism (e.g., a profile coversnetwork_policies).This complements #1070's grep-for-references approach by adding a capability-preservation check. Suggested wording for the skill (in the migration/refactoring guidance section):
Validation criteria
On the next 3 PRs where the code agent deletes or consolidates multi-section config files in this repo, the agent should (a) list all functional sections in deleted files in its PR description or commit message, and (b) verify each section has a replacement before committing. Success: zero critical review findings about missing replacement capabilities. Observable by reviewing PR descriptions and review agent output.
Generated by retro agent from #1041