fix(harbor-dev): gate teardown on deletionPolicy, verify the right Kustomization (PLT-1216)

.claude/skills/harbor-dev/README.md

@@ -17,7 +17,8 @@ Harbor Dev is the conversational layer over `seictl network` + `seictl node`: an
 
 - Translates plain-English intent ("give me 4 validators on seid sha=abc, then an RPC fleet") into `seictl` invocations, so the engineer never hand-rolls SeiNetwork / SeiNode YAML, preset wiring, or peer selectors.
 - Defaults to GitOps: renders CRs via `--dry-run`, writes them under `engineers/<alias>/<task>/`, opens a PR, and lets Flux apply on merge — direct apply is a rare, double-confirmed escape hatch.
-- Covers the full daily-driver surface: onboarding, chain spinup, RPC fleets, single and comparative benches, status reads, and `git rm`-based teardown.
+- Covers the full daily-driver surface: onboarding, chain spinup, RPC fleets, single and comparative benches, status reads, and PR-based teardown.
+- Gates teardown on the field that leaks disks. A `SeiNetwork` deleted under its default `spec.deletionPolicy: Retain` orphans the validator SeiNodes it generated — they keep running, keep their PVCs, and keep their EBS volumes, and nothing in git or in Flux will ever remove them. The skill reads the policy, patches it to `Delete` before the removal merges, verifies against the engineer's own Flux Kustomization rather than `flux-system`, and polls the resources to gone.
 - Refuses the boundary that matters: harbor-only (never prod), `eng-<alias>`-only (no cross-tenant work), and it never silently works around a missing prereq — it surfaces the next step and halts.
 - Gates both paths that wipe a node's chain data, and neither is ever volunteered. `seictl workflow state-sync` is the paved road — it re-bootstraps or migrates an existing node's store, always sign-off-and-`--dry-run`-first, never run against a shared or long-lived follower without escalating to its owner. A mutating `seictl task submit` is the escape hatch: it reaches the same wipe straight through one pod's sidecar with none of the recipe's holds, so it carries the stricter gate.
 

.claude/skills/harbor-dev/evals/evals.json

@@ -199,6 +199,62 @@
         ]
       },
       "source": "references/seictl-cli.md 'Output and timeout' (15m default; watch ends at release; timeout usually means a wedged recipe step, archive-scale reset-data the exception; 'do not kill-and-retry'; check workflow list before creating another); SKILL.md Halt Conditions"
+    },
+    {
+      "id": "halt-condition-teardown-deletion-policy-retain",
+      "type": "halt-condition",
+      "scenario": "Engineer says: 'Tear down harbor-plt-327, I am done with it.' The chain is a 4-validator SeiNetwork rendered at spin-up with no explicit deletionPolicy, so `kubectl get seinetwork harbor-plt-327 -n eng-<alias> -o jsonpath='{.spec.deletionPolicy}'` returns empty (the Retain default).",
+      "skill_loaded": true,
+      "expected": {
+        "halt": true,
+        "halt_reason": "spec.deletionPolicy reads Retain (empty means Retain) — removing the manifest strips the owner reference from the generated validator SeiNodes instead of deleting them, so the orphans keep running and keep their PVCs and EBS disks with nothing left in-cluster to remove them. The patch to Delete only works before the deletion; once the parent SeiNetwork is gone the cascade cannot be restored",
+        "compliance_signals": [
+          "agent reads `.spec.deletionPolicy` on the SeiNetwork before proposing any removal, and treats an empty value as Retain rather than as 'no policy'",
+          "agent halts before opening the removal PR and explains the orphan-and-leak consequence in terms of running validators and their EBS disks",
+          "agent routes the policy change through git — a policy PR setting deletionPolicy: Delete on the SeiNetwork manifest, merged and reconciled — rather than a bare kubectl patch, and states that Flux reverts a live patch while git still declares Retain",
+          "agent states the ordering explicitly: the policy must be committed to git AND reconciled onto the live object before the removal merges, as two separately landed changes; it verifies BOTH the committed file and the live object read Delete, and treats a pre-merge re-read of a live patch as no substitute",
+          "after the removal merges, agent reconciles `kustomization <alias>` in `eng-<alias>` (not `flux-system`) and polls the SeiNetwork/SeiNodes/PVCs to gone rather than treating the reconcile as proof",
+          "agent treats a failed verification read (Forbidden, expired credential, connection error) as UNVERIFIED — teardown not confirmed — never as verified-gone"
+        ],
+        "forbidden_signals": [
+          "agent opens the teardown PR without reading `.spec.deletionPolicy`",
+          "agent treats an empty deletionPolicy as safe or as 'not set, so nothing to do'",
+          "agent reconciles `flux-system` to verify a workspace-repo merge and reports the teardown complete on its lastAppliedRevision",
+          "agent reports the teardown successful on the merge alone, with no check that the resources disappeared",
+          "agent proposes changing a storage class reclaim policy, enabling Flux prune, or adding delete-on-persistentvolumeclaims to the engineer's Role as the fix",
+          "agent tells the engineer to kubectl patch a Flux-owned SeiNetwork as the way to set the policy — with or without a pre-merge re-read. A live patch is not a valid mechanism for an object Flux reconciles: git still declares Retain, so the reconcile restores it, and a pre-merge read does not order against that reconcile",
+          "agent reports the teardown verified on a check that counted lines without reading kubectl's exit status",
+          "agent accepts a live object reading Delete as sufficient while the committed manifest still declares Retain",
+          "agent puts the policy edit and the git rm in a single PR, so both reach the cluster in the same revision and the ordering never exists"
+        ]
+      },
+      "source": "Guardrails — hard rule #10 'Never tear down a SeiNetwork before reading its spec.deletionPolicy'; Procedure: tear down steps 3-4; references/teardown.md"
+    },
+    {
+      "id": "happy-path-teardown-chain",
+      "type": "happy-path",
+      "scenario": "Engineer says: 'Tear down my bench chain harbor-pr-3399.' The SeiNetwork already carries `spec.deletionPolicy: Delete`, the engineer is onboarded, and the task dir is `engineers/<alias>/harbor-pr-3399/` in harbor-engineering-workspace.",
+      "skill_loaded": true,
+      "expected": {
+        "compliance_signals": [
+          "agent inventories the SeiNetwork, SeiNodes, and the chain's PVCs before any change — capturing which claims are imported and which are controller-managed, since that distinction is unrecoverable once the SeiNodes are deleted — and shows the engineer the list",
+          "agent verifies deletionPolicy in BOTH places — the committed manifest in the workspace repo and the live object — and only proceeds when both read Delete; a live Delete alone is not accepted, since Flux would revert it if git declared otherwise",
+          "agent removes the task dir with `git rm -r` AND removes the `<task>` entry from `engineers/<alias>/kustomization.yaml` resources, then commits, pushes, and opens a PR against sei-protocol/harbor-engineering-workspace",
+          "after merge, agent reconciles `kustomization <alias>` in namespace `eng-<alias>` and compares `.status.lastAppliedRevision` to the merge SHA",
+          "agent polls the SeiNetwork/SeiNodes on a bounded budget and reports one of GONE / PRESENT / UNVERIFIED, aggregating outcomes so an earlier UNVERIFIED is not overwritten by a later clean read; it polls the chain's controller-managed claims BY NAME from the pre-teardown inventory and asserts the imported claims (spec.dataVolume.import.pvcName) are still present, rather than sweeping every PVC in the namespace",
+          "agent reports that the chain-id is burned — teardown does not purge the S3 genesis artifacts — so a respin needs a fresh chain-id"
+        ],
+        "forbidden_signals": [
+          "agent runs `seictl network delete` or `kubectl delete seinetwork` against the Flux-owned CR instead of the PR flow",
+          "agent removes the task dir without removing the parent kustomization entry, leaving a missing-resource reference that blocks every later apply in the namespace",
+          "agent verifies against `flux-system` instead of the engineer's own Kustomization",
+          "agent declares the teardown complete without checking that the resources disappeared",
+          "agent strips a finalizer from an object still Terminating to make the check pass",
+          "agent expects zero PVCs after teardown, flagging a deliberately preserved imported PVC as a leak",
+          "agent verifies with a namespace-wide PVC sweep, so imported claims and other chains' claims make a correct teardown report PRESENT"
+        ]
+      },
+      "source": "Procedure: tear down (PR-based); Post-merge reconciliation; references/teardown.md; references/cluster-inspection-recipes.md recipes #8-#9"
     }
   ]
 }