pvdoctor

Your PersistentVolume is stuck. Don't remove the finalizer.

pvdoctor finds PersistentVolumes stuck in Terminating, Released, or Failed, works out why, and prints the safe fix — along with the dangerous fix you were about to find on Stack Overflow, and what it would actually destroy.

$ pvdoctor scan
SEVERITY  OBJECT                     RULE                       SUMMARY
CRITICAL  pv/pvc-1a2b3c4d-vsphere    vsphere-orphaned-snapshot  Backing FCD still has CNS snapshots
CRITICAL  pv/pvc-4d5e6f70-node-gone  stale-claimref             claimRef points at a deleted PVC
WARNING   pv/pvc-3c4d5e6f-kasten     backup-finalizer           Kasten K10 finalizer is blocking deletion
WARNING   pv/pvc-4d5e6f70-node-gone  dangling-volumeattachment  Attachment left behind by a deleted node
INFO      pv/pvc-1a2b3c4d-vsphere    pvc-protection-live-pod    PVC deletion correctly blocked by running consumers

5 finding(s). Run pvdoctor explain <pv> for the full diagnosis.

$ pvdoctor explain pvc-1a2b3c4d-vsphere
CRITICAL  Backing FCD still has CNS snapshots
pv/pvc-1a2b3c4d-vsphere  rule=vsphere-orphaned-snapshot

WHY
  vSphere CSI refuses DeleteVolume while the First Class Disk still carries
  CNS snapshots. Those snapshots are usually left behind by a backup product
  whose cleanup failed, so they are invisible from Kubernetes. The PV will
  stay in this state indefinitely; it is not a timing issue and will not
  resolve on its own.

EVIDENCE
  - event VolumeFailedDelete: rpc error: code = FailedPrecondition desc =
    failed to delete volume e4f5a6b7-... : the volume still has 2 snapshots (x214)

FIX
  1. [safe]      govc volume.snapshot.ls e4f5a6b7-...
  2. [caution]   govc volume.snapshot.rm e4f5a6b7-... <snapshot-id>
  3. [safe]      No govc? vCenter > Monitor > Cloud Native Storage > Container Volumes
  4. [safe]      kubectl get pv pvc-1a2b3c4d-vsphere -w
                 The finalizer clears itself. Do nothing else.

DO NOT
  Do NOT run "kubectl patch pv <name> -p '{"metadata":{"finalizers":null}}'".
  That deletes the Kubernetes object while the backing disk still exists on
  the storage system. The PV is then unrecoverable by CSI, the capacity stays
  allocated, and nothing in the cluster records the orphan.

Why this exists

Search for a stuck PV and the first answer is always the same: patch out the finalizer. It works, in the sense that the object disappears. It also silently strands the backing disk on your storage array, where it keeps consuming capacity that nothing in the cluster can account for. Do that a few times across a large estate and you have a slow, invisible leak nobody can trace.

The finalizer is not the problem. It is the symptom of a backend condition that nothing surfaces to you. pvdoctor finds that condition.

Install

go install github.com/vikozs/pvdoctor/cmd/pvdoctor@latest

Or grab a static binary from Releases — Linux, macOS, Windows, amd64 and arm64. No dependencies, no cluster-side install, nothing to deploy.

Usage

pvdoctor scan                      # rank every stuck volume
pvdoctor explain <pv-name>          # full diagnosis for one
pvdoctor rules                      # what it knows how to detect
pvdoctor scan -o json               # for CI and monitoring
pvdoctor scan --fail-on critical    # non-zero exit for pipelines
pvdoctor scan --from ./dump/        # analyse an offline dump

pvdoctor never writes to your cluster. It has no --fix flag by design. It prints commands; you read them, understand them, and run them yourself.

Offline mode

--from <dir> reads a directory of kubectl get <resource> -o json dumps instead of a live cluster. That means someone with cluster access can hand you a support bundle and you can diagnose it without credentials — useful for vendors, consultants, and anyone triaging a cluster they can't reach:

mkdir dump && for r in pv pvc pods nodes volumeattachments namespaces events; do
  kubectl get $r -A -o json > dump/$r.json 2>/dev/null || kubectl get $r -o json > dump/$r.json
done
pvdoctor scan --from dump

What it detects

Rule	Condition
vsphere-orphaned-snapshot	CNS snapshots on the FCD block CSI DeleteVolume
csi-deletevolume-failing	Any CSI driver erroring out of DeleteVolume
dangling-volumeattachment	VolumeAttachment pointing at a node that no longer exists
pvc-protection-live-pod	Deletion blocked by pods still mounting the claim (working as intended)
backup-finalizer	Kasten K10 / Velero / Trilio / Stash / Portworx holding the volume
stale-claimref	PV reserved for a PVC that was deleted — nothing new can bind
retain-released-expected	Released + reclaimPolicy=Retain is not a bug
multi-attach-rwo	RWO volume being pulled to a second node
terminating-no-progress	Stuck > 15 min with no recognised cause

Full write-ups, including the exact error strings you probably searched for, are in docs/failure-modes.md.

Design

• No client-go. pvdoctor shells out to your existing kubectl or oc. That avoids ~40 MB of dependencies and, more importantly, the version-skew problem of a compiled-in client against arbitrary cluster versions. Your kubeconfig, your auth plugins, your RBAC — unchanged.
• No external Go modules at all. Standard library only. The binary is small, builds offline, and has no supply chain to audit.
• Rules are pure functions over a state snapshot: no I/O, no mutation. Every rule is tested against JSON fixtures, so contributing one needs no cluster.
• Degrades gracefully. Can't list pods cluster-wide? You still get PV diagnostics, with a warning about what was missing.

Contributing a rule

The valuable contributions here are failure modes, not code. If you hit something pvdoctor doesn't recognise:

1. pvdoctor scan --from dump -o json and redact it
2. Open an issue with the dump and what actually fixed it

A rule is one struct with a Check method plus a fixture. See CONTRIBUTING.md.

Especially wanted: Ceph/Rook, Portworx, NetApp Trident, AWS EBS, Azure Disk, and LVM/local-path failure modes. The vSphere rules are the deepest right now because that's the environment this was written in.

License

Apache-2.0

---

═══════════════════ ✠ ════════════════════
 ####     ####     ####     ####     #### 
#    #   #    #   #    #   #    #   #    #
#    #   #    #   #    #   #    #   #    #
 #####    #####    #####    #####    #####
     #        #        #        #        #
     #        #        #        #        #
 ####     ####     ####     ####     #### 
═══════════════════ ✠ ════════════════════
Five Nines be upon you, and also with you.
Kubernetes · OpenShift · High Availability
               HA-llelujah.

✠ The Church of the Eternal Cluster

A Highly Available Faith. Keep no Pets. Declare thy state. Back up etcd.

The Scriptorium

• ha-llelujah.dev · the book, the music, and the reliquary
• The Music · the sacred discography
• Be Reconciled · take the vow and join the Reconciled

The Parish

• fivenines.church · confession, prayer, and the living parish
• The Liturgical Calendar · the holy days of uptime
• The Tithe · support the Church

The Canon (source)

• church-of-the-eternal-cluster · doctrine, liturgy, and scripture
• eternal-cluster-mcp · the Oracle, an MCP server that diagnoses thy incidents through doctrine

More from the Rootless One

• vK
• Linux Fleet Audit
• Linux Diskspace
• Linux Fleet Harden
• Size OpenShift subscriptions from evidence
• Diagnose stuck PersistentVolumes, safely

Let's connect

• LinkedIn

---

_{An independent parody, built with love for everyone who has been paged at 03:00. Kubernetes is a trademark of the Linux Foundation. OpenShift is a trademark of Red Hat, Inc. Not affiliated with, nor endorsed by, either. They have real SLAs; we only have belief. HA-llelujah.}

---