From 029f3f85cf9c302e7c7dc2b3f1c6ad460958a932 Mon Sep 17 00:00:00 2001 From: Rick Gardner Date: Tue, 23 Jun 2026 11:04:31 -0400 Subject: [PATCH 1/6] docs: update for TrueNAS SCALE migration - README hardware row now reflects TrueNAS SCALE and 53TB usable - ADR-010 records the UGOS -> TrueNAS SCALE move and the 5x20TB RAIDZ2 + 2x SSD mirrored boot pool topology - disaster-recovery gains TrueNAS/ZFS checklist items, RTO/RPO rows for the data pool, boot pool, and TrueNAS config, and a NAS Recovery section - naming-and-ips nas.homelab row notes TrueNAS SCALE and pool layout --- README.md | 2 +- docs/adr.md | 21 +++++++++++++++++++++ docs/disaster-recovery.md | 27 +++++++++++++++++++++++++++ docs/naming-and-ips.md | 2 +- 4 files changed, 50 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index a859fd1..5921033 100644 --- a/README.md +++ b/README.md @@ -6,7 +6,7 @@ A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a | Device | Role | CPU | RAM | Storage | OS | | -------- | ------ | ----- | ----- | --------- | ---- | -| UGREEN DXP6800 Pro | NAS + Docker host | Intel i5-1235U | 40Gb | 36TB | UGOS | +| UGREEN DXP6800 Pro | NAS + Docker host | Intel i5-1235U | 40Gb | 53TB | TrueNAS SCALE | | Intel NUC #1 | RKE2 worker | Intel Core i5-8259U | 32Gb | 500Gb | Elemental OS | | Intel NUC #2 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS | | Intel NUC #3 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS | diff --git a/docs/adr.md b/docs/adr.md index 91f8c3a..cc97614 100644 --- a/docs/adr.md +++ b/docs/adr.md @@ -162,3 +162,24 @@ Deploy ARC v2 (Actions Runner Controller) into the K8s cluster to run ephemeral - The `nas-deploy` workflow splits into a `detect` job and a `deploy` job. The `deploy` job only runs when changed stacks are detected, avoiding unnecessary work on ansible-only or unrelated pushes. - Runner security: the workflow only triggers on push to `main` (not `pull_request`), so untrusted PRs cannot execute code on the homelab runner. Branch protection on `main` requiring review is the gate. - Secrets injected via the runner pod do not receive GitHub's automatic log masking, so `::add-mask::` directives are applied explicitly at the start of each deploy job. + +## 010 - TrueNAS SCALE over UGOS, with 5x20TB RAIDZ2 pool + +### Context + +The UGREEN DXP6800 Pro shipped with UGOS, which I hit several friction points on: a hardware fault that needed recovery, a few security concerns around the stock surface area, and updates that would silently undo configuration I had set on the CLI (sshd, system config, etc.) by reapplying values from the UGOS UI/config DB. UGOS is also closed enough that it's hard to manage declaratively or recover predictably. At the same time the original 5x10TB pool was running out of headroom for the media stack. + +### Decision + +Reinstall the NAS on TrueNAS SCALE (Community Edition) and rebuild storage as a 5x20TB RAIDZ2 data pool (~53TB usable) plus a 2x SSD mirrored boot pool so a single boot drive failure can't take the NAS offline. ZFS replaces the UGOS storage stack, and TrueNAS' API becomes the management surface for anything we can't keep in Git. + +### Impacts + +- ZFS + RAIDZ2 on the data pool tolerates 2 simultaneous disk failures, with scrubs and checksums catching bitrot the previous setup couldn't see. +- Mirrored boot pool means the OS survives a single SSD failure — no full reinstall path for a dead boot drive. +- Usable capacity goes from ~36TB to ~53TB, restoring headroom for the media library and giving room for ZFS snapshots/replication as a backup target. +- TrueNAS still keeps its own config DB and strongly steers you toward the UI/API for settings rather than letting you treat `/etc` as source of truth — better than UGOS (CLI edits no longer get clobbered on update, since changes go through middlewared) but a step short of fully declarative. Accepted as the cost of staying on a supported appliance OS. +- The TrueNAS API (and middlewared) gives a real automation surface for users, datasets, shares, snapshots, and replication, which UGOS did not. +- Docker Compose workloads (see [ADR-005](#005---media-on-docker)) keep running on the NAS; the dataset layout under `/mnt//...` replaces the old `/volume3/...` paths and stack mounts were updated to match. +- NFS exports for Longhorn backups and the monitoring StorageClass (see [ADR-007](#007---nfs-csi-driver-for-monitoring-storage)) are now ZFS datasets, so snapshots of those exports are cheap and atomic. +- The UGOS app store and any UGREEN-specific integrations are gone; everything on the NAS is now either a Docker Compose stack from this repo or a TrueNAS-native feature. diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md index 47984a4..2273cce 100644 --- a/docs/disaster-recovery.md +++ b/docs/disaster-recovery.md @@ -6,6 +6,10 @@ - [ ] Longhorn backup target configured (NAS NFS share) - [ ] Longhorn recurring snapshots scheduled - [X] Age private key backed up offline +- [ ] TrueNAS config backup (encrypted) saved offline + in password manager +- [ ] ZFS scrubs scheduled on data pool (RAIDZ2) and boot pool (mirror) +- [ ] ZFS snapshot tasks scheduled for media + app config datasets +- [ ] ZFS replication target configured for critical datasets - [ ] CNPG backup target configured (NAS NFS share or object storage) - [ ] CNPG scheduled backups enabled for ff-postgres (production) - [ ] NAS app config directories backed up @@ -35,6 +39,9 @@ | ff-postgres (dev) | N/A | ~5 min | Single instance, dev data is disposable — recreate from scratch | | Media library | N/A | N/A | Not backed up (re-downloadable, want to find a way to track what currently exists, will *arr config backups work here?) | | App configs (Sonarr, Radarr, etc.) | Last NAS backup | ~30 min | Restore config dirs, redeploy | +| NAS data pool | Last ZFS snapshot | ~1 hr | RAIDZ2 tolerates 2-disk loss; full pool rebuild via import or replication restore | +| NAS boot pool | N/A | ~1 hr | 2x SSD mirror tolerates 1-drive loss; reinstall TrueNAS + restore config backup on total loss | +| TrueNAS config | Last config backup | ~15 min | Restored via web UI after reinstall, before importing data pool | | SOPS encryption key (age) | Offline backup and external Password Manager | Manual | Required to decrypt all secrets | ## RKE2 Cluster Recovery @@ -85,6 +92,26 @@ ff-postgres is managed by the CloudNativePG operator, reconciled by Flux via the **Dev** instance (single replica) is treated as disposable — recreate from scratch, run migrations. +## NAS Recovery + +The NAS runs TrueNAS SCALE on a 2x SSD mirrored boot pool, with a 5x20TB RAIDZ2 data pool (~53TB usable). See [ADR-010](adr.md). + +### Single disk failure (data pool) + +RAIDZ2 keeps the pool online through 2 simultaneous disk losses. Replace the failed disk and resilver — no service interruption, no manual data restore. + +### Single disk failure (boot pool) + +The 2x SSD mirror tolerates one boot drive failure. Replace the failed SSD and let TrueNAS resilver the boot pool — NAS stays online throughout. + +### Total NAS loss (full reinstall) + +1. Reinstall TrueNAS SCALE onto fresh boot SSDs (mirrored). +2. Restore the TrueNAS config backup via the web UI — this brings back users, datasets, shares, NFS exports, snapshot tasks, and API keys. +3. Import the existing data pool if the disks survived, or recreate the pool and restore from ZFS replication target. +4. Re-run the `nas-deploy` workflow (or re-merge to `main`) to rsync all Docker Compose stacks back onto the NAS and bring them up via Ansible. +5. Verify NFS exports are reachable from the cluster — Longhorn backup target and the monitoring StorageClass depend on them. + ### PiHole recovery 1. One instance runs on k8s. If it dies, it should migrate to another node. If the entire cluster is down, we still have a second instance running on diff --git a/docs/naming-and-ips.md b/docs/naming-and-ips.md index 3c861b7..35b8f1d 100644 --- a/docs/naming-and-ips.md +++ b/docs/naming-and-ips.md @@ -16,7 +16,7 @@ | Hostname | IP | Role | Notes | | -------- | -- | ---- | ----- | -| nas.homelab | 192.168.0.233 | UGREEN DXP6800 Pro — NAS + Docker host | Media stacks, NFS, Pi-hole secondary | +| nas.homelab | 192.168.0.233 | UGREEN DXP6800 Pro — TrueNAS SCALE + Docker host | Media stacks, NFS, Pi-hole secondary; 5x20TB RAIDZ2 data pool, 2x SSD mirror boot pool | | nasbridge0.homelab | 192.168.0.235 | NAS bridge interface | Docker bridge network | | nuc1.homelab | 192.168.0.50 | RKE2 worker (i5-8259U, 32GB) | | | nuc2.homelab | 192.168.0.51 | RKE2 worker (i7-8559U, 32GB) | | From cc191c571ef1368940b1d80e05feadea018e057e Mon Sep 17 00:00:00 2001 From: Rick Gardner Date: Tue, 23 Jun 2026 11:37:10 -0400 Subject: [PATCH 2/6] docs: add lab compute tier (Nutanix, AAP, TFE agent) - README hardware table split into Core and Lab sub-sections; adds R620 and XPS 9510 rows - ADR-011 records the lab compute decision and the TFE-agent-on-Proxmox choice over RKE2 - disaster-recovery adds a single lab-tier row (rebuild from scratch, no preserved state) - naming-and-ips adds a Lab Compute table for idrac/r620/cvm/pve/aap/tfagent and flags the scattered IP block --- README.md | 11 ++++++++++- docs/adr.md | 33 +++++++++++++++++++++++++++++++++ docs/disaster-recovery.md | 1 + docs/naming-and-ips.md | 14 ++++++++++++++ 4 files changed, 58 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 5921033..ada1b8c 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,11 @@ # 🏠 Homelab GitOps -A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a NAS for media workloads, and a 3-node RKE2 Kubernetes cluster for platform services. +A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a NAS for media workloads, and a 3-node RKE2 Kubernetes cluster for platform services. A separate lab tier (Nutanix on an R620, Proxmox on an XPS) hosts AAP, a TFE agent, and other enterprise tooling for skills prep — kept outside the GitOps loop because it's expected to be temporary. ## Hardware +### Core + | Device | Role | CPU | RAM | Storage | OS | | -------- | ------ | ----- | ----- | --------- | ---- | | UGREEN DXP6800 Pro | NAS + Docker host | Intel i5-1235U | 40Gb | 53TB | TrueNAS SCALE | @@ -11,6 +13,13 @@ A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a | Intel NUC #2 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS | | Intel NUC #3 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS | +### Lab (skills prep, not GitOps-managed) + +| Device | Role | CPU | RAM | Storage | OS | +| -------- | ------ | ----- | ----- | --------- | ---- | +| Dell PowerEdge R620 | Nutanix CE host (AHV + CVM) | 2x Xeon | 128Gb | Mixed SSD/HDD | Nutanix CE (AHV) | +| Dell XPS 15 9510 | Proxmox host (AAP + TFE agent VMs) | Intel Core (11th gen) | 32Gb | 1TB SSD | Proxmox VE | + ## Repository Structure ```text diff --git a/docs/adr.md b/docs/adr.md index cc97614..27aaa75 100644 --- a/docs/adr.md +++ b/docs/adr.md @@ -183,3 +183,36 @@ Reinstall the NAS on TrueNAS SCALE (Community Edition) and rebuild storage as a - Docker Compose workloads (see [ADR-005](#005---media-on-docker)) keep running on the NAS; the dataset layout under `/mnt//...` replaces the old `/volume3/...` paths and stack mounts were updated to match. - NFS exports for Longhorn backups and the monitoring StorageClass (see [ADR-007](#007---nfs-csi-driver-for-monitoring-storage)) are now ZFS datasets, so snapshots of those exports are cheap and atomic. - The UGOS app store and any UGREEN-specific integrations are gone; everything on the NAS is now either a Docker Compose stack from this repo or a TrueNAS-native feature. + +## 011 - Lab compute tier for enterprise-tooling skills prep + +### Context + +For a new job I want hands-on time with platforms I'll see at work but don't currently run anywhere in the homelab: Ansible Automation Platform (AAP), Terraform Enterprise (specifically the agent pattern), and Nutanix. These are heavy compared to the existing workload mix — AAP wants a multi-GB VM, Nutanix CE wants a single physical host with 32GB+ RAM and multiple disks, and a TFE agent needs a persistent worker with credential access to whatever it's provisioning. The existing 3-NUC RKE2 cluster isn't sized for this, and I don't want transient learning workloads mixed with the platform services I rely on (DNS, monitoring, ingress, secrets). + +I also don't expect this tier to be permanent. Once I'm comfortable with the platforms I'd rather tear it down than carry the power bill and maintenance burden indefinitely. + +### Decision + +Add a third compute tier dedicated to enterprise-tooling labs, kept entirely separate from the NAS + RKE2 core: + +- **R620 (`r620.homelab`, iDRAC `idrac.homelab`)** runs Nutanix CE on bare metal — dual Xeon, 128GB RAM, mixed SSD/HDD fits the CE single-node requirements. AHV host at `r620.homelab`, CVM at `cvm.homelab`. +- **Dell XPS 15 9510 (`pve.homelab`)** runs Proxmox VE — 32GB RAM, 1TB SSD — and hosts the lab VMs. +- **AAP (`aap.homelab`)** runs as a Proxmox VM. +- **TFE agent (`tfagent.homelab`)** runs as a Proxmox VM. Only the agent is local; TFE itself stays hosted. + +The TFE agent could have gone into RKE2 (the ARC pattern from [ADR-009](#009---arc-v2-self-hosted-runners-for-nas-deployment) would have been reusable), but Proxmox wins here because: + +1. The point of this tier is to mirror how these platforms get deployed at work, and TFE agents in real environments overwhelmingly run on dedicated VM worker pools, not k8s. +2. Terraform runs can be noisy (long-lived plans/applies, outbound API calls to Nutanix/vCenter/etc.) and isolating them from platform workloads is worth more than declarative-ness on a temporary box. +3. Decommission story is clean — power off the XPS and the whole agent disappears, no Flux/cluster cleanup. +4. Proxmox has the headroom; RKE2 doesn't have much to spare. + +### Impacts + +- Lab compute is treated as best-effort. Nothing in this tier should hold state that matters — DR coverage stays light and recovery is "rebuild from scratch." +- Proxmox is **not** brought under Flux/GitOps. It's managed manually because it's expected to be temporary and the automation work doesn't pay back. +- The R620 is a high-draw bare-metal box; expect to keep it powered off when not actively learning. +- No new networking — these hosts join the flat 192.168.0.0/24 LAN. See [naming-and-ips.md](naming-and-ips.md) for the new rows. +- Secrets for AAP and the TFE agent stay local to each tool (AAP credentials store, TFE workspace vars) rather than going through OpenBao/ESO — neither runs in K8s, and the lab tier shouldn't take a dependency on core homelab infrastructure. +- When the lab is retired: power off the R620, shut down the Proxmox VMs, remove rows from `naming-and-ips.md`, and leave this ADR in place as historical context. diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md index 2273cce..4967e1e 100644 --- a/docs/disaster-recovery.md +++ b/docs/disaster-recovery.md @@ -42,6 +42,7 @@ | NAS data pool | Last ZFS snapshot | ~1 hr | RAIDZ2 tolerates 2-disk loss; full pool rebuild via import or replication restore | | NAS boot pool | N/A | ~1 hr | 2x SSD mirror tolerates 1-drive loss; reinstall TrueNAS + restore config backup on total loss | | TrueNAS config | Last config backup | ~15 min | Restored via web UI after reinstall, before importing data pool | +| Lab compute (Nutanix, AAP, TFE agent) | N/A | N/A | Skills-prep tier — rebuild from scratch, nothing here holds state that matters (see [ADR-011](adr.md)) | | SOPS encryption key (age) | Offline backup and external Password Manager | Manual | Required to decrypt all secrets | ## RKE2 Cluster Recovery diff --git a/docs/naming-and-ips.md b/docs/naming-and-ips.md index 35b8f1d..beb883e 100644 --- a/docs/naming-and-ips.md +++ b/docs/naming-and-ips.md @@ -24,6 +24,19 @@ | rancher.homelab | 192.168.0.48 | Rancher management UI | | | nagios.homelab | 192.168.0.190 | Nagios monitoring | | +### Lab Compute + +Skills-prep tier, not GitOps-managed and expected to be temporary. See [ADR-011](adr.md). + +| Hostname | IP | Role | Notes | +| -------- | -- | ---- | ----- | +| idrac.homelab | 192.168.0.120 | R620 out-of-band management | Dell iDRAC | +| r620.homelab | 192.168.0.38 | Nutanix CE AHV host (2x Xeon, 128GB) | Bare metal, keep powered off when idle | +| cvm.homelab | 192.168.0.39 | Nutanix CVM | Controller VM on the R620 | +| pve.homelab | 192.168.0.45 | Proxmox VE host (XPS 15 9510, 32GB, 1TB SSD) | Hosts AAP + TFE agent VMs | +| aap.homelab | 192.168.0.178 | Ansible Automation Platform VM | On `pve.homelab` | +| tfagent.homelab | TBD | Terraform Enterprise agent VM | On `pve.homelab`; IP to assign | + ### Services | Hostname | IP | Platform | Notes | @@ -46,6 +59,7 @@ | Range | Purpose | | ----- | ------- | | .1 | Gateway | +| .38–.39, .45, .120, .178 | Lab compute (R620/Nutanix, Proxmox, iDRAC, AAP) — scattered, see Lab Compute table | | .48–.52 | K8s / Rancher infrastructure | | .190 | Monitoring (Nagios) | | .233–.235 | NAS + NAS services | From abf77a446994b6b94bb630a84d9c5e6c386b60b0 Mon Sep 17 00:00:00 2001 From: Richard S Gardner <8586577+rgardner4012@users.noreply.github.com> Date: Tue, 23 Jun 2026 11:47:32 -0400 Subject: [PATCH 3/6] minor clarity update Gb to GB Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index ada1b8c..67b7bf6 100644 --- a/README.md +++ b/README.md @@ -8,10 +8,10 @@ A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a | Device | Role | CPU | RAM | Storage | OS | | -------- | ------ | ----- | ----- | --------- | ---- | -| UGREEN DXP6800 Pro | NAS + Docker host | Intel i5-1235U | 40Gb | 53TB | TrueNAS SCALE | -| Intel NUC #1 | RKE2 worker | Intel Core i5-8259U | 32Gb | 500Gb | Elemental OS | -| Intel NUC #2 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS | -| Intel NUC #3 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS | +| UGREEN DXP6800 Pro | NAS + Docker host | Intel i5-1235U | 40GB | 53TB | TrueNAS SCALE | +| Intel NUC #1 | RKE2 worker | Intel Core i5-8259U | 32GB | 500GB | Elemental OS | +| Intel NUC #2 | RKE2 worker | Intel Core i7-8559U | 32GB | 250GB | Elemental OS | +| Intel NUC #3 | RKE2 worker | Intel Core i7-8559U | 32GB | 250GB | Elemental OS | ### Lab (skills prep, not GitOps-managed) From 8afce71f7b497dc108741bffa9f0cadb6d271e9c Mon Sep 17 00:00:00 2001 From: Richard S Gardner <8586577+rgardner4012@users.noreply.github.com> Date: Tue, 23 Jun 2026 11:48:01 -0400 Subject: [PATCH 4/6] Minor clarifying correction Gb to GB Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 67b7bf6..932ccb5 100644 --- a/README.md +++ b/README.md @@ -17,8 +17,8 @@ A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a | Device | Role | CPU | RAM | Storage | OS | | -------- | ------ | ----- | ----- | --------- | ---- | -| Dell PowerEdge R620 | Nutanix CE host (AHV + CVM) | 2x Xeon | 128Gb | Mixed SSD/HDD | Nutanix CE (AHV) | -| Dell XPS 15 9510 | Proxmox host (AAP + TFE agent VMs) | Intel Core (11th gen) | 32Gb | 1TB SSD | Proxmox VE | +| Dell PowerEdge R620 | Nutanix CE host (AHV + CVM) | 2x Xeon | 128GB | Mixed SSD/HDD | Nutanix CE (AHV) | +| Dell XPS 15 9510 | Proxmox host (AAP + TFE agent VMs) | Intel Core (11th gen) | 32GB | 1TB SSD | Proxmox VE | ## Repository Structure From 8774ec22a1530f32d624d7c0c7e9c9b19e9058a7 Mon Sep 17 00:00:00 2001 From: Richard S Gardner <8586577+rgardner4012@users.noreply.github.com> Date: Tue, 23 Jun 2026 11:48:53 -0400 Subject: [PATCH 5/6] minor correction mirror/mirrored Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/naming-and-ips.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/naming-and-ips.md b/docs/naming-and-ips.md index beb883e..5ecfdb7 100644 --- a/docs/naming-and-ips.md +++ b/docs/naming-and-ips.md @@ -16,7 +16,7 @@ | Hostname | IP | Role | Notes | | -------- | -- | ---- | ----- | -| nas.homelab | 192.168.0.233 | UGREEN DXP6800 Pro — TrueNAS SCALE + Docker host | Media stacks, NFS, Pi-hole secondary; 5x20TB RAIDZ2 data pool, 2x SSD mirror boot pool | +| nas.homelab | 192.168.0.233 | UGREEN DXP6800 Pro — TrueNAS SCALE + Docker host | Media stacks, NFS, Pi-hole secondary; 5x20TB RAIDZ2 data pool, 2x SSD mirrored boot pool | | nasbridge0.homelab | 192.168.0.235 | NAS bridge interface | Docker bridge network | | nuc1.homelab | 192.168.0.50 | RKE2 worker (i5-8259U, 32GB) | | | nuc2.homelab | 192.168.0.51 | RKE2 worker (i7-8559U, 32GB) | | From 670a1e765bdda4d5372b57a1bb1cf819e77daab1 Mon Sep 17 00:00:00 2001 From: Richard S Gardner <8586577+rgardner4012@users.noreply.github.com> Date: Tue, 23 Jun 2026 11:50:18 -0400 Subject: [PATCH 6/6] Clarify nas-deploy workflow re-run drop re-merge to main wording. Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com> --- docs/disaster-recovery.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/disaster-recovery.md b/docs/disaster-recovery.md index 4967e1e..b3ed00e 100644 --- a/docs/disaster-recovery.md +++ b/docs/disaster-recovery.md @@ -110,7 +110,7 @@ The 2x SSD mirror tolerates one boot drive failure. Replace the failed SSD and l 1. Reinstall TrueNAS SCALE onto fresh boot SSDs (mirrored). 2. Restore the TrueNAS config backup via the web UI — this brings back users, datasets, shares, NFS exports, snapshot tasks, and API keys. 3. Import the existing data pool if the disks survived, or recreate the pool and restore from ZFS replication target. -4. Re-run the `nas-deploy` workflow (or re-merge to `main`) to rsync all Docker Compose stacks back onto the NAS and bring them up via Ansible. +4. Re-run the most recent successful **NAS Deploy** workflow run from GitHub Actions, or push a no-op change under `docker/` on `main` to trigger a fresh deploy (rsync + Ansible). 5. Verify NFS exports are reachable from the cluster — Longhorn backup target and the monitoring StorageClass depend on them. ### PiHole recovery