docs: TrueNAS SCALE migration + lab compute tier

README.md

@@ -1,15 +1,24 @@
 # 🏠 Homelab GitOps
 
-A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a NAS for media workloads, and a 3-node RKE2 Kubernetes cluster for platform services.
+A hybrid homelab managed entirely through GitOps — Docker Compose stacks on a NAS for media workloads, and a 3-node RKE2 Kubernetes cluster for platform services. A separate lab tier (Nutanix on an R620, Proxmox on an XPS) hosts AAP, a TFE agent, and other enterprise tooling for skills prep — kept outside the GitOps loop because it's expected to be temporary.
 
 ## Hardware
 
+### Core
+
+| Device | Role | CPU | RAM | Storage | OS |
+| -------- | ------ | ----- | ----- | --------- | ---- |
+| UGREEN DXP6800 Pro | NAS + Docker host | Intel i5-1235U | 40GB | 53TB | TrueNAS SCALE |
+| Intel NUC #1 | RKE2 worker | Intel Core i5-8259U | 32GB | 500GB | Elemental OS |
+| Intel NUC #2 | RKE2 worker | Intel Core i7-8559U | 32GB | 250GB | Elemental OS |
+| Intel NUC #3 | RKE2 worker | Intel Core i7-8559U | 32GB | 250GB | Elemental OS |
+
+### Lab (skills prep, not GitOps-managed)
+
 | Device | Role | CPU | RAM | Storage | OS |
 | -------- | ------ | ----- | ----- | --------- | ---- |
-| UGREEN DXP6800 Pro | NAS + Docker host | Intel i5-1235U | 40Gb | 36TB | UGOS |
-| Intel NUC #1 | RKE2 worker | Intel Core i5-8259U | 32Gb | 500Gb | Elemental OS |
-| Intel NUC #2 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS |
-| Intel NUC #3 | RKE2 worker | Intel Core i7-8559U | 32Gb | 250Gb | Elemental OS |
+| Dell PowerEdge R620 | Nutanix CE host (AHV + CVM) | 2x Xeon | 128GB | Mixed SSD/HDD | Nutanix CE (AHV) |
+| Dell XPS 15 9510 | Proxmox host (AAP + TFE agent VMs) | Intel Core (11th gen) | 32GB | 1TB SSD | Proxmox VE |
 
 ## Repository Structure
 

docs/adr.md

@@ -162,3 +162,57 @@ Deploy ARC v2 (Actions Runner Controller) into the K8s cluster to run ephemeral
 - The `nas-deploy` workflow splits into a `detect` job and a `deploy` job. The `deploy` job only runs when changed stacks are detected, avoiding unnecessary work on ansible-only or unrelated pushes.
 - Runner security: the workflow only triggers on push to `main` (not `pull_request`), so untrusted PRs cannot execute code on the homelab runner. Branch protection on `main` requiring review is the gate.
 - Secrets injected via the runner pod do not receive GitHub's automatic log masking, so `::add-mask::` directives are applied explicitly at the start of each deploy job.
+
+## 010 - TrueNAS SCALE over UGOS, with 5x20TB RAIDZ2 pool
+
+### Context
+
+The UGREEN DXP6800 Pro shipped with UGOS, which I hit several friction points on: a hardware fault that needed recovery, a few security concerns around the stock surface area, and updates that would silently undo configuration I had set on the CLI (sshd, system config, etc.) by reapplying values from the UGOS UI/config DB. UGOS is also closed enough that it's hard to manage declaratively or recover predictably. At the same time the original 5x10TB pool was running out of headroom for the media stack.
+
+### Decision
+
+Reinstall the NAS on TrueNAS SCALE (Community Edition) and rebuild storage as a 5x20TB RAIDZ2 data pool (~53TB usable) plus a 2x SSD mirrored boot pool so a single boot drive failure can't take the NAS offline. ZFS replaces the UGOS storage stack, and TrueNAS' API becomes the management surface for anything we can't keep in Git.
+
+### Impacts
+
+- ZFS + RAIDZ2 on the data pool tolerates 2 simultaneous disk failures, with scrubs and checksums catching bitrot the previous setup couldn't see.
+- Mirrored boot pool means the OS survives a single SSD failure — no full reinstall path for a dead boot drive.
+- Usable capacity goes from ~36TB to ~53TB, restoring headroom for the media library and giving room for ZFS snapshots/replication as a backup target.
+- TrueNAS still keeps its own config DB and strongly steers you toward the UI/API for settings rather than letting you treat `/etc` as source of truth — better than UGOS (CLI edits no longer get clobbered on update, since changes go through middlewared) but a step short of fully declarative. Accepted as the cost of staying on a supported appliance OS.
+- The TrueNAS API (and middlewared) gives a real automation surface for users, datasets, shares, snapshots, and replication, which UGOS did not.
+- Docker Compose workloads (see [ADR-005](#005---media-on-docker)) keep running on the NAS; the dataset layout under `/mnt/<pool>/...` replaces the old `/volume3/...` paths and stack mounts were updated to match.
+- NFS exports for Longhorn backups and the monitoring StorageClass (see [ADR-007](#007---nfs-csi-driver-for-monitoring-storage)) are now ZFS datasets, so snapshots of those exports are cheap and atomic.
+- The UGOS app store and any UGREEN-specific integrations are gone; everything on the NAS is now either a Docker Compose stack from this repo or a TrueNAS-native feature.
+
+## 011 - Lab compute tier for enterprise-tooling skills prep
+
+### Context
+
+For a new job I want hands-on time with platforms I'll see at work but don't currently run anywhere in the homelab: Ansible Automation Platform (AAP), Terraform Enterprise (specifically the agent pattern), and Nutanix. These are heavy compared to the existing workload mix — AAP wants a multi-GB VM, Nutanix CE wants a single physical host with 32GB+ RAM and multiple disks, and a TFE agent needs a persistent worker with credential access to whatever it's provisioning. The existing 3-NUC RKE2 cluster isn't sized for this, and I don't want transient learning workloads mixed with the platform services I rely on (DNS, monitoring, ingress, secrets).
+
+I also don't expect this tier to be permanent. Once I'm comfortable with the platforms I'd rather tear it down than carry the power bill and maintenance burden indefinitely.
+
+### Decision
+
+Add a third compute tier dedicated to enterprise-tooling labs, kept entirely separate from the NAS + RKE2 core:
+
+- **R620 (`r620.homelab`, iDRAC `idrac.homelab`)** runs Nutanix CE on bare metal — dual Xeon, 128GB RAM, mixed SSD/HDD fits the CE single-node requirements. AHV host at `r620.homelab`, CVM at `cvm.homelab`.
+- **Dell XPS 15 9510 (`pve.homelab`)** runs Proxmox VE — 32GB RAM, 1TB SSD — and hosts the lab VMs.
+- **AAP (`aap.homelab`)** runs as a Proxmox VM.
+- **TFE agent (`tfagent.homelab`)** runs as a Proxmox VM. Only the agent is local; TFE itself stays hosted.
+
+The TFE agent could have gone into RKE2 (the ARC pattern from [ADR-009](#009---arc-v2-self-hosted-runners-for-nas-deployment) would have been reusable), but Proxmox wins here because:
+
+1. The point of this tier is to mirror how these platforms get deployed at work, and TFE agents in real environments overwhelmingly run on dedicated VM worker pools, not k8s.
+2. Terraform runs can be noisy (long-lived plans/applies, outbound API calls to Nutanix/vCenter/etc.) and isolating them from platform workloads is worth more than declarative-ness on a temporary box.
+3. Decommission story is clean — power off the XPS and the whole agent disappears, no Flux/cluster cleanup.
+4. Proxmox has the headroom; RKE2 doesn't have much to spare.
+
+### Impacts
+
+- Lab compute is treated as best-effort. Nothing in this tier should hold state that matters — DR coverage stays light and recovery is "rebuild from scratch."
+- Proxmox is **not** brought under Flux/GitOps. It's managed manually because it's expected to be temporary and the automation work doesn't pay back.
+- The R620 is a high-draw bare-metal box; expect to keep it powered off when not actively learning.
+- No new networking — these hosts join the flat 192.168.0.0/24 LAN. See [naming-and-ips.md](naming-and-ips.md) for the new rows.
+- Secrets for AAP and the TFE agent stay local to each tool (AAP credentials store, TFE workspace vars) rather than going through OpenBao/ESO — neither runs in K8s, and the lab tier shouldn't take a dependency on core homelab infrastructure.
+- When the lab is retired: power off the R620, shut down the Proxmox VMs, remove rows from `naming-and-ips.md`, and leave this ADR in place as historical context.

docs/disaster-recovery.md

@@ -6,6 +6,10 @@
 - [ ] Longhorn backup target configured (NAS NFS share)
 - [ ] Longhorn recurring snapshots scheduled
 - [X] Age private key backed up offline
+- [ ] TrueNAS config backup (encrypted) saved offline + in password manager
+- [ ] ZFS scrubs scheduled on data pool (RAIDZ2) and boot pool (mirror)
+- [ ] ZFS snapshot tasks scheduled for media + app config datasets
+- [ ] ZFS replication target configured for critical datasets
 - [ ] CNPG backup target configured (NAS NFS share or object storage)
 - [ ] CNPG scheduled backups enabled for ff-postgres (production)
 - [ ] NAS app config directories backed up
@@ -35,6 +39,10 @@
 | ff-postgres (dev) | N/A | ~5 min | Single instance, dev data is disposable — recreate from scratch |
 | Media library | N/A | N/A | Not backed up (re-downloadable, want to find a way to track what currently exists, will *arr config backups work here?) |
 | App configs (Sonarr, Radarr, etc.) | Last NAS backup | ~30 min | Restore config dirs, redeploy |
+| NAS data pool | Last ZFS snapshot | ~1 hr | RAIDZ2 tolerates 2-disk loss; full pool rebuild via import or replication restore |
+| NAS boot pool | N/A | ~1 hr | 2x SSD mirror tolerates 1-drive loss; reinstall TrueNAS + restore config backup on total loss |
+| TrueNAS config | Last config backup | ~15 min | Restored via web UI after reinstall, before importing data pool |
+| Lab compute (Nutanix, AAP, TFE agent) | N/A | N/A | Skills-prep tier — rebuild from scratch, nothing here holds state that matters (see [ADR-011](adr.md)) |
 | SOPS encryption key (age) | Offline backup and external Password Manager | Manual | Required to decrypt all secrets |
 
 ## RKE2 Cluster Recovery
@@ -85,6 +93,26 @@ ff-postgres is managed by the CloudNativePG operator, reconciled by Flux via the
 
 **Dev** instance (single replica) is treated as disposable — recreate from scratch, run migrations.
 
+## NAS Recovery
+
+The NAS runs TrueNAS SCALE on a 2x SSD mirrored boot pool, with a 5x20TB RAIDZ2 data pool (~53TB usable). See [ADR-010](adr.md).
+
+### Single disk failure (data pool)
+
+RAIDZ2 keeps the pool online through 2 simultaneous disk losses. Replace the failed disk and resilver — no service interruption, no manual data restore.
+
+### Single disk failure (boot pool)
+
+The 2x SSD mirror tolerates one boot drive failure. Replace the failed SSD and let TrueNAS resilver the boot pool — NAS stays online throughout.
+
+### Total NAS loss (full reinstall)
+
+1. Reinstall TrueNAS SCALE onto fresh boot SSDs (mirrored).
+2. Restore the TrueNAS config backup via the web UI — this brings back users, datasets, shares, NFS exports, snapshot tasks, and API keys.
+3. Import the existing data pool if the disks survived, or recreate the pool and restore from ZFS replication target.
+4. Re-run the most recent successful **NAS Deploy** workflow run from GitHub Actions, or push a no-op change under `docker/` on `main` to trigger a fresh deploy (rsync + Ansible).
+5. Verify NFS exports are reachable from the cluster — Longhorn backup target and the monitoring StorageClass depend on them.
+
 ### PiHole recovery
 
 1. One instance runs on k8s. If it dies, it should migrate to another node. If the entire cluster is down, we still have a second instance running on

docs/naming-and-ips.md

@@ -16,14 +16,27 @@
 
 | Hostname | IP | Role | Notes |
 | -------- | -- | ---- | ----- |
-| nas.homelab | 192.168.0.233 | UGREEN DXP6800 Pro — NAS + Docker host | Media stacks, NFS, Pi-hole secondary |
+| nas.homelab | 192.168.0.233 | UGREEN DXP6800 Pro — TrueNAS SCALE + Docker host | Media stacks, NFS, Pi-hole secondary; 5x20TB RAIDZ2 data pool, 2x SSD mirrored boot pool |
 | nasbridge0.homelab | 192.168.0.235 | NAS bridge interface | Docker bridge network |
 | nuc1.homelab | 192.168.0.50 | RKE2 worker (i5-8259U, 32GB) | |
 | nuc2.homelab | 192.168.0.51 | RKE2 worker (i7-8559U, 32GB) | |
 | nuc3.homelab | 192.168.0.52 | RKE2 worker (i7-8559U, 32GB) | |
 | rancher.homelab | 192.168.0.48 | Rancher management UI | |
 | nagios.homelab | 192.168.0.190 | Nagios monitoring | |
 
+### Lab Compute
+
+Skills-prep tier, not GitOps-managed and expected to be temporary. See [ADR-011](adr.md).
+
+| Hostname | IP | Role | Notes |
+| -------- | -- | ---- | ----- |
+| idrac.homelab | 192.168.0.120 | R620 out-of-band management | Dell iDRAC |
+| r620.homelab | 192.168.0.38 | Nutanix CE AHV host (2x Xeon, 128GB) | Bare metal, keep powered off when idle |
+| cvm.homelab | 192.168.0.39 | Nutanix CVM | Controller VM on the R620 |
+| pve.homelab | 192.168.0.45 | Proxmox VE host (XPS 15 9510, 32GB, 1TB SSD) | Hosts AAP + TFE agent VMs |
+| aap.homelab | 192.168.0.178 | Ansible Automation Platform VM | On `pve.homelab` |
+| tfagent.homelab | TBD | Terraform Enterprise agent VM | On `pve.homelab`; IP to assign |
+
 ### Services
 
 | Hostname | IP | Platform | Notes |
@@ -46,6 +59,7 @@
 | Range | Purpose |
 | ----- | ------- |
 | .1 | Gateway |
+| .38–.39, .45, .120, .178 | Lab compute (R620/Nutanix, Proxmox, iDRAC, AAP) — scattered, see Lab Compute table |
 | .48–.52 | K8s / Rancher infrastructure |
 | .190 | Monitoring (Nagios) |
 | .233–.235 | NAS + NAS services |