I want a gitOps engine to reconcile as much of the configuration as possible. Currently the 2 front runners are Flux and Argo CD.
Use Flux
- Lower resource utilization.
- Flux natively supports SOPS integration vs requiring plugins or wrappers.
- For our single cluster the benefits of Argo's RBAC and multi-cluster UI provide no benefit.
- PR's will serve as the review mechanism.
We will need an ingress controller
Use Traefik
- Ingress-nginx is being retired in the next 2 months. Even though we could still use it and benefit from it's extensive example-set it's not worth moving forward with it.
- Traefik uses the new gateway api, and is a leading implementation of it.
- Default for rke2, reducing friction.
- Native letsEncrypt.
- Dynamic configuration.
- Built in dashboard.
- Steeper learning curve.
We need to manage secrets for at least 2 platforms: K8s and Docker compose stacks. I want it secure and recoverable.
OpenBao + External Secrets Operator for K8s secrets, SOPS for Docker secrets. OpenBao runs directly on NAS to avoid circular dependencies, ESO runs in cluster and can serve cashed secrets if NAS is down.
- One Age key for both SOPS and Flux bootstrap.
- ESO Syncs k8s secrets dynamically. Also caches secrets allowing existing workloads to run if OpenBao is temporarily down.
- OpenBao demostrates production grade secrets management with a real secrets platform, complete with auditing and auth using service acct tokens.
- SOPS benefits: K8s might be more likely to be reconfigured, I'd like the nas docker services to not be reliant on any of the k8s services.
- SOPS, simple to use, single key, and git native, enabling rollbacks and versioning.
I want ReadWriteMany functionality. With the Ugreen nas there was limited CSI capability. In general the k8s services don't require much storage.
Longhorn enables local disk speed while allowing for ReadWriteMany functionality. NFS can cause issues, but gives me greater access to space, so it's a backup option if necessary.
- About 180Gb usable with 3 replicas in longhorn, maybe 3-400 if we drop to 2 replicas.
- Media workoads (*arr/plex) stay on Docker/NAS with local storage.
- Longhorn runs a storage engine per node, consuming more RAM/CPU.
- NFS may not ever be required.
The Media stack (Plex, Radarr, Sonarr, Prowlar, Configarr) requires significant amounts of storage, and was originally setup as docker compose stacks in portainer on the NAS. I need to decide if I want to migrate it to k8s.
Keep it as Docker compose on the NAS. HA features of K8s, don't warrant lost functionality by moving it off the NAS.
- Hardlinks break, increasing data transfers.
- Kubernetes features don't add actual benefits to this stack (scheduling, scaling, etc).
- Requires 2 deployment features.
- Plex hardware encoding requires direct hw access. This can be accomplished through plugins but adds complexity.
- Prometheus will require NAS node-exporter for cross platform monitoring.
When migrating MetalLB to Flux, applying the Helm chart and its custom resource configuration (IPAddressPool, L2Advertisement) in a single Flux Kustomization causes a race condition on fresh clusters. The CRDs registered by the Helm chart don't exist yet when Flux tries to apply the CR instances, causing transient failures on first reconciliation.
Split Helm chart deployments and their post-install CR configuration into two separate Flux Kustomizations with a dependsOn relationship. The chart Kustomization uses wait: true to ensure all pods are healthy before the config Kustomization applies. Adopt this as the standard pattern for any component where CRDs are installed by a Helm chart and then consumed by separate CR manifests.
- Clean first-apply on a fresh cluster with no transient errors or retry noise in logs.
- Explicit dependency ordering makes the reconciliation sequence readable and predictable.
- Slightly more files per component (two Flux Kustomizations + two subdirectories instead of one flat directory).
- Pattern is reusable for future components like cert-manager, Traefik, and external-dns where we want CRDs -> CRs.
- Debugging is easier since each Flux Kustomization has its own status and can fail independently.
The kube-prometheus-stack will need persistent storage for Prometheus (metrics), Grafana (dashboards/config), and Alertmanager. Prometheus in particular can consume significant disk space (60-90GB at 30 days retention). Longhorn SSDs are limited (200-400GB per node) and better reserved for workloads needing IOPS.
Deploy the NFS CSI driver (csi-driver-nfs) as a second StorageClass alongside Longhorn. Use NFS for monitoring storage (Prometheus, Grafana, Alertmanager) while keeping Longhorn as the default for everything else.
- 200GB Prometheus PVC on NFS keeps SSD space free for latency-sensitive workloads.
- NFS query performance is adequate for monitoring, dashboards are read-heavy but not latency-critical.
- Requires an NFS export on the NAS configured for the cluster nodes.
- NFS CSI driver deploys a DaemonSet (node plugin) and a Deployment (controller) — lightweight resource footprint.
- StorageClass
nfsis explicitly not default — workloads must opt-in by specifyingstorageClassName: nfs. - If NAS is down, Prometheus stops writing but the cluster keeps running, acceptable.
When an application has multiple independent Flux Kustomizations that share a namespace (e.g. a CNPG database cluster managed by infra and an app deployment managed via a separate GitRepository), namespace ownership becomes ambiguous. If the namespace is created by one Kustomization and Flux prune is enabled, deleting that Kustomization would delete the namespace and take down the other workload with it.
Introduce infra/shared-namespaces/ as the single owner of any namespace that is shared between two or more independent Flux Kustomizations. Neither the app nor the database config creates or owns these namespaces — they are pre-created infrastructure.
- Deleting an app or its database config will not accidentally delete the namespace or affect other workloads in it.
- Namespaces are created early in the reconciliation order, before anything that depends on them.
- New apps with shared namespaces add their namespace entry to
infra/shared-namespaces/namespaces.yamlrather than creating their own namespace resource. - Single-namespace apps that own their namespace entirely (e.g. pihole) are unaffected and continue to manage their namespace locally.
Docker Compose stacks on the NAS need to be deployed automatically when changes are merged to main. GitHub-hosted runners cannot reach the NAS as it is on the local network. A self-hosted runner solution is required that fits within the existing K8s cluster and secrets infrastructure.
Deploy ARC v2 (Actions Runner Controller) into the K8s cluster to run ephemeral self-hosted GitHub Actions runners. The nas-deploy workflow runs on these runners and handles detecting changed stacks, rsyncing files to the NAS, and using Ansible to bring stacks up.
- Runners are ephemeral — each job gets a fresh pod with a clean workspace volume (Longhorn PVC), destroyed after the job completes. No persistent state between runs.
- Secrets (NAS SSH key, SOPS age key, NAS host/user) are stored in OpenBao, synced to K8s via ESO, and injected as environment variables into the runner pod via the ARC HelmRelease. They are not stored in GitHub Secrets and are only present in the in-cluster runner environment during job execution.
- The
nas-deployworkflow splits into adetectjob and adeployjob. Thedeployjob only runs when changed stacks are detected, avoiding unnecessary work on ansible-only or unrelated pushes. - Runner security: the workflow only triggers on push to
main(notpull_request), so untrusted PRs cannot execute code on the homelab runner. Branch protection onmainrequiring review is the gate. - Secrets injected via the runner pod do not receive GitHub's automatic log masking, so
::add-mask::directives are applied explicitly at the start of each deploy job.
The UGREEN DXP6800 Pro shipped with UGOS, which I hit several friction points on: a hardware fault that needed recovery, a few security concerns around the stock surface area, and updates that would silently undo configuration I had set on the CLI (sshd, system config, etc.) by reapplying values from the UGOS UI/config DB. UGOS is also closed enough that it's hard to manage declaratively or recover predictably. At the same time the original 5x10TB pool was running out of headroom for the media stack.
Reinstall the NAS on TrueNAS SCALE (Community Edition) and rebuild storage as a 5x20TB RAIDZ2 data pool (~53TB usable) plus a 2x SSD mirrored boot pool so a single boot drive failure can't take the NAS offline. ZFS replaces the UGOS storage stack, and TrueNAS' API becomes the management surface for anything we can't keep in Git.
- ZFS + RAIDZ2 on the data pool tolerates 2 simultaneous disk failures, with scrubs and checksums catching bitrot the previous setup couldn't see.
- Mirrored boot pool means the OS survives a single SSD failure — no full reinstall path for a dead boot drive.
- Usable capacity goes from ~36TB to ~53TB, restoring headroom for the media library and giving room for ZFS snapshots/replication as a backup target.
- TrueNAS still keeps its own config DB and strongly steers you toward the UI/API for settings rather than letting you treat
/etcas source of truth — better than UGOS (CLI edits no longer get clobbered on update, since changes go through middlewared) but a step short of fully declarative. Accepted as the cost of staying on a supported appliance OS. - The TrueNAS API (and middlewared) gives a real automation surface for users, datasets, shares, snapshots, and replication, which UGOS did not.
- Docker Compose workloads (see ADR-005) keep running on the NAS; the dataset layout under
/mnt/<pool>/...replaces the old/volume3/...paths and stack mounts were updated to match. - NFS exports for Longhorn backups and the monitoring StorageClass (see ADR-007) are now ZFS datasets, so snapshots of those exports are cheap and atomic.
- The UGOS app store and any UGREEN-specific integrations are gone; everything on the NAS is now either a Docker Compose stack from this repo or a TrueNAS-native feature.
For a new job I want hands-on time with platforms I'll see at work but don't currently run anywhere in the homelab: Ansible Automation Platform (AAP), Terraform Enterprise (specifically the agent pattern), and Nutanix. These are heavy compared to the existing workload mix — AAP wants a multi-GB VM, Nutanix CE wants a single physical host with 32GB+ RAM and multiple disks, and a TFE agent needs a persistent worker with credential access to whatever it's provisioning. The existing 3-NUC RKE2 cluster isn't sized for this, and I don't want transient learning workloads mixed with the platform services I rely on (DNS, monitoring, ingress, secrets).
I also don't expect this tier to be permanent. Once I'm comfortable with the platforms I'd rather tear it down than carry the power bill and maintenance burden indefinitely.
Add a third compute tier dedicated to enterprise-tooling labs, kept entirely separate from the NAS + RKE2 core:
- R620 (
r620.homelab, iDRACidrac.homelab) runs Nutanix CE on bare metal — dual Xeon, 128GB RAM, mixed SSD/HDD fits the CE single-node requirements. AHV host atr620.homelab, CVM atcvm.homelab. - Dell XPS 15 9510 (
pve.homelab) runs Proxmox VE — 32GB RAM, 1TB SSD — and hosts the lab VMs. - AAP (
aap.homelab) runs as a Proxmox VM. - TFE agent (
tfagent.homelab) runs as a Proxmox VM. Only the agent is local; TFE itself stays hosted.
The TFE agent could have gone into RKE2 (the ARC pattern from ADR-009 would have been reusable), but Proxmox wins here because:
- The point of this tier is to mirror how these platforms get deployed at work, and TFE agents in real environments overwhelmingly run on dedicated VM worker pools, not k8s.
- Terraform runs can be noisy (long-lived plans/applies, outbound API calls to Nutanix/vCenter/etc.) and isolating them from platform workloads is worth more than declarative-ness on a temporary box.
- Decommission story is clean — power off the XPS and the whole agent disappears, no Flux/cluster cleanup.
- Proxmox has the headroom; RKE2 doesn't have much to spare.
- Lab compute is treated as best-effort. Nothing in this tier should hold state that matters — DR coverage stays light and recovery is "rebuild from scratch."
- Proxmox is not brought under Flux/GitOps. It's managed manually because it's expected to be temporary and the automation work doesn't pay back.
- The R620 is a high-draw bare-metal box; expect to keep it powered off when not actively learning.
- No new networking — these hosts join the flat 192.168.0.0/24 LAN. See naming-and-ips.md for the new rows.
- Secrets for AAP and the TFE agent stay local to each tool (AAP credentials store, TFE workspace vars) rather than going through OpenBao/ESO — neither runs in K8s, and the lab tier shouldn't take a dependency on core homelab infrastructure.
- When the lab is retired: power off the R620, shut down the Proxmox VMs, remove rows from
naming-and-ips.md, and leave this ADR in place as historical context.