feat: audit S3 object storage immutability

CHANGELOG.md

@@ -8,6 +8,17 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 ## [Unreleased]
 
 ### Added
+- Read-only S3-compatible object-storage immutability audits: mandatory bucket
+  versioning, optional Object Lock/default retention, and optional newest-object
+  freshness through signed GET metadata and bounded `ListObjectsV2` pagination.
+  Rule IDs include `s3/versioning-disabled`, `s3/object-lock-*`,
+  `s3/newest-object-*`, and `s3/unreachable`. Includes configuration validation,
+  doctor preflight, guided setup live probes, sanitized XML parser fixtures, and
+  generated wizard transcripts. No probe object is written and no object body is
+  downloaded.
+- Moved `AwsSigV4` from RestoreGuard.Cli to RestoreGuard.Providers so both the
+  existing report sink and the new provider use the same dependency-free signer.
+  No reporting behavior changed.
 - Optional multi-cluster Kubernetes/K3s auditing with Velero, PVC, workload,
   and node protection checks plus read-only Doctor RBAC preflight.
 - Guided setup support for Kubernetes/K3s clusters, with live readiness and

README.md

@@ -62,6 +62,9 @@ infrastructure *actually is*, and prints a RED/YELLOW/GREEN report per service:
 - **Off-site freshness & capacity** — every scheduled rclone sync job actually
   ran, succeeded, and is recent (a job that never ran is RED, not invisible);
   the destination isn't silently full.
+- **S3-compatible bucket immutability** — signed read-only metadata probes for
+  versioning, optional Object Lock/default retention, and optional newest-object
+  freshness with a bounded listing (no object bodies, no probe writes).
 - **Kubernetes / K3s** — Velero coverage and freshness, PVC protection,
   deployment availability, and node readiness.
 - **Disk health** — SMART status on the hypervisors.
@@ -106,6 +109,7 @@ have. You need:
 | `zfsReplications` | `zfs list -H -p -t snapshot` on source (and replica) | `zfs` on PATH; read access (no `zfs send/recv` is ever run) |
 | `offsiteJobs` | `tail` the job's log; optional `rclone about` | read the log; with `rcloneRemote`, the host's remote must authenticate |
 | `sqliteBackupDirs` | recursive `find` for `*-wal`/`*-shm` | read access to the backup folder |
+| `objectStorageBuckets` | signed GET bucket metadata and bounded `ListObjectsV2` metadata pages | bucket credentials; permission to read bucket versioning, Object Lock, and object metadata |
 | `pbsOffsite` *(legacy — prefer `offsiteJobs`)* | `tail` the sync log, `rclone about` | read the log; the host's rclone remote must authenticate |
 | `pbsMaintenance` | `pct exec <CT> -- proxmox-backup-manager` (GC, verify + sync jobs, datastore list); `ls` of `host/<id>` snapshot dirs for `hostBackups` | root on the PVE host that runs the PBS container |
 | `kubernetesClusters` | read-only `kubectl get` for cluster, workload, Velero, PodVolumeBackup, and CSI snapshot resources | SSH access to the configured `kubectl` prefix; for an LXC prefix, root on the PVE host for `pct exec` |

docs/modules/ROOT/examples/restoreguard.sample.json

@@ -120,9 +120,34 @@
     "rcloneRemote": "onedrive:",
     "targetName": "pbs-nvme",
     "maxSyncAgeHours": 26,
-    "maxRunAgeHours": 6
+  "maxRunAgeHours": 6
   },
 
+  // S3-compatible bucket immutability: signed read-only metadata probes. Versioning
+  // is always required; Object Lock and newest-object freshness are optional. Secret
+  // files live on the RestoreGuard operator machine (never in this JSON, never logs).
+  // maxListKeys/maxListRequests bound the newest-object scan to 5 pages x 100 keys;
+  // an exhausted bound is YELLOW rather than a false green.
+  "objectStorageBuckets": [
+    {
+      "name": "offsite-backup-bucket",
+      "endpoint": "https://s3.example.com",
+      "bucket": "homelab-backups",
+      "prefix": "backup/",
+      "region": "us-east-1",
+      "forcePathStyle": true,
+      "accessKeyFile": "/secure/restoreguard/s3-access-key",
+      "secretKeyFile": "/secure/restoreguard/s3-secret-key",
+      "checkObjectLock": true,
+      "objectLockRequired": true,
+      "minRetentionDays": 30,
+      "checkNewestObject": true,
+      "maxNewestObjectAgeHours": 26,
+      "maxListKeys": 100,
+      "maxListRequests": 5
+    }
+  ],
+
   // PBS GC + verify-job hygiene, read by running proxmox-backup-manager inside the
   // PBS container from its PVE host (pct exec — needs root on the PVE host).
   "pbsMaintenance": {

docs/modules/ROOT/pages/checks.adoc

@@ -185,6 +185,25 @@ in real labs: the replica keeps its old snapshots and *looks* fine.
 | `k8s-node/not-ready` | RED | a cluster node is NotReady
 |===
 
+== S3-compatible object storage (`objectStorageBuckets`)
+
+The provider performs signed read-only GET requests for versioning, optional
+Object Lock, and optional `ListObjectsV2` metadata. It never writes or reads
+object content. A bounded freshness scan that reaches its request/page limit is
+YELLOW, not a false green.
+
+[cols="2,1,4"]
+|===
+| `s3/unreachable` | RED | endpoint, credentials, permissions, or bucket name prevented metadata discovery
+| `s3/versioning-disabled` | RED | bucket versioning is suspended or absent, so an old object can overwrite a newer backup
+| `s3/object-lock-disabled` | RED | Object Lock is required by config but is not enabled on the bucket
+| `s3/object-lock-no-default-retention` | RED | Object Lock is enabled but new objects have no default retention rule
+| `s3/object-lock-retention-too-short` | RED | the default retention is shorter than `minRetentionDays`
+| `s3/newest-object-missing` | RED | the optional freshness scan found no object under the configured prefix
+| `s3/newest-object-stale` | RED | the newest object is older than `maxNewestObjectAgeHours`
+| `s3/newest-object-unproven` | YELLOW | the bounded listing reached its page/request limit before proving which object is newest
+|===
+
 == Storage & disks (`pveNodes`, `trueNas`, `smartHosts`)
 
 [cols="2,1,4"]

docs/modules/ROOT/pages/configuration.adoc

@@ -276,6 +276,48 @@ The script's log must mark runs with `=== <ts> sync start ===` /
 *Host needs:* read access to the log; the host's rclone remote must
 authenticate non-interactively.
 
+== `objectStorageBuckets` — S3-compatible immutability *(hand-edited)*
+
+[source,json]
+----
+"objectStorageBuckets": [
+  {
+    "name": "offsite-backup-bucket",
+    "endpoint": "https://s3.example.com",
+    "bucket": "homelab-backups",
+    "prefix": "backup/",
+    "region": "us-east-1",
+    "forcePathStyle": true,
+    "accessKeyFile": "/secure/restoreguard/s3-access-key",
+    "secretKeyFile": "/secure/restoreguard/s3-secret-key",
+    "checkObjectLock": true,
+    "objectLockRequired": true,
+    "minRetentionDays": 30,
+    "checkNewestObject": true,
+    "maxNewestObjectAgeHours": 26,
+    "maxListKeys": 100,
+    "maxListRequests": 5
+  }
+]
+----
+
+`endpoint` must be an HTTP(S) origin (no path, query, fragment, or embedded
+credential). RestoreGuard signs GET requests with SigV4 and reads only bucket metadata. It
+never uploads a probe object and never downloads object content. Versioning is
+always required. Object Lock and default retention are optional; enable
+`checkObjectLock` for a bucket that must be immutable. Freshness is opt-in and
+uses `ListObjectsV2` pages under `prefix` to find the newest object.
+
+The freshness scan is deliberately bounded by `maxListKeys` (1–1000 per page)
+and `maxListRequests` (1–100). If the bound is exhausted before it can prove
+the newest object, RestoreGuard reports a YELLOW `s3/newest-object-unproven`
+finding instead of treating the scan as green. Secret values may be inline or in
+files; prefer files and keep them outside source and logs.
+
+The wizard probes the same signed metadata during setup. `doctor` evaluates the
+same configured versioning, Object Lock/default-retention, and optional
+newest-object requirements without running the full audit.
+
 == `pbsMaintenance` — PBS GC & verification *(hand-edited)*
 
 [source,json]

docs/modules/ROOT/pages/index.adoc

@@ -74,6 +74,9 @@ xref:configuration.adoc[configuration reference]):
   deployment availability, and node readiness.
 * *Off-site copies* — every scheduled rclone sync job ran, succeeded, and is
   recent; the destination has room. A job that never ran is a finding, not a gap.
+* *S3-compatible object storage* — bucket versioning, Object Lock/default
+  retention, and (optionally) a bounded newest-object freshness proof via signed
+  metadata and list requests only. It never writes a probe object or reads object bodies.
 * *Disks* — SMART health on the machines that own the physical disks.
 
 The full rule catalog is in xref:checks.adoc[What gets checked].

docs/wizard-transcripts/01-everything-correct.txt

@@ -182,7 +182,23 @@ Kubernetes cluster #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = ski
   alert when the newest backup is older than (hours) [26]: «Enter»
 Kubernetes cluster #2 SSH destination (e.g. pve or root@192.168.1.5; Enter = done): «Enter»
 
-Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 6 file-backup source(s) (2 with restore canary), 1 ZFS dataset(s) (1 replicated), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s), 1 Kubernetes cluster(s).
+--- S3-compatible object storage (versioning, Object Lock, newest object) ---
+Do you back up to an S3-compatible bucket (MinIO, Garage, R2, AWS)? [y/N]: «y»
+  a name for bucket #1 (Enter = skip): «offsite»
+  endpoint URL: «https://s3.example.com»
+  bucket name: «valid-bucket»
+  file containing the access key: «s3-access»
+  file containing the secret key: «s3-secret»
+  require Object Lock and a default retention rule? [y/N]: «y»
+  enforce a minimum default retention age? [y/N]: «y»
+  minimum default retention (days) [30]: «30»
+  check the newest object's age with a read-only listing? [y/N]: «y»
+  alert when the newest backup is older than (hours) [26]: «26»
+  S3 region [us-east-1]: «Enter»
+  checking bucket metadata (signed GET only) ... OK
+  a name for bucket #2 (Enter = skip): «EOF»
+
+Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 6 file-backup source(s) (2 with restore canary), 1 ZFS dataset(s) (1 replicated), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s), 1 Kubernetes cluster(s), 1 S3 bucket(s).
 Wrote restoreguard.json (+ suppressions.json for known exceptions later).
 
 Every audit also saves its JSON report to a per-user reports folder
@@ -323,6 +339,25 @@ Wizard result: config written
       "maxBackupAgeHours": 26,
       "checkWorkloads": true
     }
+  ],
+  "objectStorageBuckets": [
+    {
+      "name": "offsite",
+      "endpoint": "https://s3.example.com",
+      "bucket": "valid-bucket",
+      "prefix": "",
+      "region": "us-east-1",
+      "forcePathStyle": true,
+      "accessKeyFile": "s3-access",
+      "secretKeyFile": "s3-secret",
+      "checkObjectLock": true,
+      "objectLockRequired": true,
+      "minRetentionDays": 30,
+      "checkNewestObject": true,
+      "maxNewestObjectAgeHours": 26,
+      "maxListKeys": 100,
+      "maxListRequests": 5
+    }
   ]
 }
 

docs/wizard-transcripts/02-wrong-answers-rejected.txt

@@ -234,7 +234,24 @@ Kubernetes cluster #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = ski
   Check Kubernetes node and deployment availability too? [y/N]: «y»
 Kubernetes cluster #2 SSH destination (e.g. pve or root@192.168.1.5; Enter = done): «Enter»
 
-Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 1 file-backup source(s), 1 ZFS dataset(s), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s), 1 Kubernetes cluster(s).
+--- S3-compatible object storage (versioning, Object Lock, newest object) ---
+Do you back up to an S3-compatible bucket (MinIO, Garage, R2, AWS)? [y/N]: «y»
+  a name for bucket #1 (Enter = skip): «offsite»
+  endpoint URL: «https://s3.example.com/invalid»
+  PROBLEM — use an http(s) origin URL without a path, query, fragment, or embedded credentials (e.g. https://s3.example.com).
+  endpoint URL: «https://s3.example.com»
+  bucket name: «valid-bucket»
+  file containing the access key: «s3-access»
+  file containing the secret key: «s3-secret»
+  require Object Lock and a default retention rule? [y/N]: «y»
+  enforce a minimum default retention age? [y/N]: «n»
+  check the newest object's age with a read-only listing? [y/N]: «y»
+  alert when the newest backup is older than (hours) [26]: «Enter»
+  S3 region [us-east-1]: «Enter»
+  checking bucket metadata (signed GET only) ... OK
+  a name for bucket #2 (Enter = skip): «EOF»
+
+Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 1 file-backup source(s), 1 ZFS dataset(s), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s), 1 Kubernetes cluster(s), 1 S3 bucket(s).
 Wrote restoreguard.json (+ suppressions.json for known exceptions later).
 
 Every audit also saves its JSON report to a per-user reports folder
@@ -331,6 +348,24 @@ Wizard result: config written
       "checkWorkloads": true,
       "veleroNamespace": null
     }
+  ],
+  "objectStorageBuckets": [
+    {
+      "name": "offsite",
+      "endpoint": "https://s3.example.com",
+      "bucket": "valid-bucket",
+      "prefix": "",
+      "region": "us-east-1",
+      "forcePathStyle": true,
+      "accessKeyFile": "s3-access",
+      "secretKeyFile": "s3-secret",
+      "checkObjectLock": true,
+      "objectLockRequired": true,
+      "checkNewestObject": true,
+      "maxNewestObjectAgeHours": 26,
+      "maxListKeys": 100,
+      "maxListRequests": 5
+    }
   ]
 }
 

docs/wizard-transcripts/03-everything-skipped.txt

@@ -65,6 +65,9 @@ SMART host #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = skip SMART)
 --- Kubernetes / K3s (node and deployment health, plus Velero backup coverage) ---
 Do you run a Kubernetes or K3s cluster that RestoreGuard should audit? [y/N]: «n»
 
+--- S3-compatible object storage (versioning, Object Lock, newest object) ---
+Do you back up to an S3-compatible bucket (MinIO, Garage, R2, AWS)? [y/N]: «n»
+
 Nothing was configured, so there is nothing to audit yet.
 Re-run `restoreguard` to try again, or copy restoreguard.sample.json and edit it by hand.
 -------------------------------------------------------------------------------

restoreguard.sample.json

@@ -120,9 +120,34 @@
     "rcloneRemote": "onedrive:",
     "targetName": "pbs-nvme",
     "maxSyncAgeHours": 26,
-    "maxRunAgeHours": 6
+  "maxRunAgeHours": 6
   },
 
+  // S3-compatible bucket immutability: signed read-only metadata probes. Versioning
+  // is always required; Object Lock and newest-object freshness are optional. Secret
+  // files live on the RestoreGuard operator machine (never in this JSON, never logs).
+  // maxListKeys/maxListRequests bound the newest-object scan to 5 pages x 100 keys;
+  // an exhausted bound is YELLOW rather than a false green.
+  "objectStorageBuckets": [
+    {
+      "name": "offsite-backup-bucket",
+      "endpoint": "https://s3.example.com",
+      "bucket": "homelab-backups",
+      "prefix": "backup/",
+      "region": "us-east-1",
+      "forcePathStyle": true,
+      "accessKeyFile": "/secure/restoreguard/s3-access-key",
+      "secretKeyFile": "/secure/restoreguard/s3-secret-key",
+      "checkObjectLock": true,
+      "objectLockRequired": true,
+      "minRetentionDays": 30,
+      "checkNewestObject": true,
+      "maxNewestObjectAgeHours": 26,
+      "maxListKeys": 100,
+      "maxListRequests": 5
+    }
+  ],
+
   // PBS GC + verify-job hygiene, read by running proxmox-backup-manager inside the
   // PBS container from its PVE host (pct exec — needs root on the PVE host).
   "pbsMaintenance": {