Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
- Optional multi-cluster Kubernetes/K3s auditing with Velero, PVC, workload,
and node protection checks plus read-only Doctor RBAC preflight.
- Guided setup support for Kubernetes/K3s clusters, with live readiness and
optional Velero Backup-resource probes.
- New check: `dashboard-registration-drift`
- Detects Docker containers with published ports that are not registered in Homepage dashboard
- Finding codes:
Expand Down
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -157,7 +157,8 @@ restoreguard help

`-c/--config <path>` selects a config file (default `./restoreguard.json`);
`restoreguard.sample.json` is the annotated template for the advanced sections the
wizard doesn't cover.
wizard doesn't cover. The guided setup includes Kubernetes/K3s clusters: it
live-probes the `kubectl` prefix and, when selected, Velero Backup access.

Alternatively, build a standalone binary yourself with the .NET 10 SDK (swap the
RID for `win-x64` / `osx-arm64` as needed):
Expand Down
3 changes: 2 additions & 1 deletion docs/modules/ROOT/examples/restoreguard.sample.json
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,8 @@
"dashboardHostAliases": ["mydockerhost", "myotherhost"]
},

// Kubernetes / K3s clusters. This is config-file-only in the v1 wizard.
// Kubernetes / K3s clusters. The wizard live-probes the common setup; use
// this form when hand-editing or adding an advanced command prefix.
// kubectl is a command PREFIX executed on alias (for sshd-less LXC use
// e.g. "pct exec 601 -- k3s kubectl"). veleroNamespace: null explicitly
// disables Velero checks; omission defaults to "velero".
Expand Down
5 changes: 4 additions & 1 deletion docs/modules/ROOT/pages/configuration.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,10 @@ completed Velero pod-volume backup or Ready CSI snapshot from the latest success
backup; a Backup resource alone is not data-protection evidence.
Workload checks can be disabled with `checkWorkloads: false`; do not
disable both workload checks and Velero, because that would audit no surface.
Kubernetes is not included in the v1 setup wizard; configure it by hand.
The guided setup wizard live-probes the configured `kubectl` prefix against
`/readyz` and, when selected, probes Velero Backup access before writing this
section. The cluster's audit name is its proven SSH destination; edit the file
when a different display name is needed.

*Host needs:* SSH access to a machine with the configured `kubectl` command prefix;
for an LXC prefix, root access on the PVE host to run `pct exec`.
Expand Down
31 changes: 29 additions & 2 deletions docs/wizard-transcripts/01-everything-correct.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ hypervisor only; dumps live in /var/backups/db-prod (12 files); the restic
repo is /mnt/restic-repo, borg is /backups/borg with /root/.borg-pass; the
canary /etc/fstab restores, any other path restores 0 bytes; PVE has node
'pve' with storage 'pbs-store'; dataset tank/private exists; smartctl: ok on
pve/hypervisor, no disks on nas, not installed on truenas.
pve/hypervisor, no disks on nas, not installed on truenas. Kubernetes readyz
works through kubectl or the documented K3s LXC prefix; Velero is in namespace velero.

---------------------------------- dialogue ----------------------------------

Expand Down Expand Up @@ -168,7 +169,20 @@ SMART host #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = skip SMART)
checking for smartctl and physical disks ... OK
SMART host #2 SSH destination (e.g. pve or root@192.168.1.5; Enter = done): «Enter»

Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 6 file-backup source(s) (2 with restore canary), 1 ZFS dataset(s) (1 replicated), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s).
--- Kubernetes / K3s (node and deployment health, plus Velero backup coverage) ---
Do you run a Kubernetes or K3s cluster that RestoreGuard should audit? [y/N]: «y»
Kubernetes cluster #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = skip): «pve»
testing ssh pve 'echo ok' ... OK
kubectl command prefix on that host (e.g. kubectl or pct exec 601 -- k3s kubectl) [kubectl]: «pct exec 601 -- k3s kubectl»
checking ... OK — Kubernetes API is ready
Does this cluster use Velero for backup coverage? [y/N]: «y»
Velero namespace [velero]: «Enter»
checking ... OK — Velero Backup resources are readable
Check Kubernetes node and deployment availability too? [y/N]: «y»
alert when the newest backup is older than (hours) [26]: «Enter»
Kubernetes cluster #2 SSH destination (e.g. pve or root@192.168.1.5; Enter = done): «Enter»

Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 6 file-backup source(s) (2 with restore canary), 1 ZFS dataset(s) (1 replicated), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s), 1 Kubernetes cluster(s).
Wrote restoreguard.json (+ suppressions.json for known exceptions later).

Every audit also saves its JSON report to a per-user reports folder
Expand Down Expand Up @@ -299,6 +313,16 @@ Wizard result: config written
"alias": "nas",
"path": "/backups/appdata"
}
],
"kubernetesClusters": [
{
"name": "pve",
"alias": "pve",
"kubectl": "pct exec 601 -- k3s kubectl",
"veleroNamespace": "velero",
"maxBackupAgeHours": 26,
"checkWorkloads": true
}
]
}

Expand Down Expand Up @@ -338,3 +362,6 @@ Wizard result: config written
33. [nas] find '/backups/appdata' \( -name '*-wal' -o -name '*-shm' \) -type f -printf '%P\n' | head -50
34. [hypervisor] echo ok
35. [hypervisor] command -v smartctl > /dev/null 2>&1 || { echo missing-tool; exit 0; }; [ -n "$(smartctl --scan 2>/dev/null)" ] && echo ok || echo no-disks
36. [pve] echo ok
37. [pve] pct exec 601 -- k3s kubectl get --raw=/readyz > /dev/null
38. [pve] pct exec 601 -- k3s kubectl get backups.velero.io -n 'velero' -o json > /dev/null
44 changes: 42 additions & 2 deletions docs/wizard-transcripts/02-wrong-answers-rejected.txt
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,8 @@ hypervisor only; dumps live in /var/backups/db-prod (12 files); the restic
repo is /mnt/restic-repo, borg is /backups/borg with /root/.borg-pass; the
canary /etc/fstab restores, any other path restores 0 bytes; PVE has node
'pve' with storage 'pbs-store'; dataset tank/private exists; smartctl: ok on
pve/hypervisor, no disks on nas, not installed on truenas.
pve/hypervisor, no disks on nas, not installed on truenas. Kubernetes readyz
works through kubectl or the documented K3s LXC prefix; Velero is in namespace velero.

---------------------------------- dialogue ----------------------------------

Expand Down Expand Up @@ -209,7 +210,31 @@ SMART host #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = skip SMART)
checking for smartctl and physical disks ... OK
SMART host #2 SSH destination (e.g. pve or root@192.168.1.5; Enter = done): «Enter»

Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 1 file-backup source(s), 1 ZFS dataset(s), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s).
--- Kubernetes / K3s (node and deployment health, plus Velero backup coverage) ---
Do you run a Kubernetes or K3s cluster that RestoreGuard should audit? [y/N]: «y»
Kubernetes cluster #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = skip): «pve»
testing ssh pve 'echo ok' ... OK
kubectl command prefix on that host (e.g. kubectl or pct exec 601 -- k3s kubectl) [kubectl]: «notkubectl»
checking ... PROBLEM — could not reach the Kubernetes API with that command — check kubectl and its context
Keep this value anyway? [y/N]: «n»
(press Enter to skip this question)
kubectl command prefix on that host (e.g. kubectl or pct exec 601 -- k3s kubectl): «Enter»
Skipping this cluster (no working kubectl command).
Kubernetes cluster #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = skip): «pve»
testing ssh pve 'echo ok' ... OK
kubectl command prefix on that host (e.g. kubectl or pct exec 601 -- k3s kubectl) [kubectl]: «Enter»
checking ... OK — Kubernetes API is ready
Does this cluster use Velero for backup coverage? [y/N]: «y»
Velero namespace [velero]: «wrong-namespace»
checking ... PROBLEM — could not read Velero Backup resources there — check the namespace and RBAC
Keep this value anyway? [y/N]: «n»
(press Enter to skip this question)
Velero namespace: «Enter»
Skipping Velero coverage for this cluster.
Check Kubernetes node and deployment availability too? [y/N]: «y»
Kubernetes cluster #2 SSH destination (e.g. pve or root@192.168.1.5; Enter = done): «Enter»

Configured: 1 Docker host(s), DB dumps, 1 Proxmox node(s), TrueNAS, 1 file-backup source(s), 1 ZFS dataset(s), 1 off-site job(s), 1 SQLite scan folder(s), SMART on 1 host(s), 1 Kubernetes cluster(s).
Wrote restoreguard.json (+ suppressions.json for known exceptions later).

Every audit also saves its JSON report to a per-user reports folder
Expand Down Expand Up @@ -296,6 +321,16 @@ Wizard result: config written
"alias": "nas",
"path": "/backups/appdata-live"
}
],
"kubernetesClusters": [
{
"name": "pve",
"alias": "pve",
"kubectl": "kubectl",
"maxBackupAgeHours": 26,
"checkWorkloads": true,
"veleroNamespace": null
}
]
}

Expand Down Expand Up @@ -337,3 +372,8 @@ Wizard result: config written
35. [nas] command -v smartctl > /dev/null 2>&1 || { echo missing-tool; exit 0; }; [ -n "$(smartctl --scan 2>/dev/null)" ] && echo ok || echo no-disks
36. [hypervisor] echo ok
37. [hypervisor] command -v smartctl > /dev/null 2>&1 || { echo missing-tool; exit 0; }; [ -n "$(smartctl --scan 2>/dev/null)" ] && echo ok || echo no-disks
38. [pve] echo ok
39. [pve] notkubectl get --raw=/readyz > /dev/null
40. [pve] echo ok
41. [pve] kubectl get --raw=/readyz > /dev/null
42. [pve] kubectl get backups.velero.io -n 'wrong-namespace' -o json > /dev/null
6 changes: 5 additions & 1 deletion docs/wizard-transcripts/03-everything-skipped.txt
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,8 @@ hypervisor only; dumps live in /var/backups/db-prod (12 files); the restic
repo is /mnt/restic-repo, borg is /backups/borg with /root/.borg-pass; the
canary /etc/fstab restores, any other path restores 0 bytes; PVE has node
'pve' with storage 'pbs-store'; dataset tank/private exists; smartctl: ok on
pve/hypervisor, no disks on nas, not installed on truenas.
pve/hypervisor, no disks on nas, not installed on truenas. Kubernetes readyz
works through kubectl or the documented K3s LXC prefix; Velero is in namespace velero.

---------------------------------- dialogue ----------------------------------

Expand Down Expand Up @@ -61,6 +62,9 @@ Do you keep plain-copy backups of app data (vaultwarden, *arr, HA — SQLite app
--- SMART disk health (machines with PHYSICAL disks: hypervisors/bare metal; a NAS VM only sees virtual disks) ---
SMART host #1 SSH destination (e.g. pve or root@192.168.1.5; Enter = skip SMART): «Enter»

--- Kubernetes / K3s (node and deployment health, plus Velero backup coverage) ---
Do you run a Kubernetes or K3s cluster that RestoreGuard should audit? [y/N]: «n»

Nothing was configured, so there is nothing to audit yet.
Re-run `restoreguard` to try again, or copy restoreguard.sample.json and edit it by hand.
-------------------------------------------------------------------------------
Expand Down
3 changes: 2 additions & 1 deletion restoreguard.sample.json
Original file line number Diff line number Diff line change
Expand Up @@ -218,7 +218,8 @@
"dashboardHostAliases": ["mydockerhost", "myotherhost"]
},

// Kubernetes / K3s clusters. This is config-file-only in the v1 wizard.
// Kubernetes / K3s clusters. The wizard live-probes the common setup; use
// this form when hand-editing or adding an advanced command prefix.
// kubectl is a command PREFIX executed on alias (for sshd-less LXC use
// e.g. "pct exec 601 -- k3s kubectl"). veleroNamespace: null explicitly
// disables Velero checks; omission defaults to "velero".
Expand Down
84 changes: 82 additions & 2 deletions src/RestoreGuard.Cli/InteractiveMode.cs
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
using System.Text.Json;
using System.Text.Json.Nodes;
using RestoreGuard.Providers;
using RestoreGuard.Providers.Docker;
using RestoreGuard.Providers.Pve;
Expand Down Expand Up @@ -638,6 +639,65 @@ User root
}
}

io.WriteLine();
io.WriteLine("--- Kubernetes / K3s (node and deployment health, plus Velero backup coverage) ---");
var kubernetesClusters = new List<Providers.Kubernetes.KubernetesClusterConfig>();
if (AskYesNo(io, "Do you run a Kubernetes or K3s cluster that RestoreGuard should audit?"))
{
while (true)
{
var alias = await AskSshDestinationAsync(ssh, io,
$"Kubernetes cluster #{kubernetesClusters.Count + 1} SSH destination (e.g. pve or root@192.168.1.5; Enter = {(kubernetesClusters.Count == 0 ? "skip" : "done")})");
if (alias.Length == 0)
break;

var kubectl = await AskProbedAsync(io,
" kubectl command prefix on that host (e.g. kubectl or pct exec 601 -- k3s kubectl)", "kubectl",
async command =>
{
var r = await ssh.RunAsync(alias, $"{command} get --raw=/readyz > /dev/null");
return r.ExitCode == 0
? (true, "Kubernetes API is ready")
: (false, "could not reach the Kubernetes API with that command — check kubectl and its context");
});
if (kubectl.Length == 0)
{
io.WriteLine(" Skipping this cluster (no working kubectl command).");
continue;
}

string? veleroNamespace = null;
if (AskYesNo(io, " Does this cluster use Velero for backup coverage?"))
{
var namespaceName = await AskProbedAsync(io, " Velero namespace", "velero",
async ns =>
{
var r = await ssh.RunAsync(alias,
$"{kubectl} get backups.velero.io -n {Sh(ns)} -o json > /dev/null");
return r.ExitCode == 0
? (true, "Velero Backup resources are readable")
: (false, "could not read Velero Backup resources there — check the namespace and RBAC");
});
if (namespaceName.Length > 0)
veleroNamespace = namespaceName;
else
io.WriteLine(" Skipping Velero coverage for this cluster.");
}

var checkWorkloads = AskYesNo(io, " Check Kubernetes node and deployment availability too?");
if (veleroNamespace is null && !checkWorkloads)
{
io.WriteLine(" Skipping this cluster: Velero and workload checks cannot both be off.");
continue;
}

var maxBackupAgeHours = veleroNamespace is null ? 26 : AskHours(io, 26);
// Use the proven SSH destination as the stable identity. This avoids an
// unprobeable free-text label and preserves valid old-config semantics.
kubernetesClusters.Add(new(alias, alias, kubectl, veleroNamespace, maxBackupAgeHours, checkWorkloads));
}
}

var configured = new List<string>();
if (dockerHosts.Count > 0) configured.Add($"{dockerHosts.Count} Docker host(s)");
if (logicalDb is not null) configured.Add("DB dumps");
Expand All @@ -658,6 +718,7 @@ User root
if (offsiteJobs.Count > 0) configured.Add($"{offsiteJobs.Count} off-site job(s)");
if (sqliteBackupDirs.Count > 0) configured.Add($"{sqliteBackupDirs.Count} SQLite scan folder(s)");
if (smartHosts.Count > 0) configured.Add($"SMART on {smartHosts.Count} host(s)");
if (kubernetesClusters.Count > 0) configured.Add($"{kubernetesClusters.Count} Kubernetes cluster(s)");

if (configured.Count == 0)
{
Expand All @@ -675,11 +736,12 @@ User root
SuppressionsFile: "suppressions.json",
ZfsReplications: zfsReplications.Count > 0 ? zfsReplications : null,
OffsiteJobs: offsiteJobs.Count > 0 ? offsiteJobs : null,
SqliteBackupDirs: sqliteBackupDirs.Count > 0 ? sqliteBackupDirs : null);
SqliteBackupDirs: sqliteBackupDirs.Count > 0 ? sqliteBackupDirs : null,
KubernetesClusters: kubernetesClusters.Count > 0 ? kubernetesClusters : null);

var configDir = Path.GetDirectoryName(Path.GetFullPath(configPath))!;
var suppressionsPath = Path.Combine(configDir, "suppressions.json");
File.WriteAllText(configPath, JsonSerializer.Serialize(config, WizardJson));
File.WriteAllText(configPath, SerializeWizardConfig(config));
if (!File.Exists(suppressionsPath))
File.WriteAllText(suppressionsPath, "[]\n");

Expand Down Expand Up @@ -709,6 +771,24 @@ with examples in restoreguard.sample.json.
Encoder = System.Text.Encodings.Web.JavaScriptEncoder.UnsafeRelaxedJsonEscaping,
};

/// <summary>Preserves explicit Kubernetes `veleroNamespace: null`: omitting it
/// has a different meaning (the record's default is `velero`). The general wizard
/// serializer omits nulls to keep the generated file approachable, so restore this
/// one semantic null after serializing the rest of the config.</summary>
private static string SerializeWizardConfig(RestoreGuardConfig config)
{
var root = JsonSerializer.SerializeToNode(config, WizardJson)!.AsObject();
if (config.KubernetesClusters is { } clusters && root["kubernetesClusters"] is JsonArray rendered)
{
for (var i = 0; i < clusters.Count; i++)
{
if (clusters[i].VeleroNamespace is null)
rendered[i]!.AsObject()["veleroNamespace"] = null;
}
}
return root.ToJsonString(WizardJson);
}

/// <summary>Asks for an SSH destination and immediately proves it works; on
/// failure shows the cause and re-asks (with a keep-anyway escape hatch).</summary>
internal static async Task<string> AskSshDestinationAsync(ISshProvider ssh, WizardIO io, string prompt, string defaultValue = "")
Expand Down
11 changes: 11 additions & 0 deletions tests/RestoreGuard.Tests/FakeLabSsh.cs
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,17 @@ public Task<SshResult> RunAsync(string hostAlias, string command, CancellationTo
if (command.Contains("echo ok"))
return Ok("ok");

// Kubernetes wizard probes use the same read-only readiness and Velero
// command shapes that Doctor/audit depend on.
if (command.Contains("get --raw=/readyz"))
return command.StartsWith("kubectl ", StringComparison.Ordinal)
|| command.StartsWith("pct exec 601 -- k3s kubectl ", StringComparison.Ordinal)
? Ok("") : Fail("command not found");
if (command.Contains("get backups.velero.io"))
return command.Contains("-n 'velero'")
? Ok("{\"items\":[]}")
: Fail("backups.velero.io not found in that namespace");

if (command.Contains("find '/var/backups/db-prod'"))
return Ok("12\n");
if (command.Contains("find ")) // any other dump dir doesn't exist
Expand Down
Loading
Loading