restic-restore-drill

Runs a full restic recovery drill: it opens the repository with the password from your recovery envelope — never the one on the host — restores a snapshot and verifies what came back. It refuses to run on the host recorded in the snapshot, so the drill can't quietly turn into a test of the machine that already works.

A backup job proves that writing worked. It can't prove recovery: it runs on the machine that already holds the credentials, and it never reads anything back.

---

The ten weeks

A production host backed up nightly to object storage. Every job green. restic check clean. The recovery password was written down and stored safely, the way you're supposed to.

The password in the envelope did not open the repository, and hadn't for about ten weeks.

Nothing on that host could have found it. The nightly job authenticates with the credentials on the host, so it succeeded every single night. The escrowed copy was only ever going to be used somewhere else, on the worst day of the year, by someone who could no longer ask the original machine what the real password was.

It was found by sitting down at a different machine and trying to restore.

---

What it checks

1 — the escrowed password opens the repository. The drill reads the password from a file you supply separately, and opens the live repository with it. If the file it reads the repository URL from also contains RESTIC_PASSWORD, it refuses to run: verifying a host against itself is the blind spot this exists to remove.

When it fails, it is careful about why. restic reports wrong password or no key found whenever it cannot decrypt any repository key — which is not the same as "your password is wrong". Deleting a repository's key objects produces that identical message with a perfectly good password. So the drill names both possibilities rather than sending you off to re-cut an envelope that was never the problem. Any other failure — an unreachable endpoint, a typo'd bucket, DNS — is reported as what it is, with restic's own words, and explicitly not as evidence against the envelope.

2 — you are not where the snapshot came from. restic records a hostname in every snapshot. The drill compares it to this host's hostname and stops if they match.

This is a guard against the obvious mistake, not proof of machine separation. The recorded hostname is metadata — --host at backup time, an FQDN versus a short name, a rename, or a container will all make the strings differ on the same physical box. Treat it as a seatbelt, not a certificate.

Before restoring, it checks the restore will fit. Three things consume the target filesystem, and sizing the check on the first alone is what lets a preflight pass and the disk fill anyway:

counted	how
file content	restic stats --mode restore-size
allocation rounding — every entry rounded up to a whole allocation unit	a one-byte file still occupies a full unit, so a million tiny files report ~1 MiB and land as several GiB. The unit is measured, by writing one byte into the target and asking what it allocated — stat -f %S reports the filesystem's fundamental block size, which is not the same thing on ext4 bigalloc, where clusters can be a thousand times larger
restic's cache	nothing — deliberately. It lives inside RG_DRILL_DIR, but by preflight time cat config, snapshots and stats have already walked the same metadata it stores, so it is already written and df already counts it; charging for it again would be double-counting. It does not grow afterwards either: data packs are read straight from the backend and never cached (measured — a 57 MiB restore left a 20 KiB cache, unchanged even by a full --read-data), and step 4 runs --no-cache.

Free inodes are checked separately, because they run out independently of bytes and the failure looks nothing like a full disk — the restore stops with ENOSPC on a filesystem df still reports as nearly empty. Filesystems with no fixed inode table (btrfs) report none free; that is skipped rather than treated as zero. XFS is not such a case — it reports a figure computed from free blocks, and the drill uses it.

Everything above gets a further 10% of headroom, since a filesystem at 100% is its own outage. Picking a tidier path doesn't help — /var/lib is on the root filesystem too — so the check is against actual free space on whatever filesystem RG_DRILL_DIR lands on. If the size can't be determined it refuses rather than restoring blind; set RG_SKIP_SPACE_CHECK=1 to override deliberately.

Estimates, not a guarantee: quotas, reservations, a concurrent writer filling the filesystem underneath the drill, or a compressed/deduplicating target can all move the real number. It refuses when the estimate doesn't fit, which is the case that actually bites — not a promise that a disk can never fill.

3 — the restore comes back. restic restore <id> --target … --verify, so restic rereads the files it wrote and compares them to the repository. One snapshot id is resolved up front and used for the host check, the restore and the reporting — snapshots --latest 1 returns the newest snapshot per host, while restore latest resolves to the single globally newest, so reading one and restoring the other can inspect one machine's snapshot and restore another's.

4 — a sample of the repository still decrypts. check --read-data-subset=5% rereads a random 5% of repository pack files. It is separate from step 3 and much weaker than it looks: the sample is random, so it may re-read only packs the restore already covered and tell you nothing new about the rest of the repository. Treat it as a cheap smoke check for rot, not as coverage. A full --read-data is the real answer; it costs egress on every byte, so schedule one periodically.

It runs with --no-cache, which is load-bearing rather than tidiness. Left to itself, check creates its own restic-check-cache-* directory under RESTIC_CACHE_DIR — which this drill pins inside the drill directory — fills it against the whole repository, then deletes it. That happens after the free-space preflight has already passed, and it is sized by the repository's total metadata rather than the snapshot being restored, so a small snapshot in a large repository could clear the preflight and then fill the target. It also removes itself before anything could measure it, which is why measuring the cache after a run shows nothing.

What it looks like when it catches something

$ restic-restore-drill
[…] drill starting against s3:https://…/backups/restic
[…] 1/4 does the ESCROWED password open the repository?
ABORT: the escrowed password in ~/.config/restic-drill/escrow-password could not decrypt
       any key in s3:https://…/backups/restic.
       Either way recovery from the envelope alone is currently impossible. Two distinct
       causes produce this same message:
         · the envelope is stale. The backup job keeps succeeding regardless, because it
           authenticates with the host's own credentials. Re-cut the envelope, or add its
           password as a repository key from a host that CAN open the repository.
         · the repository's key objects are missing or unreadable, in which case no password
           opens it and no key can be added until they are restored.
       Look at the repository's keys/ before deciding which one you have.
$ echo $?
1

And a clean run:

[…] drill starting against s3:https://…/backups/restic
[…] 1/4 does the ESCROWED password open the repository?
    ok: the escrowed password opens the repository
[…] 2/4 selecting a snapshot
    ok: snapshot 8f3ac12, recorded as taken on 'prod-01'
    ok: this host is 'recovery-box'; the snapshot was recorded on 'prod-01'
    ok: 412 GiB free for a 106 GiB restore
[…] 3/4 restoring snapshot 8f3ac12 and verifying the restored files
    ok: restored and verified 41882 file(s)
[…] 4/4 sampling repository data for rot
    ok: a random 5% of repository packs decrypt and verify
[…] DRILL PASSED — escrowed password opens …; restored and verified 41882 file(s) (96 GiB) on recovery-box

Exit codes: 0 drill passed · 3 smoke test only — not recovery evidence · anything else, a check failed. The smoke mode has its own status so a monitor that records only success/failure can't file it as a pass.

Exit 3 has two causes, and the final line always says which one: you passed --allow-source-host and the drill ran on the host recorded in the snapshot, or the snapshot records no hostname at all so the guard could not run. Both need --allow-source-host; neither is evidence that recovery works from another machine.

Requirements

Linux, bash 4+ (associative arrays — macOS's system bash 3.2 won't run it), restic, and Python 3.7+ (datetime.fromisoformat). GNU findutils for find -printf and GNU coreutils for du -k, stat and mktemp — these are separate packages, and a minimal system can have one without the other. Plus hostname (a hard requirement — the source-host guard cannot run without it, and the drill refuses up front rather than dying mid-run), df, and a POSIX awk (used to read the byte and inode columns of df; it comes from neither of the packages above, and without it the preflight reads free space as zero and refuses). sync is used but optional — without it the allocation probe falls back rather than failing. (stat is used only by the space preflight, to measure the target's allocation unit; without it the drill assumes 4 KiB rather than failing.) The documented envelope workflow also uses shred.

Backends: local repositories — an absolute path or local:/absolute/path. A relative local:repo is refused, because restic resolves it against the current directory, so the same setting would drill ./repo from your shell and /repo under a systemd service — and S3-compatible object stores with static keys (R2, MinIO, AWS). b2:/azure:/gs: are refused — this script forwards only AWS-style credentials, and half-supporting them would fail confusingly. rest:/sftp: are refused because this script does not implement trustworthy host extraction for them — both forms do carry a host (rest:https://host:8000/repo, sftp:user@host:/repo), but so do REST-over-Unix-socket URLs, SSH aliases resolved from ~/.ssh/config, and sftp: with a custom command, and a host guard that is right most of the time is not one worth trusting.

Configuration

variable	required	what it does
RG_ESCROW_PASSWORD_FILE	yes	file holding the password from your recovery envelope. Never the host's.
RG_REPOSITORY	yes¹	restic repository URL
RG_APPROVED_HOST	for s3:	expected endpoint host; the drill refuses a repository whose endpoint doesn't match
RG_DRILL_DIR	recommended	parent directory for the restore. The drill creates and owns a unique child inside it, so pointing this at an existing directory neither takes it over nor restores on top of it. Defaults to $TMPDIR.
RG_ENV_FILE	no	credentials file, read as data (below)
RG_SKIP_SPACE_CHECK	no	proceed when the restore size can't be determined, instead of refusing
RG_RESTIC_BIN	no	path to the restic binary; the test suite uses it to substitute a stub

¹ or RESTIC_REPOSITORY inside RG_ENV_FILE.

The credentials file is parsed, never sourced. Sourcing it executes it, with this script's privileges, at the moment it is read — and that file is the one thing here an attacker is most likely to be able to write. The simplest working attack is not subtle:

echo "DRILL PASSED — escrowed password opens s3:…"; exit 0

A monitor reading the exit status and the last line then records a successful recovery drill that never happened.

Two specific overrides are tempting and don't work: restic(), because the call goes through exec, which bypasses shell functions; and the drill's own r(), because it is defined after the file is read, so the script's definition wins. Don't generalise that into "function overrides are harmless here" — hostname() is never redefined by the script, and overriding it would defeat the source-host guard outright. Arbitrary code execution is the hazard; no clever override is required. So RG_ENV_FILE is read with a strict allow-list: KEY=VALUE only, keys limited to RESTIC_REPOSITORY and the three AWS variables, values passed to restic as environment rather than evaluated. Anything else — a function definition, an unexpected key like LD_PRELOAD, a line that isn't an assignment — is refused outright. A $(…) on the right-hand side is accepted as literal text and never runs.

RESTIC_PASSWORD in that file is refused specifically. If the drill could read the live password from the same file it reads the repository from, it would be verifying the host against itself — which is the blind spot this tool exists to remove.

Every restic call then runs in a subshell whose entire inherited environment is cleared, with only the variables the drill names put back. That is an allow-list rather than a deny-list on purpose: a deny-list would have to enumerate every variable that could ever hand restic a password, including ones added by future restic versions.

It is built with shell builtins, not by passing KEY=VALUE arguments to env(1). Those arguments are argv of a real process — readable from /proc/<pid>/cmdline, and recorded verbatim and permanently by execve auditing (auditd, eBPF, Falco). That was measured, not assumed: strace showed the AWS secret key and a proxy URL's embedded password in the execve record. For the same reason the script disables shell tracing at startup — under bash -x, every expanded assignment goes to stderr, and from there into journald or a CI log.

The allow-list does include the network plumbing — HTTP(S)_PROXY, NO_PROXY, SSL_CERT_FILE, SSL_CERT_DIR, AWS_CA_BUNDLE, AWS_REGION, AWS_DEFAULT_REGION — because stripping it makes the drill unusable in exactly the hardened environments most likely to keep a recovery envelope. None of it can substitute for the escrowed repository password. That is not the same as saying it is nonsecret: an authenticated proxy URL contains a password, which is precisely why none of this goes on a command line.

RG_APPROVED_HOST pins the endpoint host the repository must live in — compared verbatim against the host portion of the URL, so if your endpoint carries a port (s3:http://minio.internal:9000/bucket) the approved value must include it too (minio.internal:9000). Where the account is in the hostname (R2: <account>.r2.cloudflarestorage.com) that is effectively account pinning. On AWS it isn't: s3:s3.amazonaws.com/right/repo and …/wrong/repo both satisfy it. It catches a mistyped or swapped endpoint, not a wrong bucket. Local repositories don't need it.

Credentials need read plus the ability to create and delete lock objects — restic locks the repository for restore and check. A strictly read-only key authenticates and lists, then fails at the lock.

Two different kinds of evidence

Be clear with yourself about which one you have.

The envelope drill. You open the physical envelope, type what's written in it, run the drill, and remove the file afterwards. This is the only thing that proves the envelope works, and it's the check that would have caught the ten weeks. Do it by hand.

Run it as a subshell, so the transcription is removed when the drill returns rather than whenever you eventually close your terminal. An EXIT trap typed at an interactive prompt fires on shell exit — leave that shell open in tmux and the envelope password sits readable for days.

shred here is best-effort, not destruction. It overwrites and unlinks, but it cannot promise the bytes are gone on a copy-on-write or data-journalled filesystem, through a snapshot, or on an SSD that has already wear-levelled the block elsewhere — GNU says as much. Writing into $XDG_RUNTIME_DIR (normally a tmpfs) keeps the transcription off an ordinary persistent filesystem, which is the improvement actually on offer. It is not a guarantee the secret never reaches disk: tmpfs pages can be swapped out unless mounted noswap, the shell variable holding the password can be swapped too, and hibernation writes memory to disk wholesale. For a stronger property you need encrypted or disabled swap and no hibernation. If $XDG_RUNTIME_DIR is unset the example falls back to /tmp; on a disk-backed /tmp, prefer /dev/shm.

git clone https://github.com/LecoMV/restic-restore-drill
cd restic-restore-drill

envelope_drill() (                       # ( ) not { } — its own subshell
  set -eu                                # -e so a failed mktemp cannot fall through
  d="$(mktemp -d "${XDG_RUNTIME_DIR:-/tmp}/drill.XXXXXX")" || return 1
  [ -n "$d" ] && [ -d "$d" ] || return 1 # without this, "$d/escrow" would be "/escrow"
  trap 'shred -u "$d/escrow" 2>/dev/null; rm -rf "$d"' EXIT   # armed BEFORE the secret exists
  set +e                                 # from here we want the drill's status, not an abort

  read -rs -p 'password from the envelope: ' pw; echo       # -s: no echo, no shell history
  printf '%s\n' "$pw" > "$d/escrow"; unset pw
  chmod 600 "$d/escrow"

  RG_REPOSITORY='s3:https://<account>.r2.cloudflarestorage.com/backups/restic' \
  RG_APPROVED_HOST='<account>.r2.cloudflarestorage.com' \
  RG_ESCROW_PASSWORD_FILE="$d/escrow" \
  RG_DRILL_DIR=/mnt/restore-scratch \
  AWS_ACCESS_KEY_ID="$AWS_ACCESS_KEY_ID" AWS_SECRET_ACCESS_KEY="$AWS_SECRET_ACCESS_KEY" \
    ./bin/restic-restore-drill
)                                        # cleanup runs here, on the drill's return

envelope_drill; echo "drill exit: $?"    # 0 passed · 3 smoke only · else failed

Give it somewhere with room. A real drill restores the whole snapshot. With RG_DRILL_DIR unset it lands under /tmp — on most machines, the root filesystem. The drill will refuse rather than fill it, but a refusal every week is not a working drill. Point RG_DRILL_DIR at a filesystem sized for the entire snapshot, and remember its contents are your data, unencrypted, until the drill exits.

The unattended drill. systemd/ has a weekly timer reading a stored transcription. To install it — on a machine that is not the backup source, since the drill refuses to certify a run on the host recorded in the snapshot:

sudo install -m 0755 bin/restic-restore-drill /usr/local/bin/
sudo install -m 0644 systemd/restic-restore-drill.{service,timer} /etc/systemd/system/
sudo systemctl edit restic-restore-drill.service   # set RG_REPOSITORY, RG_APPROVED_HOST,
                                                   # RG_DRILL_DIR + its ReadWritePaths
sudo systemctl daemon-reload
sudo systemctl start restic-restore-drill.service  # run it once, in the foreground, first
sudo systemctl enable --now restic-restore-drill.timer

Every CHANGE-ME in the unit is deliberate: it will fail loudly rather than drill the wrong repository.

Note that the unit passes object-store credentials as Environment=RG_ENV_FILE=…, not EnvironmentFile=. That is not a style preference. EnvironmentFile= puts every assignment in that file straight into the process environment without passing through the parser above — and since the script runs under #!/usr/bin/env bash, such a file could set BASH_ENV, which bash reads and executes before the script's first line. A two-line payload then prints a forged DRILL PASSED and exits 0, and a root service records a recovery drill that never happened. Handing over the path instead means the file is parsed as data, and BASH_ENV is refused like any other unexpected key. Run the service once by hand before enabling the timer — a weekly unit that has never succeeded is not a drill, it is a cron job that sends mail. It proves the repository is restorable off-host using the credential in that file. Note what it does not establish: it cannot tell whether that file still matches the paper in your safe (it never reads the paper), and it cannot tell whether the file is a distinct credential at all — if configuration management ever populates it from the live password, every run stays green while proving only that the host can open its own repository. Keeping the escrow password as a separate repository key is what makes this check meaningful.

Set your envelope password up as a second repository key so recovery never depends on the host's own credential:

restic key add     # on the host that owns the repository; enter the envelope password

The drill installs nothing. It writes into a unique directory it creates under RG_DRILL_DIR (default $TMPDIR) and removes that directory on normal exit, on failure, and on SIGHUP/SIGINT/SIGTERM — the signal handlers clean up and then re-raise, so the drill actually terminates instead of carrying on against a directory it just deleted.

Removal handles the awkward case: restic restores directory modes faithfully, so a snapshot containing a 0555 directory lands one whose children rm -rf cannot unlink when you are not root, and a 0000 one cannot even be walked to count what was restored. The drill restores owner traversal on directories first — never on symlinks — both before it inspects the restored tree and again before removing it.

Cleanup happens before the verdict, not after. If the tree cannot be removed, the drill does not print DRILL PASSED; it fails. Announcing success and leaving the removal to an exit handler meant a cleanup that failed still exited 0, so anything reading only the status recorded a pass while an unencrypted copy of the backup sat on the recovery host. That is the same class of problem this tool exists to find, so it does not get a pass here either.

It cannot promise more than that, so it doesn't: SIGKILL, a power loss, or a full disk mid-restore will leave the restored tree behind — unencrypted — and restic's own cache lives inside that directory too. The drill also does not manage the escrow password file you point it at; that one is yours, which is what the subshell above exists to handle. After an abnormal exit, check RG_DRILL_DIR yourself — the next run will point out what it finds.

What this doesn't do

It does not make backups. restic backup already does that, and a tool that both writes and blesses its own backups is the problem this one exists to solve. Point it at a repository something else fills.

One bash script, one repository at a time. systemd/ ships an example service and timer, but the project manages no scheduler, no orchestration and no multi-host fleet — backrest and autorestic do that well.

It can't verify your envelope is stored somewhere sensible, or that anyone but you can find it.

Tests

./tests/bats/bin/bats tests/

64 tests. bats itself is vendored, runtime only, and accounts for most of the 30 files here. The nine that are mine: bin/restic-restore-drill, two systemd/ units, tests/restore_drill.bats, the CI workflow, .gitignore, README, LICENSE and NOTICE. The bats-assert / bats-support helper libraries are deliberately not vendored — the suite uses plain bash conditionals, so shipping them would have been 19 files of third-party code nothing loads. See NOTICE. ShellCheck runs over the suite as well as the script in CI — SC2314 is the only thing that catches ! cmd used as a bats assertion, and bash's set -e does not act on a return value inverted with !, so such a line silently does nothing and its test passes regardless.

No Bats installation required — with restic and python3 present (see Requirements), clone and run. The suite is hermetic — it builds its own repositories in a temp directory and keeps restic's cache there, so it never touches $HOME or a real repository.

Most of them are the negative cases, because those are the ones that matter: a stale escrow password, a drill run on the source host, a multi-host repository where the snapshot inspected must be the snapshot restored, and credential files that try to smuggle a password, a shell function or a command substitution past the parser.

License

MIT