Recover gracefully from stale resume tokens

[beefstew]

Summary

When resume = yes is set and a zfs send | zfs receive -s is interrupted, pyznap saves a resume token on the destination and tries to resume on the next run. If the source snapshot referenced by that token has been cleaned up (e.g. a short-lived frequent snapshot deleted by retention policy before the next send window), every subsequent pyznap send fails with:

cannot resume send: '<pool>/<fs>@pyznap_<timestamp>_frequent' used in the initial send no longer exists
ERROR: Error while sending to <dest>: cannot receive: failed to read from stream...

All configured retries hit the same wall because they all attempt to resume the same dead token, and the next scheduled run repeats the loop. The destination filesystem is stuck indefinitely — no new snapshots can be pulled — until an operator manually runs zfs recv -A <dest>.

This PR patches pyznap/send.py so that when the resume step fails with cannot resume send in stderr, pyznap automatically aborts the resumable receive state (using the existing ZFSDataset.receive_abort() helper) and falls through to a fresh incremental from the newest common snapshot.

Reproducer

Any setup with resume = yes, aggressive short-retention snapshots (e.g. frequent = 4, pyznap snap every 15 min), and an occasional mid-stream interruption is susceptible. Concretely:

1. Trigger a partial zfs recv -s on the destination that leaves a resume token pointing at a recent frequent snapshot.
2. Wait until retention on the source cleans up that snapshot (under an hour with frequent = 4).
3. Observe pyznap send loop forever across retries and cron cycles, never advancing past the dead token.

Root cause

send_snap() collapses every CalledProcessError into return code 2 — the same code used for transient failures that should retry. send_filesystem() then treats any non-zero result from the resume branch as fatal and never reaches the fresh-incremental path. The existing abort_resume() function (and its call site) are commented out at the bottom of pyznap/send.py, suggesting this was on the author's radar but not wired up.

Fix

Two edits in pyznap/send.py, no changes elsewhere:

1. send_snap() — when the subprocess failed and we were resuming and stderr contains cannot resume send, return a new sentinel 3. All other failure paths continue to return 2 as before, so transient-error retry behaviour is unchanged.

2. send_filesystem() — the resume branch handles rc == 3 by calling dest_fs.receive_abort() (already defined on ZFSDataset in pyznap/pyzfs.py), clearing resume_token, and falling through to the existing fresh-/incremental-send logic. Non-3 failures still return immediately.

The detection substring cannot resume send is emitted by the OpenZFS kernel module for every known stale-token error variant (used in the initial send no longer exists, destination has snapshots, etc.), so one check covers the failure modes.

Why it's safe

• zfs recv -A only discards the in-flight partial stream; committed snapshots are untouched, so common (computed earlier in send_filesystem) stays valid without recomputation.
• Return code 3 is only produced when resume=True, so non-resume sends keep their existing 2-retry path.
• If receive_abort() itself fails we return 1, which send_config's retry loop does not retry — no risk of tight-looping on a broken destination.
• No new dependencies, no config surface changes, no behavior change for healthy transfers.

The patch

stale-resume-token.patch

Unified diff against master:

--- a/pyznap/send.py
+++ b/pyznap/send.py
@@ -73,7 +73,11 @@
         logger.error('Error while sending to {:s}: {}...'.format(dest_name_log, err))
         return 1
     except CalledProcessError as err:
-        logger.error('Error while sending to {:s}: {}...'.format(dest_name_log, err.stderr.rstrip().decode().replace('\n', ' - ')))
+        stderr_text = err.stderr.rstrip().decode().replace('\n', ' - ')
+        logger.error('Error while sending to {:s}: {}...'.format(dest_name_log, stderr_text))
+        if resume and 'cannot resume send' in stderr_text:
+            # resume token references a snapshot no longer on source; signal caller to abort
+            return 3
         # returncode 2 means we will retry send if requested
         return 2
     except KeyboardInterrupt:
@@ -177,11 +181,22 @@
         logger.info('Found resume token. Resuming last transfer of {:s} (~{:s})...'
                     .format(dest_name_log, bytes_fmt(base.stream_size(raw=raw, resume_token=resume_token))))
         rc = send_snap(base, dest_name, base=None, ssh_dest=ssh_dest, raw=raw, resume=True, resume_token=resume_token)
-        if rc:
+        if rc == 3:
+            logger.warning('Resume token for {:s} references a missing source snapshot; '
+                           'aborting resumable state and starting fresh incremental...'
+                           .format(dest_name_log))
+            try:
+                dest_fs.receive_abort()
+            except (DatasetNotFoundError, DatasetBusyError, CalledProcessError) as err:
+                logger.error('Failed to abort resumable receive on {:s}: {}...'.format(dest_name_log, err))
+                return 1
+            resume_token = None
+        elif rc:
             return rc
-        # we need to update common snapshots after finishing the resumable send
-        dest_snapnames = [snap.name.split('@')[1] for snap in dest_fs.snapshots()]
-        common = set(snapnames) & set(dest_snapnames)
+        else:
+            # we need to update common snapshots after finishing the resumable send
+            dest_snapnames = [snap.name.split('@')[1] for snap in dest_fs.snapshots()]
+            common = set(snapnames) & set(dest_snapnames)
 
     if not common:
         if dest_snapnames:

Verification

Apply with patch -p1 from the repository root. Verified against current master:

• patch -p1 --dry-run — clean, no fuzz
• python3 -m py_compile pyznap/send.py — valid syntax
• python3 -c "import pyznap" — imports cleanly

Synthetic end-to-end test

1. On the destination, start a slow pull into a throwaway dataset and interrupt it mid-stream so a resume token is written:

   zfs send -v <pool>/<fs>@<recent-frequent> | pv -L 1m | zfs recv -s <dest>/test_resume
   # ^C mid-stream
   zfs get -H receive_resume_token <dest>/test_resume   # non-empty

2. On the source, destroy that specific snapshot to simulate the retention-cleanup race:

   zfs destroy <pool>/<fs>@<recent-frequent>

3. Add a pyznap config stanza pointing the source at <dest>/test_resume, then pyznap send.

Expected log (patched):

INFO: Found resume token. Resuming last transfer of <dest>/test_resume ...
ERROR: Error while sending to <dest>/test_resume: cannot resume send: ... no longer exists...
WARNING: Resume token for <dest>/test_resume references a missing source snapshot; aborting resumable state and starting fresh incremental...
INFO: Updating <dest>/test_resume with recent snapshot ...
INFO: <dest>/test_resume is up to date...

Cleanup: zfs destroy -r <dest>/test_resume.

Related

Issue #71 (Invalid option 't' while resuming transfer) touches on the adjacent category of resume-token problems. This PR does not attempt to address that one.