Reclaim space from duplicate files on ZFS without turning on deduplication. It scans your datasets, finds blocks with identical content, and reflinks them with block cloning. zfs-dedup runs offline unlike zfs own deduplication, it doesn't require additional memory, when it doesn't run.
% sudo zfs-dedup ~/
scanning 1 ZFS mountpoints
found 8329094 files
hashed 8329094 files (4977666 from cache), 882.3 GiB total
pruned 97 stale cache entries
saved 79.0 GiB (9.0%) across 3860361 blocks, 0 mismatches, 0 errors
nix run github:Mic92/zfs-dedup
or cargo build --release.
- Linux
- OpenZFS 2.2.0+ with the
block_cloningpool feature enabled (zpool upgradeorzpool set feature@block_cloning=enabled) - OpenZFS 2.3.0+ recommended: hash reads use
O_DIRECTto avoid evicting your hot ARC during a cold scan. Older ZFS silently falls back to buffered reads. FIDEDUPERANGEfor in-kernel verify+clone is in OpenZFS master (openzfs/zfs#18745, not yet in a release). Older ZFS requires--forcefor the userspace verify path.- Read-only ZFS bind mounts over a writable dataset (e.g.,
/nix/storeon NixOS) are remounted read-write in a private mount namespace. The host's mounts are not modified.
# Dry run, all mounted ZFS datasets
zfs-dedup -n
# Dedup specific paths
zfs-dedup ~/photos ~/backup
# Limit hashing threads
zfs-dedup -j 4 /tank
usage: zfs-dedup [-n] [-c CACHE] [-j N] [DIR...]
DIR... directories to scan (default: all mounted ZFS datasets)
-c, --cache PATH hash cache (default: $XDG_CACHE_HOME/zfs-dedup/cache.redb)
-n, --dry-run don't modify anything
-j, --jobs N hashing threads (default: all cores)
-f, --force dedup even without FIDEDUPERANGE (racy verify+clone)
-V, --version print version
Re-runs are cheap: file hashes are cached and only files that changed get rehashed.
Deduping safely needs to compare two ranges and clone them in one
operation (FIDEDUPERANGE), otherwise something could write to one of
them in between. OpenZFS gained this ioctl in
openzfs/zfs#18745 (master, not yet released).
zfs-dedup uses it automatically when available.
On older ZFS, zfs-dedup falls back to a userspace compare followed by
FICLONERANGE and refuses unless you pass --force to accept the small
race window. Only use --force when you are sure that no process will
modify your data while zfs-dedup runs. The fallback also bumps the
mtime of deduped files, because ZFS treats a clone as a write.
FIDEDUPERANGE leaves timestamps untouched.
FIDEDUPERANGE is only a new ioctl in the kernel module. It adds no
pool feature flag and does not change the on-disk format. It uses the
same block_cloning feature FICLONERANGE has used since 2.2. For a
pool created before that:
- Enable
feature@block_cloning(zpool upgradeorzpool set feature@block_cloning=enabled <pool>). Pools without it are skipped. - On 2.2.1 through 2.2.x set the
zfs_bclone_enabled=1module parameter. It defaults to on since 2.3.0. - Blocks written before the feature was enabled clone fine, no rewrite needed.
- Encrypted datasets can only clone into each other when they share the same master key, i.e. the same encryption root.
Once your ZFS includes openzfs/zfs#18745, zfs-dedup detects
FIDEDUPERANGE automatically and --force is no longer needed. The
hash cache from earlier runs stays valid.
zfs-dedup needs roughly 240 MiB per million files in the largest dataset during the walk, dropping to about 65 MiB per million afterwards. As datasets are scanned one at a time, only the biggest one matters.
| files in largest dataset | peak RSS | after the walk |
|---|---|---|
| 1 M | ~240 MiB | ~65 MiB |
| 10 M | ~2.4 GiB | ~650 MiB |
| 50 M | ~12 GiB | ~3.2 GiB |
- ZFS only. For btrfs or XFS use bees or duperemove.
- Files must be on the same pool and share a recordsize to be cloned into each other.
- Cloned blocks share storage but show up twice in
du. Checkzpool get bcloneused,bclonesavedfor actual savings, the tool also will report savings
MIT