Issue when using old digest file with new disk, suggest clear warning in doc

[Gitsarry]

On the Blocksync-fast FAQ is written as answer to the question "What are the main uses of this program ?":
"The program can be used for simple cloning of entire disks"

So I used blocksync-fast to "clone" an existing disk to a new bought disk. But I recognized that the partition table of the new disk was not changed, and the new disk was not usable as backup disk, so it was not cloned precisely.

This was wrong from my side:

Now I learned that blocksync-fast does not copy partition table data. This is nowhere mentioned in the documentation.

This is extremely important information for people who use blocksync-fast as a precise backup program.

Could you elaborate why you skip some first bytes from the synchronisation?

You should definitely state that a tool like dd is needed first to produce a recoverable backup disk!

[nethappen]

The program should be able to clone an entire disk. If you've made a valid copy of the disk, for example:
$ blocksync-fast -s /dev/sda -d /dev/sdb
Then the entire contents, including the partition table should be copied from /dev/sda to /dev/sdb.
You can also add the --no-compare option to speed up copy.

If you cloned only the partition /dev/sda1 to /dev/sdb, it is an image of only the partition without partition table.
What command did you use exactly and are the drives the same ?

[Gitsarry]

Hello, thanks for quick reply.
Sorry, I guess the problem sits in front of my screen.
The case is, that I used to clone a complete computer device like ORIG_DEV=/dev/sda
to an external USB drive like KLON_DEV=/dev/disk/by-id/ata-Samsung_Portable_SSD_T5_S3ULNXXXXXXX
using a digest file like DIGEST_FILE=/var/cache/blocksync-fast/disk1.digest

The command is:
blocksync-fast --algo=XXH3LOW --digest="$DIGEST_FILE" --show-progress --src="$ORIG_DEV" --dst="$KLON_DEV"

Now I wanted to have two USB backup drives (stored in different locations for security), and did run the same above command with the newly bought new USB disk. And the newly cloned disk did not work, and I saw there was still the original partition table on it.

Am I right, that one must not change any data at all on the target device when using digest file, because otherwise, blocksync-fast thinks, reading the digest file, that some data was written to the target device and would be still there, while in fact the real data on the traget device is something other, and will not be corrected by blocksync-fast?

Under "Limitations and Notes" is written:
"It is also important to note that when synchronization with the Digest file is interrupted, there may be an inconsistencies [sic!] between the state of the Digest file and the target storage device. Therefore, after each such interruption, it is recommended to rebuild the Digest file from the target backup [...]"

Could you make this text clearer to understand? What exactly is "synchronization with the Digest file", it's somehow nebulous. Is such "interruption of the synchronization with the digest file" my case, when you just use a new device as target device? I would suggest to put a clear warning with examples of pitfalls, as this is very dangerous as you get corrupted and useless "backups".
And do I understand correctly, that with "rebuilding of the digest file" you mean just deleting it, so it will be freshly created?

[nethappen]

You've diagnosed the problem correctly. That's exactly how it works!
The digest file is like a "table of contents" of what is on the destination disk.
When you reuse it, the program assumes that this data is still there and skips copying it.

When you gave it a new, empty disk, the program looked at the digest file and thought:

The destination disk already has data (according to the digest file), so no need to copy it.
Except that physically, this data wasn't on the new disk - hence the problem with the partition table.

That's exactly why you need a SEPARATE digest file for EACH destination disk.
It's like having a separate map for each backup disk.

--

Regarding "interruption of synchronization with the Digest file" - this refers to a situation where something went wrong during copying (e.g., disk disconnected, power outage). In such cases, the digest file may "think" that certain blocks were written when they actually weren't. So it is perfectly to works on copy of digest file to avoid this situation:
cp --reflink /digest-file-current /digest-file-working and if the operation completes successfully, then: mv /digest-file-working /digest-file-current

But your case is different - you used the same digest file for a different disk, rather than continuing an interrupted copy.

And you're right about rebuilding the digest file. You can either let the program create a new one automatically during the next sync (by simply providing differnt file name for a digest file or by removing the old one).

Or if you want to explicitly rebuild the digest file from an existing backup disk:
$ blocksync-fast --make-digest -s "$KLON_DEV" -f "$DIGEST_FILE" --algo=XXH3LOW

--

By the way, if you don't use a digest file, the program compares blocks directly by reading from both disks. It's slower, but safer when switching between different disks.

Thanks for the suggestion. I'll definitely clarify this documentation so others don't fall into the same trap. The current description is indeed too technical and can be misleading.

[Gitsarry]

Thank you very much for the detailed explanation. Now I clearly understand (maybe I did understand some years ago when I started using blocksync-fast, but forgot...). Something like this should be added to the doc.

So this digest file is really dangereous, but on the other hand, it makes the backups so incredibly fast, so I feel like using it. Whenever there would be an interruption, I would just delete it and create it new, that's simple and safe.

[Gitsarry]

How about collision resistency when using digest file? It could be that blocks on the source device change, but have the same checksum as in the digest file, so target device won't be updated, resulting in data loss / corruption in the backup. In the docs you mention it shortly, just stating two algos (one of which I use, see above) that would be "good and sufficient" "in practice". Maybe there could be added some link to in-depth explanation of the collision problem?

[Gitsarry]

My other use case was the following:
I have an USB disc and cloned it with blocksync-fast and digest file to a second USB disc. To instantly verify the backup, I then disconnected to source disk, and instead connected the fresh clone. In the next backup cycle, the steps where the same. This worked with two disks and only exactly one digest file, because the disks where swapped when they where exactly the same.

But now with three USB disks, this would be:
Clone 1 to 2, make 2 active
Clone 2 to 3, make 3 active
Clone 3 to 1, make 1 active
I thought if this would be consistently usable with a digest file for each of the three disks, and I think that would theoretically work, but it is too complicated and nearly not understandable what happens there, so practically too vague, so I will not use digest file with three rotating disks.

[nethappen]

How about collision resistency when using digest file? It could be that blocks on the source device change, but have the same checksum as in the digest file, so target device won't be updated, resulting in data loss / corruption in the backup. In the docs you mention it shortly, just stating two algos (one of which I use, see above) that would be "good and sufficient" "in practice". Maybe there could be added some link to in-depth explanation of the collision problem?

The collisions mentioned in the documentation refer to a theoretical situation where two different data blocks could produce the same checksum.

For example, in algorithms like CRC32 or XXH3LOW, a single checksum is 4 bytes long. If a data block changes, there is a probability that this change goes undetected – meaning a collision occurs.

This probability is approximately 1/2³² = 0.000000025%. This means that one processed data block might not be updated in the backup.
The larger the data block, the more data that would not be updated in that particular backup.

In practice, however, this situation will not occur.

Additionally, in blocksync-fast you can use algorithms with longer hashes – for example, 256-bit algorithms, which produce a 32-byte hash per block.
The collision risk then becomes 1/2²⁵⁶, which is:

0.0000000000000000000000000000000000000000000000000000000000000000000000000000008636% – i.e., zero.

For context:
2²⁵⁶ is approximately 10⁷⁷, compared to the number of atoms in the observable universe, which is around 10⁸⁰.

A longer hash naturally increases the digest file size, but this can be balanced by increasing the processed block size.

In practice, for 32-bit algorithms, this situation is so extremely unlikely that it should not occur under normal conditions.
Users have been using CRC32 and similar algorithms in backup tools for decades without practical issues.

[nethappen]

My other use case was the following: I have an USB disc and cloned it with blocksync-fast and digest file to a second USB disc. To instantly verify the backup, I then disconnected to source disk, and instead connected the fresh clone. In the next backup cycle, the steps where the same. This worked with two disks and only exactly one digest file, because the disks where swapped when they where exactly the same.

But now with three USB disks, this would be: Clone 1 to 2, make 2 active Clone 2 to 3, make 3 active Clone 3 to 1, make 1 active I thought if this would be consistently usable with a digest file for each of the three disks, and I think that would theoretically work, but it is too complicated and nearly not understandable what happens there, so practically too vague, so I will not use digest file with three rotating disks.

1 -> 2 (digest = 2)
2 -> 3 (digest = 3)
3 -> 1 (digest = 1)

You need 3 digest files in this situation or not use digest files.