Formatting doesn't work for target disks bigger than 2TB

[mdcowles]

rpi-clone is excellent; thank you very much for it.

It would be convenient if the disk formatting logic were updated so that it worked with target disks bigger than 2TB, which are increasingly common. I think that the issue is that Master Boot Record (MBR) partition tables are limited to 2TB. GUID Partition Tables (GPTs) are necessary beyond 2TB.

[framps]

Agree with you. Unfortunately I'm not a mbr/gpt expert to attack this feature request. I was glad when I got a PR which added gpt support in raspiBackup.

[mdcowles]

I'm glad to know about raspiBackup.

I think that creating a GUID partition table is mostly just a matter of using gdisk rather than fdisk. I could paste a transcript of doing it "by hand" here if that would help anyone here. (Though I suspect that ChatGPT could do the same.) The shell syntax gives me a headache and I respect the programmers who are willing to deal with it.

[framps]

I could paste a transcript of doing it "by hand" here if that would help anyone here.

Would be great 👍

Please note it's not just coding - at least if you're not familiar with the problem domain. Then you have to test everything carefully.

[mdcowles]

Here's a transcript of me doing it "by hand" on a scratch 1TB SSD:

$ # Booted on SD card, plugged in scratch 1TB SSD
$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0     2G  0 loop 
sda           8:0    0 931.5G  0 disk 
├─sda1        8:1    0   512M  0 part 
└─sda2        8:2    0   931G  0 part 
mmcblk0     179:0    0 477.5G  0 disk 
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0   477G  0 part /
zram0       254:0    0     2G  0 disk [SWAP]
$ # That says USB SSD is on /dev/sda and isn't mounted
$ # Blow away what's there
$ sudo wipefs -a /dev/sda
$ sudo sgdisk --zap-all /dev/sda
Creating new GPT entries in memory.
GPT data structures destroyed! You may now partition the disk using fdisk or
other utilities.
$ sudo gdisk /dev/sda
GPT fdisk (gdisk) version 1.0.10

Partition table scan:
  MBR: not present
  BSD: not present
  APM: not present
  GPT: not present

Creating new GPT entries in memory.

Command (? for help): n
Partition number (1-128, default 1): 1
First sector (34-1953525134, default = 2048) or {+-}size{KMGTP}: 
Last sector (2048-1953525134, default = 1953523711) or {+-}size{KMGTP}: +550M
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): ef00
Changed type of partition to 'EFI system partition'

Command (? for help): n
Partition number (2-128, default 2): 
First sector (34-1953525134, default = 1128448) or {+-}size{KMGTP}: 
Last sector (1128448-1953525134, default = 1953523711) or {+-}size{KMGTP}: 
Current type is 8300 (Linux filesystem)
Hex code or GUID (L to show codes, Enter = 8300): 
Changed type of partition to 'Linux filesystem'

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): y
OK; writing new GUID partition table (GPT) to /dev/sda.
The operation has completed successfully.
$ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINTS
loop0         7:0    0     2G  0 loop 
sda           8:0    0 931.5G  0 disk 
├─sda1        8:1    0   550M  0 part 
└─sda2        8:2    0   931G  0 part 
mmcblk0     179:0    0 477.5G  0 disk 
├─mmcblk0p1 179:1    0   512M  0 part /boot/firmware
└─mmcblk0p2 179:2    0   477G  0 part /
zram0       254:0    0     2G  0 disk [SWAP]
$ sudo mkfs.fat -F 32 -n BOOT /dev/sda1
mkfs.fat 4.2 (2021-01-31)
$ sudo mkfs.ext4 -L rootfs /dev/sda2
mke2fs 1.47.2 (1-Jan-2025)
Creating filesystem with 244049408 4k blocks and 61014016 inodes
Filesystem UUID: 42127c59-5915-437f-853f-42d9cfe2b6fa
Superblock backups stored on blocks: 
	32768, 98304, 163840, 229376, 294912, 819200, 884736, 1605632, 2654208, 
	4096000, 7962624, 11239424, 20480000, 23887872, 71663616, 78675968, 
	102400000, 214990848

Allocating group tables: done                            
Writing inode tables: done                            
Creating journal (262144 blocks): done
Writing superblocks and filesystem accounting information: done     
$ # Copy files
$ sudo mount /dev/sda2 /mnt
$ sudo rsync -axHAX --info=progress2 / /mnt
  2,148,726,758  30%   27.93MB/s    0:01:13 (xfr#47, to-chk=0/142735)   
$ sudo mkdir -p /mnt/boot/firmware
$ sudo mount /dev/sda1 /mnt/boot/firmware
$ sudo rsync -axHAX --info=progress2 /boot/firmware/ /mnt/boot/firmware/
     90,036,193 100%   45.92MB/s    0:00:01 (xfr#481, to-chk=0/498)  
$ sudo blkid /dev/sda1 /dev/sda2
/dev/sda1: LABEL_FATBOOT="BOOT" LABEL="BOOT" UUID="7354-866F" BLOCK_SIZE="512" TYPE="vfat" PARTLABEL="EFI system partition" PARTUUID="8a0fdd65-74b3-4987-a58d-ab4915c90ded"
/dev/sda2: LABEL="rootfs" UUID="42127c59-5915-437f-853f-42d9cfe2b6fa" BLOCK_SIZE="4096" TYPE="ext4" PARTLABEL="Linux filesystem" PARTUUID="f2b20eae-1798-47b5-960f-bfafb5341528"
$ # Put correct PARTUUIDs in the right lines
$ sudo vi /mnt/etc/fstab
$ # Put ext4 partition's UUID after "root=PARTUUID="
$ sudo vi /mnt/boot/firmware/cmdline.txt
$ sync
$ sudo shutdown -h now

I pulled the power cord, removed the SD card, plugged the power cord back in, and it booted correctly.

Note: That worked for me. I make no representations about how well it will work for anyone else. And I am very much aware that there is a million miles of difference between "Hey, here's something that worked for me" and reliable, tested code that will work for lots of different people on lots of different machines.

[framps]

I like to build up some gpt knowledge and will check what's the effort to enable rpi-clone for gpt.

Just to make sure I get your request correctly: You want to be able to use rpi-clone on a system running for example RaspbianOS on gpt instead of mbr.

[framps]

I just converted a Trixie RaspbianOS lite from mbr to gpt and used rpi-clone to clone the system. The update of the UUID failed but the cloned system booted successfully. So in general rpi-clone can handle a gpt system.

Looks like I don't get your issue 🙄

[mdcowles]

I was using a Raspberry Pi 5 that was running from the SD card which was originally made by the app Raspberry Pi Imager on my Mac. The SD card had an MBR partition table since that's what the imager writes.

I wanted to use rpi-clone to clone the SD card to an external USB drive which I planned to then boot from and run from. The external USB drive was 4TB.

I am pretty sure that what happened was that rpi-clone wanted to clone the SD card but also expand the root partition on the target drive so as to use all the space there. So it tried to write an MBR partition table to use all of the 4TB drive and that didn't work in some way. (Instead, it would need to know that it needed to create a GPT on the target drive rather than create another MBR partition table.)

rpi-clone ran to completion, but when I tried to boot from the 4TB USB drive, the boot process didn't finish and opened an emergency shell, I'm pretty sure because it couldn't find the root partition.

Does that help or is there something else I can clarify?

[framps]

rpi-clone ran to completion, but when I tried to boot from the 4TB USB drive, the boot process didn't finish and opened an emergency shell, I'm pretty sure because it couldn't find the root partition.

Hm, sounds the UUIDs were not updated in cmdline.txt and fstab. You can try syncUUIDs to check the UUIDs and then synchronize them. Just give it a try if gpt is used on your USB disk.

[framps]

I modified rpi-clone in my fork which now clones a gpt system successfully. BUT you have to to convert your system from mbr to gpt first. Use the tool I mentioned above for this.

Use wget https://raw.githubusercontent.com/framps/rpi-clone/refs/heads/issue_73/rpi-clone to download the modified version.

[mdcowles]

I guess that duplicating a GUID partition table is more like cloning than automatically switching from MBR to GUID when necessary is.

When you say:

you have to to convert your system from mbr to gpt first. Use the tool I mentioned
above for this.

I may be missing something but it looks like the only tool you mentioned above is syncUUIDs and converting from MBR to GPT doesn't seem like what that program does. And as far as I'm aware it's somewhere between not possible and pretty foolhardy to try to rewrite the partition table of a disk you've booted from. It seems that we're talking at cross-purposes again.

The use case I'm thinking of is copying from a booted SD card to a large USB drive (or even a large M.2 drive on the 5's PCIe interface). Since the SD card will start out with an MBR partition table from the imager program, it's not clear to me what should be updated.

[framps]

I'm sorry. When I finally got rpi-clone to support gpt I updated my previous comment the last days and unfortunately deleted the link to the nice tool which converts a mbr based RaspbianOS to gpt: mb2gpt from RonR

That's what I did in order to setup the environment and enable rpi-clone to clone a gpt based RaspbianOS:

1. Booted a standard mbr based RaspbianOS lite from SD card
2. Installed RaspbianOS lite with the imager on a second SD card
3. Ran mbr2gpt against the second SD card to convert the system from mbr to gpt
4. Shutdown and rebooted the converted gpt SD card
5. Started rpi-clone, got errors, debugged the code and finally added additional code to support gpt
6. Executed my modified rpi-clone and cloned to a third SD card
7. Booted the cloned gpt SD card successfully

So that's what you have to do:

1. Download and run mbr2gpt on your system (Make sure you have a backup - just in case)
2. Download the updated rpi-clone version from my branch
3. Run rpi-clone on the gpt based system
4. Boot the cloned gpt system

If you face any issues with my updated version just let me know.

I helped another guy the last days also to get RaspianOS to run on gpt and use a whole 4TB SSD which is not possible with mbr. I have no clue which clone tool he used. Apparently it was not rpi-clone 🙄 But he also used mbr2gpt successfully to convert his system to gpt.

[mdcowles]

Got it. Unfortunately, mbr2gpt won't run on a mounted SD card:

$ sudo ./mbr2gpt /dev/mmcblk0

/dev/mmcblk0 is in use (mounted)

I wouldn't criticize anyone for that limitation. Altering the partition table of a mounted disk would be pretty crazy.

But requiring that the clone's source drive be altered (requiring booting from a different drive) when it's only the target that strictly needs a different partition table (if it's larger than 2TB) means that the clone is not a simple one-step operation, but a slightly fiddly one.

[framps]

the clone is not a simple one-step operation, but a slightly fiddly one.

Maybe somebody else has a simpler proposal.

[mdcowles]

Can you help me understand why you're reluctant to have the program create a different style of partition table on the destination disk when the size of that disk requires it?

I'm very willing to believe that you have a good reason, but I haven't been able to see it so far. Is it that that's not strictly cloning? Or is it a problem to get the relevant data out of one partition table program and into a different one? Or something else that I'm missing?

[framps]

rpi-clone was written to clone the source disk contents to the target disk including the partition table type. As the name said - it just clones and does this very well. mbr2gpt was written to convert the partition table type from mbr to gpt and does this very well. It may be there exists a pure partition table conversion tool which can be used to create gpt on the target device. That way you don't have to convert the existing source device. Maybe somebody who knows gpt very good can help you on this. You also can initialize the target device with a gpt partition table manually and create the two required partitions and format them.

There exists the dedicated tool to convert the partition table (mbr2gpt) - and maybe other tools to do the partition table conversion - and a dedicated tool to clone a system (rpi-clone). Why should anybody spend a lot of time and effort to add the partition table conversion logic into rpi-clone and test it thoroughly given the conversion tool already exists?

It's the philosophy of Unix:

1. Make each program do one thing well. To do a new job, build afresh rather than complicate old programs by adding new "features"
2. Expect the output of every program to become the input to another, as yet unknown, program. Don't clutter output with extraneous information. Avoid stringently columnar or binary input formats. Don't insist on interactive input

I hope this helps 😃