up2k (Copyparty's upload protocol) support

[scmcgowen]

Is there an existing feature request for this?

• I have searched the existing issues before opening this feature request.
• I am checking every checkbox without reading the label.

Describe the feature you would like to see.

I would like to see support in the panel for Copyparty's fast and reliable up2k upload protocol.
This protocol can be used to do parallel uploads and to bypass file upload limits from tunneling providers.

Describe the solution you'd like.

When you upload a file via drag and drop or the upload button, it should try to use up2k if possible

Additional context to this request.

https://github.com/9001/copyparty

[9001]

hey, copyparty dev here --

sorry for the wall of text, it's purely handwritten fwiw :-) and tried to keep the ranting to a minimum

first and foremost, the main reason why you may NOT want to choose up2k:

the biggest drawback of the up2k protocol is that (as far as I know) the only existing implementation of the server-side part of the algorithm is inside copyparty itself, and it will not be trivial to extract that into a separate reusable library. So you'd probably have to implement it from scratch, based on documentation in the copyparty repo, which is not very complete or detailed.

the lack of protocol documentation is simply because nobody's ever asked or shown interest, but it would probably be healthy to do something about that;

• when/if proper documentation is written (no idea when that will be!), it will be here (currently 404): up2k.md
• please ignore up2k.txt; it describes the rough prototype of the protocol, only ever used in the php-based precursor to copyparty, and the protocol has since changed/improved

the closest thing there is to documentation right now is devnotes.md#up2k, and this also mentions why copyparty itself uses up2k instead of other alternatives, and the "why chunk-hashes" section explains why the protocol does checksums of each chunk of each file individually (instead of checksumming each whole file as a whole)

other protocols to consider

I am only aware of TUS as a noteworthy alternative to up2k, but I am not very familiar with it.

still, I can say for sure that TUS has these advantages:

• TUS is massively more popular, with many implementations (both for the server and clients)
• TUS is much simpler, and thus easier to implement
• uploads can start immediately without any initial processing delay
  • up2k requres that each file must be fully hashed before the upload can begin, since the server needs to know the checksum of each chunk to expect, which is necessary to power some of up2k's advantages

if you combine the TUS-extensions checksum and concatenation then it should be a decent alternative to up2k;

• you can still send several chunks (parts of the file) to the server in parallel;

  • this is a huge boost for upload-speed when uploading one large file over a long physical distance, or through a cheapskate route / ISP / network-exchange, of which there is MANY on the internet -- but for LAN uploads this is entirely irrelevant
  • up2k writes each chunk directly into the final target file on the server, to use the HDD as efficiently as possible, and fragmentation is avoided by sparse-allocating the file to the final size before the upload begins
    • HOWEVER, due to hyrodactyl being php, it is maybe impossible to keep a pool of FDs open for the lifetime of the upload, meaning you would probably have to write each chunk to an intermediate file first... hm, maybe stitching them with reflinks could work, depending on underlying filesystem (btrfs/zfs/xfs only I think)
  • TUS: the standard TUS-extension for doing chunked uploading (concatenate) will instead write each incoming chunk to an individual file on the server's HDD, and then only stitch them together into the full final file when all chunks are uploaded, which is extremely slow and taxing on the server HDD, but there might be other TUS extensions which avoids this drawback
    • EDIT: again, with hyrodactyl being php, maybe the intermediate files are absolutely necessary anyways

• you still get data integrity verification;

  • corruption happens surprisingly often in the wild due to bad hardware, bad RAM in client or server, network equipment, buggy soundcard drivers, loose cables, ...
  • up2k will detect and automatically reupload only the corrupted chunk (not the whole file), and I think TUS can do this too (just make sure it does checksums for each chunk instead of the whole file)

the advantages of up2k

since the upload is initiated by first sending the full list of checksums to the server,

• advantage 1: the server can instantly tell whether that file already exists somewhere on the server, and tell the client that it doesn't need to do the upload since it could just make a serverside copy instead

• advantage 2: will usually detect corruption that is caused by bad RAM or other hardware issues in the uploader's PC, since the same chunk will be read from the HDD and copied into RAM at least twice; that is, assuming the file is large enough to not fit into the OS disk-cache at all, or that it gets evicted by other activity in the meantime

• advantage 3: it becomes difficult for someone with nefarious purposes to insert a bit of malicious data into the middle of an incomplete upload, and I don't think TUS offers any protection against this (mostly hypothetical) attack

on writing each chunk directly into the target file

NOTE: this section might be irrelevant for hyrodactyl, since it is written in php. If we're talking short-lived php interpreters launched by fcgi/fpm then just ignore this. However, if it is a long-lived php instance controlled by RPC or something like that, then maybe keep reading :-)

regardless if you choose up2k or something else, this is my approach to splicing the uploaded chunks directly into the final file (instead of a pool of intermediate files); it has worked well and I'd recommend something along the same lines:

• when an upload is initiated, I first create a sparse file with the correct final size
• the client includes an initial list of checksums for each chunk that it plans to upload, and the server keeps track of which checksums have not been confirmed yet
• the file is open()ed several times server-side, keeping a pool of file-descriptors which can be used to write with in parallel
• when the client POSTs (or perhaps PUTs) a chunk, a free file-descriptor is selected, and the correct offset is seeked to
• as the data is written to the file, the data is also sent through sha512 so the integrity can be verified
• if the checksum of each chunk is OK, the chunk is removed from the list of chunks-to-be-uploaded

I've simplified the outline a bit; the file-descriptor pool is initially empty, and I create and close file-descriptors as the tcp connections from the client are opened and closed. BUT, when running on Windows it's important to leave at least one file-descriptor open at all times, because otherwise you'll incur the wrath of Windows Defender as it repeatedly scans the file for each chunk written.

The only filesystem I'm aware of where this does not work (due to not supporting sparse files) is FAT32, so I have a separate codepath in the client which does sequential uploading when necessary. This is true when the copyparty server is running on android, or from a bootable flashdrive. On serverside I detect this by trying to fallocate a 1 MiB file and seeing if it ends up occupying 1 MiB worth of datablocks in the filesystem; if it does, then I refuse any POSTs of chunks which are not to be written starting from the file's current EOF.

btw, nice issue number hehe

[naterfute]

Lol, I didn't even realize the issue number. Are you in the discord? I would love to have some talks with you if your ok with it, my username is naterfute

[9001]

ah no, truth be told i just bumped into this issue since someone mentioned copyparty so it showed up in the "oh no what did I break this time" feed 😁

my attention is kinda all over the place so I'd be really intermittent on discord, plus I'm norwegian so timezones are kind of a pain :p maybe best to just continue here honestly, happy to help brainstorming ideas/concepts but can't write much code these days :>