Remove filename from storageFiles

[mrtcode]

'filename' column should be removed from storageDownloadLog and storageUploadLog too, because they depend on storageFiles table.

For those tables a default 'filename' value must be set.
storageFiles
storageDownloadLog
storageUploadLog

[mrtcode]

If we create a random dir, I think there is no difference what filename the zip file has.

[mrtcode]

We need a script to point all storageFileItems files that have the same hash, to the same (lowest storageFileID) row in storageFiles.

There are a few options how it could be implemented.

Option 1:
Get unique lowest storageFileIDs with hash:
SELECT MIN(storageFileID) AS id, hash FROM storageFiles GROUP BY hash
And loop through all shards and update:
UPDATE storageFileItems JOIN itemAttachments USING(itemID) SET storageFileID=123 WHERE storageHash = 'hash'
But this is quite inefficient because there are many shards and the file can exists only on a few of them.

Option 2:
We can use the new storageFileLibraries table.
Firstly get storageFileID, libraryID, hash. Then get the minimum storageFileID for that hash. And then update each library's storageFileItems table where storageFileID not equal to the lowest storageFileID that we have for that hash.

Option 3:
Iterate through each row in storageFileItems on every shard, get storageHash from itemAttachements, use that hash to query storageFiles for MIN(storageFileID). Update if needed.

---

There is an unused 'getFileItems' function that is unimplemented: link. Now when we have storageFileLibraries table this function would be much easier to implement if needed. Or we should remove it if no longer needed.

[dstillman]

Why the backticks? size isn't a reserved word, as far as I know.

[dstillman]

What are the possible values for this?

[mrtcode]

It returns all keys that are prefixed with a given hash. In our case:

hash
hash/file1
hash/c/file1

[mrtcode]

We should be able to eliminate zip variable too. At least from prefixing S3 keys. listObjects just needs a hash to get the key, whether it's bucket/hash, bucket/hash/file1 or bucket/hash/c/file1.

[mrtcode]

When uploading a file the original code was copying 'hash/filename1' to 'hash' on S3 if 'hash' doesn't exist. duplicateFile function was doing this. I think it's better to simplify storage logic, and just depend on listObjects to get the legacy key. If we want to save S3 queries on more popular items, we can move legacy files to new keys from external scripts like zfs-purge.

[mrtcode]

I think we should completely eliminate filename and zip fields from storageFiles because this table should only be responsible for file storage and nothing else. We should fully separate storageFiles from user specific file parameters like filename or zip, because they can be different for each user. User specific parameters like filename or zip should come from specific items in shards and not from storageFiles.

[mrtcode]

This filename and zip removal from storageFiles is quite complicated, so we can probably just create a new PR with minimal changes required for zfs-purge.