Clean up AnVIL AWS ODP mirror of HPRC files

[bvizzier-ucsc]

anvil13 will remove between 500 and 600 TB of data from HPRC due to data cleanup by the HPRC team. There are no new HPRC files introduced in anvil13.

Benedict would like us to remove those files from the AWS ODP mirror.

This ticket should be blocked by the indexing of anvil13, which we should be handed off to us this week.

[bvizzier-ucsc]

I was informed that anvil13 does not contain any new open access datasets or files. The only change to open access datasets is the removal of these files from the HPRC dataset.

[dsotirho-ucsc]

@hannes-ucsc: "We would need a list of files to be removed. It is important to note that the mirror does not store duplicate files because it is content-addressed. In other words, if the only files removed in anvil13 are duplicates, there would be nothing for us to do."

[dsotirho-ucsc]

Assignee to coordinate.

[bvizzier-ucsc]

Thought... As a preliminary pass, the file manifest includes the list of files with the md5sum and the mirrored file names. We should be able to compare the manifests from anvil12 with the manifest from anvil13 to identify which files have been removed.

[bvizzier-ucsc]

anvil12 manifest that also includes the mirrored file names.

anvil-manifest-f7cd12d3-1a79-560e-99e4-aebf4c601652.6dc47a93-a062-5a0d-be43-08a4bdca6597.tsv.zip

[dsotirho-ucsc]

@hannes-ucsc: "Once blocker #7744 lands, assignee to compare the manifest from the above comment with the new manifest for HPRC in anvil13 from #7670 (comment) and determine which files were removed in anvil13."

[dsotirho-ucsc]

PR #7755 for issue #7744 has been merged to develop.

[dsotirho-ucsc]

Spike to perform the steps outline in #7740 (comment) as part of the promotion.