Use requester-pays for mirroring in AnVIL deployments

[hannes-ucsc]

[bvizzier-ucsc]

Concerning billing projects for the DRS requests, they pointed me to this: https://data.terra.bio/swagger-ui.html#/DataRepositoryService/GetAccessURL.

So, if I’m reading this right, AnVIL mirror operations should specify an x-user-project value that is different from the standard AnVIL project.  (I’ll see about getting one set up.)

[dsotirho-ucsc]

@hannes-ucsc: "This would be a medium effort under the assumption that we can use our existing Google Cloud projects for the value of the x-user-project header when making DRS requests to TDR while mirroring. Since we currently do not incur any other Google Cloud storage charges, any such charges incurred in the future with this change in place would be attributable exclusively to mirroring. Note that we have five Google Cloud projects, one for each main deployment of Azul (tempdev, dev, anvildev, prod, anvilprod). An open question is whether we should also use requester-pays for HCA. If we can't use the existing Google Cloud projects, this would be a considerable effort."

[dsotirho-ucsc]

Assignee to verify with Benedict.

[bvizzier-ucsc]

I checked with Benedict, and we can use the existing Google Cloud project for the appropriate platform.

AnVIL is the only one that is asking us to track the mirroring egress at this time.

[hannes-ucsc]

This should be straight-forward but the devil may be in the details. Spike to create draft PR and test in personal deployment so we understand what the complications are. We only want to set the header during mirroring of AnVIL catalogs. HCA mirroring, direct downloads or curl manifests should be billed as before, to the bucket owner. We'd also need some indication that the header is effective. One way to do so would be to temporarily specify the ID of a project that the indexer SA does not have access to, e.g. that of platform-hca-dev. I'd expect TDR to return an error, or GCP when the signed URL is being accessed.