Support resumable/pre-filled upload sessions for external integrations

[Fidasek009]

We are integrating MDDB-VRE Lite into MDDash (https://mddash.dyn.cloud.e-infra.cz/, https://github.com/CERIT-SC/mddash), a separate VRE/workflow application for molecular dynamics experiments. MDDash already has an automatic publication flow for InvenioRDM repositories: our backend creates a draft deposition, uploads files, then redirects the user to the repository draft UI to review metadata and explicitly publish.

We would like to offer a similar flow for MDPosit/MDDB through VRE Lite.

What we are trying to do

From MDDash, we would like to:

1. Prepare or upload the required metadata and MD files from the MDDash backend.
2. Create a VRE Lite upload/review session.
3. Redirect the user to VRE Lite.
4. Let the user review/fill missing metadata, verify uploaded files, and continue the normal VRE Lite publication/upload flow.

Current blocker

From reading the public repository and testing the deployed API, it looks like VRE Lite currently supports:

• POST /api/upload, which uploads/stages files and returns only a success message.
• GET /api/mc?bucket=..., which returns temporary S3/MinIO upload commands for large files.
• The browser UI flow, where metadata/form state appears to be kept client-side and is not restorable through a URL/session ID.

[gbayarri]

Hi,

The VRE lite is a temporary solution for generating easily the inputs.yaml file and provide the user the instructions for uploading by him/herself the files to MinIO. Just this. We are currently working on a more complex VRE with lots of features such as launch the workflow, automatic subworkflows, connection to HPC, users, filesystem and much more. But this is still under construction we don't expect to have it in production in the short term.

If you want to add your own features to the current version, feel free to fork the repo and adapt it to your needs:

https://github.com/mmb-irb/MDDB-VRE

Best.

[Fidasek009]

Thank you for your answer. If the VRE is just for creating a user-friendly way to upload files to MinIO, would it theoretically be possible to upload data from our end directly into the S3, effectively skipping the VRE? Would MDPosit still be able to pick up the files correctly and work with them?

[gbayarri]

Yes, of course.

If you use the workflow docker container, here you have the instructions for executing it using the MinIO files as inputs:

https://github.com/mmb-irb/MDDB-docker/blob/main/readme/docker-swarm.md#usual-execution

Inside the container, the MinIO bucket is mounted in fuse as a volume in order to use the files inside the bucket as inputs:

MDDB-docker/workflow/Dockerfile

Line 72 in faa83af

echo ' /app/goofys --endpoint http://minio:$MINIO_API_PORT $BUCKET /mnt' >> entrypoint.sh && \

For further information about all the options available for executing the workflow, you can take a look to the specific documentation of the workflow:

https://mddb-workflow.readthedocs.io/en/latest/