docker always needs to run with `--generate_export_script_only`

[marrip]

If I understand your code correctly, you actually need to use --generate_export_script_only if you run the container image as the paths in the generated script would not match the paths used in the container, right?

[danielvo]

Since the file paths inside and outside the container may differ (based on how the volume mounting was done by the user), the bash script content that references specific files needs to differ based on whether it is supposed to be executed from within the image or outside of it. The "--generate_export_script_only" option should lead to: 1) creating a bash script that should be working as intended outside of the image, 2) not executing that script from within the image (executing it might not work, depending on the volume mounting). If the option is omitted, the paths within the bash script will refer to file paths as they are seen within the image, and the script will be executed (delayed execution on the host system wouldn't necessarily make sense, as the paths may differ).

[marrip]

ah, true, I didn't go back to see that you are using the bool to decide wether or not the script paths are adjusted for or not. Sorry about that. Would it make sense to open the export script up so that paths can be specified when calling it instead of hard-coding them? It could be simply positional arguments indicating the export file list and the password file and anything else required. That would allow us as well to reuse the script once it is generated. Just an idea as it would resolve the whole path handling issue a bit ☺️

[danielvo]

The downside would be that the user would be required to specify paths twice, once for the tool and once when utilizing the tool's output (that is admittedly much less of an issue if one is using a wrapper to run the tool). Regarding the path handling: The tool still needs to work with paths related to the ID list, the LocalApp/TSOPPI output and its own output, so one would only simplify the bash script generation. In that case, one would need bash script parameters for 1) the LocalApp/TSOPPI output parent directory, 2) the output prefix (this should identify the export file list path, the archive path and the md5sum file path), 3) the password file.

[marrip]

True, one would have to specify paths several times. However, it would omit a situation where a user creates an export script to run manually and then trying to run the script within the container. I don't think it is a big issue but it is a bit confusing. I guess in general it would be good to check how big the differences are timewise running the export script on the host system, inside the container, via python and actually using python packages to run the packaging/encryption. The outcome would enable us to make the best decision for how to package and encrypt and we can make an informed decision about how we want to handle the setting of paths.

[danielvo]

I don't think it's currently possible to create a bash script for host-system execution and run it within a SADET container while using the tool's main Python script. In case the same output paths are supplied twice, the bash script would be over-written with container-compatible paths if the "--generate_export_script_only" option is not utilized in the latter tool run. (If one runs the image interactively, then running any bash script within it is possible.. but I don't consider this a danger with standard use.)

Considering in-production InPreD nodes only, we have at least 4 different environments to test the tool in, Docker and Singularity image versions and multiple possible ways of running the encryption inside a container. While I would be interested in finding the most efficient within-container solution and seeing how it compares to the host-system execution in the different setups, I would now prioritize testing the core functionality (i.e., does the tool export what it is meant to export) over optimization. I don't expect that any within-container packaging/encryption execution will beat the host-system execution efficiency-wise, and if we want to rely on a solution that requires a wrapper anyway (for multiple reasons?), I guess we are fine with sticking to the host-system execution route. Should we leave the question of retiring the within-container execution option until later (until a point at which we can reasonably confidently determine that it won't be useful to anyone)?