Proposal: python package with docker container wrapper

[keller-mark]

Idea

For future workflows consisting of exclusively python code:

Rather than needing to define a docker container per-workflow, there could be a single “dummy” docker container that installs a python package that is developed independently (eg pip install portal-containers). This package could expose a CLI that is parameterized by a workflow name (eg anndata-to-ui), and the input/output directory paths. Based on the workflow name, this CLI would call a different data processing function defined in the package.

A python package for data processing developed/tested independently would substantially lower the barrier to developing/improving these workflows.

Docker would still be wrapping this package before it is used in the ingest-pipeline, but there would only need to be one dockerfile and one docker image per version of the python package.

The existing workflows could be incrementally migrated into such a package and of course the standalone dockerfile option could continue to be used in cases of dependency conflicts or non-python processing code.

[keller-mark]

We should be able to pass the parameter in the CWL files (for example in the baseCommand)

portal-containers/anndata-to-ui.cwl

Line 6 in 0b94a44

baseCommand: ['python', '/main.py', '--output_dir', './anndata-zarr', '--input_dir']

i.e., for anndata-to-ui, the above would become:

baseCommand: ['portal-containers-cli', 'anndata-to-ui', '--output_dir', './anndata-zarr', '--input_dir']

whereas for for sprm-to-anndata:

baseCommand: ['portal-containers-cli', 'sprm-to-anndata', '--output_dir', './anndata-zarr', '--input_dir']

Here, portal-containers-cli anndata-to-ui and portal-containers-cli sprm-to-anndata could each correspond to an independent sub-parser (e.g., implemented with argparse). The portal-containers python package itself could be created with Hatch.

[keller-mark]

cc @mruffalo do you forsee any issues with this proposal?