DAGonStar

DAGonStar: FAIR by design

DAGonStar, also written as DAGon*, is a lightweight Python workflow engine for running directed acyclic graph (DAG) workflows across local machines, remote servers, HPC clusters, containers, and cloud infrastructure.

DAGonStar workflows are ordinary Python programs. Tasks can depend explicitly on other tasks, or implicitly through workflow:// data references that DAGonStar resolves into task dependencies and staging operations.

Motivation

Thanks to the advent of public, private, and hybrid clouds, the democratization of Computational resources changed the rules in many scientific fields. For decades, one of the efforts of computer scientists and computer engineers was the development of tools able to simplify access to high-end computational resources by computational scientists. However, nowadays, any science field can be considered "computational" if the availability of powerful but easy-to-manage workflow engines is crucial.

Acknowledgments

The following initiatives support DAGonStar development:

Research agreement "Modelling mytilus farming at scale" (MytilX, CUP D13C24000470002, funded by the Istituto Zooprofilattico Sperimentale dell’Umbria e delle Marche) - DAGonStar orchestrates the production workflow to deliver use cases, study zones specific to weather, marine, pollutants, and farmed mussels contamination forecasts and predictions.
Research contract "Mytilus farming System with High-Performance Computing and Artificial Intelligence" (MytilEx, CUP I63C23000180002, funded by the Campania Region, Veterinary sector) - DAGonStar orchestrates the production workflow to deliver daily 168 weather, marine, pollutants, and farmed mussels contamination forecasts and predictions. PWA
EuroHPC H2020 project "Adaptative Multi-tier Intelligent data manager for Exascale" (ADMIRE, 956748-ADMIRE-H2020-JTI-EuroHPC-2019-1, funded by the European Commission) - WP7: DAGonStar orchestrates the Environmental Application, delivering on-demand weather, marine, and pollutants simulations and forecasts on the Campania Region (Italy). link

Cite DAGonStar

Workflow engine

Sánchez-Gallegos, Dante Domizzi, Diana Di Luccio, Sokol Kosta, J. L. Gonzalez-Compean, and Raffaele Montella. "An efficient pattern-based approach for workflow supporting large-scale science: The DagOnStar experience." Future Generation Computer Systems 122 (2021): 187-203. link
Barron-Lugo, J.A., Gonzalez-Compean, J. L., Carretero, J., Lopez-Arevalo, I., & Montella, R. (2021). A novel transversal processing model to build environmental big data services in the cloud. Environmental Modelling & Software, 144, 105173. link
Sánchez-Gallegos, Dante D., Diana Di Luccio, José Luis Gonzalez-Compean, and Raffaele Montella. "Internet of things orchestration using dagon workflow engine." In 2019 IEEE 5th world forum on internet of things (WF-IoT), pp. 95-100. IEEE, 2019. link
Sánchez-Gallegos, Dante D., Diana Di Luccio, J. L. Gonzalez-Compean, and Raffaele Montella. "A microservice-based building block approach for scientific workflow engines: Processing large data volumes with dagonstar." In 2019 15th International Conference on Signal-Image Technology & Internet-Based Systems (SITIS), pp. 368-375. IEEE, 2019. link
Montella, Raffaele, Diana Di Luccio, and Sokol Kosta. "Dagon: Executing direct acyclic graphs as parallel jobs on anything." In 2018 IEEE/ACM Workflows in Support of Large-Scale Science (WORKS), pp. 64-73. IEEE, 2018. link

Applications

Mellone, Gennaro, Ciro Giuseppe De Vita, Enrico Zambianchi, David Expósito Singh, Diana Di Luccio, and Raffaele Montella. "Democratizing the computational environmental marine data science: using the high-performance cloud-native computing for inert transport and diffusion lagrangian modelling." In 2022 IEEE International Workshop on Metrology for the Sea; Learning to Measure Sea Health Parameters (MetroSea), pp. 267-272. IEEE, 2022. link
De Vita, Ciro Giuseppe, Gennaro Mellone, Aniello Florio, Catherine Alessandra Torres Charles, Diana Di Luccio, Marco Lapegna, Guido Benassai, Giorgio Budillon, and Raffaele Montella. "Parallel and hierarchically-distributed Shoreline Alert Model (SAM)." In 2023 31st Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp. 109-113. IEEE, 2023. link
Montella, Raffaele, Diana Di Luccio, Ciro Giuseppe De Vita, Gennaro Mellone, Marco Lapegna, Gloria Ortega, Livia Marcellino, Enrico Zambianchi, and Giulio Giunta. "A highly scalable high-performance Lagrangian transport and diffusion model for marine pollutants assessment." In 2023 31st Euromicro International Conference on Parallel, Distributed and Network-Based Processing (PDP), pp. 17-26. IEEE, 2023. link

Surveys

Aldinucci, Marco, Giovanni Agosta, Antonio Andreini, Claudio A. Ardagna, Andrea Bartolini, Alessandro Cilardo, Biagio Cosenza et al. "The Italian research on HPC key technologies across EuroHPC." In Proceedings of the 18th ACM international conference on computing frontiers, pp. 178-184. 2021. link

License

DAGonStar is distributed under the Apache License 2.0. See LICENSE.

Documentation

The full documentation set is available in docs/:

What DAGonStar supports

Python-defined workflows.
Explicit task graphs and data-driven dependency discovery.
Local batch tasks.
Remote SSH tasks.
Slurm tasks.
Docker tasks, locally or over SSH.
Cloud-backed tasks through Apache Libcloud providers.
OpenAI-compatible LLM tasks with JSON prompts and workflow:// text inputs.
Native Python-function tasks with staged file bindings and JSON result metadata.
Web tasks for structured HTTP/HTTPS requests with scratch-local response outputs.
Provider-neutral FaaS invocation tasks for mock, HTTP/Knative, AWS Lambda, Azure Functions, and Google Cloud Run functions.
Data staging by link, copy, SCP, Globus, and SKYCDS.
Checkpoint/resume support.
Self-contained Common Workflow Language (CWL) v1.2 command-graph export.
A deterministic DAGonStar-to-CWL interoperability example, checked-in CWL artifact, reference-runner validation hook, and incremental tutorial lesson.
Native, opt-in FAIR metadata and provenance recording with RO-Crate, PROV, DataCite, CodeMeta, local output fixity, validation, and reports.
Meta-workflows that coordinate multiple workflows.

Project status and quality assessment

DAGonStar is a research-oriented workflow engine with a useful, documented core, rather than a fully polished modern library. Its strongest areas are the compact Python workflow model, explicit and workflow://-derived dependencies, checkpointing, and a broad set of execution and staging integrations. Workflows can also run in a background thread with lifecycle callbacks for local progress reporting. The opt-in FAIR recorder makes lifecycle, task, dependency, output, fixity, and checkpoint-reuse metadata available in standard local exports without adding dependencies or changing existing workflows. Workflows can also be saved as self-contained CWL v1.2 JSON documents for interchange and validation by CWL-aware tools; executor-specific DAGonStar staging and remote runtime behavior remain outside that export boundary. FaaS invocation is now a provider-neutral task type with a credential-free mock, lazy cloud adapters, structured artifact staging, sanitized provenance, and a runner-based CWL representation. Cloud deployment and automatic object-store artifact upload remain outside the implemented boundary.

The repository has a sound baseline for changes to the core behavior:

unit tests cover configuration parsing, workflow defaults and dependency discovery, cycle validation, JSON and CWL serialization, deterministic CWL example regeneration, checkpoint reuse, staging command generation, Docker and remote-container construction, SSH-port forwarding, packaging extras, optional integration boundaries, selected shell-quoting behavior, SSH host-key verification and safe remote launcher creation, and FAIR profiles, artifact declarations, recorder exports, validation, and safe environment defaults;
GitHub Actions runs that suite and source compilation on Python 3.8 and Python 3.12, plus focused Ruff and mypy checks on Python 3.12;
package extras keep Docker, cloud, Globus, and API dependencies out of the base installation;
base requests and graphviz requirements use compatible version ranges, allowing hosted notebook environments such as Google Colab to retain their platform dependencies;
documentation covers configuration, architecture, checkpoints, examples, FAIR principles, CWL export semantics, and the incremental tutorial; and
an all-factory contract test constructs a mixed workflow containing every TaskType and verifies workflow://, FAIR declarations, checkpoint reuse, portable execution, and runnable deterministic CWL export without live external services;
sample configuration avoids committed credentials and the SKYCDS path checks for required runtime configuration; and
the LLM task boundary has local tests and a fully local mock-provider example.
native Python tasks have local staging, dependency, output, and runner tests.
DynoStore has documented, tested task-level command examples; its DataMover value remains reserved rather than an automatic stager.

FAIR exports are local metadata artifacts, not repository publication: they do not copy large outputs, obtain remote checksums, or validate every optional external profile. Credential capture remains disabled by default.

The test suite is intentionally fast and local: it validates command generation, failure propagation, and integration boundaries, not live Docker, SSH, Slurm, cloud, Globus, or SKYCDS services. CI also runs focused style checks and a progressive type check over the shell-command helper. Several legacy and site-specific code paths still construct shell commands, so changes around external execution should be kept small, quoted defensively, and verified at the boundary being changed.

Installation

Use Python 3.8 or newer.

git clone https://github.com/DagOnStar/dagonstar.git
cd dagonstar
python3 -m venv .venv
. .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt

For editable development installs:

python -m pip install -e .

Optional integrations are available as install extras:

python -m pip install -e ".[docker]"
python -m pip install -e ".[cloud]"
python -m pip install -e ".[globus]"
python -m pip install -e ".[api]"
python -m pip install -e ".[faas-aws]"
python -m pip install -e ".[faas-azure]"
python -m pip install -e ".[faas-gcp]"
python -m pip install -e ".[all]"

The base package does not install Docker, cloud-provider FaaS SDKs, Globus, or Flask service libraries. If an integration is requested without its extra, DAGonStar reports the corresponding install command at the integration boundary.

Minimal FAIR workflow

FAIR recording is opt-in and uses the standard library. It emits local RO-Crate JSON-LD, PROV JSON, citation/software metadata, fixity, and a report without changing workflows that do not enable it.

from dagon import Workflow
from dagon.task import DagonTask, TaskType
from dagon.fair import Agent, Artifact, FairProfile
workflow = Workflow("FAIR-Demo")
workflow.enable_fair(FairProfile(title="FAIR demo", description="Local provenance.",
    creators=[Agent(name="DAGonStar Team")], license="Apache-2.0"))
workflow.add_task(DagonTask(TaskType.BATCH, "A", "echo hello > message.txt").declare_outputs(
    Artifact("message.txt", media_type="text/plain", license="Apache-2.0")))
workflow.run()

Outputs are written below <scratch>/.dagon/fair/<workflow>/<run>/, including run.json, ro-crate-metadata.json, prov.json, checksums.sha256, and a FAIRness report. See Lesson 15.

requirements.txt remains a full development/demo environment that installs the optional integration dependencies together.

Configuration

Copy the sample configuration before running examples:

cp dagon.ini.sample dagon.ini

Important sections:

[batch]: local scratch directory and staging thread settings.
[dagon_service]: optional DAGon service registration.
[slurm]: default Slurm partition for Slurm staging or tasks.
[ec2], [digitalocean], [gce]: optional cloud credentials.
[llm.<name>]: optional OpenAI-compatible LLM provider configuration.

Never commit real cloud keys, Globus tokens, SKYCDS tokens, SSH private keys, or site-specific passwords. SKYCDS values are read from these environment variables:

export DAGON_SKYCDS_CLIENT_TOKEN=...
export DAGON_SKYCDS_CATALOG_TOKEN=...
export DAGON_SKYCDS_API_TOKEN=...
export DAGON_SKYCDS_IP=...

Quick start

This example creates a four-task local workflow. Task D consumes outputs from tasks B and C; the workflow:// references are used to infer dependencies and stage files.

from dagon import Workflow
from dagon.task import DagonTask, TaskType


if __name__ == "__main__":
    workflow = Workflow("DataFlow-Demo")

    task_a = DagonTask(TaskType.BATCH, "A", "mkdir -p output; hostname > output/f1.txt")
    task_b = DagonTask(TaskType.BATCH, "B", "echo $RANDOM > f2.txt; cat workflow:///A/output/f1.txt >> f2.txt")
    task_c = DagonTask(TaskType.BATCH, "C", "echo $RANDOM > f2.txt; cat workflow:///A/output/f1.txt >> f2.txt")
    task_d = DagonTask(TaskType.BATCH, "D", "cat workflow:///B/f2.txt >> f3.txt; cat workflow:///C/f2.txt >> f3.txt")

    for task in (task_a, task_b, task_c, task_d):
        workflow.add_task(task)

    workflow.make_dependencies()
    workflow.run()

Running examples

Most examples expect a dagon.ini file in the current working directory or in the example directory.

cp dagon.ini.sample examples/dagon.ini
cd examples/taskflow
python taskflow-demo.py

Example groups:

examples/taskflow: explicit task dependencies.
examples/dataflow/batch: local data-driven workflows and checkpointing.
examples/dataflow/slurm: Slurm and remote Slurm workflows.
examples/dataflow/docker: Docker-backed workflows.
examples/dataflow/cloud: cloud-backed task examples.
examples/transversal: meta-workflow and transversal processing examples.
examples/hipes-tutorial: tutorial material for HiPES workflows.
examples/cwl: portable command-graph export plus a checked-in CWL v1.2 document; see the example guide and Lesson 17.
examples/envapp: environmental application workflows and utilities.
examples/async: local asynchronous launch and lifecycle-callback example.
examples/native: importable Python functions with staged workflow:// inputs.
examples/web: local HTTP request, staged upload, and downstream native processing.
examples/faas: credential-free mock invocation plus JSON and CWL export.

Generate and test the portable CWL example without a private configuration:

python3 examples/cwl/export_workflow.py --output /tmp/interoperable-workflow.cwl
python3 -m unittest tests.test_cwl_example -v

Web tasks

TaskType.WEB executes a JSON-serializable HTTP request through the task executor:

DagonTask(TaskType.WEB, "fetch", {"method": "GET", "url": "https://example.org/data", "outputs": {"body": "data.json"}})

Responses are written below outputs/ and request metadata is stored in .dagon/. Web tasks use the existing local or Slurm executor mode. Use environment-variable auth fields, never literal secrets. See the example, example guide, and tutorial.

Native Python tasks

TaskType.NATIVE runs an importable function in a task scratch directory while retaining DAGonStar staging and dependencies:

DagonTask(TaskType.NATIVE, "transform", "myworkflow.tasks:transform", inputs={"input_file": "workflow:///prepare/data.csv", "scale": 0.7}, outputs={"output_file": "clean.csv"})

File arguments are staged below inputs/, outputs are paths below outputs/, and the JSON return value is written to .dagon/native_result.json. See the native example and tutorial lesson.

FaaS tasks

TaskType.FAAS invokes an already-deployed function through a provider adapter:

DagonTask(TaskType.FAAS, "invoke", provider="mock", function="double-value",
          inputs={"document": "workflow:///prepare/outputs/input.json"},
          outputs={"result": "result.json"})

The mock provider is local and credential-free. HTTP/Knative uses a JSON envelope; AWS Lambda, Azure Functions, and Google Cloud Run functions use optional adapters and provider-standard credential chains. DAGonStar does not deploy functions. Large remote artifacts require an accessible transport such as object storage; automatic object upload is not currently implemented. See the task guide, providers, FAIR mapping, CWL mapping, examples, and Lesson 18.

Development

Useful checks:

python -m unittest discover -s tests -v
python -m py_compile dagon/*.py dagon/api/*.py dagon/communication/*.py dagon/ftp_publisher/*.py dagon/peer2peer/*.py
python -m ruff check dagon/shell.py tests
python -m mypy --follow-imports=skip dagon/shell.py
python -m pip install -e .

The repository includes a lightweight unit test suite under tests/. The tests focus on behavior that should remain stable while features evolve: configuration loading, workflow defaults, dependency discovery, cycle validation, JSON serialization, and runtime-secret validation.

GitHub Actions runs the unit tests and source compilation checks on pushes to master/main and on pull requests.

When changing workflow execution behavior, add a targeted unit test where possible. If the behavior depends on Docker, Slurm, SSH, Globus, or a cloud provider, add the smallest safe test for local logic and record any manual integration command used in your change notes.

Recommended maintenance priorities:

Add tests for checkpoint/resume behavior and staging command generation.
Add integration tests for local batch workflows and mocked tests for external-service integrations.
Replace shell string concatenation with safer quoting helpers where practical.
Split optional integrations into extras such as dagonstar[docker], dagonstar[cloud], dagonstar[globus], and dagonstar[api].
Add type hints to public workflow, task, configuration, and staging APIs.

Troubleshooting

Missing `dagon.ini`

Create one from the sample:

cp dagon.ini.sample dagon.ini

Docker examples cannot connect to Docker

Check that Docker is running and that your user can access the Docker socket:

docker info

Remote tasks fail over SSH

Verify key-based SSH manually first:

ssh -i /path/to/key user@host hostname

Globus or SKYCDS staging fails

Confirm the integration-specific credentials and endpoint IDs are configured. SKYCDS values must be supplied through environment variables, not committed in source files.

Name		Name	Last commit message	Last commit date
.github/workflows		.github/workflows
dagon		dagon
docs		docs
examples		examples
extra-scripts		extra-scripts
figures		figures
outputs		outputs
tests		tests
tools		tools
.gitignore		.gitignore
.wci.yml		.wci.yml
AGENTS.md		AGENTS.md
LICENSE		LICENSE
README.md		README.md
dagon.ini.sample		dagon.ini.sample
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
setup.cfg		setup.cfg

Navigation Menu

Uh oh!

Repository files navigation

DAGonStar

DAGonStar: FAIR by design

Motivation

Acknowledgments

Cite DAGonStar

Workflow engine

Applications

Surveys

License

Documentation

What DAGonStar supports

Project status and quality assessment

Installation

Minimal FAIR workflow

Configuration

Quick start

Running examples

Web tasks

Native Python tasks

FaaS tasks

Development

Troubleshooting

Missing `dagon.ini`

Docker examples cannot connect to Docker

Remote tasks fail over SSH

Globus or SKYCDS staging fails

About

Releases

Packages

Contributors

Languages

Uh oh!

Latest commit

History

Folders and files

Repository files navigation

DAGonStar

DAGonStar: FAIR by design

Motivation

Acknowledgments

Cite DAGonStar

Workflow engine

Applications

Surveys

License

Documentation

What DAGonStar supports

Project status and quality assessment

Installation

Minimal FAIR workflow

Configuration

Quick start

Running examples

Web tasks

Native Python tasks

FaaS tasks

Development

Troubleshooting

Missing dagon.ini

Docker examples cannot connect to Docker

Remote tasks fail over SSH

Globus or SKYCDS staging fails

About

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages

Missing `dagon.ini`