Will Cylc be added as a scheduler?

[dmentipl]

Are there any plans to add Cylc as a scheduler for DAGRunner? Presumably this would be required for using it for the IMPROVER suite at some point. Or if not are there alternative cycling schedulers considered?

I can see from the demo notebook (section "Customised graph construction") that you can parameterise generating the networkx DiGraph object, e.g. with a cycle parameter. However, it's not clear to me how this could be used as a replacement/adjunct for a cycling scheduler. I can also see in the IMPROVER suite design pack that there's a diagram that references cycling schedulers including Cylc and ECFLOW.

I'm interested in exploring the use of DAGRunner for a verification workflow. The aim here is to be able to run "offline" on an archive of IMPROVER outputs, e.g. from a hindcast, but also to be able to run in (delayed) realtime to verify outputs as observations become available. And in the long run, I suppose, we will transition our IMPROVER suite from paraflow to DAGRunner.

[cpelley]

DAGrunner is designed to abstract access to various Python schedulers such as multiprocessing, dask, and ray. Its primary function is to execute arbitrary networkx graphs using these schedulers.
Users provide DAGrunner with a callable that returns a networkx graph, meaning DAGrunner itself does not handle graph construction. That is the responsibility of the user. Paraflow was inherently concerned with and indeed tied to cylc, since it performed cylc graph construction.

Cyle isn't really a scheduler in the same sense as the above. It's more an orchestration engine.
pp_demo_workflow shows how cylc can still be utilised to provide cycle based scheduling of rose tasks, where these rose tasks call for the execution of your networkx graph (or subset of) using the DAGrunner CLI (like any other tool or script).

As an aside, in the longer run, I am wanting to explore the use of dask or ray clusters (for dynamic scaling), being better suited to cloud compute. We might need not use cylc at all in that case (as cylc isn't intended for this realm of execution). However, again, DAGrunner provides a commandline script for executing arbitrary networkx graphs, so whether you choose to continue to use cylc or not would be up to the specific project.

We should definitely pick-up offline for a chat.

[dmentipl]

DAGrunner is designed to abstract access to various Python schedulers such as multiprocessing, dask, and ray. Its primary function is to execute arbitrary networkx graphs using these schedulers. [...] pp_demo_workflow shows how cylc can still be utilised to provide cycle based scheduling of rose tasks, where these rose tasks call for the execution of your networkx graph (or subset of) using the DAGrunner CLI (like any other tool or script).

Thanks, yes, I didn't quite understand how things worked. Having had a look at pp_demo_workflow I now have a better understanding. Where in the paraflow-worldview we created a networkx graph for the entire workflow for paraflow to chop up into Cylc tasks, in the DAGRunner-worldview we create many networkx graphs as appropriate per Cylc task (that is, if we were to use Cylc).

We should definitely pick-up offline for a chat.

Yes, agreed.