You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
run_batch parallelization (Batch parallelism). For the moment, the pipeline takes one folder, and runs the specified steps with its data. The CLI can also run on batches of folder, but for the moment this is purely iterative. Since steps can be executed one by one, it should be possible to run the corresponding steps of each pipeline in parallel, and take advantage of the GPU when needed, for example for the deep models execution, which would run on batches of inputs.
parallelization of non dependent steps (DAG parallelism). This is not urgent, since the pipeline is mostly linear for the moment. Add automatic step parallelization #21
Different types of parallelization could be added :