Related to #288, functionality should be added to keep Slurm or PBS jobs running when the main process die. This is relevant for the HPCBackends and potentially could be implemented for the WorkerBackend. The main motivation behind this is that the main process could be killed while the Slurm or PBS jobs are running. Instead of cancelling all jobs when the main process die, it would be better to leave them and restart the main process that know that a calibration is already happening.
One way of implementing this is to keep a database of all Slurm jobs that are currently running. If the main process is killed, then after restarting, the main process can check the database and continue monitoring the jobs for completion.
Related to #288, functionality should be added to keep Slurm or PBS jobs running when the main process die. This is relevant for the
HPCBackends and potentially could be implemented for theWorkerBackend. The main motivation behind this is that the main process could be killed while the Slurm or PBS jobs are running. Instead of cancelling all jobs when the main process die, it would be better to leave them and restart the main process that know that a calibration is already happening.One way of implementing this is to keep a database of all Slurm jobs that are currently running. If the main process is killed, then after restarting, the main process can check the database and continue monitoring the jobs for completion.