Asynchronously request for workers for WorkerBackend

[ph-kev]

Currently, the WorkerBackend request for all the workers at once and wait for all of them to be ready. An alternative to this is to start a Task and request individual workers instead of waiting for all of them. This allows work to get started immediately when the workers are available but introduces additional complexity. For example, you can't use @everywhere because not all the workers are available.

[nefrathenrici]

This can be accomplished by making the process of adding workers non-blocking, and using a global worker pool. Workers will need to add themselves to this pool and load code to ensure they can run the given forward model:

Changed needed:

• Global worker pool
• non-blocking addprocs: We may be able to write a new addprocs method, probably will need an entirely new function. This will not use job arrays, but submit allocations individually.
• Macro for non-blocking @Everywhere. This macro will add expressions to run to a global variable that the workers will evaluate on startup.
• Function to initialize a worker: Load code, then add to pool once initialized
  • Code loading: Master-side registry (list of expressions) within ClimaCalibrate
• Worker exit / deregister mechanism

We should also add an atexit hook to ensure jobs are killed once the main process ends, or a persistent database to keep track of jobs.