combining couple of operations in one remote submission job

[jungsdao]

I have a question about wfl package.
When I want to perform couple of operations for example, minima hopping + optimization or 1st NEB (without climbing image) 2nd NEB (with climbing image), and if I want to do this series of operations by remote submission, is there any way to do it in one submission rather than separate submission?
In current AutoparaInfo with remote_info takes one remote_info per each operation which makes separate remote submissions is the only way. But more efficient way (especially when one of operation is much cheaper than the other one) would be carrying out all operations in one remote submission.
I wonder if combining multiple operations into one remote submission is possible.
I'll be appreciated for any comments on this issue.

[bernstei]

You could definitely write your own function to do the combined operations and wrap that in autoparallelization code, as described in https://libatoms.github.io/workflow/overview.parallelisation.html#creating-auto-parallelized-functions

You might also be able to use wfl.map.map (supposed to be documented in https://libatoms.github.io/workflow/wfl.html#wfl.map.map but I think this documentation is slightly broken - I need to look at that). There's an example in https://libatoms.github.io/workflow/examples.daisy_chain_mlip_fitting.html#Calculate-SOAP-descriptor

[jungsdao]

I'm trying to build my own NEB that uses autoparallelization of wfl.
However, when I try to use remote, I'm getting following error.

 15   File "/raven/u/hjung/7_NEB_protocol/prepare.py", line 529, in <module>
 16     wflNEB(in_file.groups(), out_file,
 17   File "/raven/u/hjung/Softwares/neb_protocol/wfl_stepwise_NEB.py", line 197, in NEB
 18     return autoparallelize(_run_autopara_wrappable, *args,
 19            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 20   File "/u/hjung/conda-envs/mace_py311/lib/python3.11/site-packages/wfl/autoparallelize/base.py", line 174, in autoparallelize
 21     return _autoparallelize_ll(autopara_info, inputs, outputs, func, *args, **kwargs)
 22            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 23   File "/u/hjung/conda-envs/mace_py311/lib/python3.11/site-packages/wfl/autoparallelize/base.py", line 228, in _autoparallelize_ll
 24     out = do_remotely(autopara_info, iterable, outputspec, op, rng=rng_op, args=args, kwargs=kwargs,
 25           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 26   File "/u/hjung/conda-envs/mace_py311/lib/python3.11/site-packages/wfl/autoparallelize/remote.py", line 135, in do_remotely
 27     ats_out, stdout, stderr = xpr.get_results(timeout=remote_info.timeout, check_interval=remote_info.check_interval)
 28                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 29   File "/u/hjung/conda-envs/mace_py311/lib/python3.11/site-packages/expyre/func.py", line 778, in get_results
 30     raise exc
 31 ModuleNotFoundError: No module named '__builtin__'

It seems to indicate some of components passed on cannot be pickled. Is there any idea what might have possibly caused this problem?

[jungsdao]

I think source of the problem was that I was trying to pass on a AbortCheck class for NEB with kwargs and this class object was not picklable. It seems to have worked with MD module but I don't understand why it doesn't work in this case.

[jungsdao]

It was solved by defining class in separate file.

[bernstei]

Glad it's working. The whole thing with pickling can be a bit fiddly. We could make dill an optional replacement to pickle, and that might help. Do you still have the non-working version you could use to test?

[jungsdao]

I'm having a following error message with my own modified wfl_minimahopping.py.

Error message I get is following.

Gathering results for minhop_chunk_2_0ZHCJUY_wYvl76Pq-dhhDJief6NgouDmjDzKUztFt14=_nj7nt3ar remote 15431805
Traceback (most recent call last):
  File "/raven/u/hjung/7_NEB_protocol/x_fairchem/prepare.py", line 155, in <module>
    minimahopping(in_config.groups(), out_config, calculator=calculator_tuple,
  File "/raven/u/hjung/Softwares/neb_protocol/wfl_minimahopping.py", line 226, in minimahopping
    return autoparallelize(_run_autopara_wrappable, *args,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/raven/u/hjung/miniforge3/envs/fair-chem/lib/python3.12/site-packages/wfl/autoparallelize/base.py", line 176, in autoparallelize
    return _autoparallelize_ll(autopara_info, inputs, outputs, func, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/raven/u/hjung/miniforge3/envs/fair-chem/lib/python3.12/site-packages/wfl/autoparallelize/base.py", line 230, in _autoparallelize_ll
    out = do_remotely(autopara_info, iterable, outputspec, op, rng=rng_op, args=args, kwargs=kwargs,
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/raven/u/hjung/miniforge3/envs/fair-chem/lib/python3.12/site-packages/wfl/autoparallelize/remote.py", line 167, in do_remotely
    outputspec.store(at, input_locs[at_i])
                         ~~~~~~~~~~^^^^^^
IndexError: list index out of range 

when I look into output directories:
(chunk_0)

import pickle

 d = pickle.load(open(f"_expyre_job_succeeded", "rb"))
 d.items

[[Atoms.....]]

(chunk_1)

import pickle

 d = pickle.load(open(f"_expyre_job_succeeded", "rb"))
 d.items

[[], [Atoms.....]]

(chunk_2)

import pickle

 d = pickle.load(open(f"_expyre_job_succeeded", "rb"))
 d.items

[[],[], [Atoms.....]]

I don't understand why there are empty lists are included in the result list.
This can be temporarily resolved by changing wfl/autoparallelize/remote.py

160         if ats_out is None:
161             # Skip the right number of input files. If we're here,
162             # remote_info.ignore_failed_jobs must be True, so all_items should be filled
163             at_i += len(all_items[chunk_i])
164         else:
165             for at in ats_out.groups():
166                 if at.items is not None:
167                     outputspec.store(at, input_locs[at_i])
168                     at_i += 1
169             sys.stdout.write(stdout)
170             sys.stderr.write(stderr)

But It should not be ideal solution. I'll be appreciated if you can take a look why this problem is occurring.
I hope things are reproducible.
Many thanks in advance!

input.tar.gz
output.tar.gz

[bernstei]

I suspect it has to do with the fact that your input is the result of ConfigSet.groups(), rather than a simple ConfigSet. Does minimahopping really expect the input iterable to be something that returns groups of configs rather than individual configs?

Anyway, I'll take a look, but it'll take a few days.

[bernstei]

Your tar file is missing modified_minimahopping

[bernstei]

I can reproduce the issue, and it turns out that it's very subtle. I'm not sure what to do about it yet.

[jungsdao]

Thank you for taking a look at this problem.
ConfigSet.groups() was intended for this particular setting. So that different initial seed of minima hopping can share the same history of visited minima. modified_minimahopping.py is slight modification from ASE minima hopping but it's not necessarily related to this problem.

modified_minhop.tar.gz

[bernstei]

Please test this with the new branch in pull request #351 . Remember that you might need to update workflow and expyre both on the local and remote machine, if they are different.

[jungsdao]

I just have confirmed that it works without error with this branch.