Another loch error

[mark-mackey-cresset]

Running with loch 093b7fe, I got the following. I'm assuming that the minimisation failure was something to do with the runtime error?

2026-02-09 01:55:42.034 | INFO     | somd2.runner._base:__init__:261 - Applying modifications to ghost atom bonded terms
2026-02-09 01:55:42.422 | INFO     | somd2.runner._base:_repartition_h_mass:1489 - Repartitioning hydrogen masses with factor 1.500
2026-02-09 01:55:47.316 | INFO     | somd2.runner._base:_get_gpu_devices:1394 - CUDA_VISIBLE_DEVICES set to ['0']
2026-02-09 01:55:47.316 | INFO     | somd2.runner._base:_get_gpu_devices:1409 - Number of GPUs available: 1
2026-02-09 01:55:47.317 | INFO     | somd2.runner._base:_get_gpu_devices:1410 - Number of concurrent workers per GPU: 1
2026-02-09 01:56:27.823 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.00000 on device 0
2026-02-09 01:56:36.370 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.00000 on device 0
2026-02-09 01:57:16.324 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.06250 on device 0
2026-02-09 01:57:24.820 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.06250 on device 0
2026-02-09 01:58:05.128 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.12500 on device 0
2026-02-09 01:58:13.529 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.12500 on device 0
2026-02-09 01:58:53.464 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.25000 on device 0
2026-02-09 01:59:01.823 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.25000 on device 0
2026-02-09 01:59:41.938 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.37500 on device 0
2026-02-09 01:59:50.439 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.37500 on device 0
2026-02-09 02:00:30.476 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.50000 on device 0
2026-02-09 02:00:38.827 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.50000 on device 0
2026-02-09 02:01:19.228 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.62500 on device 0
2026-02-09 02:01:27.639 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.62500 on device 0
2026-02-09 02:02:07.975 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.75000 on device 0
2026-02-09 02:02:16.336 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.75000 on device 0
2026-02-09 02:02:56.643 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.87500 on device 0
2026-02-09 02:03:05.027 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.87500 on device 0
2026-02-09 02:03:45.590 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 0.93750 on device 0
2026-02-09 02:03:54.113 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 0.93750 on device 0
2026-02-09 02:04:34.216 | INFO     | somd2.runner._repex:_create_dynamics:255 - Created GCMC sampler for lambda 1.00000 on device 0
2026-02-09 02:04:42.745 | INFO     | somd2.runner._repex:_create_dynamics:287 - Created dynamics object for lambda 1.00000 on device 0
2026-02-09 02:04:42.746 | INFO     | somd2.runner._repex:_minimise:1135 - Minimising at λ = 0.00000
2026-02-09 02:04:42.746 | INFO     | somd2.runner._repex:_minimise:1145 - Pre-equilibrating with GCMC moves at λ = 0.00000
/data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packages/loch/_sampler.py:1301: RuntimeWarning: overflow encountered in exp
  acc_prob = _np.exp(
/data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packages/loch/_sampler.py:1385: RuntimeWarning: overflow encountered in exp
  acc_prob = _np.exp(
2026-02-09 02:06:12.447 | ERROR    | somd2.runner._repex:run:738 - Minimisation failed for λ = 0.00000: SireError::invalid_state: Despite repeated attempts, the minimiser could not minimise the system while simultaneously satisfying the constraints. (call sire.error.get_last_error_details() for more info)
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /data/tmp/019a9aca-2604-475d-a010-ccedaa5683d2/86e9cdf5-4fce-4666-9a2e-aec61 │
│ 3c27d65/contents/23477~23467_Bound_Morph_0.000/run.py:11 in <module>         │
│                                                                              │
│    8   config = somd2.config.Config.from_yaml("Config.yaml")                 │
│    9                                                                         │
│   10   runner = somd2.runner.RepexRunner(sr.system.System(system._sire_objec │
│ ❱ 11   runner.run()                                                          │
│   12                                                                         │
│                                                                              │
│ /data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packag │
│ es/somd2/runner/_repex.py:741 in run                                         │
│                                                                              │
│    738 │   │   │   │   │   │   │   │   _logger.error(                        │
│    739 │   │   │   │   │   │   │   │   │   f"Minimisation failed for {_lam_s │
│    740 │   │   │   │   │   │   │   │   )                                     │
│ ❱  741 │   │   │   │   │   │   │   │   raise e                               │
│    742 │   │   │   │   │   except KeyboardInterrupt:                         │
│    743 │   │   │   │   │   │   _logger.error("Minimisation cancelled. Exitin │
│    744 │   │   │   │   │   │   _sys.exit(1)                                  │
│                                                                              │
│ /data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packag │
│ es/somd2/runner/_repex.py:1155 in _minimise                                  │
│                                                                              │
│   1152 │   │   │   │   gcmc_sampler.pop()                                    │
│   1153 │   │   │                                                             │
│   1154 │   │   │   # Minimise.                                               │
│ ❱ 1155 │   │   │   dynamics.minimise(timeout=self._config.timeout)           │
│   1156 │   │                                                                 │
│   1157 │   │   except Exception as e:                                        │
│   1158 │   │   │   return False, index, e                                    │
│                                                                              │
│ /data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packag │
│ es/sire/mol/_dynamics.py:1658 in minimise                                    │
│                                                                              │
│   1655 │   │   │   │   │   │      A value of <=0 will disable the timeout.   │
│   1656 │   │   """                                                           │
│   1657 │   │   if not self._d.is_null():                                     │
│ ❱ 1658 │   │   │   self._d.run_minimisation(                                 │
│   1659 │   │   │   │   max_iterations=max_iterations,                        │
│   1660 │   │   │   │   tolerance=tolerance,                                  │
│   1661 │   │   │   │   max_restarts=max_restarts,                            │
│                                                                              │
│ /data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packag │
│ es/sire/mol/_dynamics.py:937 in run_minimisation                             │
│                                                                              │
│    934 │   │                                                                 │
│    935 │   │   self._clear_state()                                           │
│    936 │   │                                                                 │
│ ❱  937 │   │   self._minimisation_log = minimise_openmm_context(             │
│    938 │   │   │   self._omm_mols,                                           │
│    939 │   │   │   tolerance=tolerance,                                      │
│    940 │   │   │   max_iterations=max_iterations,                            │
│                                                                              │
│ /data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packag │
│ es/sire/legacy/Convert/__init__.py:550 in minimise_openmm_context            │
│                                                                              │
│   547 │   │   max_constraint_error: float = 0.01,                            │
│   548 │   │   timeout: str = "300s",                                         │
│   549 │   ):                                                                 │
│ ❱ 550 │   │   return _minimise_openmm_context(                               │
│   551 │   │   │   context,                                                   │
│   552 │   │   │   max_iterations=max_iterations,                             │
│   553 │   │   │   tolerance=tolerance,                                       │
╰──────────────────────────────────────────────────────────────────────────────╯
RuntimeError: SireError::invalid_state: Despite repeated attempts, the minimiser
could not minimise the system while simultaneously satisfying the constraints. 
(call sire.error.get_last_error_details() for more info)

[lohedges]

I don't know whether this is specifically to do with loch or not. With somd2, minimisation convergence issues should now not raise and exception unless you have minimisation_errors set to True. Could you set that to False to see whether it proceeds? (This functionality has been present for some time.) Is there a reason why the most recent development versions of loch and somd2 aren't being used, e.g. you are still seeing the numpy warning, which was resolved. (Perhaps it's non-trivial to update your stack.)

[lohedges]

Also, just to confirm, is this error specific to this loch version, i.e. did you not see it previously?

[lohedges]

Actually, the commit you referenced should have the overflow fix, which is strange.

[lohedges]

You can't be using that commit of loch since lines 1301 and 1385 are completely different when I checkout and look at the code, i.e, not the np.exp calls that are reported in your log:

/data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packages/loch/_sampler.py:1301: RuntimeWarning: overflow encountered in exp
  acc_prob = _np.exp(
/data/cresset/Flare_12.0.0.somd2/third-party/sire/lib/python3.11/site-packages/loch/_sampler.py:1385: RuntimeWarning: overflow encountered in exp
  acc_prob = _np.exp(