Parallel writing to zarr.

[Paulie14]

Transport sampling in branch PE_trans_mesh 9a1f9df.

Running the sampling locally in interactive job:
qsub -I -q charon_2h -l select=1:ncpus=1:mem=8gb -l walltime=00:30:00
python3 sensitivity_sampling.py local
works without exceptions (did not check the written data afterwards).

Running the sampling in parallel:
python3 sensitivity_sampling.py submit
fails while opening zarr storage for samples 1,2 and 15 (out of 16):

------------------------------------------------------------
sample tags:[1 0 1]
transport_wrapper failed with exception: Consolidated metadata requested with 'use_consolidated=True' but not found in ''.
Flow123d res:  0 (18, 20, 20, 2)
TIME: 138.9432830810547
Traceback (most recent call last):
  File "/auto/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/transport_wrapper.py", line 81, in get_observations
    res = transport.transport_run(self._config, self._config.transport_fullscale.dfn_macro, tags)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/auto/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/fullscale_transport.py", line 87, in transport_run
    return parametrized_run(cfg, large_model, input_msh, tags)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/auto/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/fullscale_transport.py", line 130, in parametrized_run
    write_zarr_slice(store_path=str(work_dir / "transport_sampling"),
  File "/auto/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/fullscale_transport.py", line 161, in write_zarr_slice
    ds = xr.open_zarr(store_path, consolidated=True)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/xarray/backends/zarr.py", line 1513, in open_zarr
    ds = open_dataset(
         ^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/xarray/backends/api.py", line 715, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/xarray/backends/zarr.py", line 1587, in open_dataset
    store = ZarrStore.open_group(
            ^^^^^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/xarray/backends/zarr.py", line 664, in open_group
    ) = _get_open_params(
        ^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/xarray/backends/zarr.py", line 1791, in _get_open_params
    zarr_root_group = zarr.open_consolidated(store, **open_kwargs)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/zarr/api/synchronous.py", line 222, in open_consolidated
    sync(async_api.open_consolidated(*args, use_consolidated=use_consolidated, **kwargs))
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/zarr/core/sync.py", line 163, in sync
    raise return_result
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/zarr/core/sync.py", line 119, in _runner
    return await coro
           ^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/zarr/api/asynchronous.py", line 382, in open_consolidated
    return await open_group(*args, use_consolidated=use_consolidated, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/zarr/api/asynchronous.py", line 845, in open_group
    return await AsyncGroup.open(
           ^^^^^^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/zarr/core/group.py", line 586, in open
    return cls._from_bytes_v3(
           ^^^^^^^^^^^^^^^^^^^
  File "/storage/liberec3-tul/home/pavel_exner/workspace/endorse/apps/chodby_trans/venv/lib/python3.11/site-packages/zarr/core/group.py", line 645, in _from_bytes_v3
    raise ValueError(msg)
ValueError: Consolidated metadata requested with 'use_consolidated=True' but not found in ''.
------------------------------------------------------------

[Paulie14]

Using ds = xr.open_zarr(store_path, consolidated=False) when writing seems to fix this issue.