parallel writing to disk by the worker nodes after each grid point is modelled would significantly improve write times with minimal changes in runtime for each worker. It would require refactoring such that the empty result dataset is created prior to the start of the model run eg:
RESULT = IO.write_results_to_file() #modified so that it returns empty dask arrays to save memory
RESULT = IO.get_result()
RESULT = RESULT.chunk( {"lat": 1,'lon':1, 'time':-1})
RESULT.to_zarr(self.out_dir, mode = 'w',compute=False,
safe_chunks=True, write_empty_chunks=True)
# compute = false means that array knows the file setup but has not written anything
and then at the end of the cosipy_core loop:
region_to_write = {'lat': slice(indY, indY + 1),'lon': slice(indX, indX + 1), 'time': slice(0,nt)}
RESULT['RAIN'] = _RAIN
RESULT['SNOWFALL'] = _SNOWFALL
RESULT['LWin'] = _LWin
#other variables omitted here because it just makes a long code block, could definitely be refactored in a better way
RESULT = RESULT.expand_dims({"lat": [lat_val], "lon": [lon_val]})
RESULT.to_zarr(out_path,mode ='r+',region = region_to_write, safe_chunks = True,write_empty_chunks=True)
return True
Zarr is then easily convertible to netcdf in xarray later.
The only potential downside I can see is that some HPC filesystems like my universities largest filestore (lustre filesystem for one of the many options) does not cope well with lots of small file writes for some reason, so the user would need to know if a filesystem available to them was specially bad for this.
Happy to write up code for this change if you think it's worthwhile.
parallel writing to disk by the worker nodes after each grid point is modelled would significantly improve write times with minimal changes in runtime for each worker. It would require refactoring such that the empty result dataset is created prior to the start of the model run eg:
and then at the end of the cosipy_core loop:
Zarr is then easily convertible to netcdf in xarray later.
The only potential downside I can see is that some HPC filesystems like my universities largest filestore (lustre filesystem for one of the many options) does not cope well with lots of small file writes for some reason, so the user would need to know if a filesystem available to them was specially bad for this.
Happy to write up code for this change if you think it's worthwhile.