To support extremely large simulations with dense station grids we should introduce chunking using dask. The fastest way to do this would be:
- Create an xarray dataset consisting of all the inputs (currently we use pandas dataframes, but these are not dask-ready unlike xarray datasets).
- Chunk the inputs appropriately. Using
auto chunking behaviour is not advised because the inputs are smaller than the outputs and so it will pick chunks that are too large.
- Use xarray's map_blocks api, which allows us map over chunks in parallel. We would then iterate over each station in a for loop and dispatch to the fortran code.
While we are here I would make a further change to the hf sim code: write output to temp files in /dev/shm instead of /tmp. On some clusters /tmp is backed by a real filesystem, but we would rather write to memory (which /dev/shm is guaranteed to be).
To support extremely large simulations with dense station grids we should introduce chunking using dask. The fastest way to do this would be:
autochunking behaviour is not advised because the inputs are smaller than the outputs and so it will pick chunks that are too large.While we are here I would make a further change to the hf sim code: write output to temp files in
/dev/shminstead of/tmp. On some clusters/tmpis backed by a real filesystem, but we would rather write to memory (which/dev/shmis guaranteed to be).