Skip to content

Dask chunking in hf-sim #108

Description

@lispandfound

To support extremely large simulations with dense station grids we should introduce chunking using dask. The fastest way to do this would be:

  1. Create an xarray dataset consisting of all the inputs (currently we use pandas dataframes, but these are not dask-ready unlike xarray datasets).
  2. Chunk the inputs appropriately. Using auto chunking behaviour is not advised because the inputs are smaller than the outputs and so it will pick chunks that are too large.
  3. Use xarray's map_blocks api, which allows us map over chunks in parallel. We would then iterate over each station in a for loop and dispatch to the fortran code.

While we are here I would make a further change to the hf sim code: write output to temp files in /dev/shm instead of /tmp. On some clusters /tmp is backed by a real filesystem, but we would rather write to memory (which /dev/shm is guaranteed to be).

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions