Skip to content

Dask chunking in bb-sim #109

Description

@lispandfound

To support extremely large simulations with dense station grids we should introduce chunking using dask. The fastest way to do this would be:

  1. Chunk the hf and lf outputs. Chunking the hf outputs will be done automatically if Dask chunking in hf-sim #108 is closed, but the lf outputs will get chunked manually.
  2. Use xarray's map_blocks api, which allows us map over chunks in serial. Unlike hf sim, we would want to process these in serial because there is already a parallel layer inside the cb2014_to_fas_amplification_factors code using numba so we don't want to doubly parallelise.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions