Performance profiling results

[maxrjones]

I did some basic profiling on a simplified version of #1 (code in https://github.com/englacial/magg/blob/8fff3c8d8693049b7666eeddd2f4dcb91b150711/process-single-cell.py). Note that I'll still put together an example of how to write the indexes to Zarr rather than Parquet, but wanted to first check how the gains could relate to the goal of reducing the total time to <5 minutes as mentioned in #1 (comment). I profiled aggregating data from the cell with the most granules associated with it figuring it'd be a worst case/rase-limiting function invocation.

A vast majority of the time was spent on reading data and there was a lot of time wasted on lock contention:

A quick Claude-coded experiment reduced the execution time to ~1/3 the original processing time by using a Semaphore to limit concurrency and processing batches in parallel, but I have yet to check its work.

Is this type of use-case with needing to open hundreds of hdf5 files in a single workflow common @espg @weiji14? h5coro was definitely not designed for it, and this type of requirement was the motivation for https://github.com/developmentseed/async-tiff for COGs. A similar fully async, rust-based implementation makes a lot of sense for this style of workflow, but is a big lift. A less big lift would be contributing some examples on how to use h5coro optimally for different use-cases.

[weiji14]

Is this type of use-case with needing to open hundreds of hdf5 files in a single workflow common @espg @weiji14? h5coro was definitely not designed for it

Hundreds isn't too bad, wait until you get into >1000 😃 I thought h5coro was supposed to be thread-safe (https://slideruleearth.io/rtd/developer_guide/articles/h5coro.html#h5coro-read), are you running more processes than core counts?

A similar fully async, rust-based implementation makes a lot of sense for this style of workflow, but is a big lift

https://github.com/gauteh/hidefix. Don't think it's async, but should have multi-threading.

[maxrjones]

I thought h5coro was supposed to be thread-safe (slideruleearth.io/rtd/developer_guide/articles/h5coro.html#h5coro-read), are you running more processes tham core counts?

The Python library seems fully independent from the C++ implementation in SlideRule so that documentation may not apply here - https://github.com/SlideRuleEarth/h5coro.

[espg]

It's also heavily latitude dependent -- maybe a dozen files in the antarctic peninsula, but then 300-400 hundred as we get to the 88 south pole hole circle.

It's blowing my mind that h5coro is pure python-- I always assumed that it was importing something or at least wrapping some of the h5 libs for calls out to a compiled language. Honestly impressive that it's a performant as it is for not doing that.

As much as I like the idea of massively improving throughput with rust + async, if it's a big lift we should probably defer it until after we have the output side of how we're writing to grid structure figured out.

[weiji14]

Yeah, I could have sworn h5coro (python) called out to h5coro (C++)...

As much as I like the idea of massively improving throughput with rust + async, if it's a big lift we should probably defer it until after we have the output side of how we're writing to grid structure figured out.

@maxrjones, could you zoom in a little more on your flame graph to show the breakdown on time spent on 1) reading *.h5 files vs 2) the morton/healpix computations? I'm guessing it's (1) but just want to be sure. Edit: flamegraphs on https://drive.google.com/drive/folders/1dRoVD9F_La9NoqtyamEqvjAkOPne3ru3, and yes, the H5Coro.readDatasets call is slower than the geo2mort call by about an order of magnitude, ~0.1s vs ~0.01s (depending on the chunk size).

I've got a fairly good handle on Rust these days, so could look at replacing h5coro (Python) with hidefix (Rust). We could at least try hidefix's xarray integration first as an easy entrypoint. Otherwise I can dig into some Rust code and help with creating bindings to Python and stuff like that.

If we need more performance, we could go full Rust since we're using mortie (which is Rust-based!) for the mort2healpix stuff. Alternatively, we can get into something like cuHPX (GPU-based), but that'll be more infra to handle.