Memory profiling results

[maxrjones]

I did a small amount of memory profiling to see if we could get any gains around further "right-sizing" the lambda resources using a version of the code in #1. I was surprised to find the memory usage increasing linearly over time (see figure below and/or flamegraph stored in google drive). This was using h5coro=1.0.4, so I'm wondering if the new explicit file handle closing needs to be added to #1 or if I should pin h5coro to an older version?

[maxrjones]

After an update to explicit close files, the memory is well-controlled:

I'll leave this open because it would probably be helpful to open a docs PR to h5coro since none of their examples close the file handle.

[espg]

I remember there being a pretty significant memory usage regression from h5coro 0.8 to 1.0.3 ; I've meaning to open an issue about this with the 5hcoro folks, but it got lost in rush of AGU and the holidays...

Basically, even with explicit file closing, h5coro 0.8 is still a significantly higher memory footprint, so that's why I'm pinning it here:

https://github.com/englacial/magg/blob/fbde56505d5d18219ffdc50abc6cce2fd8bd5c66/lambda/build_layer_v14.sh#L64