Update to Parallel HDF5 and move to HDF5 C API

[pelahi]

Hello,

I am a member of the Pawsey team working with @ddeeptimahanti. I am working on migrating codes from HDF5 1.10 to 1.12 as the later version has significant performance enhancements. A useful project to test the performance enhancements is this particular package. I am also planning on working on moving this package from C++ API to the C API to enable parallel HDF5, which will also provide significant performance enhancements. I would be interested in working together with @confluence and/or @veggiesaurus to improve this package. I have forked the code and will submit pull requests but would be keen to more actively iterate on improvements.

Cheers,
Pascal

[confluence]

Hi, Pascal. Welcome to the project! We are happy in principle to switch to the C API -- all the interaction with files should be mostly confined to the Util functions, so changing this should have minimal impact on the rest of the code. We are also interested in testing the improvements in HDF5 1.12, but we would like to retain the ability to build the project with older versions, so if there are any changes that would break backwards compatibility we would like them to be made optional at compile time before we merge them in.

Our next planned feature is a more optimised version of the slow algorithm which processes channels (and/or tiles) in parallel up to some upper memory limit.

[veggiesaurus]

Hi @pelahi, glad to have you contributing to the repo. I think the switch to the C api is a good idea, since it's far more complete than the C++ API.

The one thing to take into consideration when maintaining backwards compatibility is that it should be done while still allowing the enhancements available in 1.12 to be fully utilised. Thus the compile-time flag would be a good approach. I know the HDF5 lib has its own compatibility flags you can use, but these tend to fall back to compatibility approaches that prevent the use of newer features.

If you'd like to get in contact with us on a more frequent basis to discuss code changes, we have a Slack channel we could invite you to.

[pelahi]

Hi @veggiesaurus @confluence I have several questions that likely would be best answered by slack. Could you invite pascal.elahi@pawsey.org.au to your slack channel. Just a few questions regarding the creation of data sets and how best to ensure the code can use MPI in the next update I am working on.

[confluence]

Hi, @pelahi. I have just sent you an invite.