We are working in the area of AI for science.
HDF5 is one of the file formats used extensively by scientific community to store data. Data can be organized inside HDF5 file under different dataset objects. A single HDF5 can host multiple datasets.
While exploring DVC for AI for science use case, we encountered a problem. Versioning in DVC happens at the file granularity and since these HDF5 files can be very large, it causes storage and network bottlenecks.
A small change inside any of the datasets of a file would cause the entire file to be duplicated and replicated to the storage remote.
We have a potential solution on how this problem can be addressed for Hdf5 files.
HDF5 offers a natural boundary of individual datasets inside the file. It also has a existing feature of external links by which an external datasets residing in a different file can be mounted into the file.
We are planning to split HDF5 files into separate files(using a plugin library) at dataset boundary and linking these smaller files as external links in the main file. The plugin library can be used along with the main HDF5 library, when the user wants dataset level versioning.
This will allow to version at a dataset boundary and still manage as a single unit(by creating external links inside the main file).
For eg – If earlier we have a single main.h5 file with dataset ds1 and ds2 residing inside it, we will now have main.h5, which will be light weight file and, main-ds1.split and main-ds2.split files. Dataset ds1 and dataset ds2 inside main.h5 will point(through external link) to dataset ds1 in main-ds1.split and ds2 in main-ds2.split respectively .
Any application can access the datasets ds1 and ds2 by accessing it through main file. So this split is transparent to applications and they need not be aware of the underlying split. All major operations like read and write to HDF5 will happen the same way for the applications as if it’s one single file.
This way a change in ds1 will effect only main-ds1.split file and main.h5(the main file in which ds1 is linked), leaving main-ds2.split file unchanged, thereby not needing to be duplicated and replicated again to storage remote.
To make the manageability easier with DVC, we would like to propose a feature in DVC .
We would like DVC also to have the capability to be manage these dependent external links inside the main file, along with the main file as single unit. For example, by being able to automatically find the external links mounted on the main file and version these dependent external split files associated with the main file, when the main file is added to DVC.
One scenario is dvc add, a dvc add main.h5, should automatically find the external link files (h5py has API's for this) and create .dvc files for these files, main-ds1.split and main-ds2.split and prompt git add of main.dvc , main-ds1.dvc and main-ds2.dvc
We seek discussion/ feedback on the approach, alternatives.
We are working in the area of AI for science.
HDF5 is one of the file formats used extensively by scientific community to store data. Data can be organized inside HDF5 file under different dataset objects. A single HDF5 can host multiple datasets.
While exploring DVC for AI for science use case, we encountered a problem. Versioning in DVC happens at the file granularity and since these HDF5 files can be very large, it causes storage and network bottlenecks.
A small change inside any of the datasets of a file would cause the entire file to be duplicated and replicated to the storage remote.
We have a potential solution on how this problem can be addressed for Hdf5 files.
HDF5 offers a natural boundary of individual datasets inside the file. It also has a existing feature of external links by which an external datasets residing in a different file can be mounted into the file.
We are planning to split HDF5 files into separate files(using a plugin library) at dataset boundary and linking these smaller files as external links in the main file. The plugin library can be used along with the main HDF5 library, when the user wants dataset level versioning.
This will allow to version at a dataset boundary and still manage as a single unit(by creating external links inside the main file).
For eg – If earlier we have a single main.h5 file with dataset ds1 and ds2 residing inside it, we will now have main.h5, which will be light weight file and, main-ds1.split and main-ds2.split files. Dataset ds1 and dataset ds2 inside main.h5 will point(through external link) to dataset ds1 in main-ds1.split and ds2 in main-ds2.split respectively .
Any application can access the datasets ds1 and ds2 by accessing it through main file. So this split is transparent to applications and they need not be aware of the underlying split. All major operations like read and write to HDF5 will happen the same way for the applications as if it’s one single file.
This way a change in ds1 will effect only main-ds1.split file and main.h5(the main file in which ds1 is linked), leaving main-ds2.split file unchanged, thereby not needing to be duplicated and replicated again to storage remote.
To make the manageability easier with DVC, we would like to propose a feature in DVC .
We would like DVC also to have the capability to be manage these dependent external links inside the main file, along with the main file as single unit. For example, by being able to automatically find the external links mounted on the main file and version these dependent external split files associated with the main file, when the main file is added to DVC.
One scenario is dvc add, a dvc add main.h5, should automatically find the external link files (h5py has API's for this) and create .dvc files for these files, main-ds1.split and main-ds2.split and prompt git add of main.dvc , main-ds1.dvc and main-ds2.dvc
We seek discussion/ feedback on the approach, alternatives.