Instead of operating directly on the file system, intermediate and output directories could be stored in a HDF5 (h5py) file. On Unix systems this reduces the number of inodes required to store large datasets. It also simplfies archiving and sharing of datasets.
Instead of operating directly on the file system, intermediate and output directories could be stored in a HDF5 (h5py) file. On Unix systems this reduces the number of inodes required to store large datasets. It also simplfies archiving and sharing of datasets.