Skip to content

Persist reusable vlen heap space across writer reopen #10

Description

@janosh

Problem

write_vlen_strings_slice reclaims superseded global-heap objects, but the recovered space is reusable only by the current writer session. A workload that opens a file, replaces one variable-length string, closes it, and repeats cannot reuse space freed by earlier sessions. The file therefore grows on every update even when alternating between equal-sized values.

Standalone reproducer

use rust_hdf5::H5File;

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let path = "/tmp/reopen-vlen.h5";
    let _ = std::fs::remove_file(path);

    let file = H5File::create(path)?;
    file.write_vlen_strings("notes", &["initial"])?;
    file.close()?;

    for iteration in 0..16 {
        let payload = if iteration % 2 == 0 { "a" } else { "b" }.repeat(128 * 1024);
        let file = H5File::open_rw(path)?;
        file.dataset_writer("notes")?
            .write_vlen_strings_slice(0, &[payload.as_str()])?;
        file.close()?;
        println!("{}", std::fs::metadata(path)?.len());
    }

    Ok(())
}

After the first large replacement, subsequent equal-sized replacements should ideally reuse the released space and leave the file size roughly flat.

Requested behavior

Please persist or reconstruct reusable free-space information on open_rw, so blocks released by vlen updates can be reused by later writer sessions. This should preserve the existing SWMR safety rule and remain compatible with files written by libhdf5. If persistent reclamation needs an explicit opt-in or compaction step, exposing that would still avoid downstream whole-file rebuilds.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions