Chunked hyperslab writes, chunk-block reuse, and a real v2 B-tree index

CHANGELOG.md

@@ -1,5 +1,118 @@
 # Changelog
 
+## 0.4.0
+
+### Changed
+
+Breaking, all in `format::chunk_index::btree_v2`, from giving the v2 B-tree a
+real multi-node shape and a derived filtered size-field width:
+
+- `Bt2ChunkIndex::encode` is gone. It returned one header image and one leaf
+  image, which cannot describe a tree deeper than a single node. Build the
+  tree with `Bt2ChunkIndex::build_tree`, then `Bt2Tree::encode` for the node
+  images and `Bt2Tree::header` for the header.
+- `Bt2ChunkIndex::new_filtered` and `Bt2Header::new_for_filtered_chunks` take
+  the chunk-size field width as a third argument. It is derived from the chunk
+  size (`compute_chunk_size_len`) because that is what libhdf5 recomputes when
+  it reads a version-4 layout message; a fixed width produced records libhdf5
+  misparsed.
+- `BT2_FILT_CHUNK_SIZE_LEN` is gone for the same reason — the width is derived
+  per dataset, never a constant.
+
+### Added
+
+- `write_slice` now works on chunked datasets, including compressed ones
+  (issue #2). It was previously rejected with "write_slice is only for
+  contiguous datasets", so updating one row of an appendable dataset meant
+  rebuilding and rewriting the whole thing — O(dataset) memory and I/O for an
+  O(row) change.
+
+  The selection is decomposed onto the chunk grid and only the intersecting
+  chunks are touched. A chunk the selection covers entirely is written
+  straight from the caller's buffer; a partially covered chunk is read,
+  patched and written back (libhdf5 makes the same distinction in
+  `H5D__chunk_lock`'s `relax` flag). Regions of a chunk that no write has
+  reached hold the dataset's fill value. All three chunk index types
+  (extensible array, fixed array, v2 B-tree) are supported.
+
+- Compression now works on datasets with two or more unlimited dimensions,
+  which use a v2 B-tree chunk index. The combination was previously rejected
+  with "compression of v2 B-tree (multi-unlimited-dimension) datasets is not
+  yet supported", although libhdf5 supports it (`H5D_BT2_FILT` in
+  `H5Dbtree2.c`) and h5py produces one from `maxshape=(None, None)` plus
+  `compression=`. The index now writes type-11 records carrying each chunk's
+  stored size and filter mask, so a partial `write_slice` can decompress,
+  patch and recompress a chunk and relocate it when its size changes.
+
+- `write_chunk_raw_at` — the coordinate-addressed direct chunk write
+  (`H5Dwrite_chunk`), storing already-filtered bytes verbatim under a
+  caller-supplied per-chunk filter mask. This is the form a v2-B-tree-indexed
+  dataset needs: with two or more unlimited dimensions there is no fixed chunk
+  grid for the linear `write_chunk_raw` index to mean anything against, so
+  that entry point now points here instead of rejecting the dataset outright.
+  It works on the extensible- and fixed-array indexes too. Direct writes now
+  also range-check the stored size against the index's chunk-size field on all
+  three indexes (libhdf5 `H5D_CHUNK_ENCODE_SIZE_CHECK`) rather than truncating
+  it silently.
+
+### Fixed
+
+- Appending after reopening a file no longer erases the rows already in the
+  chunk the new frame lands in (issue #3). An append that leaves its chunk
+  partial is buffered until close, and the flush built a fresh fill-value
+  chunk around the buffered frame instead of reading what the chunk already
+  held — so `append(&[1, 2, 3])`, close, reopen, `append(&[4, 5, 6])` read
+  back as `[0, 0, 0, 4, 5, 6]`. The same file written in one session was
+  correct, because the in-session append path did read-modify-write it.
+  All three append entry points now place their frames through one owner
+  that preserves everything outside the span it writes.
+
+- A v2 B-tree chunk index is now written with its records ordered by scaled
+  offsets. libhdf5 searches a B-tree node by bisection (`H5D__bt2_compare`
+  orders records with `H5VM_vector_cmp_u`), and records were appended in
+  insertion order, so a file was only correct when the caller happened to
+  write the chunk grid in ascending order. Writing it in any other order —
+  which `write_chunk_at` permits — produced a file this library read back
+  perfectly while libhdf5 and h5py saw the out-of-order chunks as fill: wrong
+  data, with no error. Records are now inserted in sorted position and
+  lookups bisect.
+
+- The width of a filtered v2-B-tree record's compressed-size field is now
+  derived from the chunk size, matching what libhdf5 recomputes for a
+  version-4 layout message (`H5D_BT2_COMPUTE_CHUNK_SIZE_LEN`). The header
+  constructor and the index previously disagreed about it (a 32- versus
+  36-byte record); both now take it from one value.
+
+- A v2 B-tree chunk index is now a real tree of fixed-size nodes, and flushing
+  it no longer leaks. Node size was previously derived from the record count
+  and everything lived in one depth-0 leaf, which caused two failures: the
+  leaf grew with every chunk, so each flush allocated a larger block and
+  stranded the previous one; and one node had to hold every record, but a
+  node's record count is a `u16`, so a dataset past 65535 chunks truncated it
+  silently. Nodes are now 2048 bytes (libhdf5's `H5D_BT2_NODE_SIZE`, which the
+  layout message already declared) and the records are bulk-loaded into as
+  many levels as they need, with internal nodes carrying the separators and
+  subtree totals libhdf5 descends. Because every node is the same size, a
+  flush overwrites the blocks already on disk and allocates only the
+  shortfall, so no block is orphaned and the addresses a reader holds stay
+  valid. A tree that loses nodes releases the surplus blocks to the allocator,
+  except under SWMR where a reader may still hold a header naming them.
+
+- Rewriting a chunk no longer leaks its old file block. Every chunk write
+  previously allocated fresh space and left the previous block stranded, so
+  repeatedly rewriting the same chunk grew the file without bound — including
+  through `append`. A chunk is now placed by consulting its index entry first:
+  an unfiltered chunk (whose size never changes) is rewritten in place, and a
+  filtered chunk that no longer fits its old block is relocated with the old
+  block released to the allocator for reuse. This mirrors libhdf5's
+  `H5D__chunk_file_alloc` / `H5MF_xfree`. Under SWMR the release is suppressed,
+  since a reader may still be following the old address.
+
+- `FileAllocator` gained a free list, so released blocks are reused before the
+  file grows. Best fit with merging of adjacent blocks; like libhdf5's default
+  strategy the list is not persisted to disk, so a block released but unused at
+  close remains slack in the file.
+
 ## 0.3.2
 
 ### Fixed

Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "rust-hdf5"
 description = "Pure Rust HDF5 library with full read/write and SWMR support"
-version = "0.3.2"
+version = "0.4.0"
 edition = "2021"
 rust-version = "1.89"
 license = "MIT"

README.md

@@ -22,7 +22,7 @@ Read and write HDF5 files with contiguous, chunked, and compressed datasets, hie
 - **Attributes** — string and numeric attributes on datasets and root
 - **SWMR** — Single Writer / Multiple Reader streaming protocol
 - **File locking** — OS-level advisory locks (`flock` / `LockFileEx`) honoring `HDF5_USE_FILE_LOCKING`
-- **Hyperslab I/O** — `read_slice` / `write_slice` for partial N-dimensional access
+- **Hyperslab I/O** — `read_slice` / `write_slice` for partial N-dimensional access, on contiguous and chunked (including compressed) datasets
 - **Buffered I/O** — BufWriter/BufReader with automatic mode switching
 - **Memory-mapped I/O** — optional zero-copy read-only access via `mmap` feature
 - **Thread safety** — optional `threadsafe` feature (`Arc<Mutex>` instead of `Rc<RefCell>`)
@@ -34,7 +34,7 @@ Read and write HDF5 files with contiguous, chunked, and compressed datasets, hie
 
 ```toml
 [dependencies]
-rust-hdf5 = "0.2"
+rust-hdf5 = "0.4"
 ```
 
 > Requires Rust 1.89+ (uses `std::fs::File::lock` for cross-platform
@@ -150,6 +150,13 @@ let region = ds.read_slice::<i32>(&[2, 3], &[2, 3])?;
 assert_eq!(region, vec![1, 2, 3, 4, 5, 6]);
 ```
 
+`write_slice` also works on chunked datasets, including compressed ones: it
+touches only the chunks the selection intersects, reading and rewriting a
+chunk in place when the selection covers it only partially. Updating one row
+of a large chunked dataset therefore costs one chunk's worth of I/O, not the
+whole dataset's. Chunk regions no write has reached read back as the fill
+value.
+
 ### Attributes
 
 ```rust
@@ -261,9 +268,16 @@ for ensuring no second writer attaches during streaming.
 ```toml
 # Enable LZ4 + Zstandard
 [dependencies]
-rust-hdf5 = { version = "0.2", features = ["lz4", "zstd"] }
+rust-hdf5 = { version = "0.4", features = ["lz4", "zstd"] }
 ```
 
+Filters apply to every chunked layout, whichever chunk index the dataspace
+selects: a fixed array (no unlimited dimension), an extensible array (exactly
+one), or a v2 B-tree (two or more). Already-filtered bytes can be handed over
+verbatim with a per-chunk filter mask — HDF5's direct chunk write — via
+`write_chunk_raw` (linear index) or `write_chunk_raw_at` (grid coordinates,
+and the form a v2-B-tree dataset uses).
+
 ## Feature flags
 
 | Feature | Description |