Write layout message v5 for filtered chunked datasets on opt-in (issue #8)

CHANGELOG.md

@@ -1,5 +1,112 @@
 # Changelog
 
+## Unreleased
+
+### Added
+
+- `H5File::set_libver_latest` opts datasets created after the call into the
+  latest file format, the equivalent of libhdf5's
+  `H5Pset_libver_bounds(low = H5F_LIBVER_V200)`: filtered chunked datasets
+  get a version-5 data layout message whose chunk indexes store on-disk
+  chunk sizes in fixed 8-byte fields, removing the overflow risk when a
+  filter expands a chunk. Off by default — a v5 file needs libhdf5 ≥ 2.0
+  (h5py bundling hdf5 1.14 rejects it), while the default v4 stays readable
+  everywhere. Independent of the knob, a chunk larger than 4 GiB forces
+  version 5, matching libhdf5, because v4 cannot represent its size field.
+  The version read from an existing file is preserved on reopen, so
+  appending to a v5 file never silently downgrades it. In-memory chunk-size
+  fields in the fixed-array and v2-B-tree indexes widened from `u32` to
+  `u64` to carry the 8-byte field. (issue #8)
+
+### Fixed
+
+- Concurrent operations on the *same* dataset serialize wholly under the
+  `threadsafe` feature. Each dataset now carries an operation lock beside
+  its metadata slot; every public write entry (`write_chunk*`,
+  `write_slice`, `append*`, `extend`/`set_extent`, `flush`) holds it for
+  the operation's full duration. The per-slot mutex only serialized each
+  individual acquisition, so two threads appending to one dataset could
+  interleave between the buffer take, the chunk writes and the extend —
+  losing or doubling rows. Same-dataset concurrency was previously
+  documented as unsupported; it is now correct, and writes to different
+  datasets still never contend.
+
+- Chunk geometry is validated at every dataset create, the rule libhdf5
+  applies in `H5D__chunk_construct`: the chunk rank must match the dataspace,
+  no chunk dimension may be zero, and a chunk dimension may not exceed a
+  fixed maximum dimension unless that dimension's current size is zero. The
+  extensible-array and compressed-vlen creators previously accepted any
+  geometry, and a chunk wider than a fixed dimension made appends land rows
+  at the chunk stride — `[1, 2, 3, 4]` read back as `[1, 2, 0, 0]`.
+
+  **Behavior change:** `SwmrWriter::create_streaming_dataset_tiled` no longer
+  accepts a chunk tile larger than the frame. libhdf5 refuses to create that
+  geometry, so no libhdf5-based writer (including the NDFileHDF5 tiling
+  controls the API mirrors) can produce such a file; previously the frame was
+  zero-padded up to the tile.
+
+- `append_vlen_strings` now applies the same character-set rule as
+  `write_vlen_strings_slice` — non-ASCII strings are rejected when the
+  dataset declares ASCII, instead of being stored mislabeled (libhdf5
+  stores the bytes unvalidated; h5py raises on the same mismatch) — and
+  refuses a dataset whose elements are not variable-length strings, which
+  it previously overwrote with vlen references as raw bytes.
+
+- The append buffer records the absolute row its frames belong to, and
+  every operation that writes rows the buffer holds flushes it to the
+  chunks first. Before this, the buffer's position was derived from the
+  current extent, and two operations broke the derivation: a typed
+  `write_slice` into the buffered tail was silently overwritten by the
+  flush at close (write 99, read back 50), and an `extend` with buffered
+  appends made the flush land them at the extended end instead of where
+  they were appended. `write_vlen_strings_slice` drops its
+  patch-the-buffer path for the same flush-first rule.
+
+- Appends work on every chunk index and chunk shape. The append paths'
+  chunk writes required the extensible-array index, so appending to a
+  fixed-array or v2 B-tree dataset buffered fine and then failed at
+  `close()` with "not a chunked dataset", losing the buffered rows. They
+  also packed rows at the frame stride, so a chunk row narrower than the
+  frame — legal, libhdf5-creatable geometry — corrupted the first chunk
+  and errored on the second. Append writes now go through the same
+  index-generic hyperslab engine as `write_slice`, which also makes
+  appends to reopened 0.4.0 files with a wider-than-row chunk land
+  correctly instead of reading back `[1, 2, 0, 0]` for `[1, 2, 3, 4]`.
+
+- Every dataset creator checks the new name is unique before registering
+  the dataset. The vlen creators and `create_chunked_dataset_compressed`
+  skipped the check, so creating two datasets under one name silently
+  emitted an invalid file with two same-named links. The check-then-push
+  pair is now a witness type (`begin_create` → `push_dataset`), so a
+  creator cannot skip it.
+
+- Chunk slots are computed against the **maximum-extent** chunk grid, the
+  libhdf5 rule (`max_down_chunks` in H5Dfarray.c/H5Dearray.c/H5Dnone.c;
+  the fixed array is sized from `max_nchunks`). Slots were computed from
+  the *current* extent, which coincides only while every dimension after
+  the first sits at its maximum — any other geometry wrote files libhdf5
+  reads differently, and extending re-scrambled the mapping. One owner
+  (`io::chunk_grid`) now serves the writer, the reader, and the dataset
+  API. Fallout fixed with it:
+  - The builder silently dropped a finite `max_shape` on the fixed-array
+    path (no unlimited dimension): the array was sized from the current
+    shape and the stored dataspace had no maximum, so the dataset could
+    never grow. `create_fixed_array_dataset_with_max` sizes the array
+    from the maximum's grid and such datasets now extend/append up to
+    their maximum. The fixed-shape creators keep their signatures and
+    now store `max_dims == dims` explicitly.
+  - **Behavior change:** growing a dataset past its stored maximum — or
+    growing one with *no* stored maximum at all — is rejected by
+    `extend_dataset`/`set_dataset_extent`, matching `H5Dset_extent`
+    (libhdf5 defaults maxdims to dims at creation). Previously the grow
+    succeeded and writes failed later, or scrambled chunk slots.
+  - **Behavior change:** creating an extensible-array dataset whose
+    unlimited dimension is not dimension 0 is rejected. Its chunks have
+    no fixed linear slot without libhdf5's swizzling (not implemented);
+    the geometry previously re-indexed — i.e. silently lost — chunks on
+    every extend. Reading such a file (libhdf5-written) now errors
+    instead of returning wrong data.
+
 ## 0.4.1
 
 ### Added

docs/threadsafe-fine-grained-locking.md

@@ -140,17 +140,28 @@ chunked dataset) becomes: lock the *one* target `DatasetSlot`, do the
 short allocate+write+record, unlock. Different datasets run fully in
 parallel.
 
-Scope of the same-dataset guarantee: a *single* slot operation — one
-`write_chunk` / `record_*` — serializes on that slot's `Mutex`, so it
-cannot tear the chunk index. It does **not** make a multi-step sequence
-atomic. `append` deliberately drops the slot between phases (read dims →
-take `append_buffer` → compress + write each chunk → `extend_dataset`) so
-compression and disk I/O run unlocked; two threads appending to the *same*
-dataset can therefore interleave those phases and corrupt the
-buffer/dimension accounting. The supported concurrency is across
-**distinct** datasets (one writer per dataset, e.g. a rayon `par_iter`
-over datasets). Concurrent writes to a *single* dataset are unsupported;
-callers must serialize them externally.
+Scope of the same-dataset guarantee: the metadata slot's `Mutex`
+serializes a *single* acquisition — one `record_*`, one buffer take — so
+it cannot tear the chunk index, but it does not make a multi-step
+sequence atomic. `append` deliberately drops the slot between phases
+(read dims → take `append_buffer` → compress + write each chunk →
+`extend_dataset`) so compression and disk I/O run unlocked. Whole
+operations are serialized one level up: each `DatasetCell` carries an
+**op lock** beside its metadata slot, every public write entry
+(`write_chunk*`, `write_slice`, `append*`, `extend`/`set_extent`,
+`flush`) takes it for the operation's full duration and delegates to a
+`_inner` variant, and multi-acquisition compositions in `dataset.rs`
+take it around their whole sequence. `_inner` variants and the
+`pub(crate)` helpers (`write_append_frames`, `flush_append_buffer*`,
+`write_chunk_at_coords`) require the caller to hold it — or to hold the
+writer exclusively via `&mut`, as close and the SWMR wrapper do. The op
+lock is not reentrant; the single-thread build's `RefCell` panics on a
+nested acquisition, so a missed entry/inner split fails in every test
+run. Lock order: `create_lock → op → registry spine → metadata slot`,
+never two datasets' op locks at once. Concurrent operations on the
+*same* dataset are therefore supported and serialize wholly
+(`tests/threadsafe_same_dataset_append.rs`); the parallelism win remains
+across distinct datasets, whose op locks never contend.
 
 ### 4.4 `SharedInner` — drop the outer `Mutex`