Tighten the stored format, open each file once, and read what ObsPy reads

docs/api/tiles.md

@@ -5,9 +5,9 @@
 # xdas.virtual.tiles
 
 Lazy tile-backed virtual arrays: the only backend of the formats that
-HDF5 virtual datasets cannot serve (Silixa TDMS, MiniSEED), the default
-one for Febus, and available on request from every other engine
-(`vtype="tiles"`).
+HDF5 virtual datasets cannot serve (Silixa TDMS, and everything ObsPy
+reads), the default one for Febus, and available on request from every
+other engine (`vtype="tiles"`).
 
 ## TileArray
 
@@ -27,7 +27,7 @@ Attributes
    TileArray.chunks
    TileArray.ntiles
    TileArray.engine
-   TileArray.attrs
+   TileArray.root
 ```
 
 Methods
@@ -40,7 +40,7 @@ Methods
    TileArray.from_variable
    TileArray.to_dataset
    TileArray.create_variable
-   TileArray.finalize_save
+   TileArray.sibling_datasets
    TileArray.concat
    TileArray.expand_dims
    TileArray.squeeze

docs/release-notes.md

@@ -3,25 +3,33 @@
 ## 0.2.9 (unreleased)
 
 ### New Features
-- **Tile-backed virtual arrays.** The new `xdas.virtual.tiles` module exposes file archives as one lazy `TileArray`. Slicing (any step, including negative), integer indexing, `np.newaxis`, concatenation, and the numpy manipulation routines (the `transpose`, `flip`, `split`, `stack` and `atleast` families, `expand_dims`, `squeeze`, `roll`, `tile`, `delete`, `append`/`insert`) all stay lazy; whole-array reductions (`sum`, `mean`, `min`, `max`, …) stream one tile row at a time; reads touch only the tiles the selection overlaps (@atrabattoni).
-- **`vtype="tiles"` on every HDF5 engine.** The open functions with `vtype="tiles"` return tile-backed arrays for the asn, febus, terra15, apsensing, prodml and native xdas engines. Silixa and MiniSEED always emit them now (replacing the serialized-dask-graph fallback, with time-axis push-down for Silixa), and Febus defaults to them — one tile per file, where the HDF5 backing needed one virtual mapping per data block. Custom engines add support by implementing `Engine.load_tile(path, selection, **params)` (@atrabattoni).
-- Tile-backed arrays round-trip through the native xdas netCDF format: the manifest is stored as a compact `__tiles__` sibling group, relocatable by editing its single root path and directly readable by the 0.3 line (@atrabattoni).
-- **Explicit engine configuration.** The open functions declare `engine`, `vtype` and `ctype`, and `engine` also accepts a configured `xdas.io.Engine` instance. Format-specific parameters (`overlaps`/`offset` for febus, `ignore_last_sample` for miniseed, `swapped_dims` for prodml, `tz` for terra15, `group` for the native format) are engine constructor parameters, validated up front (@atrabattoni).
-- `open_mfdataarray` no longer caps the number of files when the resolved vtype consolidates its scan results, which `tiles` does; the 100 000 ceiling remains for `hdf5`, which builds one virtual mapping per file and so cannot be fused into anything smaller (@atrabattoni).
-- **Streamed multi-file combining.** `open_mfdataarray` now fuses scan results every 100 000 files instead of holding one data array per file until the end, so memory no longer grows with the archive: results are accumulated without coordinate simplification (lossless in any arrival order) and sorted once at the end, giving the same result as before whatever the file naming. Acquisitions interleaved in time now group by compatibility, one array per acquisition, instead of splitting at each alternation. Since the batch size is also the ceiling, anything that opened in one call before still takes the single-batch path unchanged (@atrabattoni).
-- **`xdas.sortby`.** Sort a tile- or stack-backed data array along a dimension by coordinate value, lazily: the blocks are permuted through the manifest without reading any data. This is how the streamed combine orders shuffled archives, exposed for standalone use (@atrabattoni).
-- `simplify` runs in linear time whatever the number of gaps: the reduce stage is now a one-pass sleeve instead of Douglas-Peucker, which degenerated quadratically on gap-rich coordinates (a 100 000-file gappy archive simplified in minutes; now milliseconds). The deviation guarantee is unchanged — dropped tie points stay within `tolerance` of the curve, surviving values never move — though the surviving tie-point selection may differ slightly on jittery axes (@atrabattoni).
+- **Tile-backed virtual arrays.** The new `xdas.virtual.tiles` module exposes a file archive as one lazy `TileArray`: slicing (any step), integer indexing, `np.newaxis`, concatenation and the numpy manipulation routines stay lazy, reductions stream one tile row at a time, and a read touches only the tiles the selection overlaps. Select it with `vtype="tiles"` on any HDF5 engine — Febus defaults to it, Silixa and the ObsPy formats always use it. Tile-backed arrays round-trip through the native netCDF format as a compact `__tiles__` group, relocatable by editing the single root path of its header. Custom engines opt in by implementing `Engine.load_tile(path, selection, **params)` (@atrabattoni).
+- **The `obspy` engine**, named for the library rather than for a format: decoding is `obspy.read`, so miniSEED, SAC, GSE2, SEG-2 and everything else ObsPy supports goes through it. Each contiguous `Trace` becomes one lazy `DataArray` and the collection mirrors the `Stream`, nested `network / station / location / channel`; files the miniseed engine rejected (two sampling rates, duplicated ids, interleaved acquisitions) are now readable (@atrabattoni).
+- **`xdas.trim_overlaps`.** Resolve the overlaps of a data array or collection by dropping the duplicated samples, keeping the later copy by default (ObsPy's `merge(method=1, interpolation_samples=0)`) or the earlier one with `keep="first"`. Trimming stays at the manifest level, so a lazy array stays lazy; `xdas.split(da, "overlaps")` still keeps every copy (@atrabattoni).
+- **`DataCollection.select`**, with `obspy.Stream.select` semantics: `dc.select(station="SX00*", channel="HH?")`. It is an alias of `query`, which now applies an indexer wherever its level sits in the tree rather than only at the root (@atrabattoni).
 
-### Breaking Changes
-- Python 3.10 support is dropped and the numpy requirement is raised to 2.3: the tile manifests use `np.strings` routines introduced in numpy 2.3, which itself requires Python 3.11+. Python 3.10 reaches end of life in October 2026 (@atrabattoni).
-- Passing a bare read function as `engine` now raises a `TypeError`: subclass `xdas.io.Engine` instead (see the data-formats documentation) (@atrabattoni).
-- Misspelled or unsupported keyword arguments passed next to an engine name now raise a `TypeError` instead of being silently ignored, and combining `vtype`, `ctype` or engine keywords with an already configured engine instance raises a `ValueError` (@atrabattoni).
+### Improvements
+- **Scanning scales to archives of any size.** `open_mfdataarray` fuses scan results every 100 000 files instead of holding one data array per file, so memory no longer grows with the archive. With `vtype="tiles"` the file-count ceiling is lifted and constant tile geometry costs one element instead of one per tile: a 23-million-tile archive opens in 1.11 GB instead of 1.67 GB (@atrabattoni).
+- **Explicit engine configuration.** Every open function declares `engine`, `vtype` and `ctype`, and `engine` also accepts a configured `xdas.io.Engine` instance. Format-specific parameters (`overlaps`/`offset` for febus, `ignore_last_sample` for miniseed, `swapped_dims` for prodml, `tz` for terra15, `group` for the native format) are engine constructor arguments, validated up front — a misspelled keyword now raises instead of being silently ignored (@atrabattoni).
+- **`xdas.sortby`.** Sort a tile- or stack-backed data array along a dimension by coordinate value, lazily: the blocks are permuted through the manifest without reading any data (@atrabattoni).
+- `xdas.concat` opening a *new* dimension now checks that the inputs agree on their other coordinates and promotes the scalar ones that vary. Stacking the components of a station is `xd.concat(traces, "channel")`, lazily, with `channel` becoming a real coordinate (@atrabattoni).
+- Acquisitions interleaved in time now group by compatibility, one array each, instead of splitting at every alternation (@atrabattoni).
+- Saving and opening a data collection is linear in its size again (#81): saving 1300 events went from ~53 min to ~35 s (@atrabattoni).
+- `simplify` runs in linear time whatever the number of gaps. The deviation guarantee is unchanged, though the surviving tie points may differ slightly on jittery axes (@atrabattoni).
+- When no engine can open a file, the error now lists what each engine that recognised it said, instead of only reporting that none succeeded (@atrabattoni).
 
-### Deprecations
-- Writing dask-backed virtual arrays is deprecated and emits a `FutureWarning`; existing files still open, but no engine emits them any more (@atrabattoni).
+### Breaking Changes
+- Python 3.10 support is dropped and the numpy requirement is raised to 2.3 (@atrabattoni).
+- **Dask virtualization is removed**, reader and writer alike, along with the `xdas.dask` module: a `__dask_array__` graph can no longer be read. A Dask array remains valid `DataArray` data — it is now computed on write like any other eager array, and `virtual=True` rejects it (@atrabattoni).
+- **Opening a seismological file without naming an engine returns a nested collection, not a stacked array.** `xd.open(file)` on a three-component file used to return a `(3, 100)` array by guessing the traces were synchronized; it now returns the `network / station / location / channel / acquisition` tree. `xd.concat(traces, "channel")` is the one-liner back, and `engine="miniseed"` still gives the old shape. More generally, `xd.open` now combines whether it opened one file or many, so the shape it returns no longer depends on the file count (@atrabattoni).
+- `DataCollection.query` raises a `KeyError` on an indexer naming no level of the collection, instead of silently returning everything unchanged. `dc.query(time=slice(0, 5))`, which used to be a no-op, now raises: use `sel` to trim inside the leaves (@atrabattoni).
+- Custom engines must subclass `xdas.io.Engine`: passing a bare read function as `engine` now raises a `TypeError` (see the data-formats documentation) (@atrabattoni).
 
 ### Bug Fixes
-- The miniseed `ctype` argument is now honored: it previously routed to an unused attribute and the reader always built interpolated time coordinates. The default is unchanged (@atrabattoni).
+- Fix a STEIM-compressed `int32` miniSEED file being scanned as `float64`: the element type now comes from the file's encoding rather than from the empty array `headonly=True` returns (@atrabattoni).
+- Fix the miniseed `ctype` argument being ignored: the reader always built interpolated time coordinates. The default is unchanged (@atrabattoni).
+- Fix a data collection keyed by zero-padded codes — a SEED location such as `"00"`, say — reading back from netCDF as a sequence with its keys lost (@atrabattoni).
+- Fix miniSEED scans being forced to a single process (@atrabattoni).
 
 ## 0.2.8
 

docs/user-guide/faq.md

@@ -76,7 +76,7 @@ chunk boundaries automatically when used with {py:func}`~xdas.processing.process
 ## Can I use xdas with seismic data that is not DAS?
 
 Yes. The data model is generic: a {py:class}`~xdas.DataArray` can represent any
-labeled N-dimensional array. The [](io/miniseed.md) page shows a complete example with
+labeled N-dimensional array. The [](io/obspy.md) page shows a complete example with
 a large-N seismic array stored as miniSEED files. All signal processing routines in
 {py:mod}`xdas.signal` and {py:mod}`xdas.fft` work on any DataArray regardless of the
 physical quantity it represents.

docs/user-guide/io/data-formats.md

@@ -34,13 +34,32 @@ Xdas support the following DAS formats:
 | SINTELA           | ONYX              | `"sintela"`       | HDF5, tiles       | `hdf5`    |
 | Terra15           | Treble            | `"terra15"`       | HDF5, tiles       | `hdf5`    |
 
-It also implements its own format and support ProdML and miniSEED:
+It also implements its own format, supports ProdML, and reads every format
+ObsPy reads — miniSEED, SAC, GSE2, SEG-2 and the rest — through a single
+engine named after the library rather than after a format:
 
 | Format            | `engine` argument | Virtualization    | Default   |
 |:-----------------:|:-----------------:|:-----------------:|:---------:|
 | Xdas              | `None`            | HDF5, tiles       | `hdf5`    |
 | ProdML            | `"prodml"`        | HDF5, tiles       | `hdf5`    |
-| miniSEED          | `"miniseed"`      | tiles             | `tiles`   |
+| ObsPy formats     | `"obspy"`         | tiles             | `tiles`   |
+| miniSEED (legacy) | `"miniseed"`      | tiles             | `tiles`   |
+
+The `"obspy"` engine is the only one that describes a file as a *collection*
+rather than a single array: it emits one lazy data array per ObsPy `Trace`,
+nested on the SEED hierarchy. See [](obspy.md).
+
+```{note}
+`"miniseed"` is the engine `"obspy"` replaced, kept so that views written by
+it keep decoding and code written against it keeps running. It describes a
+whole file as one tile of stacked channels, which it classifies as
+*synchronized* or *unsynchronized* and refuses anything else — a file holding
+two sampling rates, for instance. Both engines read the same files, so
+registration order settles auto-detection: `"obspy"` is tried first, and
+`"miniseed"` is reached only through {py:func}`xdas.open_dataarray`, which
+asks for the single stacked array the new engine cannot produce. Prefer
+`"obspy"` in new code.
+```
 
 ```{note}
 A Febus file stores a stack of overlapping blocks rather than one contiguous array. The

docs/user-guide/io/index.md

@@ -7,5 +7,5 @@ This section covers reading and writing data with *xdas*.
 
 data-formats
 virtual-datasets
-miniseed
+obspy
 ```