diff --git a/.gitignore b/.gitignore index 9bbadd6a..ef61fca3 100644 --- a/.gitignore +++ b/.gitignore @@ -71,6 +71,8 @@ target/ # docs docs/api/* +# Where a local `quarto render docs` puts the built site. +docs/_site/ _autosummary .quarto/ docs/site_libs diff --git a/docs/tutorial/file_io.qmd b/docs/tutorial/file_io.qmd index 28d558e9..adb2c4ff 100644 --- a/docs/tutorial/file_io.qmd +++ b/docs/tutorial/file_io.qmd @@ -150,6 +150,8 @@ The `Patch.io` namespace also includes functionality for converting `Patch` inst ## Directory Indexer The `DBDirectoryIndexer` tracks the contents of a directory which contains fiber data. It creates a small, hidden SQLite index named `.dascore_index.sqlite3` at the top of the directory. Directory spools use this index internally and push metadata selections into SQLite before loading patch data. See the [spool index note](../notes/spool_index.qmd) for the schema and lifecycle. +`.dascore_index.sqlite3` is one of two hidden names DASCore gives a meaning at the top of a data directory. The other is `.inventory`, which is where a directory may keep the [inventory](inventory.qmd) describing the observing system its data was recorded through. Both are companions the directory keeps rather than content it holds, which is why both are hidden and neither is scanned as data. + ```{python} #| output: false diff --git a/docs/tutorial/inventory.qmd b/docs/tutorial/inventory.qmd new file mode 100644 index 00000000..3701778a --- /dev/null +++ b/docs/tutorial/inventory.qmd @@ -0,0 +1,290 @@ +--- +title: Inventory +execute: + warning: false +--- + +A DASDAE **inventory** describes the observing system a fiber archive was recorded through: the cables and fibers the light travelled down, the interrogators that lit them, where the fiber physically goes, and how all of it was configured over time. It plays the role StationXML plays in seismology, extended to the things a fiber deployment has and a seismic station does not — an optical path with components along it, and channels whose meaning is a position on that path. + +The data files stay as they are. An inventory is written once, beside the archive, and DASCore joins the two on demand: patches carry an `acquisition_key` (`network.fiber_array.location.acquisition`) which, together with the time they cover, resolves to exactly one entry in the inventory. + +Nothing here changes patch data. Everything an inventory contributes is metadata — attributes on a patch, coordinates along the fiber, and names you can select and group by. + +# The model + +An inventory is a tree of DASCore models. Its spine is containment, and each edge below is labelled with the field that holds the thing it points at. + +```{mermaid} +flowchart TD + Inventory -->|networks| Network + Network -->|fiber_arrays| FiberArray + Network -->|stations| Station + FiberArray -->|acquisitions| Acquisition + FiberArray -->|optical_paths| OpticalPath + Station -->|channels| Channel +``` + +An [`Inventory`](`dascore.core.inventory.Inventory`) holds networks; a [`Network`](`dascore.core.inventory.Network`) holds fiber arrays, and stations for the conventional instruments recorded alongside them; a [`FiberArray`](`dascore.core.inventory.FiberArray`) is the durable observing identity, which holds both [`Acquisition`](`dascore.core.inventory.Acquisition`) objects — how an interrogator was set up — and [`OpticalPath`](`dascore.core.inventory.OpticalPath`) objects — what the light actually travelled through. + +The optical path is the part with no seismological analog. It is described by four independent tracks along optical distance. + +```{mermaid} +flowchart LR + OpticalPath -->|optical_components| Components["FiberSegment · Splice · Connector · Terminator"] + OpticalPath -->|geometry| Geometry + OpticalPath -->|coupling| CouplingCondition + OpticalPath -->|annotations| OpticalPathAnnotation +``` + +The components are the ordered physical pieces the light passes through, and they are what gives the path its length: each one tiles the interval after the last, so the path ends where the final component does. The other three describe intervals of that length — the geometry says where each distance is in space, the coupling says how the fiber is attached to the ground there, and the annotations name anything else worth recording per interval. Each of those covers what it covers, and none of them has to cover the whole path. + +Objects reused in several places — interrogators, cables, enclosures, measurements — are written once under the inventory's `resources` and referred to elsewhere by their `resource_id`. Such a field accepts either the object itself or that string, which is what the dashed edges below mean. + +```{mermaid} +flowchart LR + Acquisition -.->|interrogator| Interrogator + FiberSegment -.->|container| Cable + Cable -.->|container| Enclosure + OpticalPath -.->|measurements| OpticalMeasurement +``` + +These three diagrams are the shape of the model, not the whole of it; each class's own API page lists every field it has. + +# Writing an inventory + +An inventory can be a single YAML file, but the format it is usually authored in is a **directory**, which splits the metadata along its natural grain: small heterogeneous objects as YAML (or JSON) files matching the models, and long row-shaped track data as CSV files a field crew can maintain in a spreadsheet. + +```{python} +import tempfile +from pathlib import Path + +import dascore as dc + +files = { + # The envelope: the document's own facts. Optional. + "inventory.yaml": "object_type: Inventory\n", + # Shared objects, named by the resource_id others refer to them by. + "resources/fi-1.yaml": ( + "object_type: Interrogator\n" + "manufacturer: Fake Interrogators\n" + "model: FI-1\n" + ), + # A file's name states the identity: network.fiber_array.location.code + "acquisitions/DAS.R2D1..RAW.yaml": ( + "object_type: Acquisition\n" + "data_category: DAS\n" + "data_type: velocity\n" + "gauge_length: 10.0\n" + "spatial_interval: 1.0\n" + "interrogator: fi-1\n" + # Where the interrogator's own axis lands on the optical path. + "distance_map:\n" + " instrument_distance: [0.0, 299.0]\n" + " distance: [100.0, 399.0]\n" + ), + # An entity is a file until it needs tracks, and then a directory. + "fiber_arrays/DAS.R2D1/attrs.yaml": ( + "object_type: FiberArray\nname: the north array\n" + ), + "fiber_arrays/DAS.R2D1/path/attrs.yaml": ( + "object_type: OpticalPath\nname: main\n" + ), + # The tracks along the path, as tables. + "fiber_arrays/DAS.R2D1/path/optical_components.csv": ( + "sequence,object_type,optical_length,name\n" + "1,FiberSegment,100.0,lead-in\n" + "2,Splice,0.0,wellhead splice\n" + "3,FiberSegment,400.0,trench cable\n" + ), + "fiber_arrays/DAS.R2D1/path/coupling.csv": ( + "start_distance,end_distance,coupling_type,medium\n" + "100,400,trench,soil\n" + ), + "fiber_arrays/DAS.R2D1/path/annotations.csv": ( + "start_distance,end_distance,group,value\n" + "100,250,zone,north\n" + "250,400,zone,south\n" + "150,300,noisy,true\n" + ), +} + +inventory_path = Path(tempfile.mkdtemp()) / "north_array" +for name, text in files.items(): + path = inventory_path / name + path.parent.mkdir(parents=True, exist_ok=True) + path.write_text(text) + +inventory = dc.inventory(inventory_path) +``` + +A few rules make that directory readable without a schema in front of you: + +- **The file declares what it is.** Every object file states its `object_type`, and its container checks that statement rather than supplying it. +- **The name states the identity.** `acquisitions/DAS.R2D1..RAW.yaml` *is* the acquisition `DAS.R2D1..RAW` — network `DAS`, fiber array `R2D1`, blank location code, code `RAW`. Restating a part of it inside the file is allowed, as long as the two agree; there is never a precedence rule between two spellings of one fact. +- **`path` is the one reserved *container* name.** A directory called `path` addresses an optical path rather than serializing an attribute named "path". (`attrs` and `inventory` are reserved as file stems — an entity's own object file, and the envelope — and the top-level directory names above are fixed.) +- **Files that participate in no convention are ignored.** Photos, field notes, and deployment logs can live in the inventory directory undisturbed. + +Loading checks the document, so a directory that is not a valid inventory says so at the point it is read: + +```{python} +from dascore.exceptions import InvalidInventoryError + +try: + dc.inventory(inventory_path / "acquisitions") +except InvalidInventoryError as error: + print(error) +else: + raise AssertionError("a directory of acquisitions is no inventory") +``` + +Inventories can also be built in memory from the model classes directly, which is how the tests build theirs. The examples below take one already built that way, along with a patch it describes: + +```{python} +from dascore.examples import inventory_patch_pair + +patch, example_inventory = inventory_patch_pair() +``` + +# What an inventory can contribute + +[`Inventory.get_names`](`dascore.core.inventory.Inventory.get_names`) lists the names an inventory could put on a patch, split by where each one lands. `attrs` are the observing-system facts, which are one value per patch; `coords` take a value per channel, because they describe a position along the fiber. + +```{python} +names = inventory.get_names() + +print("attrs:", names.attrs[:4], "...") +print("coords:", [x for x in names.coords if "." not in x]) + +# The annotation groups named in the CSV are among them. +assert {"zone", "noisy"} <= set(names.coords) +assert "gauge_length" in names.attrs +``` + +`zone` and `noisy` are there because the annotations CSV named them: an annotation group becomes a coordinate under its own name. `coupling` and `optical_components` are there for the same reason — a track contributes its name only where a path actually describes it. `distance` is where each channel sits on the optical path, and `x`/`y`/`z` and `longitude`/`latitude`/`elevation` are the two spellings of the axes the inventory's coordinate reference system declares, which a geometry track resolves to. + +Listing a name is not promising a value for it. This example states no geometry, so the spatial names resolve to nothing until it gets one. + +# Attaching an inventory to a spool + +[`Spool.attach_inventory`](`dascore.core.spool.Spool.attach_inventory`) carries an inventory on a spool, and touches no data. No patch gains a field, no row moves, `len` does not change. It costs nothing per patch, which is what makes it safe to do early and decide later what to use it for. Attaching a *different* inventory does clear any enrichment set up from the old one, since applying the old instructions to new metadata would rewrite every patch behind your back. + +```{python} +spool = dc.spool(patch).attach_inventory(example_inventory) + +# Attaching alone adds nothing to the patches. +assert "gauge_length" not in dict(spool[0].attrs) +``` + +What attaching *does* change is which names resolve. The coordinates the inventory defines along the fiber become selectable, and [`Spool.split_by`](`dascore.core.spool.Spool.split_by`) can expand the spool by the values of one: + +```{python} +# Keep only the channels the inventory places in the northern zone. +northern = spool.select(zone="north") + +# Or get one patch per zone, each holding that zone's channels. +zones = spool.split_by("zone") +assert len(zones) == 2 +assert set(zones.get_contents()["zone"]) == {"north", "south"} +``` + +Selecting on an inventory coordinate trims channels rather than cutting the patch at the spool level, since the inventory is what says which channel is which. A patch left with no matching channels at all drops out, there being nothing to keep. + +# Enriching patches + +[`Spool.enrich`](`dascore.core.spool.Spool.enrich`) is how the inventory's metadata actually reaches the patches. It is set up once and applied as each patch is extracted, so it stays cheap on a large spool. + +```{python} +enriched = spool.enrich() + +patch_out = enriched[0] +assert patch_out.attrs.gauge_length == 10.0 +assert "zone" in patch_out.coords.coord_map +``` + +Enrichment never removes a patch. One the inventory does not describe comes out unchanged rather than missing, with a warning, so an inventory that deliberately covers part of an archive needs no pruning first. Deciding membership is a separate step, below. + +A single patch can be enriched directly with [`Patch.enrich`](`dascore.proc.inventory.enrich`), which takes the same arguments apart from `on_unresolved` — one patch either resolves or raises, so there is no policy to state: + +```{python} +one = patch.enrich(example_inventory) +assert one.attrs.gauge_length == 10.0 +``` + +# Conforming a spool + +[`Spool.conform_to_inventory`](`dascore.core.spool.Spool.conform_to_inventory`) is the one eager step of the workflow, and the only one that changes what the spool contains. It resolves every row now, refuses patches the inventory does not describe, and *subdivides* a patch whose span crosses a change of optical path — so the spool can grow as well as shrink. Pass `on_unresolved="drop"` (or `"warn"`) to drop the undescribed patches instead of raising, which is what an inventory deliberately covering part of an archive wants. + +```{python} +# A second patch this inventory says nothing about. +stranger = dc.get_example_patch("random_das", acquisition_key="XX.NOPE..RAW") +mixed = dc.spool([patch, stranger]).attach_inventory(example_inventory) +assert len(mixed) == 2 + +# Conforming keeps only what the inventory describes. +conformed = mixed.conform_to_inventory(on_unresolved="drop") +assert len(conformed) == 1 +assert conformed[0].attrs.acquisition_key == "DAS.R2D1..RAW" +``` + +Subdivision is exact: each piece begins at the first sample at or after the change that opens it, so together the pieces hold every sample the patch held and hold none of them twice. `len` and `get_contents` describe the pieces. + +This is why attaching an inventory never implies conforming to one. Attaching is inert; conforming changes `len`, moves rows, raises by default on a patch the inventory does not describe, and raises, whatever the policy, when a patch straddles a change of acquisition. A spool that quietly changed its own length would be a bad thing to get by default. + +# An inventory a directory carries + +A directory of data can keep the inventory that describes it, under the name `.inventory`, and a spool opened on that directory starts out attached to it. The metadata is then found where it lies, rather than named again by every script that reads the archive. + +```{python} +data_path = Path(tempfile.mkdtemp()) / "archive" +data_path.mkdir() +patch.io.write(data_path / "patch.h5", "DASDAE") + +# Either form works: the directory .inventory/, or a serialized file +# naming its format -- .inventory.yaml, .inventory.yml, or .inventory.json. +example_inventory.to_yaml(data_path / ".inventory.yaml") + +carrying = dc.spool(data_path).update() +assert carrying.enrich()[0].attrs.gauge_length == 10.0 +``` + +The name is hidden for the same reason `.dascore_index.sqlite3` is: it is a companion the directory keeps rather than content it holds, so the file scanner skips it. Both forms present at once is two spellings of one fact, and is refused. The visible spelling `inventory.yaml` is deliberately *not* it — in the authoring format that name is the envelope, so a data directory holding one would be claiming to be an inventory directory itself. + +Three things happen at three different times, and keeping them apart is the point: + +- **Discovery is eager.** Whether the directory carries an inventory is settled when the spool is opened, by a stat of each spelling and nothing more. +- **Reading is lazy.** The file is read at the first question only an inventory can answer — never for `len`, `get_contents`, `sort`, `chunk`, extracting a patch, or a selection on names the index already knows. A malformed inventory therefore cannot stop you loading data; only the inventory-backed calls fail, and they name the directory the inventory came from and say that the spool picked it up on opening. +- **Refreshing is explicit.** An inventory is an input, not a cache, so it is read once and held with no modification-time check. `attach_inventory()` with no argument means "the one this directory carries, read it again", which is the whole authoring loop: + +```{python} +# Correct the gauge length in the file the directory carries. +old_acquisition = example_inventory.networks[0].fiber_arrays[0].acquisitions[0] +corrected = example_inventory.replace( + old_acquisition, old_acquisition.new(gauge_length=12.0) +) +corrected.to_yaml(data_path / ".inventory.yaml") + +# The spool which already read it still reports what it read. +assert carrying.enrich()[0].attrs.gauge_length == 10.0 + +# Asking again is what picks the correction up. +refreshed = carrying.attach_inventory() +assert refreshed.enrich()[0].attrs.gauge_length == 12.0 +``` + +`attach_inventory`, `enrich`, and `conform_to_inventory` all accept a path as well as an `Inventory`, read on the same terms. + +# Serializing + +[`Inventory.to_yaml`](`dascore.core.inventory.Inventory.to_yaml`) writes the single-file interchange form — the artifact to ship beside a data archive — and [`dc.inventory`](`dascore.core.inventory_loader.inventory`) reads any of the forms back. + +```{python} +text = inventory.to_yaml() +assert dc.inventory(text) == inventory + +# JSON works everywhere YAML does, and needs no YAML library installed. +json_path = data_path / "inventory.json" +json_path.write_text(inventory.model_dump_json()) +assert dc.inventory(json_path) == inventory +``` + +A document that cannot be parsed, cannot be decoded, or keys a field with something that is not a name (`1: 2` is legal YAML) raises `InvalidInventoryError` rather than whatever the parser happened to raise, wherever it is read from. diff --git a/docs/tutorial/patch.qmd b/docs/tutorial/patch.qmd index 9e7b9b33..f2eb0299 100644 --- a/docs/tutorial/patch.qmd +++ b/docs/tutorial/patch.qmd @@ -462,7 +462,7 @@ assert patch.get_coord("distance").step is not None assert outside.get_coord("distance").step is None ``` -That is why [`Spool.unselect`](`dascore.core.spool.Spool.unselect`) refuses the patches' own coordinates. A patch can have samples removed from its middle; at spool level the complement of a range would be a hole in every patch rather than a choice between patches. The coordinates an attached DASDAE inventory defines along the fiber are a separate case, and are accepted: removing one of those chooses which channels a patch holds. When several coordinates are named, each is complemented on its own — the true complement of a block is a frame around it, which no array can hold. +That is why [`Spool.unselect`](`dascore.core.spool.Spool.unselect`) refuses the patches' own coordinates. A patch can have samples removed from its middle; at spool level the complement of a range would be a hole in every patch rather than a choice between patches. The coordinates an attached DASDAE [inventory](inventory.qmd) defines along the fiber are a separate case, and are accepted: removing one of those chooses which channels a patch holds. When several coordinates are named, each is complemented on its own — the true complement of a block is a frame around it, which no array can hold. ## Order Order is similar to [`Patch.select`](`dascore.Patch.select`), but will re-arrange data to the order specified by a value array. This may also cause parts of the patch to be duplicated. diff --git a/docs/tutorial/spool.qmd b/docs/tutorial/spool.qmd index f52ec098..d932941f 100644 --- a/docs/tutorial/spool.qmd +++ b/docs/tutorial/spool.qmd @@ -331,3 +331,20 @@ print(agg_patch) ``` See the [parallel processing recipe](../recipes/parallelization.qmd) for more examples with `map`. + +# Inventory + +A spool can carry a DASDAE [inventory](inventory.qmd): a description of the observing system its data was recorded through — the fiber, where it goes, and how the interrogator was configured over time. Attaching one adds nothing to the patches, but makes the names the inventory defines along the fiber available to `select` and [`split_by`](`dascore.core.spool.Spool.split_by`), and lets [`enrich`](`dascore.core.spool.Spool.enrich`) copy that metadata onto patches as they are extracted. + +```{python} +import dascore as dc +from dascore.examples import inventory_patch_pair + +patch, inventory = inventory_patch_pair() +spool = dc.spool(patch).attach_inventory(inventory) + +# The inventory annotates two zones along the fiber, so they can be selected. +assert len(spool.split_by("zone")) == 2 +``` + +A spool opened on a directory which carries an inventory under the name `.inventory` — as a directory, or as `.inventory.yaml`, `.inventory.yml`, or `.inventory.json` — starts out attached to it. diff --git a/scripts/_templates/_quarto.yml b/scripts/_templates/_quarto.yml index fdf8ca1e..479a6e87 100644 --- a/scripts/_templates/_quarto.yml +++ b/scripts/_templates/_quarto.yml @@ -124,6 +124,9 @@ website: - text: Coordinates href: tutorial/coords.qmd + - text: Inventory + href: tutorial/inventory.qmd + - id: Recipes title: 'Recipes' collapse-level: 1 diff --git a/tests/test_inventory_diagrams.py b/tests/test_inventory_diagrams.py new file mode 100644 index 00000000..65ae173f --- /dev/null +++ b/tests/test_inventory_diagrams.py @@ -0,0 +1,133 @@ +""" +Tests which keep the inventory tutorial's mermaid diagrams honest. + +The diagrams are hand-written, so nothing stops them describing a model that no +longer looks like that. These tests read the diagrams back out of the page and +check each edge against the models: that the source is a model, that the field +labelling the edge exists on it, that the target is a model that field can +actually hold, and that a dashed edge is drawn exactly where the field accepts a +resource_id string in place of the object. +""" + +from __future__ import annotations + +import re +import types +from pathlib import Path +from typing import Union, get_args, get_origin, get_type_hints + +import pytest + +import dascore.core.inventory as inventory_module +from dascore.core.inventory import InventoryModel + +_REPO_ROOT = Path(__file__).resolve().parents[1] +_DOC_PATH = _REPO_ROOT / "docs" +_PAGE_PATH = _DOC_PATH / "tutorial" / "inventory.qmd" + +# Run wherever the docs are, skip where they are not, on the same terms as +# test_changelog.py: the sdist grafts tests but ships only docs/LICENSE, so the +# directory existing does not mean the real docs tree is there. Deliberately not +# keyed on the tutorial itself, or deleting the page would skip rather than fail. +_DOCS_PRESENT = (_DOC_PATH / "index.qmd").is_file() + +pytestmark = pytest.mark.skipif( + not _DOCS_PRESENT, reason="the documentation tree is not installed" +) + +_BLOCK = re.compile(r"^```\{mermaid\}\n(.*?)^```", re.MULTILINE | re.DOTALL) +# `Source -->|field| Target` or its dashed form, where a target may carry a +# label: `Components["FiberSegment · Splice"]`. +_EDGE = re.compile( + r"^\s*(\w+)\s*(-->|-\.->)\s*\|(\w+)\|\s*(\w+)(?:\[\"([^\"]+)\"\])?\s*$" +) +_ARROW = re.compile(r"-\.?->") + + +def _read_diagrams(): + """Return the page's edges, and the edge lines which could not be read. + + The unread lines matter as much as the edges: an edge this module cannot + parse is an edge it cannot check, and dropping it silently is how the whole + file goes vacuous one arrow at a time. + """ + if not _DOCS_PRESENT: # nothing to read; every test here is skipped + return (), () + edges, unread = [], [] + # Explicitly utf-8: the page is, and the separator this splits labels on is + # not ascii, so reading it under a locale which is not utf-8 -- windows -- + # decodes the separator to something else and quietly stops splitting. + for block in _BLOCK.findall(_PAGE_PATH.read_text(encoding="utf-8")): + for line in block.splitlines(): + if (match := _EDGE.match(line)) is None: + if _ARROW.search(line): + unread.append(line.strip()) + continue + source, arrow, field, node, label = match.groups() + # A labelled node stands for the several types its label names. + targets = tuple(label.split(" · ")) if label else (node,) + edges.append((source, arrow == "-.->", field, targets)) + return tuple(edges), tuple(unread) + + +def _accepted_models(model, field): + """Return the models the field holds, and whether it accepts a reference. + + A reference is a `str` in the same union as a model, which is how the + inventory spells "this may be a resource_id instead of the object". + """ + annotation = get_type_hints(model)[field] + found, referenced = set(), False + + def _walk(node, in_reference_union): + nonlocal referenced + origin = get_origin(node) + if origin in (Union, types.UnionType): + args = get_args(node) + in_reference_union = str in args + for arg in args: + _walk(arg, in_reference_union) + elif origin is not None: + for arg in get_args(node): + _walk(arg, in_reference_union) + elif isinstance(node, type) and issubclass(node, InventoryModel): + found.add(node) + referenced = referenced or in_reference_union + + _walk(annotation, False) + return found, referenced + + +_EDGES, _UNREAD = _read_diagrams() + + +class TestDiagramEdges: + """Every edge drawn in the tutorial has to be a field the models have.""" + + def test_the_page_draws_edges(self): + """A regex which quietly matched nothing would pass every test below.""" + assert len(_EDGES) >= 12 + + def test_every_edge_line_is_read(self): + """An arrow this module cannot parse is an arrow it cannot check.""" + assert not _UNREAD, f"Unparsed mermaid edges: {_UNREAD}" + + @pytest.mark.parametrize(("source", "dashed", "field", "targets"), _EDGES) + def test_an_edge_matches_the_models(self, source, dashed, field, targets): + """The source, the field, the targets, and the arrow all have to agree.""" + model = getattr(inventory_module, source, None) + assert isinstance(model, type) and issubclass(model, InventoryModel), ( + f"{source} is not an inventory model." + ) + assert field in model.model_fields, f"{source} has no field {field!r}." + + accepted, referenced = _accepted_models(model, field) + names = {x.__name__ for x in accepted} + for target in targets: + assert target in names, f"{source}.{field} cannot hold a {target}." + + arrow = "dashed" if dashed else "solid" + assert dashed == referenced, ( + f"{source}.{field} is drawn {arrow}, which says the wrong thing " + "about whether it accepts a resource_id in place of the object." + )