Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/docs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -41,7 +41,7 @@ jobs:
shell: bash -l {0}
run: |
set -e
micromamba run -n oceanarray jupyter nbconvert --to notebook --execute notebooks/demo_mooring.ipynb --output=demo_mooring-output.ipynb --ExecutePreprocessor.allow_errors=True
micromamba run -n oceanarray jupyter nbconvert --to notebook --execute notebooks/demo_mooring.ipynb --output=demo_mooring-output.ipynb
mv notebooks/*output.ipynb docs/source/
pushd docs
micromamba run -n oceanarray make clean html
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/docs_deploy.yml
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ jobs:
shell: bash -l {0}
run: |
set -e
micromamba run -n oceanarray jupyter nbconvert --to notebook --execute notebooks/demo_mooring.ipynb --output=demo_mooring-output.ipynb --ExecutePreprocessor.allow_errors=True
micromamba run -n oceanarray jupyter nbconvert --to notebook --execute notebooks/demo_mooring.ipynb --output=demo_mooring-output.ipynb
mv notebooks/*output.ipynb docs/source/
pushd docs
micromamba run -n oceanarray make clean html
Expand Down
2 changes: 1 addition & 1 deletion .github/workflows/pypi.yml
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@ jobs:
- name: Set up Python
uses: actions/setup-python@v7
with:
python-version: "3.x"
python-version: "3.13"

- name: Install build tools
run: |
Expand Down
13 changes: 13 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -6,8 +6,16 @@ All notable changes to oceanarray are documented here.

## [Unreleased]

## [0.5.0] — 2026-10-12

### Added

- **Serial-uniqueness validation** ([#101](https://github.com/ocean-uhh/oceanarray/pull/101)): `validate` refuses duplicate serials within a mooring, a blank serial, and the reserved names `stack`/`grid` (which would collide the per-instrument report filenames `{mooring}_{serial}_report.html`). A trailing `*` is stripped before every check — `1234` and `1234*` on one mooring are a duplicate, a serial that is only `*` is blank — and stays a diagram-only "pay attention to this sensor" annotation, never carried into filenames or nav labels.

### Changed

- **Report NetCDF-inventory appendix restyled** ([#103](https://github.com/ocean-uhh/oceanarray/pull/103)): the instrument, stack and grid reports share one inventory appendix — a Coordinates table (one row per coordinate variable: type, units, standard/long name, min/max, N valid, attributes), variables grouped by dimension set, QC companions shown inline as a green `(_QC)`, and global attributes split into named groups with a Type column; the per-page variable/scalar/globals templates collapse to one shared set, and the grid page gains a coordinates table it never had. The dune2 test fixture units were canonicalised in place (14 attribute edits, data byte-identical) and the hosted dune2 example suite regenerated. `demo_mooring.ipynb` is ported to `oceanarray.processors.process` and runs on the committed fixture from a clean checkout; `demo_step1.ipynb` is removed, and five old-API demo notebooks move to `notebooks/legacy/`.
- **Shared masthead nav** ([#100](https://github.com/ocean-uhh/oceanarray/pull/100)): the report masthead navigation is rebuilt from the shared vendored macro, and the six report-design files are re-vendored from ctdcast with a vendoring guard (`scripts/check_vendored.py`).
- **Canonical units at every stage** ([#102](https://github.com/ocean-uhh/oceanarray/pull/102)): one canonical spelling per quantity is now written to every file — `degree_Celsius`, `mS cm-1`, `dbar`, `m`, `m s-1`, `umol kg-1`, `g kg-1`, `kg m-3`, `1` (practical salinity and turbidity). A registry-scoped pass (`oceanarray.processors.units.canonicalise_units`) runs on the way out of every stage, rewriting the reader spellings (`degC`, `mS/cm`, `meters`, `db`, `S/m`→×10) to the canonical and recording any value conversion in `units_converted_from` / `units_conversion_factor`. Temperature variables carry CF-1.11 `units_metadata = "temperature: on_scale"`. Plot labels are derived from the canonical string by `format_units` (slash form: `mS/cm`, `m/s`, `kg/m³`, `µmol/kg`, `°C`, `PSU`, `NTU`).

- **Single netCDF writer** ([#98](https://github.com/ocean-uhh/oceanarray/pull/98)): the six write paths the processing stages used before now go through one `oceanarray.writers.write` seam. The files that were written uncompressed — the seasenselib-written stage files, whose compression parameters its writer silently discarded, and the bare-`to_netcdf` stage3 output — shrink 58–66% on the dune2 fixtures. The two that were already zlib level-5 compressed — the pressure-gridded and stacked outputs — grow about 1% as they move to the common level-4 setting. Decoded values, dtypes, attributes, and time encoding are unchanged; raw values are byte-identical, verified per stage against the old writer. The private `_get_netcdf_writer_params` methods are removed, including the `quantize` key, which was never a valid netCDF4 encoding parameter and never took effect.
Expand All @@ -27,6 +35,11 @@ All notable changes to oceanarray are documented here.
- **`None` attributes dropped, not blanked** ([#99](https://github.com/ocean-uhh/oceanarray/pull/99)): a `None`-valued attribute (global or per-variable) is now dropped with a warning instead of written as `""`. `UNK` belongs where a value is determined, not the write seam. On the dune2 fixtures no attribute is affected (0/84 globals are empty); this is a guardrail.
- **Time encoding pinned** ([#99](https://github.com/ocean-uhh/oceanarray/pull/99)): every datetime variable is written as `seconds since 1970-01-01` (float64, `proleptic_gregorian`) rather than xarray's per-file guess. Decoded instants change by at most 0.13 µs (float64 seconds have ~238 ns spacing at the 2026 epoch, so ≤119 ns rounding; ~0.5 µs by 2050) — negligible against clock drift, which is corrected in whole seconds. **Raw stored values change**, so a reader using `decode_times=False`/`decode_cf=False` sees different numbers. This also removes a pandas-version dependence in the units string (space vs `T`).

### Known issues

- **Empty-units turbidity files**: a registered variable with an empty `units` fails that one instrument at stage 1 — the handler logs `ERROR: <file>: <reason>`, skips the instrument, and the mooring completes with the others and a non-zero exit (the family rule). Six turbidity files on the drive are affected; the reader fix lands in 0.5.1.
- **dune2 test fixture stack/grid**: the committed `_stack.nc`/`_grid.nc` fixtures reference instruments not present in the committed raw and YAML — they were built from the full mooring before the fixture was trimmed to two instruments — so running `oceanarray run` on the fixture collapses the stack to those two. A known fixture inconsistency, rebuilt in 0.5.1.

## [0.4.0] — 2026-09-08

### Added
Expand Down
4 changes: 2 additions & 2 deletions CITATION.cff
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,8 @@ authors:
affiliation: University of Hamburg, Bundesstraße 53, 20146 Hamburg, Germany
title: oceanarray
doi: 10.5281/zenodo.21890564
version: 0.4.0
date-released: 2026-09-08
version: 0.5.0
date-released: 2026-10-12
url: https://github.com/ocean-uhh/oceanarray
repository-code: https://github.com/ocean-uhh/oceanarray
license: MIT
Expand Down
188 changes: 0 additions & 188 deletions MIGRATION-BASEDIR.md

This file was deleted.

2 changes: 1 addition & 1 deletion docs/source/methods/concatenation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ into a single continuous time series on a regular time-pressure grid.

Purpose
-------
- Interpolate individual deployment periods onto a unified pressure grid (e.g., every 20 dbar from 0–4820 dbar).
- Interpolate individual deployment periods onto a common pressure grid (e.g., every 20 dbar from 0–4820 dbar).
- Construct a consistent time axis (e.g., half-daily `jg` from April 1, 2004 to October 11, 2015).
- Interpolate all segments onto this unified time grid, filling short gaps between deployments.
- Output gridded `.mat` files for later use in AMOC transport code.
Expand Down
10 changes: 5 additions & 5 deletions docs/source/methods/standardisation.rst
Original file line number Diff line number Diff line change
Expand Up @@ -155,7 +155,7 @@ The standardised output is a **raw-equivalent** `xarray.Dataset` with:
- **Dimensions**: `time` (primary coordinate)
- **Variables**: Standardised names (e.g., `temperature`, `salinity`, `pressure`)
- **Coordinates**: `time`, plus instrument metadata variables
- **Attributes**: Comprehensive metadata including instrument, mooring, and deployment information
- **Attributes**: Metadata including instrument, mooring, and deployment information

Example output structure:

Expand Down Expand Up @@ -186,7 +186,7 @@ Example output structure:
5. Quality Control and Error Handling
--------------------------------------

The standardisation process includes robust error handling:
The standardisation process includes error handling:

- **Missing Files**: Graceful handling with detailed logging
- **Format Errors**: Reader-specific error catching and reporting
Expand Down Expand Up @@ -217,7 +217,7 @@ This standardisation step evolved from the RAPID programme's RDB format, which p
2021 01 05 21.00056 3.9389 33.2389 1725.4
2021 01 05 22.00056 3.9389 33.2405 1725.3

The modern NetCDF-based approach provides several advantages over the historical RDB format:
The NetCDF-based approach provides several advantages over the historical RDB format:

- **Self-describing metadata**: CF-compliant attributes and coordinate information
- **Efficient storage**: Compression and chunking for large datasets
Expand All @@ -239,9 +239,9 @@ This script performed similar functions to the current Python implementation:
:linenos:
:caption: Excerpt from `microcat2rodb_3.m`

The modern Python implementation in :mod:`oceanarray.processors.stage1` provides equivalent functionality with improved:
The Python implementation in :mod:`oceanarray.processors.stage1` provides equivalent functionality with improved:

- **Error handling**: Comprehensive logging and graceful failure modes
- **Error handling**: Logging and graceful failure modes
- **Format support**: Multiple instrument types through pluggable readers
- **Metadata management**: YAML-based configuration with validation
- **Output formats**: NetCDF with CF conventions for interoperability
Expand Down
26 changes: 26 additions & 0 deletions docs/source/migration.rst
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,32 @@
Migrating from the ``--basedir`` Layout
=========================================

Upgrading from 0.4 to 0.5
-------------------------

0.5.0 is a breaking release. If you are upgrading an existing install:

- **Reprocess from stage 1 for files you distribute.** Old stage files still
load — the canonical-units pass rewrites the reader spellings (``degC`` →
``degree_Celsius``, ``mS/cm`` → ``mS cm-1``, …) on the way in and logs one line
per variable — but files you *share* should be regenerated so they carry the
canonical strings
(`#102 <https://github.com/ocean-uhh/oceanarray/pull/102>`__).
- **``save_dataset`` / ``save_OS_instrument`` are gone.** Call
``oceanarray.writers.write`` instead
(`#99 <https://github.com/ocean-uhh/oceanarray/pull/99>`__).
- **Time encoding is pinned** to ``seconds since 1970-01-01`` (float64).
Re-encoding introduces at most 0.13 µs of rounding in the decoded instants
(≤ 119 ns at the 2026 epoch), but the on-disk representation changes, so a
reader using ``decode_times=False`` / ``decode_cf=False`` sees different raw
units and numbers
(`#99 <https://github.com/ocean-uhh/oceanarray/pull/99>`__).
- **``oceanarray process`` runs stages 1–3 by default.** Pass ``--stage 1 2``
for the old raw→trim-only behaviour
(`#102 <https://github.com/ocean-uhh/oceanarray/pull/102>`__).

----

This page describes how to move from the older instrument-first raw
directory layout (used with the ``--basedir`` flag) to the current
mooring-first layout (used with ``--raw-dir`` and ``--proc-dir``).
Expand Down
4 changes: 2 additions & 2 deletions oceanarray/cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,8 +38,8 @@ def _parse_dirs(
raise SystemExit(
"ERROR: --basedir has been removed. Use --raw-dir + --proc-dir "
"(the mooring directory is now <proc-dir>/<mooring>, not "
"<basedir>/moor/proc/<mooring>). See MIGRATION-BASEDIR.md at the "
"repository root."
"<basedir>/moor/proc/<mooring>). See the migration guide: "
f"docs/source/migration.rst or {paths.DOCS_URL}migration.html"
)

if not proc_dir:
Expand Down
Loading
Loading