0.2.8 - #79
Merged
Merged
Conversation
Increase sleep delays from 1ms to 10ms in test_asn.py to give ZMQ enough time to process subscriptions and deliver XPUB_WELCOME_MSG.
…_coords in trigger instead of local implementation.
…ection; finish assign_parent rename
Rename all public coordinate API methods to private (_isvalid, _get_value, _get_indexer, _slice, _concat, _to_dataset, _collect_from_dataset, _is_monotonic_increasing) throughout source and tests. Implement abstract methods in DefaultCoordinate, DenseCoordinate, and ScalarCoordinate. Restore removed methods (decimate, from_array, from_block) and update call sites in core, io, and picking. Adapt tests to match the revised semantics: SampledCoordinate.end now returns the inclusive last value (base-class _get_value); slice_indexer overrides are removed from Default and Dense so step raises NotImplementedError and stop is exclusive-open; decimate tests replaced by equivalent coord[::q] indexing.
Replace the median-based nominal sampling interval with the exact minimax spacing: the value minimising the worst per-segment accumulated drift, which is precisely what _is_valid_sampling_interval bounds. Fold the logic into to_regular (dropping _nominal_sampling_interval), add a tolerance="auto" mode that picks the smallest valid tolerance, and skip recomputation when the coordinate is already regular.
Detect split candidates against a running reference step that follows sustained sampling-rate changes, so a continuous axis whose rate changes is no longer reported as a discontinuity while genuine gaps still are. Also make parse_scalar_delta raise on a None value when no default is available.
… rule Add _continuous_segments to encode the CF rule (den==1 = discontinuity, see section 8.3) in one place, used by both validity checking and spacing inference. Expand the class docstring with the continuous-area/discontinuity model and regularity semantics, add short docstrings to the sampling-interval helpers, and drop the misleading tolerance=None default on _is_valid_sampling_interval.
Replace the O(n^2) pairwise height matrix in to_regular's spacing inference with the upper-envelope (convex-hull trick) of the 2n lines +/-(den*si - num); its lowest vertex is the binding pair. The inner hull loop is numba-compiled, keeping prep/lexsort vectorized.
Add a docstring explaining how the tolerance bound implicitly preserves CF 8.3 structure, and extend the test suite with cases for real and soft discontinuities, multiple runs with isolated tie points, kink preservation, and datetime tie values.
`__len__` uses the documented `tie_indices[0] == 0` invariant, `_slice` drops a Python-loop in favour of a vectorised `_get_value` call, the Chebyshev-center pair is returned as `(pos_idx, neg_idx)` rather than the opaque `(i, j)`, and `_upper_envelope_min_pair` carries an explicit invariant comment on why its scan is bounded.
Douglas-Peucker can fuse a soft discontinuity into a continuous ramp; the merged segment then carries the absorbed jump on top of the original tie-value jitter. Storing `self.tolerance + tolerance` covers that worst case and degrades to `self.tolerance` for a lossless simplify (`tolerance=0`).
Previously a regular coord short-circuited to `self.copy()` whenever no `sampling_interval` was forced, silently dropping any user-supplied `tolerance` (including `tolerance="auto"`). Default each unspecified argument to the stored value instead, so an explicit override on either axis is always respected.
`_concat` is a low-level primitive: it preserves the regular contract only when both sides advertise the exact same `sampling_interval`, otherwise the merged coord is irregular. The joining tie pair is a CF discontinuity, so each side's segments validate independently and `max(tolerance)` bounds the union; raising on a mismatch was punishing a perfectly representable result. User-facing reconciliation moves up to `concat_coords`: after the usual `simplify` step, an irregular merge gets one chance to recover a single shared rate via `to_regular(tolerance=...)`, falling through unchanged when no spacing fits.
The "auto" branch picked the smallest tolerance that kept the inferred spacing valid, which silently accepted arbitrarily large drift on pathological inputs - the very opposite of what a tolerance argument is supposed to enforce. Callers that genuinely want to absorb the worst-case drift can now pass an explicit numeric tolerance instead.
Split AxisCoordinate.simplify into two opt-in stages: reduce (drop redundant tie points, default on) and regularize (acquire a nominal sampling_interval, default off). InterpCoordinate gains the promotion logic; Sampled/Dense treat regularize as a no-op. Thread reduce/regularize through concat_coords (regularize on, so the rate-recovery path keeps working) and the public concat (regularize off, preserving round-trip equality). Make to_regular/infer_regular private.
Both the time and frequency output coordinates now use coord_cls.from_block() where coord_cls mirrors the input dimension's coordinate type, rather than falling back to a generic InterpCoordinate dict. This ensures sampling_interval is properly preserved for SampledCoordinate inputs.
Provides a minimal DataArray for use in tests and doctests. Defaults to 100 × 10 (100 Hz, 10 m spacing → 1 s × 100 m). Accepts a step argument (scalar or per-dimension tuple); float steps on the datetime dimension are auto-converted to timedelta64[ns].
Replace the standalone get_sampling_interval(da, dim) calls in signal.py with da.coords[dim].get_sampling_interval(), propagate sampling_interval through UpSample and resample_poly coordinate reconstruction, and update tests to use the new coordinate API.
Promote to_regular to the public AxisCoordinate interface: InterpCoordinate enforces or infers a spacing, SampledCoordinate validates and copies, and DenseCoordinate converts to a regular InterpCoordinate. isregular() moves to the Coordinate base (False for scalars) so the predicate exists on the whole hierarchy. Regularity now means "carries a declared sampling_interval". Accordingly DenseCoordinate.get_sampling_interval returns None instead of the end-to-end average, which was vacuously "regular" for any dense axis and fed meaningless rates to signal processing on jittery data. The module-level get_sampling_interval becomes the single choke point for signal routines (signal.py stops open-coding the check). Data written by earlier versions carries no declared rate, so rather than breaking every existing archive it falls back to inferring one and emits a FutureWarning stating the inferred value, the tolerance it requires and the migration path. It raises only when no spacing can be inferred at all. Also fix from_block for sizes below two, which built invalid tie indices.
Treat tolerance as a property of the coordinate rather than a per-call parameter: simplify(tolerance=None) now spends the coordinate's own declared jitter instead of a zero-like default, and a regular coordinate keeps its tolerance through a reduce pass unless the fused values no longer validate, in which case it widens by the spent budget only. Operations that derive a new rate declare the error they introduce: UpSample records the truncation residue of delta // factor on top of the inherited jitter. Chunk seams then land within tolerance of the nominal grid, so chunked and unchunked pipelines produce equal coordinates again. Align concat_coords defaults with concat (tolerance=None, regularize=False); regular inputs stay regular through concatenation, so promotion is only needed for irregular ones and remains opt-in. Bag compatibility checks use the coordinate-level primitive so an irregular chunk yields a CompatibilityError rather than a TypeError.
Scanners know the acquisition rate, so build coordinates that declare it: prodml and terra15 derive it from the file's own timestamps, the ASN ZMQ subscriber from its header, and from_stream via from_block at nanosecond resolution so a to_stream round trip preserves the coordinate. Per-file tolerance stays zero; cross-file jitter is reconciled at concat time. The FFT functions likewise emit regular frequency and signal axes, without which an fft/ifft round trip would leave the result unusable by any further signal processing.
Add a "Regular coordinates" section to the interpolated-coordinates guide, list to_regular and the module-level get_sampling_interval in the API reference, give xdas.testing its own page, and drop the stale synthetics.dummy entries left by the move to xdas.testing. Rewrite the 0.2.8 release notes as a net diff from 0.2.7 rather than a log of the development history: the sampling-interval change and its transition shim are stated once under Deprecations, and API surface churn that no ordinary user code touches sits under Refactoring. docs/plan_regular_coordinates.md records the design decisions behind the change and can be dropped before merging.
Replace hand-rolled DataArray construction and wavelet_wavefronts payloads with xd.testing.dummy wherever the test only cares about shapes, coordinate positions and round-tripping rather than the signal itself. Coordinate, trigger, picking and StreamWriter tests keep their explicit fixtures since the data values or datetime literals are load-bearing there.
Regular Coordinates
UpSample read coord.tolerance directly, which is None on every coordinate written before 0.2.8, so UpSample and ResamplePoly raised TypeError on all existing data — GPS-synced or not — where 0.2.7 had worked. An irregular input carries neither a rate to inherit nor a jitter bound to derive one from, so the result now stays irregular rather than claiming a precision the source never declared. simplify widened a fused coordinate's tolerance by the accuracy budget without re-checking it. Douglas-Peucker bounds how far values move, not how much drift fusing a discontinuity exposes, so that sum is no bound at all: on jittery multi-file archives it fell short by tens of nanoseconds and the constructor raised, making open_mfdataarray fail for a band of tolerances while succeeding above and below it. Widen to the least value that describes the surviving tie points instead, and stay irregular rather than raise if even that fails. Sequential.reset only reset Partial atoms, leaving every stateful atom holding its filter state, so a reused sequence silently returned wrong data. The constructor wraps non-atoms into Partial, so the inherited Atom.reset already covers every element and the override is gone. The ASN engine now builds one regular block per ROI from dx * roiDec and concatenates them, so distance declares a spacing like every other engine. Taking the step from the metadata rather than re-deriving it from each ROI's bounds keeps it bit-identical across ROIs, which is what lets concatenation preserve the axis; regularizing recovers it when the steps differ only by float rounding, while genuinely different decimations still stay irregular. resample and resample_poly likewise carry the declared jitter across the rate change rather than resetting it to zero.
Ruff 0.16 enables a much broader default rule set (B, C4, SIM, RUF, PERF,
TRY, BLE, S, DTZ, FLY, PL...), which surfaced 161 errors. This makes the
codebase pass again.
Mostly mechanical: collection literals instead of dict()/list()/tuple()
calls, dict.get, itertools.pairwise, next(iter(...)), `not x == y` ->
`x != y`, f-strings, dropped stale noqa directives.
Genuine defects caught along the way:
- a missing `assert` in test_sampled, so the assertion never ran;
- a pointless `== 0` on the call inside a pytest.raises block;
- a dead `dx` statement in the apsensing engine;
- the VirtualArray abstract stubs were bare `NotImplemented` expressions,
so they silently returned None; they now raise NotImplementedError.
Mutable argument defaults (the dim={...} mappings of fft, rfft, ifft,
irfft, stft and to_stream) became None sentinels; the defaults are
unchanged and stay documented. Class-level registries and engine specs
are annotated ClassVar. Deliberate patterns keep targeted noqa with a
reason: engine-fallback blind excepts, the long-lived TDMS handle and its
naive-UTC epoch, and the grouped __all__.
TRY004 is disabled in pyproject: it wants TypeError for type checks, but
xdas raises ValueError for all argument validation and its public API and
tests assert that. RUF012 is ignored under tests/, where class attributes
are fixture tables.
Making stft's default dim reachable exposed that stft never honoured the
"first"/"last" dimension aliases at all: it compared coordinate names
against the unresolved alias, so `stft(da)` raised a size conflict. Fixed
with the same resolution fft.py already uses, plus a regression test.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #79 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 43 44 +1
Lines 4656 4772 +116
Branches 727 748 +21
==========================================
+ Hits 4656 4772 +116 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Set the version to 0.2.8rc0 so the branch builds pre-release artifacts. test_version required every dot-separated part to be a digit, which rejects any PEP 440 pre-release marker; match the version pattern instead.
The version was duplicated in pyproject.toml and xdas/__init__.py, and docs/conf.py carried a third copy that had already drifted to 0.2.7. Declare it dynamic and let setuptools read xdas.__version__, which is now the only place to edit; conf.py derives its release from it too.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
New Features
sampling_interval(with atolerancebounding the allowed jitter). Query it withisregular()/get_sampling_interval(); promote an irregular coordinate withto_regular(). File engines,from_block, and thefft/stftoutputs produce regular coordinates out of the box (@atrabattoni).tolerance, andsimplify/concatspend the declared tolerance by default, fusing chunk seams away (@atrabattoni).simplifygainedreduceandregularizekeywords, and the gaps/overlaps API now works on every axis coordinate, including dense ones (@atrabattoni).Deprecations
FutureWarningexplains how to make the coordinate regular (da[dim] = da[dim].to_regular(tolerance=...)). A future release will raise instead (@atrabattoni).Bug Fixes
Sequential.reset()silently doing nothing: it only resetPartialatoms, so stateful atoms such asIIRFilterorResamplePolykept their state and a reused sequence returned wrong data (@atrabattoni).stftignoring the"first"/"last"dimension aliases — including its own defaultdim— which raised a size-conflict error instead of transforming the named axis (@atrabattoni).Refactoring
Coordinateis now a proper ABC and the newAxisCoordinateABC holds the axis-mapping contract shared by dense, interpolated, and sampled coordinates. Useisinstance(coord, AxisCoordinate)instead of the removedis*predicates (@atrabattoni).DefaultCoordinate,to_dict/from_dict,get_div_points,decimate, andfrom_array; made underscore-privateconcat,get_indexer,get_value,format_index,slice_index(er),isvalid, andget_query; NumPy 2.0copykeyword compliance (@atrabattoni).concat_coordsnow simplifies its result by default, likeconcat; values are unchanged, only redundant tie points are dropped (@atrabattoni).xdas.testing.dummy, a configurable fixture generator replacingxdas.synthetics.dummy(@atrabattoni).B,C4,SIM,RUF,PERF,TRY,BLE,S,DTZ,FLY,PL…). Mutable argument defaults (thedim={...}mappings offft,rfft,ifft,irfft,stft,to_stream) becameNonesentinels documenting the same defaults; class-level registries and engine specs are annotatedClassVar; deliberate patterns (engine-fallback blind excepts, the long-lived TDMS handle, the grouped__all__) carry targetednoqa.TRY004is disabled project-wide, since xdas raisesValueErrorfor all argument validation, including type checks (@atrabattoni).VirtualArraystubs (__getitem__,__array__,shape,dtype,to_dataset) now raiseNotImplementedErrorinstead of silently returningNone(@atrabattoni).