Description
Chunk planning raises AssertionError: source 0 does not overlap output 2 when a chunk boundary falls strictly inside a sub-tolerance gap between two sources of the same partition. The assert's comment claims the state "cannot be reached from the public API", but it can: any gap larger than one sample but within tolerance (so the sources still merge into one partition) leaves a value window no source covers, and a chunk boundary landing in that window trips it.
Found while chunking a real deployment whose files drift 0.3–2 s at boundaries (spool.chunk(time=600, keep_partial=True)), but keep_partial is not required.
Minimal repro (public API)
import numpy as np
import dascore as dc
p1 = dc.get_example_patch(time_min="2020-01-01")
t = p1.get_coord("time")
# 1.4-sample gap: bigger than one sample, within the default tolerance of 1.5
p2 = dc.get_example_patch(time_min=t.max() + 1.4 * t.step)
spool = dc.spool([p1, p2])
# aim a chunk boundary at the middle of the gap
span = (t.max() - t.min()) + 0.7 * t.step
length = span / np.timedelta64(1, "s") / 2
spool.chunk(time=length) # AssertionError: source 0 does not overlap output 2Or directly on the planner with a bare frame:
import pandas as pd
from dascore.utils.chunk_plan import build_chunk_plan
df = pd.DataFrame(
{"time_min": [0.0, 10.4], "time_max": [9.0, 19.4], "time_step": [1.0, 1.0]}
)
build_chunk_plan(df, time=5) # AssertionError: source 0 does not overlap output 2Mechanism
The member mapping picks each output's first source with searchsorted(src1, output_start, side="right") - 1 — the last source starting at or before the output. When the output's start lies inside an inter-source gap, that source ends before the output starts, the trim comes out inverted (lo > hi), and the defensive assert fires (dascore/utils/chunk_plan.py, "does not overlap output"). Sources within a partition are near-contiguous, not contiguous — the assert's premise only holds for gaps of at most one sample.
A possible fix: advance past a non-overlapping leading source instead of asserting. Its samples all lie before the output, and the next source (which searchsorted on the stops already bounds) does overlap, so nothing is silently dropped — but that reasoning deserves more care than the perf PR (#892) wanted to take on, which is why #892 preserves the assert exactly.
Example versions
Description
Chunk planning raises
AssertionError: source 0 does not overlap output 2when a chunk boundary falls strictly inside a sub-tolerance gap between two sources of the same partition. The assert's comment claims the state "cannot be reached from the public API", but it can: any gap larger than one sample but withintolerance(so the sources still merge into one partition) leaves a value window no source covers, and a chunk boundary landing in that window trips it.Found while chunking a real deployment whose files drift 0.3–2 s at boundaries (
spool.chunk(time=600, keep_partial=True)), butkeep_partialis not required.Minimal repro (public API)
Or directly on the planner with a bare frame:
Mechanism
The member mapping picks each output's first source with
searchsorted(src1, output_start, side="right") - 1— the last source starting at or before the output. When the output's start lies inside an inter-source gap, that source ends before the output starts, the trim comes out inverted (lo > hi), and the defensive assert fires (dascore/utils/chunk_plan.py,"does not overlap output"). Sources within a partition are near-contiguous, not contiguous — the assert's premise only holds for gaps of at most one sample.A possible fix: advance past a non-overlapping leading source instead of asserting. Its samples all lie before the output, and the next source (which
searchsortedon the stops already bounds) does overlap, so nothing is silently dropped — but that reasoning deserves more care than the perf PR (#892) wanted to take on, which is why #892 preserves the assert exactly.Example versions
dev(reproduced at both dddf0f7 and with Vectorize chunk planning over partitions #892 applied)