Description
Chunking the same dimension twice (the "collapse" re-plan path) assembles one patch far larger than the plan says it should be. The ChunkPlan is correct — every output row spans exactly the requested number of samples — but one assembled patch spans from the start of the whole spool instead of from its own planned envelope.
This is not specific to any argument form; it reproduces with plain numeric lengths on dev.
Example
import dascore as dc
sp = dc.get_example_spool("random_das")
first = sp.chunk(time=833 * 0.004) # 833-sample patches
second = first.chunk(time=416 * 0.004) # re-chunk the SAME dim
[x.shape[1] for x in second]
# [416, 416, 416, 416, 2082, 416, 416, 416, 416, 416, 416, 416, 416, 416]
# ^^^^ should be 416
The plan for that same call is uniform:
plan = first.chunk_plan(time=416 * 0.004)
step = plan.outputs["time_step"].iloc[0]
spans = ((plan.outputs["time_max"] - plan.outputs["time_min"] + step) / step).round()
sorted(set(spans)) # [416.0] -- every planned output is 416 samples
Output 4's planned envelope is 00:00:06.656 – 00:00:08.316, but the assembled patch covers 00:00:00.000 – 00:00:08.316.
Two observations that may localize it:
-
Exactly one member row in the whole plan is marked _modified=False (i.e. "load this member whole, no trim), and it belongs to exactly the one output that assembles wrong:
output_id _patch_id time_min time_max
6 4 2 2020-01-03 00:00:06.664 2020-01-03 00:00:07.996
Loading that member whole rather than trimming to the 1.332 s window accounts for the extra samples.
-
The re-plan's _patch_id values live in a different namespace than either spool. For the example above:
member _patch_id values: [0, 1, 2, 3, 4, 5, 6, 7, 8]
root _patch_id values: [1, 2, 3]
parent _patch_id values: [1, 2, 3, 4, 5, 6, 7]
They look like positional indices assigned by _ensure_patch_id over the collapsed frame (which has 9 member rows), rather than ids of the patches they name. Whether that is intentional (the plan resolver may interpret them against the parent's member rows) or the source of the mis-trim, I did not determine.
Expected behavior
An assembled patch should match its plan row: second should contain 14 patches of 416 samples each. More generally, len(plan.outputs) and each output's envelope should agree with what assembly produces, since chunk_plan is documented as the plan chunk executes.
Notes
Found while adding size-based chunking (chunk(time=25 * dc.units.megabytes)), where it breaks the "patch data never exceeds the requested size" guarantee. Verified on a clean worktree with no local changes, so it is independent of that work. The test TestChainedChunk::test_size_then_size in that branch asserts on the plan rather than the assembled patches and references this issue; it should be tightened to assert nbytes once this is fixed.
Versions
- OS: Linux 6.8.0 (Ubuntu, glibc 2.39)
- DASCore Version: 0.1.21.dev73+g37a034562 (
dev)
- Python Version: 3.13.7
Description
Chunking the same dimension twice (the "collapse" re-plan path) assembles one patch far larger than the plan says it should be. The
ChunkPlanis correct — every output row spans exactly the requested number of samples — but one assembled patch spans from the start of the whole spool instead of from its own planned envelope.This is not specific to any argument form; it reproduces with plain numeric lengths on
dev.Example
The plan for that same call is uniform:
Output 4's planned envelope is
00:00:06.656 – 00:00:08.316, but the assembled patch covers00:00:00.000 – 00:00:08.316.Two observations that may localize it:
Exactly one member row in the whole plan is marked
_modified=False(i.e. "load this member whole, no trim), and it belongs to exactly the one output that assembles wrong:Loading that member whole rather than trimming to the 1.332 s window accounts for the extra samples.
The re-plan's
_patch_idvalues live in a different namespace than either spool. For the example above:They look like positional indices assigned by
_ensure_patch_idover the collapsed frame (which has 9 member rows), rather than ids of the patches they name. Whether that is intentional (the plan resolver may interpret them against the parent's member rows) or the source of the mis-trim, I did not determine.Expected behavior
An assembled patch should match its plan row:
secondshould contain 14 patches of 416 samples each. More generally,len(plan.outputs)and each output's envelope should agree with what assembly produces, sincechunk_planis documented as the planchunkexecutes.Notes
Found while adding size-based chunking (
chunk(time=25 * dc.units.megabytes)), where it breaks the "patch data never exceeds the requested size" guarantee. Verified on a clean worktree with no local changes, so it is independent of that work. The testTestChainedChunk::test_size_then_sizein that branch asserts on the plan rather than the assembled patches and references this issue; it should be tightened to assertnbytesonce this is fixed.Versions
dev)