Skip to content

Re-chunking the same dimension assembles one patch larger than its plan row #832

Description

@d-chambers

Description

Chunking the same dimension twice (the "collapse" re-plan path) assembles one patch far larger than the plan says it should be. The ChunkPlan is correct — every output row spans exactly the requested number of samples — but one assembled patch spans from the start of the whole spool instead of from its own planned envelope.

This is not specific to any argument form; it reproduces with plain numeric lengths on dev.

Example

import dascore as dc

sp = dc.get_example_spool("random_das")
first = sp.chunk(time=833 * 0.004)          # 833-sample patches
second = first.chunk(time=416 * 0.004)      # re-chunk the SAME dim

[x.shape[1] for x in second]
# [416, 416, 416, 416, 2082, 416, 416, 416, 416, 416, 416, 416, 416, 416]
#                     ^^^^ should be 416

The plan for that same call is uniform:

plan = first.chunk_plan(time=416 * 0.004)
step = plan.outputs["time_step"].iloc[0]
spans = ((plan.outputs["time_max"] - plan.outputs["time_min"] + step) / step).round()
sorted(set(spans))          # [416.0]  -- every planned output is 416 samples

Output 4's planned envelope is 00:00:06.656 – 00:00:08.316, but the assembled patch covers 00:00:00.000 – 00:00:08.316.

Two observations that may localize it:

  1. Exactly one member row in the whole plan is marked _modified=False (i.e. "load this member whole, no trim), and it belongs to exactly the one output that assembles wrong:

       output_id  _patch_id                time_min                time_max
    6          4          2 2020-01-03 00:00:06.664 2020-01-03 00:00:07.996
    

    Loading that member whole rather than trimming to the 1.332 s window accounts for the extra samples.

  2. The re-plan's _patch_id values live in a different namespace than either spool. For the example above:

    member _patch_id values: [0, 1, 2, 3, 4, 5, 6, 7, 8]
    root   _patch_id values: [1, 2, 3]
    parent _patch_id values: [1, 2, 3, 4, 5, 6, 7]
    

    They look like positional indices assigned by _ensure_patch_id over the collapsed frame (which has 9 member rows), rather than ids of the patches they name. Whether that is intentional (the plan resolver may interpret them against the parent's member rows) or the source of the mis-trim, I did not determine.

Expected behavior

An assembled patch should match its plan row: second should contain 14 patches of 416 samples each. More generally, len(plan.outputs) and each output's envelope should agree with what assembly produces, since chunk_plan is documented as the plan chunk executes.

Notes

Found while adding size-based chunking (chunk(time=25 * dc.units.megabytes)), where it breaks the "patch data never exceeds the requested size" guarantee. Verified on a clean worktree with no local changes, so it is independent of that work. The test TestChainedChunk::test_size_then_size in that branch asserts on the plan rather than the assembled patches and references this issue; it should be tightened to assert nbytes once this is fixed.

Versions

  • OS: Linux 6.8.0 (Ubuntu, glibc 2.39)
  • DASCore Version: 0.1.21.dev73+g37a034562 (dev)
  • Python Version: 3.13.7

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions