Skip to content

dc.scan silently returns nothing for a generator input #818

Description

@d-chambers

Description

dc.scan (and dc.scan_to_df / dc.scan_payloads) silently returns nothing when given a one-shot iterable. No warning, no error — just an empty result, which reads as "none of these are fiber data".

The cause is in _iter_scan_results (dascore/io/core.py), which walks its input twice: once through _count_generator to size the progress bar, and once to actually scan.

    # Unfortunately, we have to iterate the scan candidates twice to get
    # an estimate for the progress bar length. Maybe there is a better way...
    _generator = _iterate_scan_inputs(
        path, ext=ext, mtime=timestamp, include_directories=False
    )
    length = _count_generator(_generator)
    generator = _iterate_scan_inputs(path, ext=ext, mtime=timestamp)

_iterate_scan_inputs passes any iterable straight through via iterate(), so the counting pass drains a generator and the scanning pass sees an exhausted one. A list or tuple is fine because it can be walked twice.

Example

import dascore as dc

patches = [dc.get_example_patch(), dc.get_example_patch("random_das")]

print(len(dc.scan(patches)))              # 2
print(len(dc.scan(tuple(patches))))       # 2
print(len(dc.scan(p for p in patches)))   # 0  <-- silently empty
print(len(dc.scan(iter(patches))))        # 0  <-- silently empty
print(len(dc.scan_to_df(p for p in patches)))  # 0

Expected behavior

Either of these would be fine; the current silent-empty result is not:

  1. Scan a one-shot iterable correctly — e.g. materialize the input once, or drop the counting pass and let the progress bar run without a known total (track already handles an unknown length by skipping the bar).
  2. Reject it explicitly, so the caller finds out rather than getting an empty list.

The same double-walk also means a directory tree is traversed twice per scan, which the TODO above _count_generator already notes.

Found while typing the scan dispatchers in #816, which added a ScanInput alias. That alias deliberately says Collection rather than Iterable to keep generators out until this is resolved, so fixing this should also widen the alias.

Versions

  • OS: Linux 6.8.0 (Ubuntu 24.04)
  • DASCore Version: 0.1.20.dev71+g775eaf546 (dev)
  • Python Version: 3.14.0

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions