Description
dc.scan (and dc.scan_to_df / dc.scan_payloads) silently returns nothing when given a one-shot iterable. No warning, no error — just an empty result, which reads as "none of these are fiber data".
The cause is in _iter_scan_results (dascore/io/core.py), which walks its input twice: once through _count_generator to size the progress bar, and once to actually scan.
# Unfortunately, we have to iterate the scan candidates twice to get
# an estimate for the progress bar length. Maybe there is a better way...
_generator = _iterate_scan_inputs(
path, ext=ext, mtime=timestamp, include_directories=False
)
length = _count_generator(_generator)
generator = _iterate_scan_inputs(path, ext=ext, mtime=timestamp)
_iterate_scan_inputs passes any iterable straight through via iterate(), so the counting pass drains a generator and the scanning pass sees an exhausted one. A list or tuple is fine because it can be walked twice.
Example
import dascore as dc
patches = [dc.get_example_patch(), dc.get_example_patch("random_das")]
print(len(dc.scan(patches))) # 2
print(len(dc.scan(tuple(patches)))) # 2
print(len(dc.scan(p for p in patches))) # 0 <-- silently empty
print(len(dc.scan(iter(patches)))) # 0 <-- silently empty
print(len(dc.scan_to_df(p for p in patches))) # 0
Expected behavior
Either of these would be fine; the current silent-empty result is not:
- Scan a one-shot iterable correctly — e.g. materialize the input once, or drop the counting pass and let the progress bar run without a known total (
track already handles an unknown length by skipping the bar).
- Reject it explicitly, so the caller finds out rather than getting an empty list.
The same double-walk also means a directory tree is traversed twice per scan, which the TODO above _count_generator already notes.
Found while typing the scan dispatchers in #816, which added a ScanInput alias. That alias deliberately says Collection rather than Iterable to keep generators out until this is resolved, so fixing this should also widen the alias.
Versions
- OS: Linux 6.8.0 (Ubuntu 24.04)
- DASCore Version: 0.1.20.dev71+g775eaf546 (dev)
- Python Version: 3.14.0
Description
dc.scan(anddc.scan_to_df/dc.scan_payloads) silently returns nothing when given a one-shot iterable. No warning, no error — just an empty result, which reads as "none of these are fiber data".The cause is in
_iter_scan_results(dascore/io/core.py), which walks its input twice: once through_count_generatorto size the progress bar, and once to actually scan._iterate_scan_inputspasses any iterable straight through viaiterate(), so the counting pass drains a generator and the scanning pass sees an exhausted one. Alistortupleis fine because it can be walked twice.Example
Expected behavior
Either of these would be fine; the current silent-empty result is not:
trackalready handles an unknown length by skipping the bar).The same double-walk also means a directory tree is traversed twice per scan, which the
TODOabove_count_generatoralready notes.Found while typing the scan dispatchers in #816, which added a
ScanInputalias. That alias deliberately saysCollectionrather thanIterableto keep generators out until this is resolved, so fixing this should also widen the alias.Versions