spool.update() permanently skips files whose mtime predates the last index update

[d-chambers]

Description

DirectoryIndexer.update() will never index a file whose mtime predates the last index update, even though the file is present on disk and was not indexed before. The omission is permanent: subsequent update() calls keep skipping it.

_get_mtime derives the scan cutoff from the index's stored last_updated_timestamp (indexer.py:263-272), and update() passes it as the timestamp floor to scan_to_df (indexer.py:326), where _iter_filesystem drops any entry with st_mtime < timestamp.

The watermark is on file mtime, but the thing it stands in for is index membership. Those agree only while mtime is monotonic with arrival time, which fails whenever a file is placed in the directory with its original timestamp preserved:

• cp -p, rsync -t / -a, tar -x, mv from another directory
• restoring a directory from backup
• clock skew between the writing host and the indexing host (NFS)
• a file that failed to scan on an earlier pass (partial or corrupt transfer) and later became readable without its mtime changing

Nothing self-heals. Recovery requires touching the file, deleting the index, or calling update(paths=...) with the file path.

Note the asymmetry: the cutoff is only applied while walking a directory. Passing a file path directly bypasses it, because _iter_filesystem raises NotADirectoryError from os.scandir and yields the path unconditionally (misc.py:220-221). So indexer.update(paths=[missed_file]) recovers the file, while indexer.update() on its parent directory does not.

Example

import os
import tempfile
import time
from pathlib import Path

import dascore as dc

with tempfile.TemporaryDirectory() as tmp:
    path = Path(tmp)
    patch = dc.get_example_patch()

    # One file indexed normally.
    patch.io.write(path / "a.h5", "dasdae")
    dc.spool(path).update()

    # A second file arrives afterwards but carries an older mtime, as it would
    # from `cp -p`, `rsync -a`, `tar -x`, or `mv` from elsewhere.
    b = path / "b.h5"
    patch.io.write(b, "dasdae")
    old = time.time() - 3600
    os.utime(b, (old, old))

    dc.spool(path).update()
    found = sorted(Path(p).name for p in dc.spool(path).get_contents()["path"])
    print("on disk:", sorted(p.name for p in path.glob("*.h5")))
    print("indexed:", found)

Output:

on disk: ['.dascore_index.h5', 'a.h5', 'b.h5']
indexed: ['a.h5']

b.h5 stays missing from the index no matter how many times update() is called.

Expected behavior

update() should index every readable file in the directory that is not already in the index, regardless of its mtime relative to the last update. The mtime watermark is a sound optimization for re-scanning already-indexed files, but it should not be the sole gate on discovering new ones — that check wants to be against the set of indexed paths.

A cheaper interim fix, if a full listing diff is considered too expensive for large trees: document the limitation prominently on update() and note update(paths=...) as the recovery path.

Versions

• OS: Ubuntu 24.04 (Linux 6.8.0)
• DASCore Version: 0.1.18 — the relevant code in indexer.py and hdf5.py is byte-identical on master as of filing, so this is not fixed in 0.1.20
• Python Version: 3.14.0

[d-chambers]

Follow-up: this appears to be already fixed on dev, which replaces dascore/io/indexer.py with the SQL-backed index under dascore/io/index/. That update() compares each source's stored (mtime_ns, size_bytes) against the filesystem and states the change explicitly:

Change detection compares each source's stored (mtime_ns, size_bytes) against the filesystem — never a global watermark

So this report is scoped to the released line (0.1.18–0.1.20, and master at 6cee97c), not to dev. Filing it anyway in case the released line gets a patch before the index rewrite lands, and as a regression test to carry over — feel free to close as superseded if dev is the only forward path.

Two things worth preserving as test cases whichever way this goes:

• A file arriving with an mtime older than the last index update (cp -p, rsync -a, tar -x) must still be indexed.
• Comparing size alongside mtime — as dev does — additionally catches a rewrite landing within the same mtime granularity, which an mtime-only comparison misses.

On the cost of dropping the watermark, measured over 20k files:

with mtime filter (current)    yielded=     0  walk=0.047s
no filter                      yielded= 20000  walk=0.007s

The unfiltered walk is faster. _iter_filesystem short-circuits at misc.py:206 (if timestamp is None or entry.stat().st_mtime >= timestamp), so passing a timestamp forces a stat() on every entry while passing None skips it. Removing the filter costs nothing at the traversal layer.