Add hive-style path attributes to directory spools - #776
Conversation
key=value segments in paths under a directory spool (each directory segment, plus __-separated pairs in any segment including the extension-stripped file name) become string attrs: they appear in get_contents(), work with select (equality/collections/glob/regex), and are stamped onto loaded patches. A path attr overrides a same-named attr declared inside the file, so renaming a directory is a cheap way to attach or correct metadata. The applied dict is persisted per source (sources.path_attrs, JSON) and surfaced privately as _path_attrs, since derived/union catalogs absolutize paths. INDEX_VERSION bumps to 4; old indexes rebuild automatically. Directory updates now detect renames/moves by exact (mtime_ns, size_bytes) identity (1:1 both sides) and rewrite paths and path-derived attrs in batched SQL instead of rescanning contents; renames that remove a hive key, or ambiguous stat matches, fall back to a rescan. Renaming 12 partition dirs over 600 files went from 4.1s (full rescan) to 0.12s; steady-state indexing, selection, and patch loading are unchanged within noise.
|
Warning Review limit reached
Next review available in: 20 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (3)
📝 WalkthroughWalkthroughChangesHive-style Hive path attributes
Possibly related PRs
Suggested labels: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
coderabbitai
Bot
added
documentation
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@dascore/io/index/ingest.py`:
- Around line 228-257: Add stacklevel=2 to both warnings.warn calls in the
file-attribute and hive_path_attrs handling, including the calls near typed
attribute processing and inside hive_path_attrs, so warnings point to the
caller.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 032b5374-60fc-40e8-9855-500cff18c5d3
📒 Files selected for processing (11)
benchmarks/test_spool_benchmarks.pydascore/io/index/backend.pydascore/io/index/catalog.pydascore/io/index/indexer.pydascore/io/index/ingest.pydascore/io/index/query.pydascore/io/index/schema.pydascore/utils/paths.pydocs/notes/spool_index.qmddocs/tutorial/spool.qmdtests/test_io/test_index/test_hive_attrs.py
sqlite3's context manager only wraps the transaction, so the test's connection stayed open and blocked the rebuild's index unlink on Windows; close it explicitly. The tutorial's rmtree likewise deleted a directory holding the spool's open index; drop the cleanup like the other tutorial cells.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## dev #776 +/- ##
==========================================
Coverage 100.00% 100.00%
==========================================
Files 164 164
Lines 17356 17471 +115
==========================================
+ Hits 17356 17471 +115
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Drop the redundant empty-moves guard (the syncer already checks), test a plain no-hive rename as a move, and unit-test _detect_moves skipping unstatted sources and the root unit.
|
✅ Documentation built: |
1 participant
Description
Directory spools now parse
key=valuepairs out of the paths inside the spool, Hive-style. Each directory segment can hold one pair (network=XX/), and any segment — including the file name, with its extension stripped — can hold several separated by__(the same separatorget_patch_namesuses), e.g.station=A__tag=raw.h5. Parsed values become string attrs: they appear inget_contents(), work withselect(equality, collections, glob, regex), participate in chunk grouping, survive unions/pickling, and are stamped onto loaded patches. When a path attr and a file-declared attr share a name, the path wins — renaming a directory (or file) attaches or corrects metadata without rewriting data.Details:
dascore.utils.paths.parse_hive_path_attrs(percent-decoding, deepest-key-wins, Hive NULL sentinel skipped, letter-led extension stripping sodepth=1.5survives). Reserved/underscore key names follow the same skip/warn rules as file attrs.sources.path_attrs, JSON) and surfaced privately as_path_attrs, since derived/union catalogs absolutize paths.INDEX_VERSION3 → 4; existing indexes rebuild automatically.(mtime_ns, size_bytes)identity (unique 1:1 on both sides; the rename-invariant manifest signature for directory-format units) and rewrite paths + path-derived attrs in batched SQL, never re-reading contents. Renames that remove a hive key, or ambiguous stat matches, fall back to a rescan.Benchmarks vs
dev(600 files / 12 partition dirs): cold index build, no-op update, time select, and patch loading unchanged within noise; renaming one partition dir + update went 457 ms → 72 ms, renaming all 12 dirs 4.13 s → 0.12 s (~34×, and the gap grows with archive size since dev rescans contents). Path parsing costs ~1.5 µs/path. Two codspeed benchmarks added (TestHivePathAttrBenchmarks: hive indexing, patch loading with stamped attrs).Behavior note for release: directories whose names happen to contain
=now silently produce attrs, and the version bump forces a one-time index rebuild.Changelog
key=valuepath segments into string attrs, usable inget_contents(),select, and chunk grouping; a path attr wins over a file-declared attr of the same name, so renaming a directory attaches metadata without rewriting data. Any path segment containing=is parsed this way, so a directory named for something else that happens to contain one now yields an attr.Checklist
I have (if applicable):
Summary by CodeRabbit
key=valuepath attributes in directory-based spools.__-separated pairs, repeated-key precedence) and efficient rename/move handling that updates attributes without rescanning when safe.