Attempt at a Datatoad-like FactLSM #39
Open
FiV0 wants to merge 2 commits into
Open
Conversation
Add org.hooray.facts: the in-memory fact-store representation from frankmcsherry/datatoad — Strides offsets, byte-string Terms columns, layered Forest tries, and the FactLSM/FactSet log-structured merge — with the layer-at-a-time kernels (sort with width-4 radix path, union via report queues, intersect, retain, filter, gallop). Query-side machinery (act_on/permute, join_cols, comms) is out of scope for now; width specializations beyond sort are deferred with comments at the dispatch points. Tests port datatoad's model-based forest_model.rs oracle suite in both width regimes, plus kernel-level tests for the porting hazards. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Since the engine is purely in-memory, drop datatoad's byte-string representation from org.hooray.facts: Terms is now a flat object column ordered by UniversalComparator, with item equality defined as compare == 0 so deduplication stays consistent with the sort order. Strides remains solely as the layers' list partitioning. ByteList, RadixSort, the width-4 u32 sort, and colSort's fixed-width fast path disappear; the representation-agnostic kernels (union, intersect, retain, filter, gallop) are untouched. Tests swap the two byte-width regimes for uniformly-typed and mixed-type object domains, each equals-consistent with the comparator so the model oracles hold. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
TODO: explaination