Consider advanced search/query-surface additions beyond the near-term FTS fixes

[coreyt]

Summary

The near-term FTS/query-surface improvements should stay focused on the highest-leverage missing primitives: explicit search modes, score/hit metadata, recursive property extraction, and a narrow fallback helper.

Separately, there is a cluster of larger additions that could plausibly improve the long-term query/search surface, but each of them adds real API and implementation complexity. These should be reviewed together as an intentional design track rather than accreted piecemeal.

This issue groups four such candidates:

• full query union() / branch composition
• full staged search/traverse/hydrate API
• fielded/path-aware property FTS materialization
• rich snippets/highlighting/explanations

The point is not that all four must be built. The point is that these are the next layer of search/query-surface power that Memex pressure surfaced, and each one changes the long-term shape of the product enough that it deserves explicit consideration.

Why These Belong In One Design Item

These are tightly related, even though they sit at different layers.

• union() / branch composition is about expressing multi-strategy retrieval in the query surface.
• staged search/traverse/hydrate is about expressing multi-phase retrieval and graph lifting in the query surface.
• fielded/path-aware property FTS is about preserving the structural search information those richer query flows would need.
• snippets/highlighting/explanations are about exposing enough result semantics for applications and operators to understand why a search hit was returned.

If these are evaluated independently, there is a risk of solving one layer while blocking or distorting the others. For example:

• adding query-stage composition without carrying hit provenance or path information forward would produce a more complex query API with weak observability
• adding snippets/highlighting against today’s flattened property FTS would create a shallow explanation surface on top of a weak indexing shape
• adding fielded/path-aware indexing without deciding whether the public query surface wants staged retrieval or branch composition could lead to internal capability with no coherent way to consume it

These should therefore be treated as one review topic: what is the appropriate long-term search/query surface for fathomdb, beyond the near-term fixes?

Background

Memex’s FTS integration worked functionally, but it exposed a distinction between:

• missing substrate primitives that probably belong in fathomdb
• application-owned ranking and semantic logic that should remain in Memex

The near-term deficiencies appear to be mostly:

• weak score/hit metadata
• an ambiguous text_search() contract
• no first-class fallback/union primitive
• overly flat property FTS for nested payloads
• weak search debuggability

Those can be addressed without committing to the larger additions in this issue.

This issue is specifically about the next layer beyond that: the more powerful but more opinionated additions that could make fathomdb’s query/search surface meaningfully more expressive.

1. Full Query union() / Branch Composition

What it would mean

Allow a query to be composed from multiple branches, likely with explicit dedup and merge semantics.

Examples of the kind of thing this could enable:

• strict search branch plus relaxed search branch
• chunk FTS branch plus property FTS branch with explicit weighting or precedence
• direct node hit branch plus traversed-parent branch
• multiple retrieval strategies combined in one compiled surface rather than assembled in SDK/application code

Why it should be considered

Today, best-effort retrieval strategies are awkward to express. Memex’s “full-query first, then per-term fallback” pattern is the clearest example. A narrow fallback helper may solve the immediate case, but it does not answer the more general question of whether the query model should support branch composition as a first-class concept.

There is a real product question here:

Should fathomdb remain a single-driver, single-branch query compiler with a few packaged best-effort search helpers, or should it grow into a query surface that can express multi-branch retrieval explicitly?

That is a foundational choice. If the answer is yes, union()-style composition becomes a core part of the query language. If the answer is no, then best-effort search should stay narrow and highly packaged.

Why it is risky / complex

This is not just “add UNION support.” It creates design questions around:

• dedup identity rules
• branch precedence
• score normalization across branches
• bind/shape hashing semantics
• explain/debug surfaces for merged result sets
• whether branch composition is generic or search-specific

It would be easy to add complexity faster than value here. That is why it should be reviewed explicitly instead of smuggled in through tactical fixes.

2. Full Staged Search/Traverse/Hydrate API

What it would mean

Allow a query to explicitly say:

1. search one object/scope/kind
2. traverse or lift those hits to related nodes
3. hydrate and return a different public result shape
4. optionally carry forward hit metadata from the originating match

This is stronger than today’s flat search plus traversal or grouped expansion.

Why it should be considered

Memex exposed an awkward but important pattern: sometimes the thing you want to search is not the thing you want to return.

Example shape:

• search WMObservation
• traverse to parent WMAction
• filter/rank at the action level
• still preserve the fact that the originating hit came from a specific observation

That is not a Memex-only pattern. It is a generic graph/search pattern: search in one scope, return another scope.

Today this is possible only indirectly, and the semantics become blurry:

• where does ranking happen?
• what object owns the score?
• what do we dedup on?
• how does the caller know why a returned parent node matched?

A staged API could make this explicit and coherent.

Why it is risky / complex

This would push fathomdb closer to a richer retrieval language rather than a simpler node-first query builder. It introduces questions like:

• is staging generic across all query operators, or only for search?
• are intermediate stages public in the API or hidden in the compiler?
• how are scores aggregated when multiple child hits map to one parent?
• how much result-shaping belongs in the engine versus SDK versus application?

This could become one of the most powerful features in the query surface, but also one of the easiest places for the API to become too clever and too hard to reason about.

3. Fielded / Path-Aware Property FTS Materialization

What it would mean

Move beyond today’s flattened “concatenate extracted values into one FTS text blob” model toward derived search state that preserves more structure, such as:

• field identity
• object scope
• path identity
• possibly multiple indexed text columns or multiple derived rows per node/scope

The exact storage shape is open, but the principle is that property FTS should retain more of the structure needed for ranking, filtering, and explanation.

Why it should be considered

This is the strongest candidate here because it addresses a real retrieval-quality limit in the current design.

Today’s property FTS is intentionally narrow and simple, but it loses information that matters for both relevance and usability:

• a match in $.name is not distinguishable from a match in $.payload.notes
• a match deep in one nested branch is not distinguishable from a match elsewhere in the same flattened node
• nested objects are still an awkward fit
• explanation surfaces are weak because the indexed representation itself does not preserve much explainable structure

If fathomdb is going to support richer search over payload-heavy nodes, this is likely the direction that matters most. It is also the cleanest substrate for future score controls, field weighting, scoped matching, and match explanations.

Why it is risky / complex

This is a material change to derived-state shape and query semantics.

Questions include:

• one row per node, one row per object scope, or normalized side tables?
• multi-column FTS versus auxiliary tables?
• how to preserve rebuildability and admin integrity guarantees?
• how much write amplification is acceptable?
• what public query semantics should become possible once this structure exists?

This work has clear upside, but it should be chosen intentionally because it increases both implementation and conceptual complexity.

4. Rich Snippets / Highlighting / Explanations

What it would mean

Expose more than “here is the matched node.” Potential surfaces include:

• matched term/phrase metadata
• matched path/field/scope
• snippets or excerpts for chunk/property hits
• highlighting ranges
• a structured explanation of why a result matched and how it was ranked

Why it should be considered

Search without explanation is workable for simple retrieval, but it becomes much harder to operate and trust as search semantics get richer.

Memex’s integration pressure already showed this in miniature:

• without score access, application ranking has to guess
• without hit provenance, it is harder to understand whether a node matched because of chunk text or structured properties
• without match-path information, nested payload search becomes opaque
• without snippets or explanation, debugging bad results is slower and application code ends up reconstructing weak approximations

If fathomdb wants a stronger “unified query API” story, results need to be inspectable, not just retrievable.

This also matters for operators and tests. Better explanation surfaces make it easier to verify search behavior and understand regressions.

Why it is risky / complex

This can easily turn into an output-format rabbit hole.

Questions include:

• what should be stable public API versus best-effort diagnostics?
• should explanations be lightweight metadata or a richer separate surface?
• can meaningful snippets/highlighting be generated from today’s flat property FTS, or do they depend on better materialization first?
• how much of this belongs in engine results versus SDK sugar?

The main risk is exposing a broad explanation surface before the underlying search representation is strong enough to support it cleanly.

Relationship To Near-Term Work

This issue should not block the near-term fixes.

The near-term path can remain:

• explicit search modes
• score/hit metadata
• recursive property extraction
• a narrow strict-vs-relaxed fallback helper

Those are high-value and comparatively low-risk.

This issue is about deciding whether fathomdb should go meaningfully beyond that and, if so, in what order.

Decision Questions

A design review for this issue should answer:

1. Should fathomdb support general query branch composition, or only narrow packaged fallback/search helpers?
2. Should the query surface support explicit staged retrieval where the searched object and returned object differ?
3. Is flattened property FTS an intentionally narrow permanent model, or should the product move toward fielded/path-aware derived search state?
4. How much match explanation belongs in core engine results versus SDK/debug tooling?
5. Which of these additions depend on the others?

Suggested Outcome

This issue does not assume all four features should be implemented.

A good outcome would be one of:

• explicitly reject some of them and document why
• adopt one as the real next design track and defer the others
• define a dependency order, such as:
  1. fielded/path-aware property FTS first
  2. then richer explanations
  3. then decide whether staged retrieval or branch composition still justify their complexity

That would be a useful result even if no immediate implementation follows.