feat(search): explicit modes, match provenance, and strict filtering

[dimittal]

Three changes to the search contract, landed together because they touch the same seam — splitting them would migrate both backends twice.

mode — hybrid | keyword | semantic

On the backend contract, not sugar over semantic_weight. It decides which candidates exist, not merely how they are weighted.

This fixes a wrong answer rather than adding a knob. A weight of 0.0 still admitted semantically-matched documents to the candidate set at score 0 — so a keyword search returned documents that matched no keyword. The explorer's Keyword/Semantic toggle had exactly this bug, because it set the weight.

keyword  'card declined at till'   4 results, all type=keyword
hybrid   'card declined at till'  11 results

semantic_weight still blends the two arms within hybrid. Asking for semantic mode now runs the vector arm whatever the configured weight says — a configured 0.0 silently answering a different question than the one asked was the same class of bug.

match — why a document came back

{"type": "keyword" | "semantic" | "both" | "filter" | "none",
 "keyword_score": 0.0-1.0,
 "semantic_score": 0.0-1.0}

Membership is tracked, not inferred from the scores. Cosine rescaled to [0,1] is positive for very nearly every embedded document, so semantic_score > 0 would label the whole index a semantic match. "semantic" means the document was among the K nearest — what the arm actually retrieved.

A listing or a pure filter reports none / filter rather than claiming a keyword hit at score 0. Surfaced in the explorer as a badge, with both arm scores on hover.

filters — a hard predicate, not a ranking hint

Exact-match constraints pushed into the backend query, on every path, in every mode. Fields must declare filterable: true, and filtering on one that does not raises.

Both of those are deliberate, and both target the same failure mode: a filter may be carrying a tenant or user boundary, and the dangerous outcome is not an error but a silent one.

• Filtering only the keyword arm would let a semantic search return precisely the documents the filter existed to exclude. A parametrised test pins this across all three modes.
• Ignoring an unknown or non-filterable field would return the whole index. The refusal names what can be filtered.
• An entity of a doc_type that does not model the field never matches — that falls out of comparing a missing value, and is why a filter cannot leak across doc_types that do not model it.

Two bugs found while building this

The filter matched nothing at first. I had written the JSON path as $.fields.<name>; schema fields are stored flat, so it silently matched zero rows — a fail-open-shaped mistake that happened to fail closed. Caught by testing against real data rather than trusting the shape.

filterable was dropped when a doc_type was serialised back to YAML. A field declared filterable came back non-filterable, and every filter on it was then refused as undeclared — a failure arriving long after its cause. Now covered by a round-trip test.

Notes

score keeps each backend's native ranking value; the normalised, comparable numbers live in match. An existing test deliberately pins the keyword scale as an exact ratio, and normalising then rounding breaks it — so the comparable numbers went where they do no harm. The retrieval audit in the next phase records all three.

OpenSearch already ran the two arms as separate queries and kept both score maps, so provenance there was mostly a matter of reading membership off the dicts it already had. Both backends move together; the mirrored candidate-set contract between them is preserved.

545 tests pass. storage/base.py and ui/view.py at 100%. New coverage for mode membership, the K-bound on semantic matching, filter enforcement in all three modes, fail-closed validation, and the YAML round trip.

🤖 Generated with Claude Code