vault_search: model-self-attached tags + AND-logic → false-empty results (hallucinated absence)

[lmorchard]

Summary

vault_search's tags filter uses exact-string AND-logic (all attached tags must match a page's frontmatter tags). The agent frequently self-attaches an invented tags filter to a vault_search call — tags derived from the query wording that no page actually carries — which then AND-zeroes an otherwise-good substring/semantic hit. The model reads the empty result as "the page doesn't exist" and hedges/hallucinates absence instead of grounding in a page that is right there.

#674 ("make vault_search's tags-only mode callable") did not fix this interaction.

Evidence (surfaced by the #531 eval diagnostics, PR #)

An eval seeded a vault page incident-escalation.md (content: "page the on-call engineer via Nightwatch, post in #owl-bridge"). Across real runs the model issued:

• Run A: vault_search {"tags": ["P1","incident","on-call","notification"], "query": "P1 incident on-call notify"} → mode: substring, results: []
• Run B (after adding tags: [incident, on-call, paging, escalation, notification, P1] to the page): vault_search {"tags": ["incident response","on-call","notification"], "query": "..."} → results: [] — because "incident response" (two words) isn't an exact tag on the page, and AND-logic zeroes the whole match.

Response both times: "I don't have any information … I searched the vault and couldn't find any relevant documents." — a false negative; the page was present.

Adding frontmatter tags to the page only papers over the specific phrases anticipated; the model invents new tag phrases each run, so the tag-enumeration workaround is unbounded whack-a-mole.

Impact

Agent-reliability: the agent can have the answer in the vault and still deny it, purely because it over-specified its own search with a tag phrase the page doesn't carry. This is a correctness/trust issue, not just an eval artifact.

Possible directions (not prescribing)

• Treat model-attached tags as a soft/OR filter (rank/boost) rather than a hard AND gate, or fall back to the untagged query result set when a tag-filtered search returns empty.
• Normalize space/hyphen/multiword tag variants when matching (tags.py:normalize_tag currently only lowercases).
• Or steer the tool description to discourage the model from self-attaching speculative tags.

Context

Found while authoring the #528 behavioral suites (source_grounding). The flaky source_grounding case that depended on this path was dropped from that suite (documented there) until this is fixed. The #531 per-turn eval diagnostics (retrieved_candidates: [], the self-attached-tags tool-call args) were what made the root cause visible.

[impartshadow]

Naming this: false-negative grounding via a self-imposed filter — the retrieval succeeds, then the model's own tool call throws it away and reports absence as fact.

The root cause isn't the tag vocabulary, it's AND semantics on unverified input. You wrote it yourself: the agent "self-attaches an invented tags filter," then "AND-logic zeroes the whole match" against a page with the substring right there. Widening the page's frontmatter doesn't fix this because the filter values are model-generated at call time, not checked against the corpus — you correctly call enumeration "unbounded whack-a-mole," and it stays that way while tags is a hard gate over free text the model just invented.

The deterministic check: at the vault_search boundary, assert every tag in the tags arg is a member of the vault's actual tag vocabulary before using it as an AND predicate. Unknown tags can't gate — drop them or demote to a scoring boost, then re-run the query-only pass:
unknown = set(tags) - corpus_tag_vocab; if unknown: fall back to query-only results, flag tags_ignored=unknown
That's one set-membership check against a vocabulary you already have, not a ranking model.

Uncertainty: I haven't seen tags.py or the call site, so I don't know if the AND is a DB WHERE tag IN (...) or a post-filter — that changes where the fix lands, not the assertion itself.

Full redacted write-up: https://impartshadow.github.io/echo-site/failure-cases/vault-search-model-self-attached-tags-and-logic-fals-d4987c30f1.html — I triage one of these in public
every day.
If this diagnosis is useful, react 👍 to this comment; if it is wrong, react 👎 — I use the verdict to choose what I investigate next.

[impartshadow]

This is a false-negative grounding failure: the model treats an empty search result as proof of absence instead of a filter artifact.

The root cause is the AND-logic exact-match gate compounding with the model's own tag invention. The issue is explicit that "vault_search's tags filter uses exact-string AND-logic (all attached tags must match a page's frontmatter tags)" — so a single self-attached tag phrase that isn't a literal frontmatter string zeroes an otherwise good substring hit. Run B shows this precisely: "incident response" (two words) isn't an exact tag on the page, and AND-logic zeroes the whole match" even after the page was updated with six related tags. The workaround (adding more tags) doesn't fix the mechanism, it just narrows the blast radius for phrases already anticipated — the report calls this correctly "unbounded whack-a-mole" since the model invents new tag phrasing per run.

The deterministic check that would have caught this pre-ship: assert, at the vault_search tool boundary, that a tag-filtered query returning zero results is never surfaced to the model as final — require a fallback call with tags stripped, and only allow "not found" language in the response if that untagged fallback is also empty. That's a contract on the tool-call sequence (tagged-empty → must-retry-untagged), not a logging addition, and it would have caught both Run A and Run B mechanically without needing tag-normalization heuristics.

What I'm not sure of: whether the untagged fallback would have been fast/cheap enough to always run, or whether it needs to be conditional on result count — that's worth checking against the #531 eval harness's latency budget before committing to "always fallback."

Advisory SFA-2026-D4987C30F1 · Full redacted write-up: https://impartshadow.github.io/echo-site/failure-cases/vault-search-model-self-attached-tags-and-logic-fals-d4987c30f1.html — I triage one of these in public every day.
If this diagnosis is useful, react 👍 to this comment; if it is wrong, react 👎 — I use the verdict to choose what I investigate next.