fix(mcp): rank tool_search on whole words weighted by rarity

[groeimetai]

Fixes #298

Stacked on #322 — the corpus it scores is the 445 tools that PR makes reachable, not 436. Review after it merges, or read only the second commit.

What was wrong

The word-level loop kept every query word longer than two characters and matched it with includes(). Three-letter English words therefore matched inside real tokens. snow_sp_theme_manage collected +10 (id contains "the"), +5 (description), and +8 twice (keywords "theme", "theming") — +31 for the word the — and was the top hit for 24 of the eval's 101 queries.

What replaces it

Whole tokens	ids split on _, text on non-alphanumeric runs. A query term that is a strict prefix of an indexed token still scores, at 40% — that is what reaches "SyntaxErrors" from "syntax".
IDF	a term is worth log(1 + (N+1)/(df+1)). Field weights are the original 10/8/5; rarity does the separating.
Coverage	score × 1 + (terms matched / terms asked). Breadth beats repeating one term.
Synonyms	query-side only, at 70%: ticket→incident, column→field, shift→oncall, mail→email.

The smoothing on IDF is not cosmetic. log((N+1)/(df+1)) is 0 for a term every tool carries — correct as a ranking weight, fatal as a filter, because score > 0 decides whether a tool is returned at all. On a one-tool index every term is carried by every tool, which is exactly what the list_changed tests build: searching "query table" against snow_query_table returned nothing.

Numbers

101 queries over 445 tools, from tool-search-eval.test.ts:

	recall@1	recall@5	recall@20	MRR
before	0.267	0.505	0.663	0.384
after	0.426	0.663	0.802	0.536

snow_sp_theme_manage is rank 1 for 1 query now instead of 24, and does not appear at all for "close the incident and fill in the resolution notes".

The synonym map was written after reading the queries the old ranker missed, so the headline number is partly fitted. Split by whether I inspected a query while tuning:

subset		recall@1	recall@5	recall@20	MRR
unseen (67)	before	0.403	0.761	1.000	0.572
unseen (67)	after	0.582	0.881	0.955	0.719
seen (34)	before	0.000	0.000	0.000	0.014
seen (34)	after	0.118	0.235	0.500	0.174

The gain is larger on the queries I never looked at than on the ones I did.

Regressions, all three of them

was	now	query
2	25	"our calls out to the vendor keep timing out" — out ⊂ outbound
6	58	"my script uses arrow functions and the instance throws a syntax error" — syntax ⊂ SyntaxErrors, now a prefix match rather than a substring one, so it survives but lower
9	54	"list everything that runs on the sc_req_item table"

Each was found by an accidental substring. Against them: 17 of the 34 previously unreachable queries now land in the top 20.

Measured and rejected

• Indexing use_cases (all 445 tools have them). Buys recall@20 0.802→0.832 on the full set, costs recall@1 and MRR — and on the unseen subset it is worse on every metric. That is the signature of a change fitted to the hard cases.
• Lifting the 10-word keyword cap. Worse across the board.

Both are recorded in the source so the next attempt does not repeat them.

Other

• Floors raised (0.24/0.45/0.63/0.35 → 0.39/0.63/0.77/0.51), sitting under the re-measured tie band across 24 index orderings. The band is narrower than before because IDF-weighted scores are floats.
• tool-search-enablement.test.ts needed a fixture tool sharing no word with its query. It held snow_query_incidents against the query "snow_query_table" — no substring match, but "query" is a real token hit now, so the registry-fallback branch it guards had stopped running.
• Two no-module-state allowlist entries for the derived token/IDF cache, which is rebuilt from toolIndex and cleared with it.
• mcp-tool-discovery claimed "table names mostly do not work". They work now, because a table name is words — the guide says so, and says which case still fails (sys_user, where the distinguishing word is one the catalog uses constantly).

424 tests pass, typecheck clean.