Investigate search options besides fuse.js

[SamWilsn]

• https://github.com/nextapps-de/flexsearch
• https://github.com/oramasearch/orama

Fuse seems to have some issues with recall/precision. Perhaps one of these is better?

[ritorhymes]

I chose to replace Fuse.js with Pagefind's full-text search capabilities starting in #40 because I believe it best matches the shape of this static site and Rust-based build system. It gives us a lower-maintenance production search path by solving more of the indexing, static asset generation, browser data delivery, and rendered-page metadata/filtering concerns directly. The biggest trade-off is the lack of built-in semantic/vector search, but since this project is not operating a hosted search backend or embedding pipeline where that capability could be used to full effect, I do not think that should drive the production search architecture.

The existing Fuse.js path was built around Atom output. That was not an ideal foundation for site search because Atom is a feed/metadata artifact, not a purpose-built representation of the rendered documentation pages.

Pagefind still produces generated search artifacts, but its production index is derived from the final rendered HTML and the data-pagefind-* hooks emitted by the built site. That lets Pagefind own the production indexing and static delivery path, while the separate corpus in #41 can remain an auxiliary artifact for debugging and external consumers (including for semantic search). Mistakes or iteration in that auxiliary corpus have a smaller blast radius because they do not directly become the browser search pipeline for the production site.

Pagefind also fits the toolchain better here. The build-side integration can run through the Rust crate inside build-eips, instead of adding a separate Node.js-based search indexing path to the preprocessor workflow.

FlexSearch would improve the client-side search engine, but it would not remove the corpus-generation, delivery, and browser-runtime ownership problem. We would still need to decide what content goes into the index, how it is generated, how much search data the browser has to load, how results are rendered, and how that search pipeline is maintained.

Orama is capable too, including vector/hybrid search, but that extra capability is not a free win for this use case. To make semantic/vector search useful here, we would still need to define the corpus and chunking model, generate or provide embeddings, decide how query embeddings are produced, tune ranking/filtering behavior, and own the generated index and loading path. That is significant complexity to manage for a production site feature that mainly needs robust static documentation search.

We can still build around Pagefind by adding a dedicated corpus in #41 that reuses the same rendered-page hooks and metadata for debugging and external consumers such as semantic search systems, without making that extra complexity part of the production site search pipeline.

---

Rollout tracked in #39 and demo site available: https://wg-eips.ritovision.com/search