[Bug]: Search indexes zero tokens for non-Latin scripts (CJK, Cyrillic, Thai, …)

[Guohao1020]

Preflight

• I searched existing issues and this isn't a duplicate
• I'm on the latest version (or I've noted my version below)

What happened?

What happened?

Workspace search (omnibar and the MCP search tool, both omnibar and full_text intents) cannot find content written in any non-Latin script. The BM25 layer indexes such text to zero tokens, and a query in those scripts also tokenizes to [] — so document bodies are unsearchable in roughly half the world's writing systems. In a mixed-script query only the Latin half does any work, which makes the failure easy to miss: results come back, but every non-Latin term was silently dropped.

Root cause. packages/core/src/search/workspace-search.ts builds the index with Orama's defaults:

const db = create({ schema: WORKSPACE_SEARCH_SCHEMA });

Orama's default english tokenizer splits on /[^A-Za-zàèéìòóù0-9_'-]+/gim — every character outside that class is a separator. Any run of text in another script is therefore treated as pure separators and deleted, on both the indexing side and the query side. All of the following index to nothing:

import { tokenizer } from '@orama/orama/components';
const t = tokenizer.createTokenizer({ language: 'english' });

t.tokenize('Русский текст');            // []   Cyrillic
t.tokenize('αναζήτηση κειμένου');       // []   Greek
t.tokenize('بحث النص');                 // []   Arabic
t.tokenize('ค้นหาข้อความ');              // []   Thai
t.tokenize('検索テキスト');              // []   Japanese
t.tokenize('텍스트 검색');               // []   Korean
t.tokenize('访问控制');                  // []   Chinese

t.tokenize('поиск tenant isolation');   // ['tenant', 'isolation']  ← only the Latin half survives

The lexical name/path ladder still works for non-Latin file names (it compares whole strings), which partially masks the problem — body content is where it fully breaks.

Suggested direction (happy to turn this into a PR): keep the default tokenizer for everything it already handles, and only additionally segment runs of word characters the default splitter would delete, using the runtime's built-in Intl.Segmenter (word granularity — plain word splitting for spaced scripts, dictionary-based segmentation for unspaced ones like Thai, Japanese, or Chinese):

• zero new dependencies (Intl.Segmenter ships with Node ≥ 16 and Electron),
• zero config, works for all scripts at once rather than per-language,
• pure-Latin input takes the original code path unchanged, so existing tokenization and BM25 scores stay byte-identical (the flag-OFF parity fixture keeps passing),
• previously-dropped text goes from unsearchable to searchable — a strict addition, no regression surface.

An alternative is Orama's own @orama/tokenizers via components.tokenizer, but it adds a dependency, covers only two languages (both marked experimental), and needs config plumbing.

Steps to reproduce

1. ok init a project; add a doc whose body contains a sentence in any non-Latin script, e.g. Русский текст or 検索テキスト.
2. Search it: omnibar or MCP search with { query: "Русский текст", intent: "full_text" }.
3. Zero hits, even though grep finds the text on disk. Same with intent: "omnibar" (body tier). A mixed query like поиск tenant isolation returns only what tenant isolation alone would return.

Platform

Web app

How did you install OpenKnowledge?

CLI (npm / npx)

Version

0.41.0

Logs, errors, or screenshots

[mike-inkeep]

Thanks for the report @Guohao1020, I'm looking into this