Windows 0.7.10 搜索索引重建卡在 indexing_memories 5%，Python 后端内存涨到 22GB 后断连

[9550816-droid]

Nowledge Mem Search Reindex Issue Report

GitHub bug report URL:

https://github.com/nowledge-co/community/issues/new?template=bug_report.yml

Issues list:

https://github.com/nowledge-co/community/issues

Official support page:

https://mem.nowledge.co/docs/community

Discord:

https://nowled.ge/discord

Email:

hello@nowledge-labs.ai

Chinese Version

Title

Windows 0.7.10 搜索索引重建卡在 indexing_memories 5%，Python 后端内存涨到 22GB 后断连

Body

## 问题描述

在 Windows 上使用 Nowledge Mem v0.7.10 时，搜索索引提示需要重建。运行 `nmem models reindex` 后，重建过程卡在 `indexing_memories` 阶段，进度停在 5%，随后 CLI 报错：

```text
httpx.RemoteProtocolError: Server disconnected without sending a response.

本地后端仍显示 reindex active，但 updated_at 不再变化。

环境

• OS: Windows
• Nowledge Mem: v0.7.10
• nmem CLI: v0.7.10
• Search embedding model: BAAI/bge-m3
• Installer: website installer build
• Data path: %LOCALAPPDATA%\NowledgeGraph
• Search index path: %LOCALAPPDATA%\NowledgeGraph\search_index

复现步骤

1. 打开 Nowledge Mem
2. 运行：

nmem status
nmem models status --verify
nmem models reindex

3. nmem models reindex 开始后停在重建搜索索引阶段
4. 约 129 秒后 CLI 断开并报 Server disconnected without sending a response

实际结果

nmem status 显示：

server   v0.7.10
status   ok
database connected
search   rebuild needed

nmem models status --verify 显示：

Search Embedding   ready   BAAI/bge-m3
Search Index       Action needed: rebuild search index

本地接口 /search-index/reindex/status 返回：

{
  "active": true,
  "run_id": "60b217f5-db74-479b-b24e-82a578b9fd25",
  "stage": "indexing_memories",
  "message": "Batch indexing 190 memories...",
  "progress": 5.0,
  "updated_at": "2026-04-29T07:01:35.611720+00:00",
  "success": null
}

同时 Python 后端进程内存异常升高：

PrivateMemorySize64: ~22.3GB
FreePhysicalMemory: ~133MB

CLI 报错：

httpx.RemoteProtocolError: Server disconnected without sending a response.

日志关键点

相关 request_id：

f6a7b605896e4c57a30528ff1e0771ba

日志中停在：

Full reindex: Starting memory reindex...
Generating embeddings for memory chunk
Generating batch embeddings (FastEmbed)

之前另一次重建也出现：

/search-index/reindex status_code=500
timeout graceful shutdown exceeded
Backend process force killed

对应 request_id：

aa4f4a34ed844afa8b24d475f9653c4c

期望结果

搜索索引重建应能完成，或者在内存不足/embedding 失败时能降级、分批处理、释放内存并返回明确错误，而不是卡在 5% 并导致后端内存涨到 22GB。

[enderzcx]

Additional repro on Windows with Nowledge Mem 0.8.3.

Environment:

• Nowledge Mem / CLI / server: 0.8.3
• OS: Windows 11 Home, 64-bit, version 10.0.26200
• RAM: 32 GiB
• Search embedding backend/model: local FastEmbed / BAAI/bge-m3
• Data volume at successful rebuild: 1,093 memories, 344,471 thread messages, 19 Library sources, 1,052 source chunks, 360 communities, 4,953 entities

Symptoms:

• Search index status was rebuild_needed with:
  Search index storage is missing LanceDB data fragments. Rebuild the search index to restore semantic memory search.
• Rebuild repeatedly appeared stuck at 5% / indexing_memories.
• System memory rose to ~99%; the user had to kill nowledge-mem.exe and the bundled Python backend process to recover.
• One failed run ended with backend virtual/commit memory around 42 GB:
  vms_bytes=42418806784, rss_bytes=10542600192, uss_bytes=11246985216
• The old index contained missing LanceDB fragments; renaming/removing only the search index did not by itself fix the rebuild stall.

Relevant log pattern from the failing/default rebuild:

Marked full search reindex needed
reason="Search index storage is missing LanceDB data fragments..."

Full reindex memory checkpoint before_rebuild:
rss_bytes ~= 2.0 GB

Generating embeddings for memory chunk:
chunk_size=200, chunk_start=0, total=500

The process stalled during the first memory embedding chunk. The problematic part appears to be local BGE-M3 embedding inference for large/long memory text batches, not the Kuzu graph data itself.

Workaround that allowed rebuild to complete:

1. Backed up the graph data first.
2. Stopped Nowledge Mem.
3. Renamed the broken/partial search_index directories instead of deleting user data.
4. Patched bundled FastEmbed locally to make indexing conservative:
   • cap document embedding batch size to 1 by default
   • cap input text used for embeddings to 200 characters by default
   • cap tokenizer max length to 128 tokens by default
   • force ONNX Runtime threads to 1 by default
5. Restarted Nowledge Mem and ran nmem --json models reindex.

With those limits, rebuild completed successfully:

{
  "stage": "completed",
  "message": "Indexed 1093 memories, 344471 messages, 19 sources",
  "details": {
    "memories": 1093,
    "messages": 344471,
    "sources": 19,
    "source_chunks": 1052,
    "communities": 360,
    "entities": 4953,
    "semantic_fallback_batches": 0,
    "semantic_fallback_items": 0,
    "errors": [],
    "success": true
  }
}

After completion, nmem --json status reported:

{
  "search_index": {
    "status": "ready",
    "action_required": false,
    "message": "Search index is ready."
  }
}

Observed memory during the successful workaround run:

• after memories: RSS ~2.18 GB, USS ~2.06 GB
• after messages: RSS ~2.83 GB, USS ~2.71 GB
• after completion/cleanup: RSS ~1.10 GB, USS ~1.00 GB

Possible fixes to consider:

• Make the search reindex embedding batch size configurable and more conservative on Windows/BGE-M3.
• Truncate or summarize very long texts before semantic indexing, or stream them in smaller inference windows.
• Expose reindex as a background job with a status endpoint, because the current CLI/API call can time out while the server-side rebuild continues.
• Detect and recover missing LanceDB fragments by automatically moving aside the damaged search index and rebuilding from graph data.
• Add clearer progress reporting for the long indexing_messages scan phase; it stayed at 35% while scanning hundreds of thousands of rows, even though it was still progressing.