Troubleshooting

Troubleshooting

Common issues, diagnostics, and solutions for Memory-Spark.

Quick Diagnostic Commands

# Check Spark service health
curl -s http://<SPARK_HOST>:18091/v1/embeddings -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SPARK_BEARER_TOKEN" \
  -d '{"input": "test", "model": "nvidia/llama-embed-nemotron-8b"}' | jq .

# Check reranker health
curl -s http://<SPARK_HOST>:18096/v1/rerank -X POST \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer $SPARK_BEARER_TOKEN" \
  -d '{"model": "nvidia/llama-nemotron-rerank-1b-v2", "query": "test", "documents": ["test doc"], "top_n": 1}' | jq .

# Check LanceDB status
ls -la ~/.openclaw/data/memory-spark/lancedb/

# Enable debug logging
VERBOSE=1 MEMORY_SPARK_DEBUG=1 DEBUG_PIPELINE=1

---

Spark Connection Issues

Embedding service unreachable

Symptom: memory_search returns { disabled: true, unavailable: true, error: "fetch failed" }

Causes & Solutions:

Cause	Check	Fix
Spark host down	ping <SPARK_HOST>	Restart Spark services
Wrong host configured	Check sparkHost in plugin config	Update openclaw.json or set SPARK_HOST env
Bearer token invalid	Check SPARK_BEARER_TOKEN in ~/.openclaw/.env	Update token
Unexpanded template variable	Log shows sparkBearerToken looks like an unexpanded template	Configure secrets.providers or env.shellEnv
Port conflict	ss -ltnp | grep 18091	Restart the embedding service
Tailscale/VPN not connected	tailscale status	Connect to Tailscale

Graceful degradation: When embedding is unavailable, Memory-Spark falls back to FTS-only search (keyword matching without semantic understanding).

Embedding returns wrong dimensions

Symptom: Error: vector dimension mismatch on upsert

Cause: The LanceDB table was created with a different embedding model/dimensions than the current configuration.

Fix:

# Check current table dimensions
# The table schema is locked at creation time
# Option 1: Delete and rebuild the table
rm -rf ~/.openclaw/data/memory-spark/lancedb/memory_chunks.lance

# Option 2: Use the rebuild script
npx tsx scripts/rebuild-table.ts

Prevention: The dims-lock.ts module prevents dimension changes after initial indexing. If you switch embedding models, you must rebuild the table.

---

Embedding Failures

Quota exhausted

Symptom: insufficient_quota or HTTP 429 from embedding provider

Resolution:

1. Check which provider is configured: embed.provider in config
2. For Spark (self-hosted): Usually means vLLM OOM — restart the embedding service
3. For OpenAI/Gemini: Check billing/quota, or switch to Spark
4. Memory-Spark queues failed embeddings and retries with exponential backoff

Embedding returns NaN or zero vectors

Symptom: All search results have score 0 or NaN

Diagnosis:

# Check if vectors are valid
VERBOSE=1 # Enable verbose logging to see embedding responses

Common causes:

• vLLM model not fully loaded (returns zeros during warmup)
• Wrong queryInstruction config (Nemotron-8B requires the instruction prefix)
• API key mismatch (returns error that's silently parsed as empty)

---

Reranker Issues

Reranker timeout

Symptom: [reranker] ERROR: 504 Gateway Timeout in logs

Causes:

• Too many candidates sent to reranker (default max: 40)
• Reranker model not loaded or GPU memory pressure
• Network latency to Spark host

Solutions:

// Reduce candidate pool size
{ "rerank": { "topN": 20 } }

// Or skip reranker entirely when not needed
{ "rerank": { "rerankerGate": "hard", "rerankerGateThreshold": 0.05 } }

Reranker produces flat scores

Symptom: [reranker] spread guard: logitSpread=0.3 < minSpread=0.5 — falling back to input order

Explanation: The cross-encoder couldn't distinguish between candidates. This is normal for queries where all candidates are equally relevant (or irrelevant).

Solutions:

• Lower minScoreSpread if this happens too often: { "rerank": { "spark": { "minScoreSpread": 0.3 } } }
• The spread guard is a safety net — falling back to vector order is usually correct

Reranker hurts results (reshuffles good vector ranking)

Symptom: Relevant results appear lower after reranking

Diagnosis: Enable verbose logging to see per-candidate scores:

VERBOSE=1  # Shows origScore vs sigmoid vs logit vs blended for each candidate

Solutions:

1. Use hard gate to skip reranker when vector is confident:

   { "rerank": { "rerankerGate": "hard", "rerankerGateThreshold": 0.08 } }

2. Increase vector weight in RRF blend:

   { "rerank": { "rrfVectorWeight": 1.5, "rrfRerankerWeight": 0.8 } }

3. Use score blend alpha to preserve original ranking signal:

   { "rerank": { "scoreBlendAlpha": 0.3 } }

---

LanceDB Query Errors

"Table not found" / empty results on fresh install

Symptom: memory_search returns empty results immediately after installation

Cause: The LanceDB table is created on first upsert, not on open(). Until files are indexed, there's no table to search.

Fix: Trigger indexing:

# Memory-Spark indexes on boot by default (watch.indexOnBoot: true)
# If not, manually trigger by modifying a memory file
touch ~/.openclaw/workspace-<agent>/MEMORY.md

Arrow schema errors on mergeInsert

Symptom: Error: Schema mismatch when upserting chunks

Cause: The table was created before the RAG overhaul added new columns (content_type, quality_score, pool, etc.). LanceDB's addColumns() creates columns with different Arrow nullability than seed-record columns.

Fix:

# Rebuild the table with the new schema
npx tsx scripts/rebuild-table.ts

# Or delete and let it recreate
rm -rf ~/.openclaw/data/memory-spark/lancedb/memory_chunks.lance
# Then restart OpenClaw to trigger re-indexing

Commit conflicts

Symptom: Commit conflict detected in logs (usually at startup)

Cause: Boot scanner and file watcher both try to write simultaneously.

Resolution: Automatic — Memory-Spark retries up to 3 times with exponential backoff. If conflicts persist:

// Increase retry count
{ "search": { "maxWriteRetries": 5 } }

FTS search returns all scores near 1.0

Symptom: All FTS results have score > 0.98, making minScore non-discriminative

Cause: BM25 sigmoid midpoint is too low for your corpus. With midpoint=3.0, most BM25 scores (typically 5-20) saturate the sigmoid.

Fix:

// Increase sigmoid midpoint (default: 10.0)
{ "fts": { "sigmoidMidpoint": 15.0 } }

Note: FTS scores are not used for minScore filtering in the pipeline — RRF uses rank positions only. This mainly affects display/debugging.

---

Pool Management Issues

Chunks routed to wrong pool

Symptom: Tool definitions appearing in agent_memory instead of agent_tools

Diagnosis:

# Check pool distribution
# Use LanceDB stats endpoint or enable debug logging
MEMORY_SPARK_DEBUG=1

Cause: Pool routing depends on content_type and file path. If content_type is not set correctly at ingest time, chunks fall through to agent_memory.

Fix: Ensure files are named correctly:

• TOOLS.md or tools-*.md → agent_tools
• MISTAKES.md or files in mistakes/ → agent_mistakes
• Or set content_type explicitly on the chunk

Reference pool chunks appearing in auto-recall

Symptom: Large PDF chunks injected into prompts, consuming token budget

Cause: This should never happen — reference pools (reference_library, reference_code) are excluded from auto-inject. If it does, check that the pool column is set correctly.

Fix: Verify pool routing:

// In pool.ts, reference docs must have content_type="reference"
// Set at ingest time for files in reference.paths directories

---

Memory Quality Issues

Too many low-quality chunks indexed

Symptom: Search returns irrelevant boilerplate, headers, or near-empty chunks

Solutions:

{
  "ingest": { "minQuality": 0.5 },  // Raise quality threshold (default: 0.3)
  "chunk": { "minTokens": 30 },     // Raise minimum chunk size (default: 20)
  "watch": {
    "excludePatterns": ["**/archive/**", "**/*.bak"],
    "excludePathsExact": ["memory/learnings.md"]
  }
}

Auto-capture storing noise

Symptom: Trivial messages being captured as "knowledge"

Solutions:

{
  "autoCapture": {
    "minConfidence": 0.8,       // Raise from 0.6
    "minMessageLength": 50,     // Raise from 30
    "useClassifier": true       // Ensure Spark classifier is enabled
  }
}

---

Performance Issues

Slow search (> 1s without HyDE)

Possible causes:

1. Large index without IVF_PQ: Vector search falls back to brute-force. Ensure index is created (needs ≥ numPartitions rows).
2. Low refineFactor: Set search.refineFactor to 20+ for better ANN accuracy
3. Too many pools searched: Each pool search is parallel, but total candidates increase processing time

HyDE adding 5+ seconds per query

Solutions:

• Disable HyDE for interactive queries: { "hyde": { "enabled": false } }
• Reduce timeout: { "hyde": { "timeoutMs": 4000 } } (faster local models)
• HyDE is most useful for short, ambiguous queries; disable for chatbot-style interactions

Memory usage growing unbounded

Cause: LanceDB table growing without cleanup

Solutions:

# Check table size
du -sh ~/.openclaw/data/memory-spark/lancedb/

# Delete old session data (if indexed)
# Typically sessions are handled by LCM, not memory-spark

---

Environment Variables

Variable	Purpose	Default
SPARK_HOST	DGX Spark hostname/IP	localhost
SPARK_BEARER_TOKEN	API authentication token	From ~/.openclaw/.env
MEMORY_SPARK_DATA_DIR	Override data directory	~/.openclaw/data/memory-spark/
VERBOSE	Enable verbose pipeline logging	Off
DEBUG_PIPELINE	Enable per-stage debug output	Off
MEMORY_SPARK_DEBUG	Enable hybrid merge debug logging	Off