NIMA Benchmark Suite

Five benchmarks covering different aspects of NIMA's cognitive architecture.

Benchmarks

1. synthetic/ — R@K Retrieval Quality (MemPalace-style)

What it tests: Keyword-proxy retrieval against the live production DB — "does a top-K memory mention the answer words?" (rank-aware). For reproducible, gold-ID R@K see §1b.
Metrics: Recall@1, Recall@5, Recall@10, Recall@20, MRR
Independent of: LLM synthesis, emotional context
Why it matters: Foundation layer — if retrieval fails, nothing else works

Run:

cd ~/.openclaw/workspace/PROJECTS/nima-bench
python3 -c "from synthetic.benchmark import TESTS, run_synthetic_benchmark; run_synthetic_benchmark()"

1b. synthetic/gold.py — Gold-ID R@K (reproducible, adversarial)

What it tests: The same question as §1, done correctly — builds a fixed, self-contained corpus in a throwaway NIMA DB, embeds it with the real embedder, runs the real RRFRetriever, and scores against known gold memory IDs.
Corpus is adversarial by design: within each topic cluster, near-duplicate distractors share vocabulary with the gold memory but answer a different question, so keyword overlap alone loses. Question tiers: easy/medium/hard plus adversarial (lexical traps, paraphrase with no shared words, contrast/negation, multi-hop). Multi-hop questions set mode="all" — every gold id must land in the top K, not just the first.
Metrics: Recall@1/5/10/20, MRR, per-difficulty R@10 — deterministic, isolated from whatever is in production.
Baseline & A/B (48-memory corpus, 26 questions): nomic-embed-text (768-d) R@1 ≈ 77%, MRR ≈ 0.88; bge-m3 (1024-d) R@1 ≈ 73%, MRR ≈ 0.85. The adversarial tier (~75% R@10 vs 100% easy/medium/hard) is the headroom — enough to separate two embedders and catch regressions.
Why it matters: §1's keyword-substring proxy inflates recall (any memory mentioning a keyword counts) and shifts with your live corpus. This measures whether the right memory (or all required memories) land in the top K — with enough headroom to A/B embedding models and catch retrieval regressions.
Requires: Ollama running with the embedder pulled (nomic-embed-text by default).

Run:

python3 synthetic/gold.py --top-k 20
python3 synthetic/gold.py --min-mrr 0.75                  # CI gate: non-zero exit if mean MRR drops
python3 synthetic/gold.py --model bge-m3                  # A/B any embedder — width auto-probed

--model swaps the embedder for corpus and query and rebuilds the throwaway corpus at the model's width (auto-probed from Ollama, or pass --dim; nima-core reads NIMA_EMBED_DIM). Switching the width of an existing production DB needs a re-embed — memories_vec is a vec0 table created once at NIMA_EMBED_DIM.

2. speed_benchmark/ — SQLite vec Extension Performance

What it tests: How fast does NIMA retrieve at scale?
Metrics: Latency (ms) at 1K, 10K, 100K memories
Tests: Vector-only, Vector+Graph, Full RRF pipeline
Why it matters: Personal AI needs to be fast. 100ms or less per query.

Run:

python3 speed_benchmark/benchmark.py --scales 1000 10000 100000

3. emotional_synthesis/ — Emotional Context Understanding

What it tests: Does NIMA understand the EMOTIONAL context of memories?
Metrics: Human-judged 1-5 on: emotional context, factual accuracy, synthesis depth
What makes NIMA different: No other memory system measures this
Why it matters: Raw retrieval wins on synthetic benchmarks. Emotional synthesis is NIMA's edge.

Run (interactive):

python3 emotional_synthesis/benchmark.py

4. personal_continuity/ — Coherent Model of David

What it tests: Does NIMA maintain a consistent, evolving model of David?
Metrics: Human-judged on: consistency, accuracy, drift detection
What it probes: Relationships, belief evolution, stale understanding
Why it matters: A good memory system remembers who you ARE, not just what you said

Run (interactive):

python3 personal_continuity/benchmark.py

Unified Runner

# List all benchmarks
python3 runner.py --list

# Run everything
python3 runner.py --all

# Run specific benchmarks
python3 runner.py --speed --synthetic
python3 runner.py --emotional --continuity

# Save results
python3 runner.py --all --output ./results/2026-04-10

Key Insight

MemPalace proved: raw verbatim + embeddings beats LLM-extracted structured memories on synthetic benchmarks.

NIMA's cognitive layers (affect, synthesis) aren't tested by R@K benchmarks. They're tested by the emotional_synthesis and personal_continuity benchmarks — which require human judgment.

This is NIMA's frontier. Push there.