Engram v0.1.3: 93.9% R@5 on LoCoMo — zero-LLM retrieval, $0/query

[Nitin-Gupta1109]

Hi! We benchmarked Engram, an open-source conversational memory library, on LoCoMo.

Engram is a retrieval system (no LLM, no answer generation), so results are reported as R@5 / R@10 / NDCG@5 rather than end-to-end F1. We evaluate whether the correct evidence turns are retrieved in the top-k — the downstream answerer is up to the application.

Results (v0.1.3, 1982 questions, 10 conversations)

Metric	Score
R@5	93.9% (1862/1982)
R@10	95.0%
NDCG@5	0.894
NDCG@10	0.896

Per category

Category	R@5	R@10	n
single-hop (factual)	90.4%	93.3%	282
temporal (dates)	93.1%	94.7%	321
multi-hop (inference)	75.0%	78.3%	92
contextual (details)	97.1%	97.5%	841
adversarial (speaker)	94.6%	94.8%	446

Setup

• Embedding: BAAI/bge-large-en-v1.5 (1024d), local
• Sparse: BM25 (rank-bm25)
• Fusion: Reciprocal Rank Fusion (RRF)
• Reranker: cross-encoder/ms-marco-MiniLM-L-6-v2 (free, local)
• No LLM used at query time or ingestion
• Hardware: CPU-only (no GPU required)
• Cost: $0/query

Key techniques

1. Session chunking (~6 turns, 1-turn overlap) to avoid embedding dilution on long sessions
2. Timestamp prefix ([2024-01-15] ) prepended to docs for temporal matching
3. Speaker-name injection at ingestion — prepend each speaker's name to their turns (Nate: I got a PS5) so entity-attribute queries (What console does Nate own?) match first-person facts. This alone lifted multi-hop from 63.0% → 75.0%.
4. Synthetic docs for preferences and topics to bridge vocabulary gaps

How to reproduce

pip install engram-search
curl -fsSL -o locomo10.json \
  https://raw.githubusercontent.com/snap-research/locomo/main/data/locomo10.json
python benchmarks/locomo_bench.py locomo10.json --mode rerank

Full benchmark harness: https://github.com/Nitin-Gupta1109/engram/blob/main/benchmarks/locomo_bench.py

Notes

• These are retrieval numbers, not end-to-end F1. They aren't directly comparable to the F1-based results reported in EXIA GHOST Results: 89.94% (Cat 1-4), first Cat 5 evaluation (71.52%) #31, NEXO Brain benchmark results on LoCoMo — first MCP memory server evaluation #33, EXIA GHOST Full Pipeline: 95.27% overall (cats 1-5), 100% Category 5 #34 — those measure the full QA pipeline including LLM answer generation. We're measuring whether the right evidence makes it into the context window.
• Engram is pip-installable and cloud-optional (FAISS local / Qdrant cloud backends).

Thanks for the benchmark — it's been a great stress test for retrieval quality on long conversations.

[xiongqiangcs]

Is this evaluation valid without LLM-Generated answers or LLM-Based judgment？