Evaluation results — EXIA GHOST V2.7.2 on LoCoMo (84.16%, full pipeline, strict scoring)

[francisdu53]

Hi LoCoMo team,

First, thank you for releasing LoCoMo — it has become a key reference benchmark for long-term conversational memory systems, and the per-category structure (single-hop / temporal / multi-hop / multi-source / adversarial) is genuinely useful for diagnosing where memory pipelines fail.

We wanted to share the results of an independent evaluation of EXIA GHOST V2.7.2 on the full 10-conversation LoCoMo set, in case it is of interest to the community.

Headline result

• Score (Cat 1-4, strict) : 1 296 / 1 540 = 84.16 %
• Standard : EverMemOS / Mem0 community protocol (gpt-4o-mini judge, 3 independent runs per question, majority vote)
• Configuration : Full cognitive pipeline (not memory-only) — every answer is produced by the cognitive system or the system explicitly abstains. No silent fallback to template responses.

Per-category breakdown

Cat	Type	Score
1	Single-hop	217 / 282 = 77.0 %
2	Temporal	272 / 321 = 84.7 %
3	Multi-hop	51 / 96 = 53.1 %
4	Multi-source	756 / 841 = 89.9 %
5	Adversarial	tracked separately as a hallucination-resistance metric, excluded from headline per community convention

Honesty note

This is the corrected re-run of a Full Pipeline configuration that we publicly retracted in April 2026. A previous score of 95.27 % was inflated by a silent graceful-degradation path that returned template responses instead of cognitive answers (the LLM judge was lenient on those templates). We fixed the bug, rebuilt the pipeline, and the corrected score is the 84.16 % above. Lower number, real number.

Reproducibility

conv-42 was independently re-benchmarked one month apart on the same configuration :

Version	Date	conv-42 strict score
V2.7.0	April 2026	158 / 199
V2.7.2	May 2026	158 / 199

Aggregate identical. One question swapped CORRECT ↔ WRONG between runs (consistent with documented ~ 1 / 200 gpt-4o-mini judge non-determinism at temperature=0.0). The pipeline itself is deterministic.

Raw data & methodology

Per-question JSON, per-conversation breakdown, scoring methodology, and the cleaned benchmark script are public so that any reviewer can re-score independently :

• Results document : https://github.com/francisdu53/exia-ghost-benchmarks/blob/main/locomo/RESULTS_FULL_PIPELINE_V272.md
• Raw per-question data : https://github.com/francisdu53/exia-ghost-benchmarks/tree/main/locomo/results/exiaghost-full-v272
• Bench script : https://github.com/francisdu53/exia-ghost-benchmarks/blob/main/locomo/eval_locomo_full_v272.py
• Public stack (versions, models, hardware) : https://github.com/francisdu53/exia-ghost-benchmarks/blob/main/methodology/STACK_V272.md

About EXIA GHOST

EXIA GHOST is a cognitive middleware that sits between a user and an LLM : the LLM verbalizes, a deterministic cognitive pipeline decides, reasons causally, and enforces ethical and consistency constraints. Architectural details are intentionally not disclosed pending patent filing — the benchmark is fully reproducible without them.

Why we posted this

Two reasons :

1. Transparency. We retracted a result publicly ; we want the corrected re-run to be discoverable from the same place researchers go to find LoCoMo, not only on our own repo.
2. Possible interest to the community. The 53.1 % on Cat 3 (multi-hop) is, by some margin, our weakest category and is the most interesting research direction for the next version. If anyone has worked on Cat 3-specific evaluation issues with LoCoMo, we would be happy to discuss.

Happy to answer questions about methodology, reproducibility, or the protocol. No action required from the maintainers — this is informational, not a PR or a request.

Thank you again for the benchmark.

— Francis (https://github.com/francisdu53)