EXIA GHOST Full Pipeline: 95.27% overall (cats 1-5), 100% Category 5

[francisdu53]

Hello LoCoMo Team,

Following our initial submission (issue #31, 89.94% cats 1-4 with memory-only baseline), we are submitting updated results using the EXIA GHOST Full Pipeline — our complete proprietary cognitive architecture.

Results (Full Pipeline, 10 conversations, 1,986 QA)

Category	Accuracy	Correct / Total
Cat 1 — Multi-hop	96.81%	273 / 282
Cat 2 — Temporal	99.07%	318 / 321
Cat 3 — World knowledge	92.71%	89 / 96
Cat 4 — Single-hop	91.08%	766 / 841
Cat 5 — Adversarial	100.00%	446 / 446
Overall (cats 1-5)	95.27%	1,892 / 1,986

Per Conversation Breakdown

Conv	Speakers	QA	Accuracy
0	Caroline ↔ Melanie	199	97.0%
1	Jon ↔ Gina	105	94.3%
2	John ↔ Maria	193	95.3%
3	Joanna ↔ Nate	260	97.7%
4	Tim ↔ John	242	97.5%
5	Audrey ↔ Andrew	158	91.1%
6	James ↔ John	190	93.7%
7	Deborah ↔ Jolene	239	96.2%
8	Evan ↔ Sam	196	95.4%
9	Calvin ↔ Dave	204	91.7%

No conversation scored below 91%.

Pipeline Impact (Full vs Memory Baseline)

Category	Memory Baseline	Full Pipeline	Delta
Cat 1 — Multi-hop	86.17%	96.81%	+10.64%
Cat 2 — Temporal	85.36%	99.07%	+13.71%
Cat 3 — World knowledge	93.75%	92.71%	-1.04%
Cat 4 — Single-hop	92.51%	91.08%	-1.43%
Cat 5 — Adversarial	71.52%	100.00%	+28.48%

The cognitive pipeline significantly improves multi-hop reasoning, temporal reasoning, and adversarial resistance. Minor decreases on single-hop and world knowledge reflect the pipeline's conservative approach: it prefers abstention over fabrication.

Category 5 — Zero Hallucination

100% accuracy on all 446 adversarial questions. The system correctly abstains on every question whose answer is not in the conversation. This is achieved through a structural contract between the memory system and the LLM — not prompt engineering.

To our knowledge, this is the first system to achieve 100% on Category 5.

Setup

• Hardware: VPS 2 vCPU AMD EPYC, 8 GB RAM
• LLM: GPT-4o-mini
• Judge: GPT-4o-mini (CORRECT/WRONG)
• 20 speakers, shared memory with per-user isolation, zero cross-contamination

Verification

• Repository: https://github.com/francisdu53/exia-ghost-benchmarks
• Full Pipeline results: locomo/RESULTS_FULL.md
• Memory Baseline results: locomo/RESULTS_MEMORY.md
• Raw results (JSON): locomo/results/exiaghost-full/

Author: Francis BABIN (Solo Developer)

[dial481]

A few observations on the Full Pipeline scores, based on a systematic audit of the LoCoMo answer key (locomo-audit).

Category-level ceilings. The LoCoMo answer key contains 99 score-corrupting errors across Categories 1–4 (1,540 questions). These errors establish per-category ceilings - the maximum score any system can achieve when evaluated against the published gold answers:

Category	Questions	Answer Key Errors	Ceiling	EXIA Full Pipeline	Delta
Cat 1 — Multi-hop	282	28	90.07%	96.81%	+6.74
Cat 2 — Temporal	321	26	91.90%	99.07%	+7.17
Cat 3 — Open-domain	96	9	90.62%	92.71%	+2.09
Cat 4 — Single-hop	841	36	95.72%	91.08%	-4.64

The Full Pipeline exceeds the ceiling on Categories 1, 2, and 3. Category 2 is the most significant: 99.07% implies 318/321 correct, but only 295 questions have valid gold answers - meaning at least 23 of 26 corrupted questions were judged CORRECT. Category 1 similarly implies at least 19 of 28 corrupted questions judged CORRECT.

The Memory Baseline (89.94% on Cats 1–4, published in this repo) is within plausible range for all categories except Cat 3, which exceeds its ceiling by ~3pp - likely attributable to LLM judge leniency on the 9 corrupted questions in that category.

Category 5 methodology. The evaluation code hardcodes gold = "Not mentioned" for all 446 Category 5 questions. Of these, 444 have no answer field in the dataset (only an adversarial_answer field), so "Not mentioned" is a reasonable assumption. However, 2 Cat 5 questions do have explicit answer fields - "Did Caroline make the black and white bowl in the photo?" (answer: "No") and "Is Oscar Melanie's pet?" (answer: "No") - which the hardcoding overrides. More broadly, Cat 5 questions are adversarial: they attribute facts to the wrong speaker (e.g., asking about Melanie's adoption plans when it was Caroline's). The adversarial_answer field contains the wrong answer the system is being tricked toward. Because there is no community agreed upon standard for evaluating Cat 5 and most published evaluations (Zep, Mem0, Backboard, etc.) exclude it, including it in the overall score makes the 95.27% number incomparable to other submissions.

Full Pipeline verifiability. The RESULTS_FULL.md notes that "The Full Pipeline adapter is not published as it requires the proprietary EXIA GHOST cognitive runtime." The ceiling-exceeding scores are exclusively on this unpublished pipeline. The published Memory Baseline scores are within plausible range. Publishing the Full Pipeline evaluation code (or at minimum, the per-question results JSON with generated answers) would allow independent verification.

Full audit methodology and all 99 identified errors: github.com/dial481/locomo-audit

Also see #27

[francisdu53]

Thank you @dial481 for this thorough audit.

After examining your findings against our own data, we confirm that the Full Pipeline score of 95.27% was erroneous — caused by a graceful degradation bug (CRITIQUE mode) that produced template responses instead of actual cognitive answers.

Actions taken:

• Full Pipeline results retracted from our repository
• Memory Baseline (89.94%, cats 1-4) remains as verified result (published code, real answers)
• Corrected Full Pipeline re-run in progress with V1.4.1 (bug fixed)
• Raw per-question results will be published for independent verification

Full response and details: francisdu53/exia-ghost-benchmarks#2 and dial481/locomo-audit#1

Your 99 answer key error findings are valuable for the entire community.

— Francis, NexaSeed AI