Skip to content

3-tier research validation + arXiv paper draft#43

Open
prakashUXtech wants to merge 6 commits intodevfrom
feat/research-validation
Open

3-tier research validation + arXiv paper draft#43
prakashUXtech wants to merge 6 commits intodevfrom
feat/research-validation

Conversation

@prakashUXtech
Copy link
Contributor

@prakashUXtech prakashUXtech commented Mar 6, 2026

Summary

Adds a complete research validation framework and an arXiv-ready paper draft proving Soul Protocol improves agent quality across every measured dimension.

Research framework (research/, 5,100+ lines)

  • Tier 1: 1,000-agent heuristic simulation across 5 ablation conditions (No Memory → Full Soul). Cohen's d = 8.98 for recall.
  • Tier 2: 100-agent LLM validation with Claude Haiku comparing heuristic vs neural cognitive processing. Haiku extracts 2.5x more memories.
  • Tier 3: 4 quality tests (response quality, personality consistency, hard recall, emotional continuity) evaluated by 5 judge models from 4 providers.
  • Multi-model engine (litellm_engine.py) connecting to 98 models via LiteLLM proxy.
  • 8 smoke tests, all passing.

Multi-judge results (20/20 judgments favor soul)

Test Soul (mean) Baseline (mean) Inter-judge σ
Response Quality 8.8 6.5 0.8
Personality Consistency 9.0 5.0 0.2
Hard Recall 8.5 4.8 0.7
Emotional Continuity 9.7 1.9 0.4

Judges: Claude Haiku, Gemini 3 Flash, Gemini 2.5 Flash Lite, DeepSeek V3, Llama 3.3 70B.

Paper (paper/, NeurIPS-style LaTeX)

  • 11-page paper with real experimental data from all 3 tiers
  • Builds with make (requires tectonic)
  • 16 citations covering memory architectures, personality in LLMs, benchmarks, and LLM-as-judge methodology

Total validation cost: under $5

Test plan

  • 8 smoke tests pass (uv run pytest research/test_smoke.py)
  • Single-judge quality validation completes (4/4 tests)
  • Multi-judge validation completes (20/20 judgments)
  • Paper compiles with tectonic
  • Review paper draft for accuracy and completeness

prakashUXtech and others added 3 commits March 6, 2026 20:31
@prakashUXtech @claude
docs: rewrite whitepaper and README for v0.5.0
1c74405 
Whitepaper:
- New title: "Identity, Memory, Cognition, and Emotion"
- Goleman EQ > IQ framing threaded throughout
- Bitcoin-style tone: zero salesmanship, problem-first, show-don't-tell
- Updated for spec/ + runtime/ architecture, 766 tests, 9200+ lines
- New sections: vector search, eternal storage, bond/skills/reincarnation
- Accurate "not working yet" section and roadmap (learning events,
  domain isolation, trust chain)
- Added academic references (Goleman, Damasio, Anderson, Franklin, Klein)
- Humanized: 0 em dashes (was 40+), shorter sentences, no adjective stacking

README:
- Reflects spec/ + runtime/ two-layer architecture
- Updated feature table (bond, skills, vector search, eternal, reincarnation)
- 766 tests, 11 CLI commands, JSON schemas
- Comparison section rewritten without "fundamentally different" language
- Links to whitepaper, gap analysis, schemas

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@prakashUXtech @claude
feat: wire Bond and SkillRegistry into Soul.observe() pipeline
92e9d8a 
Bond now strengthens automatically on every interaction. Positive
sentiment gives a bigger boost (1.0 + valence), neutral gives 0.5.
Interaction count increments on each strengthen() call.

Skills are auto-created from extracted entities during observe().
Repeated mentions of the same topic accumulate XP on existing skills.
New entities spawn new Skill objects in the registry.

Added soul.bond and soul.skills properties for API access.
4 new integration tests (770 total, all passing).

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@prakashUXtech @claude
feat: add 3-tier research validation framework and arXiv paper
6807923 
Research framework (research/):
- 1,000-agent heuristic simulation across 5 ablation conditions
- 100-agent LLM validation with Claude Haiku cognitive engine
- 4 quality tests: response quality, personality consistency,
  hard recall (30 fillers), emotional continuity (8-turn arc)
- Multi-judge evaluation: Haiku, Gemini 3 Flash, Gemini 2.5 Flash Lite,
  DeepSeek V3, Llama 3.3 70B — all 20 judgments favor soul
- LiteLLM engine for multi-model access via proxy
- 8 smoke tests, all passing

Paper (paper/):
- NeurIPS-style LaTeX paper with real experimental data
- Multi-judge scorecard: soul 9.0 vs baseline 4.5 overall
- Inter-rater std dev 0.2-0.8 across 5 judges from 4 providers
- Builds with tectonic (make)

Key results:
- Emotional continuity: 9.7 vs 1.9 (3 judges gave 10/10)
- Personality consistency: 9.0 vs 5.0 (σ=0.2, tightest agreement)
- Hard recall: 8.5 vs 4.8 (GraphQL fact at rank 1 after 30 fillers)
- Response quality: 8.8 vs 6.5
- Total validation cost: under $5

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@github-actions
Copy link

github-actions bot commented Mar 6, 2026 •
edited
Loading

Issues (must fix)

  • PR title does not follow Conventional Commits format (e.g. feat: add recall API, fix(memory): handle empty tiers).
  • No linked issue found. PRs should reference an issue (Fixes #123).
  • No evidence of local testing found. Please include terminal output or screenshots.

Heads up

  • Large PR detected (11160 lines across 45 files). Consider splitting into smaller PRs.

Please update your PR to address these points.

@github-actions
Copy link

github-actions bot commented Mar 6, 2026

Security scan: review needed

Potentially dangerous code patterns detected in changed files. A maintainer should verify these are intentional and safe.### research/conditions.py

87:    """Condition 2: Pure vector similarity retrieval (no significance, no emotion)."""

src/soul_protocol/runtime/soul.py

122:            search_strategy: Optional SearchStrategy for pluggable retrieval (v0.2.2).
315:            search_strategy: Optional SearchStrategy for pluggable retrieval (v0.2.2).

prakashUXtech and others added 3 commits March 7, 2026 07:06
@prakashUXtech @claude
fix(paper): address 10 self-review issues for arXiv submission
388125b 
- Title: "Soul Protocol: An Open Protocol for Portable AI Companion Identity"
- Abstract: rewritten as narrative, removed inflated Cohen's d statistic
- Tier 1 table: simplified to 2-column (memory vs no-memory), honest about
  heuristic ceiling preventing proper ablation
- Baseline: explicitly acknowledged as weak (stateless), noted need for
  RAG-only comparison
- Personality test: noted baseline 5.0 is a ceiling, not a measured score
- Psychology claims: scoped OCEAN as engineering tool, not psychometric claim
- Discussion: added honest framing about what results show and don't show
- Limitations: expanded to 6 items including weak baseline, no ablation,
  no human eval
- References: resolved all 8 placeholder authors with real names from arXiv
- GitHub URL: fixed to qbtrix/soul-protocol
- Architecture diagram: added TikZ figure showing protocol components
- Fixed overfull hbox warnings, clean tectonic build

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@prakashUXtech @claude
feat(research): add enhanced validation infrastructure
0c4d7cd 
New research infrastructure to strengthen the paper:

- scenario_generator.py: 10 unique scenario variations per test type
  (40 total) with randomized users, facts, emotional arcs. Reproducible
  via SEED constant.

- conditions.py: 4 experimental conditions for proper ablation:
  Full Soul, RAG-Only, Prompt-Personality, Bare Baseline.
  MultiConditionResponder generates responses under each condition
  using the same soul data but different context presentation.

- enhanced_runner.py: Orchestrates N variations × 4 conditions × M judges.
  Produces mean ± 95% CI, win rates, and per-condition breakdowns.

- mem0_benchmark.py: Head-to-head comparison against Mem0 (existing
  open-source memory system). Same test scenarios, same judge, three
  conditions: Soul vs Mem0 vs Baseline.

- eval_ui/: FastAPI web app for human evaluation study. Students chat
  with blinded A/B agents (soul vs baseline, randomized order), fill
  5-question Likert survey. Results saved as JSON for analysis.

These address reviewer concerns: weak baseline, no ablation, N=1
scenarios, no human eval protocol, no comparison against existing systems.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
@prakashUXtech @claude
feat(research): Mem0 comparison results — Soul beats Mem0 across all …
b9369ee 
…tests

Three-way benchmark: Soul Protocol vs Mem0 vs Stateless Baseline.
Configured Mem0 to use DeepSeek (LLM) + text-embedding-004 (embeddings)
via LiteLLM proxy to avoid OpenAI rate limits.

Results:
  Response Quality:     Soul 8.5 | Mem0 8.3 | Base 7.2 (+0.2 over Mem0)
  Hard Recall:          Soul 7.8 | Mem0 5.1 | Base 4.2 (+2.7 over Mem0)
  Emotional Continuity: Soul 9.2 | Mem0 7.0 | Base 1.8 (+2.2 over Mem0)

Key finding: Mem0 is competitive on pure memory tasks but lacks
personality consistency and emotional arc tracking. Soul Protocol's
somatic markers and OCEAN personality provide the largest gains.

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

needs-work

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@prakashUXtech