GraphRagQualityCheck

Use this tool to check the quality between Vector RAG and Graph RAG

A .NET 8 console application that builds a dual-layer knowledge graph in Neo4j — combining a domain graph (events, sessions, speakers, organizations) with a lexical graph (documents, sections, chunks, concepts) — and lets you run side-by-side comparisons of Vector RAG vs Graph RAG on the same data.

Why This Tool?

Everyone says "Graph RAG is better than Vector RAG." But how much better? At what? And when does it matter?

This tool lets you answer those questions with your own data:

What you can do	Why it matters
🔍 Run the same question through both Vector RAG and Graph RAG	See the difference side-by-side — same query, same data, different retrieval strategies
📊 Compare what each approach retrieves	Vector RAG returns anonymous text chunks. Graph RAG returns chunks + who wrote them, where they work, which event they presented at
🔗 See multi-hop reasoning in action	Ask "Which organizations have expertise in both graphs AND AI?" — Vector RAG can't answer it. Graph RAG traverses 4 hops and nails it
🏥 Test attribution & provenance	In regulated industries (healthcare, legal, finance), "some AI said so" isn't enough. Graph RAG gives you the full citation chain
🧪 Experiment with your own documents	Drop in your own DOCX files, build the graph, and test how each approach handles YOUR domain
📐 Understand DFS vs BFS traversal	See how depth-first and breadth-first graph strategies retrieve fundamentally different context

Who is this for?

• AI/ML Engineers evaluating retrieval strategies for production RAG systems
• Solution Architects deciding whether to invest in a knowledge graph layer
• Developers building demos or proof-of-concepts for Graph RAG
• Technical Leaders who need data (not opinions) on Vector RAG vs Graph RAG trade-offs

What you'll learn

After running this tool, you'll understand:

• Why Vector RAG fails at multi-hop reasoning (and when that matters)
• How graph traversal adds attribution, provenance, and structural context
• The real latency cost of graph enrichment (~20-80ms on top of vector search)
• When Vector RAG alone is "good enough" vs when you need the graph

Architecture

┌─────────────────────────────────────────────────┐
│                  Domain Layer                    │
│  Event ── Session ── User ── Organization       │
│  Venue ── Sponsor ── Ticket ── CFP ── Room      │
└────────────────────┬────────────────────────────┘
                     │  User (bridge node)
                     │  fromDomain=true, fromLexical=true
┌────────────────────┴────────────────────────────┐
│                  Lexical Layer                    │
│  Document ── Section ── Chunk ── Concept         │
│  Chunk ──SIMILAR_TO──▶ Chunk (cosine > 0.85)     │
└─────────────────────────────────────────────────┘

Users who are both event speakers AND document authors become bridge nodes, enabling Graph RAG to traverse from text content to real-world context (who wrote it, where they work, which conference they presented at).

Prerequisites

Requirement	Version	Purpose
.NET SDK	8.0+	Build and run the application
Neo4j	5.x+	Graph database (local or AuraDB Free)
Azure OpenAI	—	GPT-4 (concept extraction) + text-embedding-3-small (embeddings)

Neo4j Setup

1. Option A — Neo4j AuraDB (cloud, recommended for quick start):

   • Create a free instance at console.neo4j.io
   • Save the connection URI, username, and password

2. Option B — Neo4j Desktop (local):

   • Download from neo4j.com/download
   • Create a new project and database
   • Install the APOC plugin

3. Create the vector index (run in Neo4j Browser):

   CREATE VECTOR INDEX chunk_embedding IF NOT EXISTS
   FOR (c:Chunk)
   ON (c.embedding)
   OPTIONS {indexConfig: {
     `vector.dimensions`: 1536,
     `vector.similarity_function`: 'cosine'
   }}

Azure OpenAI Setup

You need two deployments in your Azure OpenAI resource:

• GPT-4 (or GPT-4o) — used for concept extraction from documents
• text-embedding-3-small — used to generate 1536-dimensional embeddings for chunks

Getting Started

1. Clone and navigate

git clone <repo-url>
cd EventKernel-Backend/EventKernelGraphBuilder

2. Create configuration

cp appsettings.template.json appsettings.json

Edit appsettings.json with your credentials:

{
  "AzureOpenAI": {
    "Endpoint": "https://your-resource.openai.azure.com/",
    "ApiKey": "your-api-key",
    "DeploymentName": "gpt4",
    "EmbeddingEndpoint": "https://your-resource.cognitiveservices.azure.com/",
    "EmbeddingModelName": "text-embedding-3-small",
    "EmbeddingDeployment": "embedding"
  },
  "Neo4j": {
    "Uri": "neo4j+s://your-instance.databases.neo4j.io",
    "Username": "your-username",
    "Password": "your-password",
    "Database": "your-database"
  },
  "Processing": {
    "DocsFolder": "path/to/your/docs",
    "ChunkSize": 600,
    "ChunkOverlap": 50,
    "MaxDegreeOfParallelism": 3,
    "BatchSize": 10
  }
}

⚠️ Never commit appsettings.json — it contains secrets. The .gitignore already excludes it. Only appsettings.template.json is tracked.

3. Restore dependencies

dotnet restore

4. Run

# Process a single document (lexical graph)
dotnet run

# Process all unprocessed documents
dotnet run -- --all

# Seed domain data (events, sessions, speakers, organizations, venues, sponsors, etc.)
dotnet run -- --seed-domain

# Show speaker bridge/unification statistics
dotnet run -- --bridge

Project Structure

EventKernelGraphBuilder/
├── Program.cs                      # Entry point, DI setup, CLI routing
├── EventKernelGraphBuilder.csproj  # .NET 8 project file
├── appsettings.template.json       # Configuration template (safe to commit)
├── appsettings.json                # Your local config (⚠️ git-ignored)
├── Models/
│   ├── DocumentMetadata.cs         # Document processing metadata
│   ├── Domain/                     # Domain entity models
│   └── RagComparison/              # RAG comparison result models
├── Services/
│   ├── Neo4jGraphService.cs        # Core Neo4j operations
│   ├── DocumentProcessor.cs        # Document ingestion pipeline
│   ├── TextChunker.cs              # Text chunking with overlap
│   ├── EmbeddingService.cs         # Azure OpenAI embedding generation
│   ├── ConceptExtractor.cs         # GPT-4 concept extraction
│   ├── DomainGraphService.cs       # Domain graph CRUD
│   ├── SeedDataGenerator.cs        # Synthetic domain data generation
│   ├── GraphBridgeService.cs       # Speaker unification (domain ↔ lexical)
│   ├── GraphRagService.cs          # Graph RAG retrieval (vector + traversal)
│   ├── VectorRagService.cs         # Vector-only RAG retrieval
│   ├── RagComparisonService.cs     # Side-by-side RAG comparison
│   ├── RagDemoUiService.cs         # Interactive demo UI
│   └── DemoScenarioService.cs      # Pre-built demo scenarios
└── logs/                           # Runtime logs (git-ignored)

Key NuGet Packages

Package	Version	Purpose
Neo4j.Driver	5.18.0	Neo4j Bolt driver
Azure.AI.OpenAI	2.0.0	Azure OpenAI SDK (embeddings + GPT-4)
DocumentFormat.OpenXml	3.0.2	DOCX file parsing
Serilog	3.1.1	Structured logging
Spectre.Console	0.49.1	Rich console output

How It Works

Document Processing Pipeline

DOCX file → Parse text → Split into Sections → Chunk (600 tokens, 50 overlap)
         → Generate embeddings (text-embedding-3-small, 1536 dim)
         → Extract concepts (GPT-4)
         → Create graph: Document → Section → Chunk → Concept
         → Build SIMILAR_TO edges (cosine > 0.85)
         → Unify speakers (bridge domain ↔ lexical)

Graph RAG vs Vector RAG

• Vector RAG: Finds the top-K similar chunks by cosine similarity. Returns text only.
• Graph RAG: Same vector search, then traverses the graph to enrich results with speaker, organization, event, and concept context.

Both are implemented and can be compared side-by-side using RagComparisonService.

Sample Queries

Find Bridge Users (exist in both layers)

MATCH (u:User)
WHERE u.fromDomain = true AND u.fromLexical = true
OPTIONAL MATCH (u)-[:SPEAKS_AT]->(sess:Session)
OPTIONAL MATCH (doc:Document)-[:PRESENTED_BY]->(u)
RETURN u.name, u.company, count(DISTINCT sess) AS sessions, count(DISTINCT doc) AS documents

Full Speaker Profile (both worlds from one person)

MATCH (u:User {name: 'Emma Smith'})
OPTIONAL MATCH (u)-[:SPEAKS_AT]->(sess:Session)<-[:HAS_SESSION]-(evt:Event)
OPTIONAL MATCH (doc:Document)-[:PRESENTED_BY]->(u)
OPTIONAL MATCH (doc)-[:HAS_CONCEPT]->(c:Concept)
OPTIONAL MATCH (u)-[:WORKS_AT]->(org:Organization)
RETURN u, collect(DISTINCT sess) AS sessions, collect(DISTINCT doc) AS documents, collect(DISTINCT c.name) AS topics, org.name AS organization

License

MIT