🔍 LLM Citation: Eliminating Hallucinations and Verifying Sources

A hybrid AI system that combines local LLM inference with symbolic source credibility scoring to ensure factual accuracy.

🎯 What It Does

Traditional language models suffer from hallucinations—they generate plausible-sounding but false information. This project solves that problem by implementing a trustworthy citation pipeline:

1. Pre-verified Sources: Articles are scraped and scored before the model ever sees them
2. Symbolic Credibility Scoring: Uses a Truth Maintenance System (TMS) with logical rules to evaluate source reliability
3. Source Injection: Only high-scoring articles are injected into the model's context
4. Factual Constraints: The LLM is explicitly instructed to only use provided sources

Key Features

• 🔐 Multi-factor Credibility Assessment: Domain authority, author verification, reference quality, recency, DOI/peer review status, and cross-source consensus
• 🧠 Symbolic AI Integration: Truth Maintenance System for logical reasoning about source credibility
• 💬 Local LLM Inference: Uses Ollama's qwen2.5:3b model for privacy and cost efficiency
• 📊 Smart Source Retrieval: Keyword-based search across verified articles in SQLite database
• 🌐 Multi-format Support: Handles news articles, academic papers (DOI), and general web content

📁 Project Structure

CS4811/
├── chatbot.py           # Interactive chat interface with source injection
├── server.py            # MCP server that handles intractin between LLM and source database
├── scraper.py           # Web scraper and article processor
├── evaluator.py         # Symbolic credibility scoring engine
├── cltre.py             # Logical Truth Rule Engine (LTRE)
├── cltms.py             # Truth Maintenance System (TMS)
├── sources.db           # SQLite database of scored articles (auto-generated)
├── .gitignore           # Python cache and database exclusions
└── README.md            # This file

Core Components

🤖 chatbot.py - Chat Interface

• Interactive command-line chat with the local LLM and MCP server
• Automatic source retrieval based on user queries
• Context window management (sliding window for conversation history)
• Source citation injection into model prompts using MCP tools
• Streaming responses with real-time token generation

🤖 server.py - MCP server

• Tool host for LLM to use when getting sources
• used when LLM deems so
• calls the scraper and database through this file
• searchs the web via duckduckgo if sources need to be found

🕷️ scraper.py - Article Scraper

• Multi-format article extraction (news, DOI, general web)
• Metadata extraction: authors, publish dates, abstracts, references
• Support for major news outlets via Newspaper4k
• DOI resolution through CrossRef API
• Chrome/Selenium fallback for dynamic content
• Automatic database population

⚖️ evaluator.py - Credibility Scoring Engine

• Multi-dimensional scoring system (0-100 scale)
• Domain Scoring: .gov/.edu = 30pts, .org = 20pts, .com = 15pts
• Author Verification: 10pts for identified authors
• Reference Quality: Up to 20pts based on reference count
• Recency Factor: Up to 20pts, decays over 16 years
• DOI/Peer Review: 20pts for academic sources
• Cross-Source Consensus: NewsAPI validation for news articles
• Penalties: -50pts for social media sources

🧠 cltre.py - Logical Truth Rule Engine

• Pattern matching and unification for logical rules
• Forward chaining inference engine
• Integration with Truth Maintenance System
• Fact assertion with dependency tracking
• Rule-based evaluation triggers

🔗 cltms.py - Truth Maintenance System

• Justification-based belief tracking
• Assumption management and retraction
• Propagation of logical consequences
• Dependency tracking for fact support
• Explanation generation (why facts are believed)

🚀 Getting Started

Prerequisites

• Python 3.12 or higher
• WSL 2 (Windows) or native Linux/macOS
• Chrome and ChromeDriver
• Ollama (for local LLM)

Installation (WSL)

Step 1: Install WSL

wsl --install

Step 2: Install System Dependencies

sudo apt update && sudo apt upgrade -y
sudo apt install -y python3 python3-pip python3-venv wget curl

Step 3: Install Chrome + ChromeDriver

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install -y ./google-chrome-stable_current_amd64.deb
sudo apt install -y chromium-chromedriver

Step 4: Install Ollama

sudo apt-get install zstd
curl -fsSL https://ollama.com/install.sh | sh

Step 5: Pull the Model

ollama serve &
ollama pull qwen2.5:3b

Installation (Anaconda)

Step 1: Install Anaconda

Download from https://www.anaconda.com/download

Step 2: Create Conda Environment

conda create -n cs4811 python=3.12
conda activate cs4811

Step 3: Install Chrome + ChromeDriver

wget https://dl.google.com/linux/direct/google-chrome-stable_current_amd64.deb
sudo apt install -y ./google-chrome-stable_current_amd64.deb
sudo apt install -y chromium-chromedriver

Step 4: Install Python Dependencies

pip install requests beautifulsoup4 selenium python-dateutil llama-index-core llama-index-llms-ollama llama-index-tools-mcp mcp ddgs 'newspaper4k[nlp]' lxml lxml_html_clean

Step 5: Install Ollama

Download from https://ollama.com/download and run:

ollama serve
ollama pull qwen2.5:3b

💻 Usage

Step 0: Run Ollama

in order for any of this to work, Ollama must be running in the background, you can verify this by running

curl http://localhost:11434

you will see the message Ollama is Running if it is running, otherwise use

ollama serve

to start the Ollama server

Step 1: Start the Chatbot

python3 chatbot.py

You'll see:

Initializing LLM...
Connecting to local MCP server...
Processing request of type ListToolsRequest
Discovered 3 tools from MCP server.
 - search_internet
 - fetch_sources
 - add_url
------------------------------------------------------------
Research Chatbot Ready. Type 'quit','exit', or 'bye' to exit.
------------------------------------------------------------

Step 2: Interact

Ask questions about topics covered in your sources or give the bot a source to add to the database

You: What are the latest developments in AI research?

thinking/Reasoning
...

Agent: assistant: [Retrieves relevant sources and provides cited answer]
------------------------------------------------------------

Commands:

• Type your question and press Enter
• Type quit, exit, or bye to exit

🧪 How It Works

1. Source Credibility Scoring

The evaluator uses a logical reasoning approach with these scoring components:

Factor	Maximum Points	Criteria
Domain Authority	30	.gov/.edu (30), .org (20), .com (15), other (5)
Author Verification	10	Identified authors present
Reference Quality	20	10+ refs (20), 5-9 (10), 1-4 (5)
Recency	20	Decays over 16 years (5840 days)
DOI/Peer Review	20	Academic sources with DOI
Cross-Source Consensus	20	News articles validated across multiple outlets
Penalties	-50	Social media domains

2. Truth Maintenance System

The symbolic AI engine maintains logical consistency:

• Justifications: Each fact has supporting reasons
• Assumptions: Base premises that can be retracted
• Propagation: Truth values cascade through the system
• Explanation: Can explain why any fact is believed Example rules:

IF social-media-domain THEN unreliable-source
IF historical-content THEN ignore-recency-requirement

3. Chatbot Source Injection

When you ask a question:

1. Query Analysis: Extract keywords from your question
2. Source Retrieval: Call fetch_sources in the MCP server to Search sources.db for matching articles
3. Context Injection: Add retrieved sources to the conversation
4. LLM Generation: Model prioritizes provided sources, but will search the internet for more if needed
5. Citation: Model includes source links in responses

🔧 Configuration

Model Parameters (chatbot.py)

OPTIONS = {
    "temperature": 0.5,      # Lower = more focused answers
    "top_p": 0.9,
    "top_k": 40,
    "num_ctx": 4096,        # Context window size in tokens
    "num_predict": 2512,    # Max tokens in reply
    "repeat_penalty": 1.1,
}

Adjusting these affects:

• temperature: Creativity vs. focus (0.0-2.0)
• num_ctx: Memory capacity (tokens)
• num_predict: Maximum response length

Source Credibility Threshold (server.py)

THRESHOLD = 50 # what the minimum score should be for a source being used in the responce

Adjusting this affects:

• score: Credibility of sources

Scoring Thresholds (evaluator.py)

Modify these in _get_newspaper_consensus_score():

threshold = 0.15  # Jaccard similarity threshold
if match_count >= 4: return 20  # 4+ matching articles
if match_count >= 3: return 15  # 3 matching articles

🌐 Adding New Sources

Method 1: Direct URLs in chatbot

asking the chatbot to add a source to the database will call the needed functions in the MCP server

Method 2: News API Integration

The evaluator already uses NewsAPI for consensus checking. To add more sources:

1. Get an API key from newsapi.org
2. Update api_key in evaluator.py
3. Modify search queries in _get_newspaper_consensus_score()

📊 Database Schema

The sources.db SQLite database contains:

CREATE TABLE sources (
    url TEXT PRIMARY KEY,
    score INTEGER,           -- Credibility score (0-100)
    authors TEXT,            -- Comma-separated author names
    domain TEXT,             -- Extracted domain
    publish_date DATE,       -- YYYY-MM-DD format
    abstract TEXT,           -- Article summary/abstract
    has_doi BOOLEAN          -- DOI presence flag
)

Only articles with score > THRESHOLD are used by the chatbot.

🐛 Troubleshooting

Ollama Connection Issues

# Check if Ollama is running
curl http://localhost:11434/api/tags
# Restart Ollama
ollama serve

Chrome/ChromeDriver Errors

# Check ChromeDriver version
chromedriver --version
# Update ChromeDriver if needed
sudo apt install --only-upgrade chromium-chromedriver

Database Issues

# Reset database (deletes all sources)
rm sources.db
python3 scraper.py  # Will recreate and populate

Import Errors

Ensure all Python dependencies are installed:

pip install requests beautifulsoup4 selenium python-dateutil llama-index-core llama-index-llms-ollama llama-index-tools-mcp mcp ddgs 'newspaper4k[nlp]' lxml lxml_html_clean

📈 Performance Considerations

• Scraping: ~2-5 seconds per article (depends on site complexity)
• Evaluation: ~1-3 seconds per article (NewsAPI calls)
• Chatbot: Near real-time responses with streaming
• Memory: ~100MB for typical source database

🎓 Use Cases

• Academic Research: Verify claims against peer-reviewed sources
• Journalism: Fact-check articles with credible sources
• Education: Teach critical thinking and source evaluation
• Policy Analysis: Support decisions with verified information
• General Knowledge: Get answers that cite trustworthy sources

🤝 Contributing

To extend this system:

1. Add New Scoring Factors: Modify evaluator.py scoring logic
2. New Rules: Add rules in evaluator._setup_rules()
3. New Tools: Add MCP tools that the LLM could use in server.py
4. Source Types: Extend scraper.py for new content formats
5. Model Choice: Change MODEL in chatbot.py (Ollama-compatible)

📝 License

This project is part of CS4811 coursework. Use and modify freely for educational purposes.

🔬 Technical Highlights

• Hybrid AI: Combines neural networks (LLM) with symbolic AI (TMS)
• Local Inference: No API costs or data sent to external services
• Logical Consistency: Truth Maintenance System ensures coherent reasoning
• Extensible Architecture: Easy to add new rules and scoring factors
• Privacy-Preserving: All processing happens locally

---

Built with ❤️ by CS4811 | Combining Neural and Symbolic AI for Trustworthy Information