Multi-Source RAG for Technical Support

Original project. An evaluation-backed RAG pipeline over product documentation, customer forums, and technical blogs.

This project treats sources as different evidence classes rather than mixing every chunk into one index. Each source gets its own loader, BM25 and dense indexes, authority weight, and provenance. Results are fused with weighted reciprocal-rank fusion, reranked with a cross-encoder, checked for contradictory claims, and returned with source-aware citations.

Measured results

The repository includes a 12-query evaluation set and four ablation conditions. Full outputs and analysis are committed under eval/results/.

Condition	Source-overlap hit rate	Forum citations	Mean pipeline latency
Retrieval only	75.0%	0	6.8 s
No reranker	75.0%	0	20.2 s
No conflict detection	91.7%	14	8.9 s
Full pipeline	91.7%	14	20.5 s

The ablation exposed two practical findings:

• The cross-encoder is load-bearing: without it, forum content disappears from every final answer.
• Claim-level conflict detection surfaces planted disagreements, but its six LLM round trips dominate latency and occasionally flag paraphrases as conflicts.

See the full project report for per-query outputs, planted contradictions, false positives, and design tradeoffs.

Architecture

docs loader  -> BM25 + dense --\
forum loader -> BM25 + dense ----> weighted RRF -> cross-encoder -> claim conflicts -> answer
blog loader  -> BM25 + dense --/                         |                |             |
                                                       scores         arbitration    citations

Source-specific evidence handling

Source	Chunking strategy	Default authority
Product documentation	Markdown hierarchy with section-path metadata	1.0
Customer forum	One question-answer pair per chunk with votes and acceptance metadata	0.5
Technical blogs	Recursive paragraph-aware chunks with overlap	0.7

Retrieval and ranking

1. Run BM25 and dense retrieval independently for each source.
2. Fuse six ranked lists using weighted RRF, avoiding incomparable raw-score scales.
3. Rerank the top fused candidates with BAAI/bge-reranker-base.
4. Add a small authority bias without suppressing highly relevant community evidence.

Conflict handling

The pipeline extracts atomic claims from the top chunks, asks an LLM to identify contradictory pairs, and selects the higher-authority claim while preserving the disagreement in the final answer. The system is designed to say which source disagreed and why one side was trusted, rather than silently dropping inconvenient evidence.

Quick start

python -m venv .venv
# Windows: .venv\Scripts\activate
# macOS/Linux: source .venv/bin/activate
pip install -e .

Create local configuration:

# Windows
copy .env.example .env

# macOS/Linux
cp .env.example .env

Set ANTHROPIC_API_KEY in .env, then:

python -m multi_source_rag.cli index
python -m multi_source_rag.cli ask "How do I rotate API keys without downtime?"

Reproduce the evaluation

python eval/run_eval.py --condition all
python eval/analyze.py
python -m pytest -q

The evaluation corpus is fictional and intentionally includes stale or conflicting forum and blog claims. This makes authority weighting and contradiction handling observable without exposing private customer data.

Repository map

src/multi_source_rag/
  ingestion/       source-specific loaders and chunkers
  retrieval/       BM25, dense retrieval, and weighted RRF
  rerank/          cross-encoder reranking
  conflict/        claim extraction and contradiction arbitration
  generation/      cited answer composition
  pipeline.py      end-to-end orchestration
  logging_config.py structured JSONL audit logging
data/              fictional docs, forum threads, and blogs
eval/              queries, ablations, raw outputs, and analysis
tests/             configuration, stable-ID, and fusion smoke tests
report.md          methodology, results, failures, and next steps

Known limitations

• The 12-query evaluation is diagnostic, not a statistically complete benchmark.
• Conflict detection is expensive and should batch claim extraction before interactive deployment.
• LLM-based contradiction detection can miss subtle disagreements or flag near-duplicate claims.
• Authority weights and the reranker blend coefficient were not tuned on a held-out set.
• Dense and cross-encoder models are downloaded on first use; production deployment needs explicit model and index lifecycle management.