Grimoire Retriever

A RAG-powered lore notes search utility that retrieves fantasy world-building content with cited source snippets.

Features

• 📚 Ingest Markdown Lore: Automatically chunk and embed your world-building notes
• 🔍 Natural Language Search: Query your lore with plain English questions
• 📝 Cited Answers: Get answers with source snippets and relevance scores
• 🔌 REST API: FastAPI-powered endpoint for integration with other tools
• 📓 Jupyter Support: Explore embeddings and test retrieval interactively

Installation

# Clone the repository
git clone https://github.com/yourusername/grimoire-retriever.git
cd grimoire-retriever

# Install dependencies
pip install -e .

# Copy environment configuration
cp .env.example .env

Quick Start

1. Ingest Your Lore

grimoire ingest ./path/to/your/lore/

2. Query Your Knowledge Base

grimoire query "Who is the Dragon Queen?"

3. Or Start the API Server

grimoire serve

Then query via HTTP:

curl -X POST http://localhost:8000/query \
    -H "Content-Type: application/json" \
    -d '{"question": "What are the magical properties of moonstone?"}'

Documentation

• API Documentation - REST API reference
• Usage Guide - Detailed usage instructions and best practices
• Notebook Guide - Interactive exploration with Jupyter

Project Structure

grimoire-retriever/
├── src/grimoire_retriever/
│   ├── __init__.py
│   ├── __main__.py        # Entry point
│   ├── api.py             # FastAPI endpoints
│   ├── cli.py             # Command-line interface
│   ├── config.py          # Configuration management
│   ├── embeddings.py      # Embedding generation
│   ├── ingest.py          # Document ingestion
│   ├── models.py          # Data models
│   ├── retriever.py       # Vector search
│   ├── schemas.py         # API schemas
│   └── synthesizer.py     # Answer synthesis
├── notebooks/
│   ├── exploration.ipynb  # Interactive exploration
│   └── README.md
├── tests/
│   └── test_synthesizer.py
├── docs/
│   ├── API.md
│   └── USAGE.md
├── .env.example
├── pyproject.toml
└── README.md

Configuration

Configure via environment variables or .env file:

Variable	Default	Description
CHROMA_PERSIST_DIR	./data/chroma	ChromaDB storage location
EMBEDDING_MODEL	sentence-transformers/all-MiniLM-L6-v2	HuggingFace model for embeddings
DEFAULT_CHUNK_SIZE	512	Tokens per chunk
API_PORT	8000	API server port

See .env.example for all options.

Development

# Install dev dependencies
pip install -e ".[dev]"

# Run tests
pytest

# Format code
black src/ tests/
ruff check src/ tests/

How It Works

1. Ingestion: Markdown files are parsed, split into overlapping chunks, and embedded using sentence-transformers
2. Storage: Embeddings are stored in ChromaDB for efficient similarity search
3. Retrieval: Queries are embedded and matched against stored chunks using cosine similarity
4. Synthesis: Retrieved chunks are combined with the question to generate a cited answer

Use Cases

• 🎮 TTRPG Campaign Management: Search your campaign notes during sessions
• ✍️ Novel Writing: Keep track of world-building details across your manuscript
• 🎲 Game Development: Query game lore documentation
• 📖 Wiki Management: Search personal knowledge bases

License

MIT License - see LICENSE file for details.

Contributing

Contributions welcome! Please read the contributing guidelines before submitting PRs.