Shadow of the Torturer — RAG Assistant

A Retrieval-Augmented Generation system for exploring Gene Wolfe's Shadow of the Torturer.

Why This Corpus?

Most RAG demos use documents where the LLM already knows the answers. Wolfe's Book of the New Sun is different:

• Unreliable narrator — Severian claims perfect memory but contradicts himself constantly
• Archaic vocabulary — Words like fuligin, aquastor, cacogen challenge embedding models
• Dense cross-references — Meaningful answers require synthesis across distant passages

This makes it a genuine test of retrieval quality: if your RAG can't find the right passages, the LLM has no hope.

Features

• Vector similarity search over the full text of Shadow of the Torturer
• Structured outputs via Instructor — responses include source attribution and narrator reliability analysis
• Comparison mode — structured vs. simple chat to show the difference
• CLI and Gradio web interface

Tech Stack

Component	Technology
Embeddings	OpenAI text-embedding-3-small
Vector Store	PostgreSQL + pgvector
Generation	OpenAI GPT-4o-mini
Structured Output	Instructor + Pydantic
UI	Gradio

Setup

Prerequisites

• Python 3.8+
• PostgreSQL 12+ with pgvector extension
• OpenAI API key

Installation

cd wolfe-rag
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt

Database Setup

CREATE EXTENSION IF NOT EXISTS vector;
CREATE DATABASE wolfe_rag;

Configuration

Edit .env with your settings:

OPENAI_API_KEY=your_key_here
DB_HOST=localhost
DB_PORT=5432
DB_NAME=wolfe_rag
DB_USER=steward
DB_PASSWORD=

Ingest the Text

Place the text of Shadow of the Torturer in the data/ directory, then run:

python src/pre_processing.py data/shadow_of_the_torturer.txt

This will chunk the text, generate embeddings, and store everything in PostgreSQL.

Run

Web Interface:

python src/main_gradio_interface.py

Command Line:

python src/main_command_line.py

Project Structure

wolfe-rag/
├── data/                           # Place the novel text here
├── src/
│   ├── pre_processing.py           # Chunking + embedding + ingestion
│   ├── rag_tools.py                # Retrieval + generation logic
│   ├── main_gradio_interface.py    # Gradio web UI
│   └── main_command_line.py        # CLI interface
├── requirements.txt
├── .env                            # API keys and DB config
├── .gitignore
└── README.md

Demo Questions

Try these to see the system in action:

• "What is the Guild of Torturers and how does Severian feel about it?"
• "Describe the Claw of the Conciliator. What does it do?"
• "Is Severian a reliable narrator? Find evidence of contradictions."
• "Who is Dorcas and what is strange about her?"
• "What happens during the duel between Severian and Agilus?"
• "What is fuligin and why is it significant?"

About

Built for Launch School Capstone — AI Engineering module. Demonstrates RAG architecture with structured outputs for literary analysis of an unreliable narrator.