NaNsense

A retrieval-augmented generation (RAG) system built by the NaNsense team for the Datathon 2025 Gieni / Orderfox challenge ("RAG to GPT-4o"). It answers natural-language questions about a large corpus of company websites by retrieving relevant page content and grounding GPT-4o's responses in it, with source attribution.

Aim

The challenge provides ~17 GB of scraped company-website text and asks for a system that can answer questions about those companies accurately and verifiably. Our goal was an end-to-end RAG pipeline that:

• shrinks the raw corpus to something tractable without losing useful signal,
• retrieves the most relevant passages for a query, and
• generates concise, context-grounded answers that cite their source pages (minimising hallucination).

Approach

1. Data cleaning (src/preprocessing.py) — filter out resource URLs (CSS/JS/images), remove stopwords, and trim the first 10% of each page (navigation/boilerplate). This reduced the dataset from 17 GB to 1.9 GB.
2. Chunking — split cleaned pages into 500-character chunks with 100-character overlap, preserving source-URL metadata (main.ipynb). Optional metadata tagging (technologies / services / materials / products / industries / regions) via fuzzy keyword matching (src/fuzzy_metadata.py).
3. Knowledge base — embed chunks with sentence-transformers/all-MiniLM-L6-v2 and persist them to a Chroma vector store.
4. Retrieval (src/evaluate.py, src/keyword_retrieval.py) — semantic top-k similarity search, with an optional hybrid mode that combines vector search with fuzzy keyword retrieval.
5. Generation (src/prompts.py) — assemble retrieved passages into context and prompt GPT-4o to answer using only that context, with source attribution.
6. Interface (src/interface.py) — a Gradio app with streaming answers and adjustable number of retrieved documents.
7. Evaluation (src/benchmark_generator.py, src/evaluate.py) — a 50-pair question/answer benchmark and an exact-match evaluator (see Results).

Results

• Corpus reduction: the cleaning pipeline cut the dataset ~9× (17 GB → 1.9 GB) while retaining the content-bearing pages, making embedding and retrieval practical.
• Knowledge base: thousands of embedded chunks, each carrying its source URL so answers can be attributed back to a specific page.
• Grounded answers: prompts constrain GPT-4o to the retrieved context, which keeps answers on-source and reduces hallucination; the Gradio demo shows this interactively.
• Evaluation method: we built a benchmark of 50 question/answer pairs generated with o1, each with a single-word ground-truth answer drawn from a randomly selected page (benchmark.json). The system is scored by exact match between its answer and the ground truth (src/evaluate.py). See report.md and presentation.pptx for the full write-up of the pipeline and evaluation.

The benchmark is intentionally strict (single-word, exact match), so it is best read as a relative signal for comparing retrieval/prompt variants rather than an absolute accuracy figure.

Stack

Python · LangChain · Chroma · sentence-transformers (MiniLM-L6-v2) · OpenAI GPT-4o · Gradio · thefuzz / rapidfuzz · PyTorch.

Repository layout

.
├── src/                      # Pipeline and app
│   ├── preprocessing.py      # URL filtering, stopword removal, trimming
│   ├── fuzzy_metadata.py     # Fuzzy metadata tagging of chunks
│   ├── keyword_retrieval.py  # Keyword extraction + hybrid retrieval
│   ├── evaluate.py           # Vector DB loading, retrieval, exact-match eval
│   ├── prompts.py            # Prompt loading + GPT-4o answer generation
│   ├── prompts.json          # RAG / follow-up prompt templates
│   ├── benchmark_generator.py# o1-based QA-pair benchmark generation
│   └── resources_words.py    # Stopword / URL-path resource lists
├── notebooks/
│   └── data_cleaning/        # Data exploration and cleaning notebooks
├── main.ipynb                # End-to-end pipeline (clean → chunk → embed → eval)
├── benchmark.json            # 50 QA-pair evaluation benchmark
├── chroma_db/                # Bundled (subsampled) vector store — see "Data layout"
├── report.md                 # Project report
├── presentation.pptx         # Datathon presentation
└── video-demo.gif            # Demo recording

Data layout

The full competition dataset (~20 GB) is not committed — it is competition-owned and far too large for GitHub. It lives locally under data/ (gitignored). The expected structure is:

data/
├── hackathon_data/   # Raw competition JSON, one file per company website (~17 GB)
│                     #   keys: url, timestamp, text_by_page_url, doc_id
├── clean/            # Cleaned JSON after preprocessing (~1.9 GB)
└── chunked_documents.pkl   # Chunked documents produced by main.ipynb

To keep the demo runnable without that dataset, a subsampled Chroma vector store is committed at chroma_db/. It is loaded directly by the app, so the demo works from a clean clone. The trade-off is that it covers only a subset of the corpus; the full-quality store is rebuilt from the complete dataset via main.ipynb.

Note on Git LFS: every committed file is under GitHub's 100 MB limit, so LFS is not required. If you ever commit a full-size vector store, move the Chroma *.bin files to Git LFS first.

How to run

The app loads the bundled vector store and calls GPT-4o, so you need an OpenAI API key. All commands are run from the repository root.

1. Create and activate a virtual environment:

   python3 -m venv .venv
   source .venv/bin/activate        # Windows: .venv\Scripts\activate

2. Install dependencies:

   pip install -r requirements.txt

3. Provide your OpenAI API key — either export it or put it in a .env file at the repo root:

   echo "OPENAI_API_KEY=sk-..." > .env

4. Launch the Gradio interface:

   python src/interface.py

Rebuilding the knowledge base (optional)

To rebuild the vector store at full quality you need the competition dataset under data/ (see "Data layout"). Then work through main.ipynb, which cleans the raw data, chunks it, builds the Chroma store, and runs the evaluation. The evaluation can also be run on its own:

python src/evaluate.py

Known limitations

• The bundled chroma_db/ is a subsample for a runnable demo; full-quality retrieval requires rebuilding from the complete (undistributed) dataset.
• The dataset is competition-owned and is not redistributed here.
• The exact-match benchmark is strict (single-word answers) and small (50 pairs); treat it as a relative signal rather than an absolute score.

License

Released under the MIT License.