Document QA Experiments

A simple tool to test different ways of answering questions from PDF documents. It lets you swap out chunking strategies (how the document is split) and QA models to see which combination works best.

1. 🚀 How to Run It

Step 1: Install

pip install -r requirements.txt

Step 2: Run

cd src
python run_experiment.py

Step 3: See Results

The script saves a JSON file (e.g., T5MPDocVQA_results.json) containing the answers, confidence scores, and detailed metrics.

---

2. 📊 Example Output

Here is what the results file looks like. It includes the predicted answer, the ground truth, and automatic evaluation scores (BERTScore).

[
  {
    "docId": "a27ff3bf5afa77fabc9e97b3350e6c9a",
    "answer_type": "extractive",
    "question": "On what page are the matters uncovered during the investigation?",
    "ground_truth": [
      "Page 6"
    ],
    "Predicted Answer": "Page 4",
    "page_retrieval": "concat",
    "strategy": "RecursiveContextStrategy",
    "bert_precision": 0.5354930758476257,
    "bert_recall": 0.5822024345397949,
    "bert_f1": 0.5578716993331909
  }
]

---

3. 🔬 How to Customize

You don't need to rewrite the code to try new ideas. Just change the configuration in run_experiment.py.

Look for this block of code:

adapter = ModularRAGAdapter(
    name="My_Experiment",

    # CHANGE 1: The Chunking Strategy
    # Replace 'RecursiveContextStrategy()' with your own class from strategies.py
    strategy=RecursiveContextStrategy(),

    # CHANGE 2: The Model Settings
    model=T5MPDocVQA({
        "model_weights": "rubentito/t5-base-mpdocvqa",

        # "logits" = Check every page and pick the best one (Default)
        # "concat" = Glue chunks together and read at once
        "page_retrieval": "logits",

        "device": "cuda",  # Use "cpu" if you don't have a GPU
        "batch_size": 1
    }),

    # CHANGE 3: Experiment Size
    config={"limit": 3} # How many documents to test (set to None for all)
)

Want to add your own strategy?

1. Open strategies.py.
2. Add your new class (copy the existing one and modify the chunk method).
3. Use it in the code block above.

---

4. 🧠 How It Works (Simply)

Pipeline 1.DUDE_loader (Amazon_original) → PDF retrieval → Docling conversion 2.Chunking strategy produces context chunks/pages 3.Model predicts answer from chunk contexts (predict_from_chunks_debug) 4.Scores prediction vs ground truth using BERTScore (P/R/F1) 5.Saves per-sample outputs to JSON + prints mean scores by answer type

Troubleshooting

• Out of Memory? Change "device": "cuda" to "device": "cpu".
• PDF Errors? Make sure you have the internet connected to download the dataset the first time.