Scientific Document Retrieval on CORD-19

Key Project Deliverables

For evaluation purposes, the most important files are:

• Source Code:
  • src - Contains the source code for the information retrieval system, and the backend for the web application.
  • web - Contains the source code for the web application based user interface to the information retrieval system.
  • evaluation.ipynb - Contains code for the evaluation of the system against the TREC-COVID relevance judgments.
• Project Report: report/main.pdf - The comprehensive project report.
• Final Presentation Slides: presentation1/main.pdf - The slides for the final project presentation.

A version of the system is hosted at http://cord19.rolandb.com/, but note that performance can be slow since it is running on relatively weak hardware.

Project Overview

This project focuses on scientific document retrieval using the CORD-19 dataset, a dataset containing a large number of scientific articles about and related to the COVID-19 pandemic. The primary goal of this project is to implement an information retrieval system that is able to quickly retrieve documents relevant to answering user queries about coronaviruses in general and COVID-19 in particular. The project demonstrates the application of traditional information retrieval using and inverted index and the BM25 ranked retrieval algorithm, implemented from scratch in Python for this project. In addition, the project also explores two advanced techniques in an attempt to improve query results, namely LLM based query expansion and neural re-ranking. Both of these advanced techniques have been attempted using pre-trained models. Finally, the project also implements a RAG to summarize the top documents returned by the search.

Features

• Implementation of a custom inverted index construction in Python.
• Custom WAND based BM25 ranked retrieval implementation.
• Custom implementation of tokenization and the Porter stemming algorithm.
• Support for different field weights for title, abstract, and body of papers.
• LLM based query expansion using pre-trained language models.
• Neural re-ranking using pre-trained cross-encoders.
• RAG to summarize the top search results for the user.
• Experimental evaluation of the system using TREC-COVID provided relevance judgments.
• Structured project report and presentations documenting the methodology and results.

Project Structure

The project is organized into the following directories and key files:

├── presentation0/          # Initial one-slide project proposal slides (LaTeX).
├── presentation1/          # Final project presentation slides (LaTeX).
├── report/                 # Full project report (LaTeX).
├── scripts/                # Scripts that automate the setup and running of the application.
├── src/                    # Contains the source main code of the IR system
│   ├── bin/
│   │   ├── download.py     # Program to download the raw CORD-19 dataset.
│   │   ├── prepare.py      # Program to perform some basic pre-processing.
│   │   ├── build.py        # Program to build the inverted index.
│   │   └── serve.py        # Program to start a web server to serve the user interface.
│   └── ...
├── web/                    # Contains the source code for the user interface.
├── data/
│   ├── eval.csv            # Result of the evaluations using default settings.
│   └── eval2.csv           # A second set of test results using a larger pre-trained re-ranking model.
├── docker-compose.yml      # A docker compose configuration to run th project.
├── Dockerfile
├── evaluation.ipynb        # Jupyter Notebook for the evaluation of the IR system.
├── requirements.txt        # List of required Python dependencies to run the Project.
└── README.md               # This file.

Setup, Installation, Usage

Before starting, you should download the repository locally by running the following commands:

git clone https://github.com/rolandbernard/ir-project
cd ir-project

You can run this project either outside Docker, or by using the provided Docker Compose configuration file. Using the Docker Compose configuration is most likely going to be easier, but less suitable for development.

Note: The project has only been tested on Linux running on x86-64 hardware.

Using Docker Compose

Assuming you have Docker and Docker Compose installed on your machine, getting this project up and running should be as easy as running docker compose up in the root directory of this project. The Docker Compose file contains a service to download the raw dataset, preprocess it and constructing the inverted index. Subsequently, it will automatically start the web server that will be available at http://localhost:8099.

The default settings will get you up and running, but if you want to change some configuration parameters copy the .env.example file to .env and change the defined variables as you see fit.

Running Outside Docker

Setup

To set up and run this project, follow these steps after downloading the project:

1. Create a virtual environment (recommended):

   python -m venv venv
   source venv/bin/activate  # On Windows, use `venv\Scripts\activate`

2. Install dependencies: The project relies on a set of Python libraries. Install them using pip:

   pip install -r requirements.txt

   Note: The requirements.txt includes specific versions for reproducibility. Ensure your environment is compatible or remove the versions.

   Alternatively, run the following:

   pip install accelerate bitsandbytes fastapi matplotlib numba numpy pandas seaborn torch torchvision transformers uvicorn

Usage

To get the project up and running execute the following steps:

1. Download the CORD-19 dataset:

   python src/bin/download.py

2. Perform some minimal pre-processing:

   python src/bin/prepare.py

3. Build the inverted index on the data:

   python src/bin/build.py

   Note: you can stop at this point if you are only interested in running the evaluation in evaluation.ipynb or use only the IR system in some other way. If you want to also run the user interface continue with the next steps.
4. Build the web application:

   cd web
   npm run install
   npm run build
   cd ..

5. Start the web server:

   python src/bin/serve.py

After following all the above steps, the user interface will be accessible to you at http://localhost:8099.

Screenshots of the User Interface

Home Page

Search Result Page

Article Page