Faqy: Your FAQ to RAG Chatbot Converter

Faqy is a powerful and easy-to-use tool that converts your existing FAQ data into a fully functional Retrieval-Augmented Generation (RAG) chatbot. Simply upload a CSV file with your questions and answers, and Faqy will automatically build and deploy a RAG pipeline that you can interact with through a clean and intuitive web interface.

✨ Key Features

• Intuitive Web Interface: A multi-page Streamlit application that provides a seamless user experience for uploading data, chatting with the RAG pipeline, and evaluating its performance.
• Flexible Data Ingestion: Easily upload your own FAQ data in a simple two-column CSV format (questions, answers).
• Advanced RAG Pipeline: The core of the application is a robust RAG pipeline that leverages state-of-the-art techniques to provide accurate and context-aware answers.
• Vector-based Search: Uses FAISS for efficient and scalable vector search, allowing for semantic understanding of user queries.
• Reranking for Accuracy: Incorporates a cross-encoder model to rerank the search results, ensuring that the most relevant context is passed to the language model.
• LLM-powered Answer Generation: Uses a large language model (LLM) to generate a final, synthesized answer based on the retrieved context and the user's query.
• Configurable Backend: Easily switch between different LLM backends (OpenAI or local Ollama) and embedding models via environment variables.
• LLM-based Evaluation: A sophisticated evaluation framework that uses an LLM to assess the correctness of the generated answers, providing a more accurate measure of the pipeline's performance.
• Centralized Logging: A comprehensive logging system that provides detailed insights into the application's behavior, with the option to log to both the console and a file.

🛠️ Tech Stack

Component	Technology
Language	Python 3
Package Manager	uv
Web Framework	Streamlit
RAG Orchestration	LlamaIndex
Vector Store	FAISS
Embeddings	OpenAI or Ollama
LLM	OpenAI or Ollama
Reranker	cross-encoder/ms-marco-MiniLM-L6-v2
Evaluation	LLM-based evaluation

🏛️ Architecture

The application follows a Retrieval-Augmented Generation (RAG) architecture. Here is a high-level overview of the pipeline:

graph TD
    A[CSV Upload] -->|Validate & Clean| B[Embed Q&A Pairs]
    B --> C[Store in FAISS Vector DB]
    C --> D[User Query]
    D --> E[Vector Search]
    E --> F[Reranker]
    F -->|Top-k Results| G[LLM Response Generation]
    G --> H[Streamlit Chat UI]

Loading

1. Data Ingestion: The user uploads a CSV file containing question-answer pairs. The application cleans and preprocesses this data and then embeds each Q&A pair into a single vector representation.
2. Indexing: The vector embeddings are stored in a FAISS vector store for efficient similarity search.
3. Retrieval: When a user asks a question, the application first converts the query into a vector embedding and then uses FAISS to retrieve the most similar Q&A documents from the index.
4. Reranking: The retrieved documents are then passed through a cross-encoder model to rerank them based on their relevance to the user's query.
5. Answer Generation: The top-ranked documents are then passed to a large language model (LLM) along with the user's original query. The LLM is instructed to generate a final answer based solely on the provided context.

🖥️ Application Pages

The application consists of four main pages:

• Home: The landing page of the application, which provides a brief introduction and overview.
• Upload: This page allows you to upload your own FAQ data in CSV format. It provides real-time feedback on the indexing process and includes safeguards to prevent accidental overwrites of the index.
• Chat: The main interface for interacting with the RAG pipeline. You can ask questions and get answers based on the indexed FAQ data.
• Evaluation: This page allows you to evaluate the performance of the RAG pipeline by uploading a validation dataset. It provides an accuracy score and a detailed breakdown of the incorrect answers.

🚀 Setup and Usage

1. Clone the repository:

   git clone git@github.com:oxi-p/faqy.git
   cd faqy

2. Create a virtual environment and install dependencies:

   uv sync

3. Configure the application:

   • Create a .env file from the .env.example file.
   • Populate the .env file with your desired configuration (see the Configuration section below).

4. Run the application:

   uv run streamlit run app.py

⚙️ Configuration

The application can be configured using the following environment variables in the .env file:

• LLM_PROVIDER: The language model provider to use. Can be OPENAI or OLLAMA.
• LARGE_LANGUAGE_MODEL_NAME: The name of the language model to use (e.g., gpt-4o, gpt-3.5-turbo).
• EMBEDDING_MODEL_PROVIDER: The embedding model provider to use. Can be OPENAI or OLLAMA.
• EMBEDDING_MODEL_NAME: The name of the embedding model to use.
• OPENAI_API_KEY: Your OpenAI API key (if using the OpenAI provider).
• TOP_K_CANDIDATES: The number of candidates to retrieve from the vector store.
• CONFIDENCE_THRESHOLD: The confidence threshold for the reranker.
• ENABLE_RERANKING: Set to true to enable reranking.
• LOG_TO_FILE: Set to true to enable logging to a file.
• LOG_FILE: The path to the log file.
• LOG_LEVEL: The logging level (e.g., INFO, DEBUG, ERROR).

📊 Evaluation

The evaluation process is designed to provide an accurate measure of the RAG pipeline's performance. It works as follows:

1. Upload a validation dataset: The validation dataset should be a CSV file with questions and answers columns.
2. Run the evaluation: The application will then iterate through each question in the validation dataset, pass it through the RAG pipeline, and get the generated answer.
3. LLM-based correctness check: For each generated answer, the application uses an LLM to determine if it is a correct and faithful representation of the ground truth answer.
4. Calculate accuracy: The final accuracy score is then calculated based on the number of correct predictions.

🧪 Testing

To run the test suite, use the following command:

uv run pytest tests/

🤝 How to Contribute

We welcome contributions to this project! Please follow these steps to contribute:

1. Fork the repository: Click the 'Fork' button in the top right corner of this page to create your own copy of the repository.

2. Create a new branch:

   git checkout -b your-feature-branch-name

3. Make your changes: Make your desired changes to the code.

4. Commit your changes:

   git add .
   git commit -m "A clear and descriptive commit message"

5. Push your changes to your fork:

   git push origin your-feature-branch-name

6. Create a pull request: Go to the original repository on GitHub and click the 'New pull request' button. Provide a clear title and description for your pull request, and our team will review it as soon as possible.

---