diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e976f47..d2acbe0 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -4,8 +4,8 @@ First of all, thank you for considering contributing to FireForm! It's people li FireForm is a Digital Public Good (DPG) designed to reduce administrative overhead for firefighters and other emergency services. By contributing, you are helping us build a more efficient future for emergency response. ->**NOTE: Before you contribute!** ->Please note that this project is still under early development. Because of this you may find many bugs caused either by cases yet unaddressed that we are aware of. Reports on these bugs will be dismissed to remove clutter, so please don't feel discouraged if your issue is dismissed quickly, it just means we were already aware of the bug. +> **NOTE: Before you contribute!** +> Please note that this project is still under early development. Because of this you may find many bugs caused either by cases yet unaddressed that we are aware of. Reports on these bugs will be dismissed to remove clutter, so please don't feel discouraged if your issue is dismissed quickly, it just means we were already aware of the bug. ## 🌈 Code of Conduct @@ -15,7 +15,7 @@ This project and everyone participating in it is governed by the [FireForm Code ### Reporting Bugs -Before creating bug reports, please check the [issues list](https://github.com/juanalvv/FireForm/issues) to see if the problem has already been reported. +Before creating bug reports, please check the [issues list](https://github.com/fireform-core/FireForm/issues) to see if the problem has already been reported. When you are creating a bug report, please include as many details as possible: * **Use a clear and descriptive title** for the issue to identify the problem. @@ -39,12 +39,141 @@ If you have a great idea for FireForm, we'd love to hear it! Please open an issu 4. Make sure your code lints. 5. Issue that pull request! -## πŸ› οΈ Local Development Setup +## πŸ› οΈ Development Setup -FireForm uses Docker and Docker Compose for development to ensure a consistent environment. +For detailed, platform-specific setup instructions (Windows, macOS, Linux), see the [Developer Setup Guide](docs/SETUP.md). ### Prerequisites -- [Docker](https://docs.docker.com/get-docker/) -- [Docker Compose](https://docs.docker.com/compose/install/) -- `make` (optional, but recommended) +- [Python 3.11+](https://www.python.org/downloads/) +- [Docker](https://docs.docker.com/get-docker/) & [Docker Compose](https://docs.docker.com/compose/install/) +- [Ollama](https://ollama.com/download) (for running the LLM locally) +- `make` (optional, but recommended β€” included with most Linux/macOS systems) + +### Quick Start (Docker) + +```bash +# 1. Clone your fork +git clone https://github.com//FireForm.git +cd FireForm + +# 2. Build and start all containers +make build +make up + +# 3. Pull the Mistral model into Ollama +make pull-model + +# 4. Open a shell in the app container +make shell +``` + +### Quick Start (Local β€” without Docker) + +```bash +# 1. Clone and set up virtual environment +git clone https://github.com//FireForm.git +cd FireForm +python -m venv venv + +# Activate (Linux/macOS) +source venv/bin/activate +# Activate (Windows) +venv\Scripts\activate + +# 2. Install dependencies +pip install -r requirements.txt + +# 3. Initialize the database +python -m api.db.init_db + +# 4. Start the API server +uvicorn api.main:app --reload + +# 5. In a separate terminal, start Ollama +ollama serve +ollama pull mistral +``` + +### Verifying Your Setup + +Once the API is running, verify it: + +```bash +# Should return {"detail":"Not Found"} +curl http://localhost:8000/ +``` + +Then open [http://localhost:8000/docs](http://localhost:8000/docs) in your browser to see the Swagger UI. + +## πŸ§ͺ Running Tests + +```bash +# Local +python -m pytest tests/ -v + +# Docker +docker compose exec app python3 -m pytest tests/ -v +``` + +> **Note:** The `make test` Makefile target currently points to `src/test/`. To run the test suite under `tests/` from Docker, use the explicit command shown above. + +## 🌿 Branch Naming Convention + +Please follow these branch naming conventions: + +| Type | Format | Example | +|------|--------|---------| +| Feature | `feat/` | `feat/batch-extraction` | +| Bug fix | `fix/` | `fix/json-decode-error` | +| Refactor | `refactor/` | `refactor/llm-client` | +| Documentation | `docs/` | `docs/setup-guide` | +| Tests | `test/` | `test/llm-unit-tests` | + +## πŸ“ Commit Message Convention + +Use [Conventional Commits](https://www.conventionalcommits.org/) format: + +``` +(): + +Examples: +feat(api): add GET /forms/download endpoint +fix(filler): correct multi-page field index reset +docs: add developer setup guide +test(llm): add batch extraction unit tests +``` + +## πŸ“‚ Project Structure + +``` +FireForm/ +β”œβ”€β”€ api/ # FastAPI backend +β”‚ β”œβ”€β”€ db/ # Database models, repositories, init +β”‚ β”œβ”€β”€ routes/ # API route handlers (forms, templates) +β”‚ β”œβ”€β”€ schemas/ # Pydantic request/response models +β”‚ β”œβ”€β”€ errors/ # Custom error classes and handlers +β”‚ └── main.py # FastAPI app entry point +β”œβ”€β”€ src/ # Core pipeline logic +β”‚ β”œβ”€β”€ controller.py # Orchestrates extraction β†’ filling +β”‚ β”œβ”€β”€ file_manipulator.py # Manages LLM + PDF coordination +β”‚ β”œβ”€β”€ filler.py # PDF form filling with pdfrw +β”‚ β”œβ”€β”€ llm.py # Ollama/Mistral LLM client +β”‚ β”œβ”€β”€ main.py # Standalone CLI entry point +β”‚ β”œβ”€β”€ inputs/ # Sample input files (PDFs, transcripts) +β”‚ └── outputs/ # Generated filled PDFs +β”œβ”€β”€ tests/ # Test suite (pytest) +β”œβ”€β”€ docs/ # Documentation +β”œβ”€β”€ Dockerfile # Container image definition +β”œβ”€β”€ docker-compose.yml # Multi-container orchestration +β”œβ”€β”€ Makefile # Development convenience commands +└── requirements.txt # Python dependencies +``` + +## πŸ”— Useful Links + +- [Project Board](https://github.com/users/juanalvv/projects/1) β€” Task tracking +- [Swagger API Docs](http://localhost:8000/docs) β€” Interactive API explorer (when server is running) +- [Docker Guide](docs/docker.md) β€” Docker-specific documentation +- [Database Guide](docs/db.md) β€” Database setup and management +- [Developer Setup Guide](docs/SETUP.md) β€” Comprehensive setup instructions diff --git a/docs/SETUP.md b/docs/SETUP.md new file mode 100644 index 0000000..018bf58 --- /dev/null +++ b/docs/SETUP.md @@ -0,0 +1,329 @@ +# FireForm Developer Setup Guide + +This guide walks you through setting up FireForm for local development on Windows, macOS, and Linux. + +## Prerequisites + +| Dependency | Version | Installation | +|-----------|---------|-------------| +| **Python** | 3.11+ | [python.org/downloads](https://www.python.org/downloads/) | +| **Docker** | 20.10+ | [docs.docker.com/get-docker](https://docs.docker.com/get-docker/) | +| **Docker Compose** | 2.0+ | Included with Docker Desktop | +| **Ollama** | Latest | [ollama.com/download](https://ollama.com/download) | +| **Git** | Any | [git-scm.com](https://git-scm.com/) | +| **Make** | Any | Included on Linux/macOS. Windows: install via [chocolatey](https://community.chocolatey.org/packages/make) (`choco install make`) or [GnuWin32](https://gnuwin32.sourceforge.net/packages/make.htm) | + +--- + +## Option A: Docker Setup (Recommended) + +Docker provides a consistent environment across all platforms. + +### Step 1 β€” Clone the Repository + +```bash +# Fork the repo on GitHub first, then: +git clone https://github.com//FireForm.git +cd FireForm +``` + +### Step 2 β€” Build Containers + +```bash +make build +``` + +This builds the `fireform-app` container image from the Dockerfile (Python 3.11-slim base). The Ollama image will be pulled automatically in the next step. + +### Step 3 β€” Start Containers + +```bash +make up +``` + +This starts two containers: +- `fireform-app` β€” runs the FastAPI backend via Uvicorn (mapped to host port `8000`) +- `fireform-ollama` β€” the Ollama LLM server (mapped to host port `11434`) + +> **Note:** The `docker-compose.yml` starts Uvicorn automatically with `--reload`. If port `8000` is not accessible, see the [Port 8000 troubleshooting section](#port-8000-not-accessible-from-host) below. + +### Step 4 β€” Pull the LLM Model + +```bash +make pull-model +``` + +This downloads the Mistral model (~4 GB) into the Ollama container. This only needs to be done once β€” the model persists in a Docker volume. + +> **Note:** This download can take several minutes depending on your connection. Do not interrupt it. + +### Step 5 β€” Verify + +```bash +# Check containers are running +docker compose ps + +# Check the API is responding (should return JSON) +curl http://localhost:8000/ +``` + +Open [http://localhost:8000/docs](http://localhost:8000/docs) in your browser to see the Swagger UI. + +### Useful Docker Commands + +| Command | Description | +|---------|-------------| +| `make fireform` | Build, start, and open a shell (all-in-one) | +| `make shell` | Open a bash shell inside the app container | +| `make exec` | Run `src/main.py` inside the container | +| `make logs` | View live logs from all containers | +| `make logs-app` | View logs from the app container only | +| `make logs-ollama` | View logs from the Ollama container only | +| `make down` | Stop all containers | +| `make clean` | Stop and remove all containers + volumes | +| `make help` | Show all available commands | + +--- + +## Option B: Local Setup (Without Docker) + +Use this if you prefer running everything natively. + +### Step 1 β€” Clone and Create Virtual Environment + +```bash +git clone https://github.com//FireForm.git +cd FireForm +python -m venv venv +``` + +Activate the virtual environment: + +```bash +# Linux / macOS +source venv/bin/activate + +# Windows (PowerShell) +venv\Scripts\Activate.ps1 + +# Windows (CMD) +venv\Scripts\activate.bat +``` + +### Step 2 β€” Install Dependencies + +```bash +pip install -r requirements.txt +``` + +> **Note:** On some systems you may also need to install `python-multipart` and `pypdf` manually if they are not yet listed in `requirements.txt`: +> ```bash +> pip install python-multipart pypdf +> ``` + +### Step 3 β€” Install Ollama and Pull Mistral + +1. Download and install Ollama from [ollama.com/download](https://ollama.com/download) +2. Start Ollama: + ```bash + ollama serve + ``` +3. In a **new terminal**, pull the Mistral model: + ```bash + ollama pull mistral + ``` + +### Step 4 β€” Initialize the Database + +```bash +python -m api.db.init_db +``` + +You should see a `fireform.db` file created in the project root. + +### Step 5 β€” Start the API Server + +```bash +uvicorn api.main:app --reload +``` + +You should see: +``` +INFO: Uvicorn running on http://127.0.0.1:8000 (Press CTRL+C to quit) +INFO: Started reloader process +``` + +### Step 6 β€” Verify + +Open [http://localhost:8000/docs](http://localhost:8000/docs) in your browser to see the Swagger UI. + +--- + +## Running Tests + +```bash +# Run all tests (local) +python -m pytest tests/ -v + +# Run all tests (Docker) +docker compose exec app python3 -m pytest tests/ -v + +# Run a specific test file +python -m pytest tests/test_templates.py -v + +# Run with coverage (if pytest-cov is installed) +python -m pytest tests/ --cov=api --cov=src -v +``` + +--- + +## Environment Variables + +| Variable | Default | Description | +|----------|---------|-------------| +| `OLLAMA_HOST` | `http://localhost:11434` | Ollama API endpoint. Set to `http://ollama:11434` in Docker. | +| `PYTHONPATH` | β€” | Set to `/app/src` in Docker. For local dev, run from project root. | +| `PYTHONUNBUFFERED` | β€” | Set to `1` in Docker for real-time log output. | + +--- + +## Common Issues and Fixes + +### `ImportError: libGL.so.1: cannot open shared object file` + +**Cause:** OpenCV requires system-level graphics libraries not included in `python:3.11-slim`. + +**Fix (Docker):** Ensure your `Dockerfile` includes: +```dockerfile +RUN apt-get update && apt-get install -y \ + curl \ + libgl1 \ + libglib2.0-0 \ + && rm -rf /var/lib/apt/lists/* +``` + +**Fix (Linux local):** +```bash +sudo apt-get install libgl1 libglib2.0-0 +``` + +--- + +### `NameError: name 'Union' is not defined` + +**Cause:** `src/main.py` uses `Union` in a type hint but doesn't import it. + +**Fix:** Add to the top of `src/main.py`: +```python +from typing import Union +``` + +> This is a known issue tracked in [Issue #187](https://github.com/fireform-core/FireForm/issues/187). + +--- + +### `Form data requires "python-multipart" to be installed` + +**Cause:** `python-multipart` is required by FastAPI for file uploads but is not listed in `requirements.txt`. + +**Fix:** +```bash +pip install python-multipart +``` + +> This is a known issue tracked in [Issue #204](https://github.com/fireform-core/FireForm/issues/204). + +--- + +### Port `11434` already in use + +**Cause:** Another instance of Ollama is already running on your system. + +**Fix:** +```bash +# Linux / macOS +sudo lsof -i :11434 + +# Windows (PowerShell) +netstat -aon | findstr :11434 + +# Kill the process or stop the existing Ollama instance +``` + +--- + +### Port `8000` not accessible from host + +**Cause:** The Docker Compose file may be missing the port mapping. + +**Fix:** Ensure `docker-compose.yml` has the following under the `app` service: +```yaml +ports: + - "8000:8000" +``` + +> This is a known issue tracked in [Issue #224](https://github.com/fireform-core/FireForm/issues/224). + +--- + +### `ModuleNotFoundError: No module named 'pypdf'` + +**Cause:** `pypdf` is used in template routes but may not be in `requirements.txt`. + +**Fix:** +```bash +pip install pypdf +``` + +--- + +## Resetting the Database + +If your database gets into a bad state: + +```bash +# Delete the existing database +rm fireform.db + +# Recreate it +python -m api.db.init_db +``` + +--- + +## Architecture Overview + +FireForm follows this pipeline: + +``` +Voice Memo / Text Input + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ FastAPI (API) β”‚ POST /forms/fill +β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ Controller β”‚ Orchestrates the flow +β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β” +β”‚ FileManipulator β”‚ Coordinates LLM + PDF +β””β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”˜ + β”‚ + β”Œβ”€β”€β”€β”€β”΄β”€β”€β”€β”€β” + β”‚ β”‚ + β–Ό β–Ό +β”Œβ”€β”€β”€β”€β”€β”€β”€β” β”Œβ”€β”€β”€β”€β”€β”€β”€β” +β”‚ LLM β”‚ β”‚Filler β”‚ +β”‚(Ollamaβ”‚ β”‚(pdfrw)β”‚ +β”‚Mistralβ”‚ β”‚ β”‚ +β””β”€β”€β”€β”¬β”€β”€β”€β”˜ β””β”€β”€β”€β”¬β”€β”€β”€β”˜ + β”‚ β”‚ + β–Ό β–Ό + JSON Filled PDF +``` + +For more details, see `src/controller.py`, `src/file_manipulator.py`, `src/llm.py`, and `src/filler.py`.