🌋 Bicolingo (Bicol‑to‑English Machine Translation)

Bicolingo is an LSTM-based Bicolano (bcl) → English (en) neural machine translation (NMT) system tailored for low-resource language processing. It implements a Seq2Seq model with a bidirectional LSTM encoder, a unidirectional LSTM decoder, and Bahdanau attention, enhanced with SentencePiece subword tokenization.

This repository contains the preprocessing pipelines, alignment utilities, and training notebooks to transition from word-level baselines to enhanced subword-tokenized neural translation.

---

🎯 Project Objectives

• Low-Resource Neural Machine Translation: Translating from Bicolano (bcl) to English (en) using a highly limited parallel dataset.
• Architecture: Seq2Seq framework with a bidirectional LSTM encoder, unidirectional LSTM decoder, and Bahdanau (additive) attention.
• Low-Resource Optimization: Integrated SentencePiece unigram subword tokenization, subword regularization, and length-normalized beam search.
• Evaluation Metrics: Evaluated primarily using METEOR, chrF, and sacreBLEU, supplemented by human adequacy and fluency reviews.

---

🛠️ Installation & Setup

Bicolingo is powered by the high-performance uv package manager and requires Python 3.14+.

1. Install uv

If you do not have uv installed, get it via:

# macOS/Linux
curl -LsSf https://astral.sh/uv/install.sh | sh

2. Set Up Virtual Environment & Dependencies

Initialize and sync the project environment:

# Run from the project root
uv sync

This command automatically reads the pyproject.toml and locks standard dependencies (like nltk, pdfreader, pypdf2, spacy, sentence-transformers, sqlalchemy, rapidfuzz) into a clean, isolated .venv.

Alternative: Automated Setup via setup.py

If you want an automated installation that automatically detects whether an NVIDIA GPU is present and installs the correct PyTorch build (GPU vs. CPU-only), you can run:

# Run from the project root
python3 setup.py

This script will:

• Detect NVIDIA GPU availability (via nvidia-smi) and install the appropriate PyTorch build (torch, torchaudio, torchvision).
• Automatically detect if uv is available. If so, it installs dependencies using uv pip install inside the environment; otherwise, it falls back to standard pip.
• Install all required project packages (fastapi, spacy, sentence-transformers, sqlalchemy, etc.).

3. Run Preprocessing or Training

Execute scripts or launch your notebook environment within the uv environment:

# Start Jupyter Lab for model training
uv run jupyter lab

---

📁 Repository Structure

├── aligned_text/          # Aligned parallel outputs (e.g., wikimedia-aligned.csv)
├── cleaned_text/          # Intermediate clean plain-text exports
├── raw_text/              # Raw source texts for Bicolano & English datasets
├── COPYING                # GNU Affero General Public License Version 3 (AGPL-3.0)
├── pyproject.toml         # Python environment configuration and project dependencies
├── uv.lock                # Deterministic lockfile managed by uv
├── AGENTS.md              # Detailed training diagnostic specs & debugging context
└── src/
    ├── notebooks/         # Google Colab / local training & alignment notebooks
    │   ├── bicol_en_seq2seq_v2_3.ipynb  # Active subword-level LSTM+Attention model
    │   ├── bicol_en_seq2seq_v2_2.ipynb  # Previous word-level baseline
    │   ├── allignment-check.ipynb        # Merging, EDA & sequence length analytics
    │   └── transformer_allignment.ipynb  # Sentence-laBSE transformer alignment checker
    └── preprocess/        # Preprocessing, PDF extraction, and text alignment utilities
        ├── pdf_extract.py      # Extracts raw text from PDFs
        ├── wikipedia-align.py  # Aligns sentences from wikitext datasets
        └── alignment.py        # Aligns parallel sentences across corpora

---

Web Dashboard

This repo now includes a local-first model comparison website:

• FastAPI backend in src/app/
• React + Vite frontend in web/

The dashboard loads saved run artifacts from outputs/ and supports:

• side-by-side translation for the unidirectional and bidirectional V2.3 models
• run metadata inspection
• test-set sample browsing and comparison
• uploaded CSV evaluation with bcl,en columns for both models
• saved training curve display

Run the dashboard from one local server

python3 setup.py

# compiling the react components 
cd web
npm install
npm run build
cd ..

# running the web dashboard
python3 setup.py web

Then open http://127.0.0.1:8000.

If you prefer frontend-only development, the React app still lives in web/, but the built bundle can now be served directly by FastAPI for local demos and model evaluation.

Running in Debug Mode (Development)

For local development with hot-reloading enabled:

• FastAPI Backend (API only with auto-reload):

  python3 setup.py debug api

• Vite Frontend Dev Server (only frontend with hot-module replacement):

  python3 setup.py debug web

---

👥 Contributors

This project is made by the following people:

• Alano, Ruzel Luigi
• Bajit, Giancarlo
• Del Rosario, Jimalyn
• Francisco, Meinard Adrian P.
• Hinay, Anthony John
• Maravilla, Ernest Matthew
• Sta Ines, Jhonder

📜 License

This project is licensed under the GNU Affero General Public License Version 3 (AGPL-3.0). See the full license text in the COPYING file.