Author: Bartek Borzyszkowski
AlpineLLM is a lightweight base language model pre-trained from scratch on raw text corpora drawn primarily from public-domain literature on alpinism.
This repository contains the source code for all stages of the model's development pipeline, including data preparation, training, evaluation, and deployment.
Currently supported version of the model: AlpineLLM-Tiny-10M-Base
Pre-trained model weights and an interactive live demo are available in the related repositories listed below.
The software has been tested on Ubuntu 22.04 with CUDA 12.2 and Python3.10.
Please use a Python virtual environment to install the dependencies:
python3.10 -m venv venv_AlpineLLM
source venv_AlpineLLM/bin/activate
pip install -r requirements.txt
-
Download dataset
The dataset consists of raw text corpora, including public-domain literature related to alpinism. It is based on several books available through Project Gutenberg and can be downloaded using the provided script:
bash core/utils/download_dataset.sh -
Pre-processing
The repository provides offline pre-processing stage that aims to serialize data to speed up training. Program parameters can be specified in the pre-processing config file.
To start the pre-processing, please run:
python3 run_preprocessing.py -
Training
The repository provides a comprehensive training program allowing to schedule multiple training experiments with hyperparameter search. Program parameters can be specified in the training config file.
To start the training, please run:
python3 run_training.py -
Evaluation (offline, inference-only)
The repository provides an evaluation program to quickly test the models. It allows to read the model's checkpoint and evaluate it on a desired test set. Program parameters can be specified in the evaluation config file.
To start the evaluation, please run:
python3 run_evaluation.py -
Demo program (text generation)
The repository provides a demo program which allows to use the language model for text generation. It loads the model's checkpoint and allows the user to type the input prompt for completion by the model. Program parameters can be specified in the demo config file.
To start the demo, please run:
python3 run_demo.py -
Deployment to Hugging Face
This project supports deployment to Hugging Face Model Hub for sharing the model weights.
To start the deployment, please run:
python3 run_deployment.pyLive web demo of AlpineLLM at Hugging Face Spaces is developed as a separate repository:
This repository uses GitHub Actions for automated testing and continous integration.
In addition to end-to-end pipelines, individual components can be started and tested independently:
-
Data loader component test
python3 -m core.dataloaders.dataloader_llm -
Evaluator component test
python3 -m core.evaluators.evaluator_llm
- Bartek Borzyszkowski
Web: borzyszkowski.github.io