nanoLLMs

Implementation of three different type of LLMs that are GPT2 , Mamba and xLSTM.

Some examples of Mamba models are Mistral's Codestral Falcon series, xLSTM's model from NXAI.

All models were trained on Kaparthy's educational example on Shakespeare text.

Model implementation was done and partially inspired by :

• GPT2 : https://github.com/karpathy/nanoGPT
• MAMBA : https://github.com/alxndrTL/mamba.py
• xLSTM : https://github.com/NX-AI/xlstm

Installation

Please run, the following command to install relevant dependencies:

pip install .

To prepare the dataset, please run (which is the same command as in nanoGPT).

python data/shakespeare_char/prepare.py

You may want to run small LLM to generate Shakespeare-like text, please check out the notebook notebooks/shakespeare_char.ipynb that trains GPT, Mamba and xLSTM models from scratch.