Thanos - Method of Pruning LLMs

Official PyTorch implementation of Thanos, as presented in our paper:

Thanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compression
Ivan Ilin
GenAI Center of Excellence, King Abdullah University of Science and Technology, Thuwal, Saudi Arabia
Paper

@article{ilin2025thanos,
  title={Thanos: A Block-wise Pruning Algorithm for Efficient Large Language Model Compression},
  author={Ilin, Ivan and Richtarik, Peter},
  journal={arXiv preprint arXiv:2504.05346},
  year={2025}
}

You can also look my YouTube video about this method,

...or see my Blog post on it.

---

Setup

Installation instructions can be found in INSTALL.md.

Usage

The scripts directory contains all the bash commands to replicate the main results.

Below is an example command for pruning LLaMA-7B with Thanos, to achieve unstructured 50% sparsity.

python main.py \
    --model decapoda-research/llama-7b-hf \
    --prune_method thanos \
    --sparsity_ratio 0.5 \
    --sparsity_type unstructured \
    --save out/llama_7b/unstructured/thanos/ 

We provide a quick overview of the arguments:

• --model: The identifier for the LLaMA model on the Hugging Face model hub.
• --cache_dir: Directory for loading or storing LLM weights. The default is llm_weights.
• --prune_method: We have implemented four pruning methods, namely [magnitude, wanda, sparsegpt, thanos].
• --sparsity_ratio: Denotes the percentage of weights to be pruned.
• --sparsity_type: Specifies the type of sparsity [unstructured, 2:4, 4:8].
• --save: Specifies the directory where the result will be stored.

For structured N:M sparsity, set the argument --sparsity_type to "2:4" or "4:8". An illustrative command is provided below:

python main.py \
    --model decapoda-research/llama-7b-hf \
    --prune_method thanos \
    --sparsity_ratio 0.5 \
    --sparsity_type 2:4 \
    --save out/llama_7b/2-4/thanos/ 

Pruning LLaMA-2

For LLaMA-2 models, replace --model with meta-llama/Llama-2-7b-hf (take 7b as an example):

python main.py \
    --model meta-llama/Llama-2-7b-hf \
    --prune_method thanos \
    --sparsity_ratio 0.5 \
    --sparsity_type unstructured \
    --save out/llama2_7b/unstructured/thanos/

Acknowledgement

This repository is build upon the Wanda and SparceGPT repository.

License

This project is released under the MIT license. Please see the LICENSE file for more information.