falcon-nix

Environment for running Falcon-40B on NixOS

Instructions

Download TheBloke's experimental 3-bit AutoGPTQ https://huggingface.co/TheBloke/falcon-40b-instruct-3bit-GPTQ
Everything should be nixified so on NixOS just do: nix run

By default, the falcon test app in this repo expects to find the dataset in ../falcon-40b-instruct-3bit-GPTQ

Using Geforce RTX 4090, expect it to take few minutes to generate a lama story.

Name		Name	Last commit message	Last commit date
Latest commit History 4 Commits
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
auto_gptq.nix		auto_gptq.nix
falcon		falcon
falcon.nix		falcon.nix
flake.lock		flake.lock
flake.nix		flake.nix
format.sh		format.sh
rouge.nix		rouge.nix
setup.py		setup.py