A high-performance, from-scratch implementation of Llama 3.2 (1B/3B) with instant startup times and optimized inference.
- From-scratch implementation of Llama 3.2 architecture
- Instant startup after first load (similar to Ollama) - 17,000x speedup
- High-performance inference - 17+ tokens/second on Apple Silicon
- Multiple model sizes - 1B and 3B parameter variants
- Optimized for Apple Silicon with MPS acceleration
- KV caching for efficient generation
- Clean, educational code based on Sebastian Raschka's "Build a Large Language Model From Scratch"
| Metric | Performance |
|---|---|
| Startup Time | 0.000s (after first load) |
| First Load | ~20-25s |
| Generation Speed | 17-25 tokens/second |
| Memory Usage | ~6-7GB for 3B model |
# Clone the repository
git clone https://github.com/yourusername/cogai4sci.git
cd cogai4sci
# Install dependencies
pip install torch safetensors tiktoken huggingface_hub blobfilefrom fast_llm import FastLLM
# Initialize model - first time loads model, subsequent times are instant!
model = FastLLM("meta-llama/Llama-3.2-1B-Instruct", use_float16=True)
# Generate text with streaming
prompt = "What is the future of AI?"
print(prompt, end="")
for token in model.generate(
prompt,
max_new_tokens=100,
temperature=0.8,
top_k=40,
use_kv_cache=True
):
print(token, end="", flush=True)Open comprehensive_llama_demo.ipynb for a complete interactive guide with all features, benchmarks, and educational content.
cogai4sci/
├── fast_llm.py # Main FastLLM class with instant startup
├── comprehensive_llama_demo.ipynb # Complete interactive demo notebook
└── README.md # This documentation
- Class-level model caching for instant subsequent loads (similar to Ollama)
- Optimized weight loading for both 1B and 3B models
- Streaming token generation with proper tokenization
- KV cache optimization for faster inference
- Complete architecture including RoPE, GroupedQueryAttention, FeedForward
- Proper dtype handling for Apple Silicon compatibility
- Comprehensive tokenizer with special token support
- Transformer-based with grouped query attention
- RoPE (Rotary Position Embedding) with proper scaling
- SwiGLU activation in feed-forward networks
- RMSNorm for layer normalization
- Weight tying between embedding and output layers
- Mixed precision (float16/bfloat16) for memory efficiency
- KV caching for faster autoregressive generation
- Repetition penalty with sliding window
- Top-k sampling for better text quality
- MPS acceleration on Apple Silicon
- Italian text generation bug - Fixed tokenizer chat mode triggering
- 3B model loading - Added support for multi-file safetensors
- Performance optimization - Achieved 17+ tokens/second
- Memory efficiency - Reduced memory usage with proper dtype handling
- Startup time - Implemented model caching for instant subsequent loads
All benchmarks, tests, and demonstrations are included in the comprehensive notebook:
jupyter notebook comprehensive_llama_demo.ipynb- Educational purposes - Learn LLM internals
- Research experiments - Modify architecture easily
- Local inference - Run models without internet
- Performance testing - Benchmark different configurations
- Prototype development - Quick iteration on LLM applications
This project is based on educational materials from Sebastian Raschka's "Build a Large Language Model From Scratch". Contributions welcome!
- Build a Large Language Model From Scratch by Sebastian Raschka
- Llama 3.2 Model Card
- RoPE: Rotary Position Embedding
This project is released under the Apache 2.0 License. See LICENSE for details.
# Complete interactive demo with all features
jupyter notebook comprehensive_llama_demo.ipynb
# Quick Python usage
python -c "from fast_llm import FastLLM; model = FastLLM('meta-llama/Llama-3.2-1B-Instruct'); print(''.join(model.generate('Hello world', max_new_tokens=20)))"Note: First model load requires downloading weights from Hugging Face (~2.5GB for 3B model). Subsequent loads are instant thanks to our caching system!