Mistral-7b-PyTorch

• Implementation of Mistral 7b using PyTorch

• Look at the model params

Sliding window attention - Rolling buffer cache - Prefill and chunking - MoE -

Big thanks for Mr. Umar Jamil for providing the necessay help to a thorough implementation

https://www.youtube.com/watch?v=UiX8K-xBUpE