Feature Request: Support ZAYA1-8B (Sparse MoE) and Markovian RSA Sampling

[Juste-Leo2]

Prerequisites

• I am running the latest code. Mention the version if possible as well.
• I carefully followed the README.md.
• I searched using keywords relevant to my issue to make sure that I am creating a new issue that is not already open (or closed).
• I reviewed the Discussions, and have a new and useful enhancement to share.

Feature Description

ZAYA1-8B is a sparse Mixture-of-Experts (MoE) model with 760M active parameters (8.4B total) developed by Zyphra. It introduces a reasoning method called Markovian RSA (Rational Speech Acts) sampling, which optimizes the chain-of-thought during inference to improve logical accuracy.

Motivation

It is an ultra-sparse model, making it ideal for local use.
Its performance is SOTA for its size

Possible Implementation

the GitHub fork of vllm with the following commit (Zyphra/vllm@641dc7a)
Zyphra's Technical Report: https://www.zyphra.com/zaya1-8b-technical-report

[mamama1]

oh yes, please! can't wait

[Ramachandrajoshi]

I also came here to create a feature request, and saw this.

[Putnik24]

I can't wait

[Juste-Leo2]

I'm excited too! However, we have to keep in mind that supporting this model involves quite a few technical challenges. For my part, I'm just trying to implement the missing building blocks to help pave the way for a future implementation.

[Juste-Leo2]

I created a branch using OpenCode to support the GGUF conversion, basing the overall implementation on the vLLM repository. The conversion script and a part of the C implementation are done. However, I am currently stuck at the LLM inference stage.

As it stands, this branch is probably not ready for a PR. The code heavily relies on AI generation, and I cannot guarantee that it hasn't inadvertently broken the logic for other existing models.

Here is the current state of the error I am facing:

~/github/llama.cpp$ ./build/bin/llama-cli -m zaya-test.gguf

Loading model... |llama_model_load: error loading model: error loading model hyperparameters: key not found in model: zaya.attention.layer_norm_rms_epsilon
llama_model_load_from_file_impl: failed to load model
common_fit_params: encountered an error while trying to fit params to free device memory: failed to load model
/llama_model_load: error loading model: error loading model hyperparameters: key not found in model: zaya.attention.layer_norm_rms_epsilon
llama_model_load_from_file_impl: failed to load model
common_init_from_params: failed to load model 'zaya-test.gguf'
srv    load_model: failed to load model, 'zaya-test.gguf'

Failed to load the model

I am dropping the branch here. Feel free to rework it, build upon it, or use it as a reference if anyone wants to push this implementation forward in the future.

https://github.com/Juste-Leo2/llama.cpp/tree/CCA

[nanduruganesh]

Thank you for the initial push @Juste-Leo2! I haven't worked with llama.cpp yet but I would also like to get this merged so people can use the model with GGUF. I forked your branch and have HF conversion + generation working in this draft PR, though there is definitely more work to be done. Very open to collaboration with anyone interested in this, please reach out or PR to our fork.

[Juste-Leo2]

Thank you for the initial push @Juste-Leo2! I haven't worked with llama.cpp yet but I would also like to get this merged so people can use the model with GGUF. I forked your branch and have HF conversion + generation working in this draft PR, though there is definitely more work to be done. Very open to collaboration with anyone interested in this, please reach out or PR to our fork.

Yes, thank you very much for your feedback, I would be delighted to collaborate with you. I'll reach out to you so we can integrate the model together :)

[kgable11]

i cannot wait for this! unfortunately i am not a coder so i can not really contribute to the effort... but thanks for the hard work!

[hg0428]

MLX has a open PR that makes it work and so does vLLM! Hopefully Llama.cpp doesn't fall behind.

[Juste-Leo2]

i cannot wait for this! unfortunately i am not a coder so i can not really contribute to the effort... but thanks for the hard work!

If you'd like, you can already try out this working PR; I think the draft on llama.cpp will be available soon :)