Would you train this model on more corpus?

[yucc-leon]

According to the blog, EvaByte only consumed 0.5T tokens but matched the performance of models trained on 5x corpus. Thus, the new architecture is at least more efficient in utilizing the training data.
Will it be much better if it uses the same amount of corpus?

[LZhengisme]

Thanks for your interest! Yes, with more training data, EvaByte would likely perform even better :) We capped it at 1.5T bytes (~0.5T tokens) due to compute constraints, but our evals during pretraining showed steady improvement without any sign of plateauing (more details can be found in the Empirical Results section of our blog post). Hope this helps!

[win10ogod]

@LZhengisme Hello, is there any new news?
Why still no training code?

[LZhengisme]

@LZhengisme Hello, is there any new news? Why still no training code?

Hey @win10ogod totally get the interest, but migrating from specialized hardware to GPUs takes time. We're on it and plan to drop a few triton kernels for training support (hopefully in a few days). Appreciate your patience :)

[win10ogod]

@LZhengisme Hello, is there any new news? Why still no training code?

Hey @win10ogod totally get the interest, but migrating from specialized hardware to GPUs takes time. We're on it and plan to drop a few triton kernels for training support (hopefully in a few days). Appreciate your patience :)

I'm willing to wait for an excellent project.

[pszemraj]

@LZhengisme any updates?? It's been 8+ months

[LZhengisme]

@LZhengisme any updates?? It's been 8+ months

Hey, thanks for the ping. Since my last comment, we’ve been working on and open-sourced the Triton kernels for training (see this commit from Feb 28), so it should now be possible to run and train EvaByte-style models on GPUs.

[pszemraj]

Cool! From what I remember you guys said more training was in-progress (as well as a paper).

Can you share any updates/details there?

[LZhengisme]

Thanks for the interest :)) The work is still ongoing, we’ve been experimenting with some new ideas on top of the EvaByte-style architecture, but they are not quite ready to release yet

We’re not currently planning a standalone paper just on the original EvaByte run, but if you’d like a more detailed overview than the blog post, here’s the slide deck from my April talk, which was part of the ASAP seminar series. ASAP is a really nice virtual seminar focused on efficient, algorithmic perspectives on sequence modeling and foundation models — lots of great talks and resources there. Hope this helps!

[pszemraj]

thanks for the transparency and the deck! no problem & understandable, iterating on ideas/experiments take time-especially when it seems there is little/no new published work on byte-level modeling from others.

Will give the deck a more detailed read over the next few days. Hopefully a useful suggestion: have you guys compared/looked at the RAT (Recurrent Attention Transformer) model as a solution for the long context modeling needs? It came out in June, but I didn't come across it till recently (when checking for updates on megalodon-related papers). Seems worth considering/comparing against-decoupled from the multibyte prediction part-as it's refreshingly simple and (apparently) performant.

Can create a new issue on this repo if you prefer