Use portable simd as a backup

[trimental]

Hello, impressive project. Maybe you could consider using Rust's portable simd library as a backup if avx2 is not available.

I made the following file however I think my system runs out of memory (8gb m1 MacBook) before I can test it, so maybe you could try it out and make sure it works.

It requires an unstable feature but you're already using one so it shouldn't be any dramas.

Thanks.

[Noeda]

TIL std::simd::* is a thing.

I have a few experiments I want to try first before I attend to porting rllama to other platforms but this looks interesting and I want to try it later. I have an 16GB M1 MacBook so I should also be able to test this, as this should be just about enough to run the LLaMA-7B in f16.

I'll get back to this. Thanks for showing that.

[trimental]

@Noeda No worries. Yep definitely better to get it right with one simd implementation before porting early and having to update them all 👍

[Chrusciki]

would you mind explaining what to do with that portable file?

[Noeda]

You would replace src/simd_support.rs with it. Currently that file in the repo is hard-coded to use AVX2 instructions only found on AMD64 platform. The code provided uses Rust's portable SIMD which (at least in theory) will work well on any platform that the std::simd::* operations support.

[Noeda]

@trimental I was revisiting this to look for portable SIMD options: https://gist.github.com/trimental/ef99199539f650775bc1d975387b657b

Do you happen to know if there's a "standard" way to cast f16 values to f32 with std::simd? I look at your code and it looks a lot like e.g.:

pub fn i16x8_as_f16_to_f32x8(a: I16x8) -> F32x8 {
    a.cast::<f32>()
}

Would cast an i16x8 tuple to f32x8 tuple, interpreting them as integers when the operation is actually supposed to interpret the values as f16 half precision floats.

I did some quick search in std::simd and not sure this exists. I would be okay switching to 3rd party SIMD library too, as long as it has the necessary support. I will be doing some more searching later.