Llama 3.2

[kunwarsahni01]

Would it be possible to convert/use Llama 3.2?

[smpanaro]

Yes, should be. It's on my short list of things to try.

[smpanaro]

@kunwarsahni01 Here's one if you want to try:

swift run LLMCLI --repo-id smpanaro/Llama-3.2-1B-Instruct-CoreML

Would be curious how many tokens/sec you get. I see ~16 on M1 Max.

[kunwarsahni01]

Getting some issues.

On MacOS 14.6.1:
Error: notFound

On MacOS: 15.0.1:
Error: The data couldn’t be read because it isn’t in the correct format.

[smpanaro]

On 14.6.1, have you pulled the latest main branch?

Not sure about 15.0.1, I haven't actually tried it there. Maybe delete the files in ~/Documents/huggingface/smpanaro/Llama-3.2-1B-Instruct-CoreML and try again?

[kunwarsahni01]

On 14.6.1, have you pulled the latest main branch?

Not sure about 15.0.1, I haven't actually tried it there. Maybe delete the files in ~/Documents/huggingface/smpanaro/Llama-3.2-1B-Instruct-CoreML and try again?

After pulling latest main on 14.6.1. Now seeing Error: The data couldn’t be read because it isn’t in the correct format.

[smpanaro]

Hm, I've never seen this. At least one person on twitter got it to work too.

Do you have Xcode? If you run it from within Xcode (hardcode the model name in CLI.swift), you can set an exception breakpoint and see where it's happening. That might help.

Another thing to try is to see if Llama 2 7B still works for you.

[adonishong]

Hm, I've never seen this. At least one person on twitter got it to work too.

Do you have Xcode? If you run it from within Xcode (hardcode the model name in CLI.swift), you can set an exception breakpoint and see where it's happening. That might help.

Another thing to try is to see if Llama 2 7B still works for you.

Thanks for your fantastic work, I have an M3 Max, with 40 cores GPU, 128GB Memory, system version is 15.2 (24C101),

Llama 2 7B works for me, here is the performance number
First run

swift run -c release LLMCLI --repo-id smpanaro/Llama-2-7b-coreml --max-new-tokens 80
Building for production...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'LLMCLI' complete! (0.13s)
ModelPipeline Llama-2-7b-hf (13 chunks)
Compiling models: *************
Loading models : *************
1 21882 6606 310 14653 431 310 278 16106 315 2696 11248 7738 278 7655 2722 515 607 26935 338
... ...

Compile + Load: 0.85 sec
Generate : 71.84 +/- 3.64 ms / token
13.95 +/- 0.51 token / sec

Second run
Building for production...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'LLMCLI' complete! (0.11s)
ModelPipeline Llama-2-7b-hf (13 chunks)
Compiling models: *************
Loading models : *************
1 21882 6606 310 14653 431 310 278 16106 315 2696 11248 7738 278 7655 2722 515 607 26935 338
... ...

Compile + Load: 0.81 sec
Generate : 71.95 +/- 3.46 ms / token
13.92 +/- 0.50 token / sec

For power consumption, asitop shows 8watt no matter with battery or AC power

WIth Llama-3.2-1B-Instruct-CoreML and Llama-3.2-3B-Instruct-CoreML, both get error
swift run LLMCLI --repo-id smpanaro/Llama-3.2-3B-Instruct-CoreML
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'LLMCLI' complete! (0.14s)
ModelPipeline Llama-3.2-3B-Instruct (16 chunks)
Error: The data couldn’t be read because it isn’t in the correct format.

swift run LLMCLI --repo-id smpanaro/Llama-3.2-1B-Instruct-CoreML
Building for debugging...
[1/1] Write swift-version--58304C5D6DBC2206.txt
Build of product 'LLMCLI' complete! (0.11s)
ModelPipeline Llama-3.2-1B-Instruct (6 chunks)
Error: The data couldn’t be read because it isn’t in the correct format.

[smpanaro]

Awesome! I've added your numbers to the README.

I'm a little suspicious of the first run load and compile time (have never seen the first and second so similar), so leaving it out until I get another data point.

The tokens/second is super interesting though, thanks for trying it!