MacBook M4 Pro performance on MacOS

[heborras]

Hi,
thank you for this nice project! It's really cool to get an impression of the performance of the Neural Engine!

Below is a bit of data to add to the readme table :)
Run with the following commit from main: commit 9d141fc6a5182c0ce8befb2711284205cc928e10 (HEAD -> main, origin/main, origin/HEAD)

swift run -c release LLMCLI --repo-id smpanaro/Llama-2-7b-coreml --max-new-tokens 80
Building for production...
[23/23] Linking LLMCLI
Build of product 'LLMCLI' complete! (37.58s)
ModelPipeline Llama-2-7b-hf (13 chunks)
Compiling models: *************
Loading models  : *************
1 21882 6606 310 14653 431 310 278 16106 315 2696 11248 7738 278 7655 2722 515 607 26935 338 23892 29889 450 1023 1667 6606 7825 5584 18834 630 526 315 2696 11248 508 29872 561 2207 313 13716 29664 29897 322 315 2696 11248 25352 983 313 25822 983 467 13 1576 26935 8024 338 263 2319 3926 12692 14653 431 470 5447 393 338 7531 304 278 21881 12786 310 10557 29892 14325 29892 322 278 2163 5070 29889 450 8024 13880 2319 4796 18281 393 6668 290 297 24554 322 526 1248 1915 630 491 367 267 29889 450 18281 526 5643 491 2654 470 13328 7655 2722 393 1712 278 26935 367 550

<s> Several species of shrub of the genus Coffea produce the berries from which coffee is extracted. The two main species commercially cultivated are Coffea canephora (robusta) and Coffea arabica (arabica).
The coffee plant is a small evergreen shrub or tree that is native to the tropical regions of Africa, Asia, and the Americas. The plant produces small white flowers that bloom in clusters and are pollinated by bees. The flowers are followed by red or yellow berries that contain the coffee beans

Compile + Load: 73,75 sec
Generate      : 74,98 +/- 8,41 ms / token
                13,43 +/- 0,84 token / sec

Looking at asitop during execution gives exactly 8 W peak power utilization on the AIE.

With the following hardware info form the System Report, but all M4 MacBooks should have the same AIE version and size installed:

Hardware Overview:

  Model Name:	MacBook Pro
  Model Identifier:	Mac16,8
  Model Number:	Z1FE000ZMD/A
  Chip:	Apple M4 Pro
  Total Number of Cores:	12 (8 performance and 4 efficiency)
  Memory:	24 GB
  System Firmware Version:	11881.61.3
  OS Loader Version:	11881.61.3

[heborras]

Just for comparison. On this hardware ollama gives me the following for the same model:

$ ollama run llama2:7b --verbose
>>> Short story please
Of course! Here is a short story:

The sun was setting over the ocean, casting a warm orange glow over the waves. A young woman named Sarah sat on the beach, watching the water lap at her toes. She had just finished a long day of work and was looking forward to
relaxing with a good book and a cold drink.

As she sat there, lost in thought, a small bird landed nearby. It chirped and fluttered its wings, clearly agitated. Sarah looked up to see a group of children laughing and playing in the distance, their shouts and laughter
echoing across the beach.

The bird hopped closer to Sarah, chirping insistently. She reached out her hand and gently stroked its feathers, trying to calm it down. The bird seemed to understand, and soon it was sitting peacefully on her lap, looking up
at her with bright eyes.

Sarah smiled and stroked the bird's feathers again, feeling a sense of peace wash over her. She closed her eyes and listened to the sound of the waves and the bird's soft chirping, letting the stresses of the day melt away. As
the sun dipped lower in the sky, Sarah felt her worries disappear, replaced by a deep sense of contentment and tranquility.

total duration:       8.157335167s
load duration:        15.87675ms
prompt eval count:    23 token(s)
prompt eval duration: 2.432s
prompt eval rate:     9.46 tokens/s
eval count:           289 token(s)
eval duration:        5.707s
eval rate:            50.64 tokens/s

Note that this runs on the GPU and uses about 16 W of power. Though I'm not sure if the quantization of the model is really the same...

[Davidqian123]

Here is the benchmark data of Llama-3.2-3B-Instruct

ModelPipeline Llama-3.2-3B-Instruct (16 chunks)
Compiling models: ****************
Loading models  : ****************
128000 35862 9606 315 14362 392 315 279 64677 76233 12791 8356 279 63494 505 902 11033 374 28532 13 578 1403 1925 9606 54453 67166 527 76233 12791 649 24453 6347 320 14804 82979 8 323 76233 12791 52412 3074 320 7098 370 3074 570 10989 3074 374 6646 311 387 315 5190 4367 323 374 15042 520 1579 4902 21237 304 13918 449 23900 20472 323 1664 19158 2692 17614 13 4997 82979 374 810 8624 47056 323 8831 311 3139 11 719 706 264 25984 261 17615 323 374 

<|begin_of_text|>Several species of shrub of the genus Coffea produce the berries from which coffee is extracted. The two main species commercially cultivated are Coffea canephora (Robusta) and Coffea arabica (Arabica). Arabica is considered to be of higher quality and is grown at high altitudes in regions with mild temperatures and well-drained soil. Robusta is more disease-resistant and easier to grow, but has a harsher flavor and is

Compile + Load: 1.49 sec
Generate      : 210.45 +/- 9.56 ms / token
                4.76 +/- 0.16 token / sec

Looking at asitop during execution gives exactly 2.1 W peak power utilization on the AIE.
My hardware information is the same as above.