Any benchmark compare with mlx?

[mzbac]

thanks for the great project, just wondering if ANE doesn't have the fused kernel, does the attention layer faster than mlx? my understanding the ANE speed up should be more on the compute side but the mlx fused kernel speed up a lot on the memory part, I haven't get the around the ANE benefits yet. I have tried to follow Anemll's repo's example but I think I missed something which cause the coreml pipeline is 1.4 slower than mlx pipeline..

[johnmai-dev]

Not yet. I’m still experimenting. From the benchmarks, it seems quite hard to significantly improve utilization. https://github.com/maderix/ANE/blob/main/benchmarks/ANE_BENCHMARK_REPORT.md

[loretoparisi]

These were my latest benchmarks. I think on this M4 very few improvements can ben done, unless kernel fusion is implemented to reduce ANE calls. I tried already several attempt, but kernel panic occurred due to unspecified api calls issues.
We're essentially at the ANE dispatch ceiling (88 × 0.5ms ≈ 44ms): no amount of CPU optimization can push past ~22–23 tok/sec without reducing the dispatch ...as far as I know at this point.

 Current Performance — M4 MacBook Air, Qwen3.5-0.8B                                                                                                          
                                                                                                                                                              
  ┌──────────────────────────┬────────────────────────────────────────────────────────────┐                                                                   
  │          Metric          │                           Value                            │                                                                   
  ├──────────────────────────┼────────────────────────────────────────────────────────────┤                                                                   
  │ Generation speed         │ ~20–22 tok/sec                                             │                                                                   
  ├──────────────────────────┼────────────────────────────────────────────────────────────┤                                                                   
  │ Prompt speed (TTFT rate) │ ~19–21 tok/sec                                             │                                                                   
  ├──────────────────────────┼────────────────────────────────────────────────────────────┤
  │ TTFT @ 13 tokens         │ ~0.65s                                                     │                                                                   
  ├──────────────────────────┼────────────────────────────────────────────────────────────┤                                                                   
  │ TTFT @ 220 tokens        │ ~10.5s                                                     │                                                                   
  ├──────────────────────────┼────────────────────────────────────────────────────────────┤                                                                   
  │ ANE dispatches/token     │ 88 (24×first_proj + 24×o_proj + 24×fused_ffn + 16×LM head) │                                                                   
  ├──────────────────────────┼────────────────────────────────────────────────────────────┤                                                                   
  │ ANE dispatch ceiling     │ ~44ms/token = ~22 tok/sec theoretical max                  │                                                                   
  └──────────────────────────┴────────────────────────────────────────────────────────────┘                                                                   
                          

[mzbac]

Not yet. I’m still experimenting. From the benchmarks, it seems quite hard to significantly improve utilization. https://github.com/maderix/ANE/blob/main/benchmarks/ANE_BENCHMARK_REPORT.md

Interesting, maybe the training backprop has some issues there. For inference, the Qwen ASR I'm currently working on, I can see the ANE utilization around 40%.