Replicating Apple's LLM benchmarking

[meghsat]

Hey all,

I came across this article: https://machinelearning.apple.com/research/exploring-llms-mlx-m5, where Apple claims to have achieved 2.87 sec TTFT on the MacBook Pro M5-24GB for the GPT-OSS-20B-MXFP4-Q4 model using MLX. However, I can’t seem to replicate those numbers — I’m getting a TTFT of ~8 sec.
Note: None of the models listed in the article are performing as claimed.
Here’s my benchmarking setup:

1. To measure TTFT, I had to modify the mlx_lm/generate.py script. Here’s the PR containing those changes: https://github.com/ml-explore/mlx-lm/pull/633/files
2. Once you add the TTFT logic, please run this code:

import mlx.core as mx
from mlx_lm import load, generate

model, tokenizer = load(
    "mlx-community/gpt-oss-20b-MXFP4-Q4", # You can replace with any model ID mentioned in the article.
    tokenizer_config={"trust_remote_code": True}
)

mx.eval(model.parameters())

vocab_size = tokenizer.vocab_size
prompt_length = 4096

mx.random.seed(0)

dummy_tokens = mx.random.randint(0, vocab_size, (prompt_length,)).tolist()

tokenizer._eos_token_ids = {}

# warmup
response = generate(
    model, 
    tokenizer, 
    prompt=dummy_tokens, 
    max_tokens=128, 
    verbose=True,
    prefill_step_size=4096 
)

# Actual run
response = generate(
    model, 
    tokenizer, 
    prompt=dummy_tokens, 
    max_tokens=128, 
    verbose=True,
    prefill_step_size=4096  
)

It would be great if anyone has observed similar or different results and could share their setup here. Thanks in advance.

[awni]

What's your hardware / OS? To reproduce those numbers you need the latest MLX (0.30.0) on macOS 26.2 (beta release) on the M5.

[meghsat]

Thank you for pointing it out. I'm on macOS 26.1.

[meghsat]

I am able to reproduce the numbers using MacOS 26.2. Thanks!

[guruswami-ai]

If you want to compare those numbers to M3 Ultra (stand-alone or clustered) or run some more tests for comparision I've published a bunch at https://github.com/guruswami-ai/mlx-benchmarks. I haven't got my hands on an M5 system yet, but your results seem impressive. I'm looking forward to building a distributed M5 mesh if/when they are available. To date, the lesson seems to be fit everything into one node if you can. An M5 Ultra with 512GB RAM will be impressive and likely bridge the gap to NVIDIA GPU hardware even more.

I need to update my benchmarks https://github.com/guruswami-ai/mlx-benchmarks/blob/main/docs/APPLE_SILICON_GUIDE.md and the 'cluster simulator'at https://chakra.guruswami.ai with your M5 results.