OpenAI o1-preview and o1-mini

[EwoutH]

Very curious about OpenAI o1-preview and o1-mini MMLU-Pro scores. Opening this issue as a tracking issue that people can follow and updates can be shared in.

[billbradley]

I headed to the issues page to post the same request, so instead I will cheer on this one :)

In case it may be helpful: as compared with other LLMs, o1-preview is (1) remarkably slow, (2) remarkably expensive and (3) remarkably different (and sometimes better) than any other LLM I've used. I'm extremely interested in finding out how it performs on MMLU-Pro.

[billbradley]

Hmm... This tweet would seem to suggest that the researchers at TIGER have already evaluated o1-preview last month:
https://x.com/WenhuChen/status/1834605218018754581

We would love to see this data reflected in the GitHub repo!

[wenhuchen]

We were capped by the request/day before. But we can definitely rerun it now with higher quota.

[EwoutH]

Curious if this still can be done, even if only for a historical comparison once the full o1 is out.

These are the first "reasoning" models in history, so it would be nice to have them for future reference.

[EwoutH]

Thanks for adding all the models in the last few days!

Regular o1 is now also available in the API. Might be a nice moment to test all three.

[wenhuchen]

It's very expensive. We are not sure whether we have the fund to run it.

[billbradley]

You know, if the accuracy for o1 is in the neighborhood of 80%, then with 1600 samples I think the standard deviation should be about:

+/- sqrt(0.8 * 0.2 / 1600) = +/- 1%

So, what do you think about evaluating on a random subsample of 1600? The approximate answer might still tell us a lot.

[EwoutH]

Would using the Batch API also help?

[billbradley]

Of course, the landscape has changed since this question was first asked (with o1 fairly accessible, o3 in the wings, and other reasoning models in play), but it would still be very interesting to learn a little more about how these models perform on MMLU-Pro.

[billbradley]

OK, at this point o1-mini looks increasingly like it's in the rearview mirror (and o3-mini would be a better candidate). But thinking about this more broadly, perhaps the real request is "how does a frontier reasoning model perform?", regardless of the particular model.

For example, I noticed this figure claiming to show performance across the DeepSeek R1 Distilled models, and Hugging Face claims that the full model achieves 84%. If I'm interpreting Amazon Bedrock pricing correctly, it looks like DeepSeek R1 costs about half as much as GPT-4o, so perhaps it might be possible to replicate the result (or grab someone else's run, if they're trustable)?

[billbradley]

OK, in case anyone else finds this thread and is interested comparing MMLU-PRO performance between models, this leaderboard may be of interest:
https://www.vals.ai/benchmarks/mmlu_pro-03-24-2025
It includes some of the models discussed above, in addition to other reasoning models (e.g., Claude 3.7 Sonnet).

That said, TIGER Lab's repository contains much more thorough information on the evaluation (e.g., the specific responses). This is pretty important for my own research purposes and I imagine for a number of other people, so there's still unmatched value in this repo.