How reproduce the perfermance mentioned in the paper?

[jianuo-huang]

I tried running the command python evals/eval_tasks.py --date 4-14 --eval-tasks ./evals/datasets/frames_test_set_200.csv --model-type LiteLLMModel --model-id deepseek/deepseek-reasoner --parallel-workers 1 --search-model-id deepseek/deepseek-reasoner, but the results I got were not as good as those reported in the paper.

[jianuo-huang]

DeepSeek Reasoner Evaluation Discrepancy and Tool-Calling Error
Problem Description
I'm trying to evaluate DeepSeek Reasoner on the Frames benchmark, but I'm encountering two issues:
Issue 1: Performance Discrepancy
I'm using the following command to evaluate DeepSeek Reasoner on the Frames benchmark:
bashpython evals/eval_tasks.py --date 4-14 --eval-tasks ./evals/datasets/frames_test_set_200.csv --model-type LiteLLMModel --model-id deepseek/deepseek-reasoner --parallel-workers 1 --search-model-id deepseek/deepseek-reasoner
This command only achieves 55.5% accuracy, which is significantly lower than the reported 75.3% accuracy. I'm wondering if I've missed some configuration details or if there are additional parameters needed to replicate the reported performance.
Issue 2: Tool-Calling Error
When trying to reproduce ODS-v1 with DeepSeek Reasoner using the following command:
bashpython evals/eval_tasks.py --date 4-14 --eval-tasks ./evals/datasets/frames_test_set --model-type LiteLLMModel --model-id deepseek/deepseek-reasoner --parallel-workers 1 --search-model-id deepseek/deepseek-reasoner --agent-action-type tool-calling
I receive the following error:
Give Feedback / Get Help: https://github.com/BerriAI/litellm/issues/new
LiteLLM.Info: If you need to debug this error, use `litellm._turn_on_debug()'.
Provider List: https://docs.litellm.ai/docs/providers
Error while generating or parsing output:
litellm.BadRequestError: DeepseekException - {"error":{"message":"deepseek-reasoner does not support Function
Calling","type":"invalid_request_error","param":null,"code":"invalid_request_error"}}
Questions

What parameters or configurations should I use to achieve the reported 75.3% accuracy on Frames with DeepSeek Reasoner?
Is it possible to use DeepSeek Reasoner with tool-calling for ODS-v1 evaluation, or is this functionality not supported by the model?

Environment Information

Model: deepseek/deepseek-reasoner
Evaluation script: evals/eval_tasks.py
Test set: frames_test_set_200.csv

[nxphi47]

how much you get on the full frames dataset ?

[jianuo-huang]

I achieved 55.5% accuracy on the first 200 samples of frames.

[nxphi47]

have you evaluated on the full set ?

[jianuo-huang]

not yet, but we will evaluate random 500 samples soon.

[NTU-P04922004]

not yet, but we will evaluate random 500 samples soon.

Hi @jianuo-huang,
Are you able to reproduce the results successfully?

[Wolfgang9999]

@jianuo-huang When evaluating on frames dataset, are you using the seach APIs or searching on offline dataset?