Reproducibility Issues with MCPMark Benchmark

[H-Sompalle]

Reproducibility Issues with MCPMark Benchmark

Specific Issues Encountered

1. Notion Server: Zero reproducibility achieved
2. GitHub Server: Fails to run properly
3. Playwright Server: Docker image size causes practical deployment issues
4. Repository Navigation Tasks: Difficult to reproduce consistently

Error traces suggest the benchmark primarily tests file structure navigation rather than MCP functionality

Based on error patterns, the benchmark appears to reduce to: "Can an LLM navigate file structures?" rather than testing the broader capabilities that MCP servers are designed to provide.
While sandboxing as a benchmark concept is excellent in theory, the current implementation challenges raise questions about whether it's measuring the right capabilities.

Has anyone successfully:

• Reproduced MCPMark results with >80% success rates?
• Created a working setup they can share?
• Identified specific environment/configuration requirements not covered in the docs?

This issue is opened in good faith to help the community understand and properly use MCPMark. Shared reproduction steps, configurations, or insights would benefit everyone trying to benchmark their MCP implementations.

[zjwu0522]

pls make sure you read the readme thoroughly.

[H-Sompalle]

This is going on: https://github.com/H-Sompalle/mcpmark-issues

[zjwu0522]

@H-Sompalle I went though your log, seems there is no problem except the pass rate (3/28). I am currently rerun o3 for notion and will get back to you asap. Notion team does changed their api after we release mcpmark and some legacy experiments may be affected. We are also investigating into if we need to update the notion part to fit the new notion mcp/api server.

BTW, do let me know if you still cannot run pg/gh/pw. Make sure you follow the guidance in the readme. It does take some time because we have local/remote services so cannot deliver it using one single dockerfile.

[H-Sompalle]

Updated it. Playwright is 64gb so I can't evaluate it right now. GH is 3/23 and PG is higher than expected with 47%. I used O3 for everything. Filesystem seems to be fine now.