Klavis Sandbox on Fireworks EP

[zihaolin96]

Note

Integrates Klavis sandbox lifecycle into EP pytest utilities and provides a working Gmail sandbox eval.

• New KlavisSandboxRolloutProcessor manages full sandbox flow: create/init (initialize_*_sandbox), generate temp MCP config, run Agent, export/dump sandbox state, attach results to row.execution_metadata.extra, cleanup and delete sandbox
• Exposes KlavisSandboxRolloutProcessor via eval_protocol.pytest.__init__
• Adds optional klavis dependency group (klavis>=2.18.0) and lockfile updates
• New test test_pytest_klavis_sandbox.py and dataset klavis_gmail_sandbox_test.jsonl: adapts JSONL to EP rows, runs against server_name="gmail", and scores via Fireworks LLM by comparing dumped sandbox state to provided ground_truth

^{Written by Cursor Bugbot for commit 7f35e24. This will update automatically on new commits. Configure here.}

See our slack channel for demo

[cursor]

Missing try/except for optional klavis dependency import

The KlavisSandboxRolloutProcessor is imported unconditionally at line 3, but it depends on the optional klavis package. This will cause an ImportError for any user who imports from eval_protocol.pytest without having klavis installed. Other optional dependency imports like PydanticAgentRolloutProcessor and LangGraphRolloutProcessor are correctly wrapped in try/except blocks (lines 16-22 and 25-31), but KlavisSandboxRolloutProcessor lacks this protection. The import and __all__ export at line 35 both need to be made conditional like the other optional dependencies.

Additional Locations (1)

• eval_protocol/pytest/__init__.py#L34-L35