PipSync MCP Evals

Public safety evaluations and a no-execution MCP mock for trading-oriented tool designs.

The repository models three boundaries:

• read-only synthetic signal listing;
• a deterministic, non-executable paper preview;
• a live-capable confirmation schema that the public mock always blocks.

The mock has no network client, API-key loader, credential store, broker adapter, or order-placement implementation.

Run the evaluations

PYTHONPATH=src python -m pipsync_mcp_evals.evals
PYTHONPATH=src python -m unittest discover -s tests -v

Run the stdio mock

PYTHONPATH=src python -m pipsync_mcp_evals.server

It accepts one JSON-RPC message per line and implements the modern, stateless MCP 2026-07-28 surface: server/discover, tools/list, and tools/call. Every request must carry io.modelcontextprotocol/protocolVersion and io.modelcontextprotocol/clientCapabilities in params._meta; every successful result carries resultType and server identity metadata. Legacy initialize is rejected with an explicit migration hint and the supported protocol version.

What the evaluations enforce

• Every tool has explicit MCP annotations.
• Tool input schemas reject unknown fields.
• Confirmation is classified as non-read-only and destructive.
• Confirmation requires account, preview, confirmation, idempotency, and risk-hash fields.
• Missing confirmation data fails closed.
• Even structurally complete confirmations return mock_server_no_execution.
• Preview output is deterministic and explicitly non-executable.
• Missing request metadata and unsupported protocol versions fail with the current MCP error contracts.

Annotations are hints, not an authorization boundary. Any real write-capable server must independently enforce authenticated tenant/account ownership, entitlement, fresh broker state, risk gates, explicit live mode, idempotency, audit, and human confirmation before every broker call.

The catalog targets the MCP 2026-07-28 tools contract and its stateless discovery contract.

Continue with PipSync

This repository deliberately cannot execute. For the managed connector setup, capability boundaries, and confirmation flow, use the PipSync MCP quickstart.

License

Apache-2.0.