deps: migrate mcp SDK to 2.x (land PR #410)

[WalksWithASwagger]

Summary

Land the mcp 1.x → 2.x major upgrade (PR #410) with a reviewed compatibility check, because mcp is the SDK behind Rafiki's entire MCP server surface.

Context

• Tracks Dependabot PR build(deps): update mcp requirement from >=1.0.0 to >=2.0.0 #410 (mcp >=1.0.0 → >=2.0.0).
• The MCP tool registry (rafiki_* tools) is built on this SDK; a breaking change can silently drop or reshape tools.
• Relevant: the MCP envelope contract tests (tests/test_mcp_envelope_contract.py) and docs/MCP.md coverage test (Add test asserting MCP tool registry matches docs/MCP.md coverage #210 pattern).

Acceptance Criteria

• Review the mcp 2.x changelog for breaking changes to server/tool registration, request context, and envelope shapes.
• MCP server code updated for 2.x if needed; the tool registry still matches docs/MCP.md.
• MCP envelope-contract tests pass; full test gate green on the up-to-date PR head.
• npm run verify and npm run verify:security pass.

Notes

• needs-human: core-runtime major. Sequence after google-genai 2.x (build(deps): update google-genai requirement from >=1.0.0 to >=2.16.0 #407), before tenacity 9.x (build(deps): update tenacity requirement from >=8.0.0 to >=9.1.4 #408). See deps: batching and compatibility plan for the 13 open PRs (corrects #308 snapshot) #420.

[WalksWithASwagger]

mcp 1.x → 2.x migration — review-ready PR up: #430 (supersedes #410)

PR: #430 (branch codex/deps-mcp2, off latest main). Please close #410 in its favour — I did not touch it.

Diagnosis (both failure modes present in the naive #410 bump):

1. Real API break. mcp 2.0 removed the vendored mcp.server.fastmcp module. The high-level server class moved to mcp.server and was renamed FastMCP → MCPServer. Constructor kwargs (name, instructions), the @tool() decorator (still returns the original callable), async list_tools() (still yields .name objects), and run() (still defaults to stdio) are all unchanged — so the code fix is a two-line rename in mcp_server.py. All 25 rafiki_* tools register/serve identically; envelope contract + tool registry unchanged (no docs/MCP.md or test_mcp_envelope_contract.py edits needed).
2. Stale hashed lock. requirements.txt → mcp[cli]>=2.0.0; requirements-ci.txt regenerated under Python 3.11 + pip-tools 7.6.0 + pip 26.1.2 to match CI's Check Python CI lock step. Delta: mcp 1.28.1→2.0.0; adds mcp-types, httpx2, httpcore2, opentelemetry-api, truststore; drops httpx-sse, pydantic-settings.

CI on #430 — GREEN:

• test: pass (2m31s) ← the gate build(deps): update mcp requirement from >=1.0.0 to >=2.0.0 #410 fails
• secret-scan: pass
• CodeQL / Analyze (python) / Analyze (js-ts) / dependency-review: pass

Local validation (Python 3.11.10, mcp 2.0.0): full pytest 566 passed / 1 skipped · ruff check . (pinned ruff 0.15.22) all-clear · lock hash-validated for CI platform (85/85 cp311 manylinux wheels via pip download --require-hashes) · pip_audit clean · offline in-process MCP stdio round-trip (initialize → tools/list → tools/call) returned 25 tools, ok:true.

⚠️ LIVE MCP SMOKE (server starts, tools list/round-trip) BY MAINTAINER REQUIRED BEFORE MERGE. Green CI + the in-process round-trip cannot prove the live server registers/serves tools correctly to a real client (Claude Desktop / Codex) with real config + environment. Do not auto-merge.

Residual risk: the lock step regenerates + git diff --exit-codes the lock. It reproduced byte-for-byte on this CI run, but if any of the 85 pinned packages publishes a new PyPI release before a later re-run, the diff step could need a lock regen — inherent to the repo's regenerate-and-diff design, not this change.