Skip to content

PERFORMANCE.md

Latest commit

 

History

History
73 lines (54 loc) · 3.08 KB

PERFORMANCE.md

File metadata and controls

73 lines (54 loc) · 3.08 KB
Raw

MCP performance and observability

The MCP retrieval path keeps one initialized client per warm Next.js/Vercel runtime. Concurrent requests share initialization and can use the same MCP session. A closed session is retired after active calls finish, and a read-only query reconnects once for connection, session, or transport failures. Authorization and invalid-query failures are never retried.

Every /api/chat response includes phase metrics under meta:

  • elapsedMs: complete API-route duration
  • upstreamMs: retrieval and generation duration
  • retrievalMs and generationMs
  • mcpConnectMs and mcpQueryMs
  • mcpConnectionReused and mcpReconnects
  • mcpToolCalls and fallbackCount
  • hitCount and retrievalBytes

Do not treat these fields as a stable public analytics API without versioning them first.

Cold and warm benchmark

Configure .env.local with a dedicated read-only development key, then run:

npm run benchmark:mcp

Run npm run test:mcp for the localhost integration test that verifies concurrent initialization deduplication, warm reuse, read-only querying, and emitted metrics without external credentials.

Optional controls:

ANTFLY_BENCHMARK_COLD_ROUNDS=5
ANTFLY_BENCHMARK_WARM_ROUNDS=20
ANTFLY_BENCHMARK_LIMIT=6
ANTFLY_BENCHMARK_QUESTION="What is Antfly and how does hybrid search work?"

The benchmark compares new-session initialization plus retrieval with repeated queries on one warm session. It invokes only listTools and the read-only query tool. It exits with status 3 when a read-only credential advertises any write or administration tools.

Run from the same region as the deployed Vercel function. Track p50 and p95 separately for cold connection, warm retrieval, generation, and complete /api/chat latency.

Latency is a release gate, not the only objective. Before shipping a retrieval optimization, run broad product, technical setup, insufficient-evidence, and unrelated-question evaluations and compare answer coverage and citations with the current production baseline. A faster result that drops required concepts or promotes low-priority sources is a regression.

Retrieval is intent-sensitive: broad conceptual questions use semantic-first retrieval, while exact technical questions retain hybrid BM25 + semantic RRF. Track latency and quality separately for these paths rather than averaging away a regression.

Development baseline

Measured on 2026-08-05 with five cold sessions and twenty warm queries against antfly_docs:

Phase p50 p95
Cold MCP connect 633 ms 1,013 ms
Query after cold connect 463 ms 1,466 ms
Query on one warm session 336 ms 351 ms

The former per-question lifecycle cost approximately 1,096 ms p50 for connection plus query. Warm session reuse reduces that path to approximately 336 ms p50, a measured reduction of about 760 ms or 69% before answer generation.

The read-only development key still advertised batch, backup/restore, and create/drop tools. The benchmark did not invoke them, but Antfly Cloud should filter tools/list by effective key tier.