Community Update — Two Weeks (July 20 – August 3, 2026)

[missBerg]

Two weeks in one update. The through-line on main is token counting: /tokenize reached three more provider families and the Responses API got an input-token endpoint, so context-window budgeting no longer depends on which backend you routed to. The MCP spec Release Candidate landed on July 28 with a phased conformance plan behind it, a streaming parser bug was found to be silently zeroing usage on any provider that omits the space after data:, and both maintainer calls kept circling one structural question — how much MCP handling should stay in extproc at all. Thanks to everyone who shipped, reviewed, and filed.

🛣️ MCP: the spec RC landed, and a question about where handling lives

The next MCP protocol revision reached Release Candidate on July 28. Conformance work is tracked in #2323 by @Hritik003: fourteen sub-items covering stateless Streamable HTTP sessions, the removal of initialize/ping/logging/setLevel, InputRequiredResult and required resultType, cacheable results with ttlMs, and JSON-RPC error-code renumbering — all while staying compatible with clients and backends still on 2025-11-25. The phased proposal is now open as a PR in #2431.

The direction from the calls is to sequence this rather than land it in one go. Phase 0 is foundations with no behavioral change. Phase 1 leaves the modern-to-modern and legacy-to-legacy paths as they are and restructures the code so cross-era work has somewhere to go. Cross-era translation and TTL caching follow in Phase 2 and beyond. Two risks were flagged up front — the gateway has to recompute the outgoing MCP name header when it validates header-against-body, and x-ai-eg-mcp-route has to be present on every method, not just initialize. A POC comes before the full implementation.

The newer question, from this week's call: enough MCP functionality now exists natively in Envoy that some — possibly all — of it could migrate out of the extproc service. The argument for it isn't purity, it's config distribution. Once MCP behavior is expressed as routes rather than as extproc configuration, the per-route config a multi-tenant deployment has to ship stops being the binding constraint. Tracking issue: forthcoming.

The scale evidence that prompted it, from the multi-tenant MCP work in #2351: at 100 MCPRoutes with 100 backends each, the extproc config secret runs about 80% full, and larger configurations have pushed past 5MB even after the 8MB limit landed. MCPRoute carries far more per-route detail than AIGatewayRoute — the earlier scale work took 2,000 AIGatewayRoutes without the control plane straining, so the MCP ceiling arrives much earlier than that number suggests. If you're running MCP at multi-tenant scale, your numbers are worth adding to the thread.

On the AIGatewayExtensionPolicy proposal in #2364, the targeting model settled on headers plus target refs, with the reasoning for why section-name targeting alone isn't enough written into the proposal. It has since moved again: the proposal is being split so the API and use cases can be agreed before the implementation mechanics are argued, with explicit pre-route and post-route phases to pin down where in the filter chain each extproc runs. Worth waiting for the re-split before reviewing — which is why it isn't in the review list below. The underlying constraint driving it — the filter chain is fixed at the landing route, before AI Gateway recomputes the real route — is also what #2461 is circling.

✨ What's new

Provider & translation coverage

• /tokenize reached three more provider families — #2390 (GCP and AWS Anthropic), #2403 (Vertex AI Gemini) and #2404 (the Bedrock Converse API), all by @hustxiayang. Token counting through the chat-completion schema now works against Vertex and Bedrock Anthropic, Gemini on Vertex, and Bedrock's own CountTokens — so context-window budgeting works the same whichever provider you routed to.
• Input-token counting for the Responses API — #2297 by @hustxiayang. Implements /v1/responses/input_tokens, which takes a Responses API body and returns the input token count — Codex and similar clients can check context-window fit and estimate cost without paying for an inference call.
• Gemini function-call follow-ups stop getting rejected — #2366 by @a-nych. The translator now accepts thinking_blocks signatures and falls back to skip_thought_signature_validator, so standard OpenAI clients can complete a tool call against Gemini 3.x instead of hitting a 400 on the second turn. Fixes #2365.
• Correct streaming finish reason on newer Gemini models — #2399 by @hustxiayang. Tool calls on models like gemini-3.5-flash now report tool_calls rather than stop in streamed responses, which is what OpenAI-compatible clients branch on.
• additional_tools accepted on /v1/responses — #2459 by @xiaolin593. The field was missing from the schema, so requests carrying it came back as a 400 with cannot unmarshal unknown input type. It now parses.

Observability & rate limiting

• SSE data lines without a space after the colon — #2470 by @CodePrometheus. The space after data: is optional per the SSE spec, but the translators only matched the spaced form — so against a backend that omits it, responses passed through fine while the gateway observed nothing: zero token usage, no resolved response model, no tracing chunks. Anyone running usage-based metrics, cost attribution, or quota on such a backend was reading zeros. Related to #2149.
• reasoning_tokens in the access log — #2438 by @Aias00. Reasoning-model spend is no longer invisible to whatever reads your access logs. Closes #2437.
• Malformed rate-limit service addresses fail fast — #2363 by @0YHR0. quotaRateLimitServiceAddr used to treat every parse error as "no port supplied" and quietly default it; now hostnames and bare IPv6 literals still get the default port, and genuinely malformed values stop the extension server at startup instead of at request time. Fixes #2362.

Auth, security & stability

• Backend credentials redacted in debug logs regardless of the redaction flag — #2435 by @Aias00. With AIGW_DEBUG on and enableRedaction off — the aigw run default — the upstream filter's injected Authorization header was logged in cleartext even though the incoming one was already redacted. Header mutations are now always redacted when debug logging is on, matching what the RequestBody phase already did. Partially addresses #2436; Envoy's own debug logs remain out of extproc's reach.
• Malformed float response IDs no longer panic the MCP proxy — #2377 by @0YHR0. The proxy validates the decoded payload before reconstructing a JSON-RPC ID, so a client can't take down the proxy with a syntactically valid but malformed ID. Fixes #2376.

💡 New design proposals & feature requests

Two weeks of threads, clustered around MCP, quota, and the Responses API. Nine worth a read — and several already have a PR attached:

MCP & routing

• Dynamic multi-tenant MCP gateway — #2432 by @PraveenKumarInjam. The design behind #2351: one MCPRoute serving many tenants, with backends and allowed tools resolved per request instead of enumerated in a CRD. The open question raised on the call is precedence — whether a dynamically supplied tool list overrides a static include, and whether a static exclude should stay a global deny regardless. Worth a comment if you have an opinion on those semantics.
• Auto-derived OAuth resource metadata — #2406 by @ricjsan. Make protectedResourceMetadata.resource optional and derive it from the intersected Gateway/Listener hostname plus the route path, so staging and production stop needing separate MCPRoute manifests just to change a hostname.
• MCPRoute referencing a BackendTrafficPolicy — #2448 by @luisalves00. Would let MCP traffic reuse Envoy Gateway's existing rate-limit machinery instead of needing a parallel path.
• Configurable MCP tool name prefixing — #2425 by @walsm232. A prefixMode field to control how aggregated backends' tool names get namespaced, for setups where the current prefixing collides with what clients expect.

Model routing, quota & the API surface

• Gateway-side anthropic-beta filtering — #2411 by @hustxiayang. Anthropic clients send all their enabled betas in one comma-separated header, and a single value the upstream doesn't recognize fails the whole request — which is why tool search can't be turned on against Vertex today. Proposes a per-backend filter, denylist by default with alerting on dropped values, switchable to allowlist behind a config flag. Implementation is open in #2420.
• Model capability based translation — #2368 by @hustxiayang. Separating models into backends by capability, with header-based filtering in front. Explicitly looking for concerns and feedback before it goes further.
• Portable reasoning across Responses API backends — #2451 by @hustxiayang. Reasoning content and signatures behave differently per provider today; this proposes a shape that survives a backend switch.
• QuotaPolicy durations beyond 1d — #2469 by @stephane-segning. The duration enum caps at a day, which makes monthly and billing-period quotas inexpressible — a common ask from anyone charging back by tenant.
• Per-request AI usage event export — #2424 by @shiv4289. A proposal for emitting per-request usage events, aimed at the billing and chargeback cases that aggregate metrics can't answer.

👀 PRs looking for review

Two weeks' worth, so this list is longer than usual. All of these are open and waiting:

• #2092 — OAuth token exchange for upstream MCP auth by @nacx
• #2351 — Per-request backend fan-out subset and allowed tools via header by @PraveenKumarInjam — still the first building block of the multi-tenant MCP work
• #2405 — OpenTelemetry GenAI semantic conventions for tracing by @dergraf
• #2415 — Backend-only authorization pre-check for the MCP initialize phase by @pbonuboinasnow
• #2420 — Per-backend anthropic-beta filter by @hustxiayang — the implementation of #2411
• #2423 — Forward api-key client-id header to backends by @mohitgurnani — gives api-key auth the same caller attribution OAuth already has on MCP metrics, spans, and access logs (#2422); it's been waiting a fortnight now
• #2427 — Trigger extproc pod rollout on injected-config drift by @Aias00 — hashes the injected sidecar config so changes to extProcExtraEnvVars actually reach running pods (#2395)
• #2431 — Proposal for the new MCP Release Candidate by @Hritik003 — the phased conformance plan behind #2323
• #2446 — Allow excluding models from /v1/models by @Killusions — hidden model aliases, from #2445
• #2468 — Model OpenAI prompt cache fields for transparent forwarding by @Aias00 — the implementation of #2454

If your own PR has been sitting: MAINTAINERS.md lists which maintainer owns which area of the codebase. Tagging the owner directly on the PR is welcome, not an imposition.

🙏 Thanks to the last two weeks' contributors

@hustxiayang, @a-nych, @0YHR0, @Aias00, @mohitgurnani, @siddharth1036, @Hritik003, @nacx, @mathetake, @shiv4289, @walsm232, @ricjsan, @toffentoffen, @VadimEisenberg, @PraveenKumarInjam, @pbonuboinasnow, @kanurag94, @bannu8615, @ajac-zero, @sivanantha321, @dergraf, @Killusions, @nlenepveu, @MVS-source, @yuehaii, @baiyutang, @Sarthak160, @pyhp2017, @FAUST-BENCHOU, @mksinghtx, @CodePrometheus, @xiaolin593, @jeremypng, @luisalves00, @stephane-segning, @AyushSawant18588, @rafaelgaspar, @darox, @Xunzhuo, @yanavlasov, @johnugeorge, @botengyao, @missBerg, @nuthalapativarun, @Flgado, @dragon3 — and everyone who showed up to triage, review, and discuss.

Back to the usual weekly cadence next week!