feat(eve): capture gateway correlation details on model-call failures by shaper · Pull Request #1502 · vercel/eve

shaper · 2026-08-01T04:42:47Z

Summary

Diagnosing a gateway stream failure from an eve log is not currently possible. The error ids in the log are local crypto.randomUUID() values, and the structured details payload — which already extracts generationId — was attached only when the semantic-error catalog recognized the failure. On unrecognized failures, the ones hardest to diagnose, eve logged a raw inspector dump with no correlation fields at all.

Surfaced by AIG-387: the agent-tools / zai-glm e2e job failed with gateway_stream_terminated, and pinning down the cause took a day of timestamp-correlating gateway logs. The actual cause was an alibaba mid-stream rate limit.

Pairs with vercel/ai-gateway#2934, which puts generation_id, provider, and model on the synthesized frame. Merge that first — these extractions are only exercised against the enriched payload.

Changes

findGatewayError recognizes gateway-synthesized frames. They key on code + origin, not a Gateway* class name or a *_error type, so they were never matched — meaning the new fields would have been silently dropped even after the gateway started sending them.
Extract provider and model, and read generation_id alongside generationId since the frame is snake_case. provider matters because a canonical slug like zai/glm-5.2 is served by many upstreams (alibaba, novita, fireworks, nebius, …), so the failing one is not inferable from the model id.
Always log details when non-empty. Recognized failures still omit the raw error to bypass the [object Object] inspector rendering of APICallError cause chains; unrecognized ones keep it for the stack.
New gateway-stream-terminated catalog rule tagged transient.

The behavior change worth reviewing

That last item is not just observability. The frame carries no status code and is not marked retryable, so it fell through classifyModelCallError to recoverable — which means zero in-process retries and an immediate session park. A transient upstream throttle killed the turn on first contact.

The transient tag reclassifies it to retry, earning the standard 3 attempts with 500ms exponential backoff plus jitter. This is very likely what stops static-tools/web-search-parallel from flaking on glm-5.2.

I called this out separately rather than folding it in silently, since it changes runtime behavior and not just log content. Happy to split it into its own PR if you would rather land the extraction first.

What this deliberately does not capture

An earlier revision also consumed an upstream_error field carrying the raw provider message. That has been dropped on both sides. eve's logs reach public CI output, and provider error bodies echo request content — per the #help-ai-gateway investigation, ~5,442 rows in three days had a customer's live Anthropic API key land verbatim in a recorded upstream error. That detail stays gateway-side until the redacting extractor on AIG-394 lands.

generationId is the handle instead, and it recovers most of the value: it turns a timestamp-correlation hunt across a high-volume production log into a single query. The rule's hint points operators at it.

Testing

pnpm test:unit — 5819 passed, 1 skipped
pnpm test:integration — 575 passed
pnpm typecheck, pnpm fmt, pnpm lint, pnpm guard:invariants all clean
New coverage: correlation-field extraction off a synthesized frame, the retry classification, the catalog rule, and a regression guard that details survives on unrecognized failures

E2E: the agent-tools / zai-glm job is the real check. Once the gateway side deploys, a surviving failure should log generationId + provider directly instead of needing timestamp correlation.

🤖 Generated with Claude Code

vercel · 2026-08-01T04:42:53Z

The latest updates on your projects. Learn more about Vercel for GitHub.

Project	Deployment	Actions	Updated (UTC)
eve-docs	Ready	Preview	Aug 1, 2026 5:08am
eve-docs-4759	Ready	Preview, v0	Aug 1, 2026 5:08am

github-actions · 2026-08-01T04:45:52Z

Bundle + Package Summary: `apps/fixtures/weather-agent`

Key takeaways

No notable deltas vs main (6ad578a).

Delta vs `main (6ad578a)`

Area	Metric	Baseline	Current	Delta
Package	Packed tarball	7.59 MB	7.59 MB	-398 B ✅
Package	Unpacked publish size	28.44 MB	28.44 MB	+763 B ⚠️
Package	Installed footprint	85.43 MB	85.43 MB	+763 B ⚠️
Package	Published files	2952	2952	0
Package	Installed files	6747	6747	0
Runtime	Unique function payloads	2	2	0
Runtime	Total function bytes	16.91 MB	16.91 MB	+1.6 kB ⚠️
Runtime	Public routes	11	11	0

Changed function payloads vs main (6ad578a) (2)

Function	Status	Baseline	Current	Delta	Route changes
`functions/__server.func`	changed	8.46 MB	8.46 MB	+795 B ⚠️	none
`functions/.well-known/workflow/v1/flow.func`	changed	8.46 MB	8.46 MB	+795 B ⚠️	none

`eve init` install

Metric	Baseline	Current	Delta
Installed footprint	123.84 MB	123.84 MB	+763 B ⚠️
Installed packages	128	128	0
dependencies	4	4	0
devDependencies	2	2	0
Dependency package bytes	42.82 MB	42.82 MB	+763 B ⚠️
devDependency package bytes	5.04 MB	5.04 MB	0 B ➖

Build Metadata

Preset: vercel
Nitro: nitro@3.0.260610-beta
Output directory: apps/fixtures/weather-agent/.vercel/output
Build metadata timestamp: 2026-08-01T05:10:02.413Z
Route aliases: 11 public, 1 internal (12 total aliases)
Vercel routes in config: 14
Severity legend: 🔴 dominant/large, 🟠 notable, 🟡 watch, ⚪ small

Package Drill-Down

Package Details

Package: eve@0.29.4
Package directory: packages/eve
Tarball: 7.59 MB (eve-0.29.4.tgz)
Unpacked payload: 28.44 MB across 2952 published files
Installed footprint: 85.43 MB across 6747 installed files
Installed root package: 27.09 MB
Installed dependencies: 58.34 MB
Runtime dependencies: 2
Peer dependencies: 5 (4 optional)

Installed footprint is measured from an isolated temporary npm install of the packed tarball.

Heavy installed dependencies

eve: 27.09 MB (31.7%)
@rolldown/binding-linux-x64-gnu: 19.28 MB (22.6%)
@rolldown/binding-wasm32-wasi: 10.66 MB (12.5%)
ai: 6.57 MB (7.7%)
zod: 5.07 MB (5.9%)

Publish payload breakdown

Published file size
🔴 dist/src/compiled/shadcn-registry/index.js       [#################.......] 9.68 MB 34.0%
🟠 dist/src/compiled/@photon-ai/chat-adapter-ime... [####....................] 2.42 MB 8.5%
🟠 dist/src/compiled/experimental-ai-sdk-code-mo... [###.....................] 1.51 MB 5.3%
🟡 dist/src/compiled/_chunks/node/undici-DWL_MYm... [#.......................] 502.4 kB 1.8%
🟡 dist/src/compiled/_chunks/workflow/undici-DWL... [#.......................] 502.4 kB 1.8%
🔴 Other published files                            [########################] 13.83 MB 48.6%

Installed footprint breakdown

Installed package size
🔴 eve                             [########################] 27.09 MB 31.7%
🔴 @rolldown/binding-linux-x64-gnu [#################.......] 19.28 MB 22.6%
🔴 @rolldown/binding-wasm32-wasi   [#########...............] 10.66 MB 12.5%
🔴 ai                              [######..................] 6.57 MB 7.7%
🔴 zod                             [####....................] 5.07 MB 5.9%
🟠 undici                          [###.....................] 3.50 MB 4.1%
🔴 Other installed packages        [############............] 13.26 MB 15.5%

Runtime dependencies (2)

Package	Range	Notes
`nitro`	`3.0.260610-beta`
`undici`	`8.9.0`

Peer dependencies (5)

Package	Range	Notes
`@opentelemetry/api`	`^1.0.0`	optional peer
`ai`	`catalog:`
`braintrust`	`^3.0.0`	optional peer
`just-bash`	`^3.0.0`	optional peer
`microsandbox`	`^0.5.0`	optional peer

eve init install drill-down

`eve init` install details

Command: eve init my-agent
Package manager: npm
Installed footprint: 123.84 MB across 8623 installed files
Installed packages: 128 total (122 transitive-only)
dependencies: 4 direct packages totaling 42.82 MB
devDependencies: 2 direct packages totaling 5.04 MB
Other transitive package files: 75.98 MB

Installed footprint is measured from an isolated temporary eve init my-agent using the current packed eve tarball.

Heavy installed dependencies

@typescript/typescript-linux-x64: 27.95 MB (22.6%)
eve: 27.09 MB (21.9%)
@rolldown/binding-linux-x64-gnu: 19.28 MB (15.6%)
@rolldown/binding-wasm32-wasi: 10.66 MB (8.6%)
zod: 9.02 MB (7.3%)

Installed footprint breakdown

Installed package size
🔴 @typescript/typescript-linux-x64 [########################] 27.95 MB 22.6%
🔴 eve                              [#######################.] 27.09 MB 21.9%
🔴 @rolldown/binding-linux-x64-gnu  [#################.......] 19.28 MB 15.6%
🔴 @rolldown/binding-wasm32-wasi    [#########...............] 10.66 MB 8.6%
🔴 zod                              [########................] 9.02 MB 7.3%
🔴 ai                               [######..................] 6.57 MB 5.3%
🔴 Other installed packages         [####################....] 23.27 MB 18.8%

dependencies (4)

Package	Range	Installed size	Share
`@vercel/connect`	`0.4.3`	141.0 kB	0.1%
`ai`	`^7.0.38`	6.57 MB	5.3%
`eve`	`file:eve-0.29.4.tgz`	27.09 MB	21.9%
`zod`	`4.4.3`	9.02 MB	7.3%

devDependencies (2)

Package	Range	Installed size	Share
`@types/node`	`24.x`	2.54 MB	2.1%
`typescript`	`7.0.2`	2.50 MB	2.0%

Function Drill-Down

Payload Size Graph

Unique function payload size and share of total
🔴 functions/.well-known/workflow/v1/flow.func     [########################] 8.46 MB 50.0%
🔴 functions/__server.func                         [########################] 8.46 MB 50.0%

Top Function Payloads

🟠 functions/.well-known/workflow/v1/flow.func • 1 public route • 8.46 MB

Metric	Value
Public routes	`/.well-known/workflow/v1/flow`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	8.46 MB
Function files	8.46 MB across 43 files
Traced dependencies	0 B
Signal	🟠 Bundled file `index.mjs` is 2.31 MB (27.4%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [#######################.] 2.31 MB 27.4%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.59 MB 18.9%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.6%
🟡 _chunks/sandbox.mjs             [########................] 769.0 kB 9.1%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.1%
🟠 Other bundled files             [########################] 2.37 MB 28.0%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x",
  "maxDuration": "max",
  "experimentalTriggers": [
    {
      "type": "queue/v2beta",
      "topic": "__eve776561746865722d6167656e74_wkf_workflow_*",
      "consumer": "default",
      "retryAfterSeconds": 5,
      "initialDelaySeconds": 0
    }
  ],
  "environment": {
    "WORKFLOW_PRECONDITION_GUARD": "1"
  }
}

🟠 functions/__server.func • 10 public routes, 1 internal alias • 8.46 MB

Metric	Value
Public routes	`/` `/eve/v1/callback/[token]` `/eve/v1/connections/[name]/callback/[token]` `/eve/v1/health` `/eve/v1/info` `/eve/v1/session` `/eve/v1/session/[sessionId]` `/eve/v1/session/[sessionId]/cancel` `/eve/v1/session/[sessionId]/stream` `/eve/v1/session/reset`
Internal aliases	`/__server`
Runtime	`nodejs24.x`
Handler	`index.mjs`
Payload	8.46 MB
Function files	8.46 MB across 43 files
Traced dependencies	0 B
Signal	🟠 Bundled file `index.mjs` is 2.31 MB (27.4%)

🟠 🔎 Dependency Analysis

📦 Bundled files:

Bundled file size
🟠 index.mjs                       [#######################.] 2.31 MB 27.4%
🟠 _chunks/runtime-artifacts.mjs   [################........] 1.59 MB 18.9%
🟡 _libs/undici.mjs                [##########..............] 980.5 kB 11.6%
🟡 _chunks/sandbox.mjs             [########................] 769.0 kB 9.1%
🟡 _libs/@ai-sdk/gateway+[...].mjs [####....................] 432.8 kB 5.1%
🟠 Other bundled files             [########################] 2.36 MB 28.0%

🧾 Vercel Config

{
  "handler": "index.mjs",
  "launcherType": "Nodejs",
  "shouldAddHelpers": false,
  "supportsResponseStreaming": true,
  "runtime": "nodejs24.x"
}

Build Timing: `e2e/fixtures/agent-tools-sandbox`

This is an informational timing measurement inside eve build, from preflight through publication. Output-size measurement and profile writing are excluded.

Build mode: deployable Vercel build with sandbox template prewarm included.

Build pipeline: 2.18 s -> 2.09 s (-92.1 ms) vs main (6ad578a).
Timing is informational: shared GitHub runners are too variable for a hard timing budget.

Detailed phase timings vs `main (6ad578a)`

Phase	Baseline	Current	Delta
`extension.check`	7.5 ms	9.5 ms	+2.0 ms
`project.resolve`	3.9 ms	1.7 ms	-2.2 ms
`workspace.create`	1.2 ms	1.3 ms	+0.1 ms
`host.prepare`	207.2 ms	175.9 ms	-31.3 ms
`vercel.service-prefix.resolve`	2.6 ms	2.3 ms	-0.3 ms
`nitro.create`	216.9 ms	209.5 ms	-7.4 ms
`sandbox.prewarm`	291.7 ms	265.1 ms	-26.6 ms
`nitro.cache.prepare`	0.3 ms	0.3 ms	0.0 ms
`nitro.prepare`	0.9 ms	0.9 ms	0.0 ms
`nitro.public-assets`	0.8 ms	0.8 ms	0.0 ms
`nitro.prerender`	0.5 ms	0.6 ms	+0.1 ms
`nitro.bundle`	1.41 s	1.39 s	-25.4 ms
`nitro.cache.write`	0.4 ms	0.4 ms	0.0 ms
`vercel.workflow-function.materialize`	23.5 ms	24.2 ms	+0.7 ms
`agent-summary.emit`	0.6 ms	0.5 ms	-0.1 ms
`nitro.close`	0.1 ms	0.1 ms	0.0 ms
`output.publish`	5.2 ms	3.6 ms	-1.6 ms
`workspace.remove`	2.3 ms	2.3 ms	0.0 ms

Diagnosing a gateway stream failure from an eve log was not possible. The error ids in the log are local `crypto.randomUUID()` values, and the structured `details` payload — which already carries `generationId` — was attached only when the semantic-error catalog recognized the failure. On unrecognized failures, the ones that are hardest to diagnose, eve logged a raw inspector dump with no correlation fields at all. Pairs with the gateway change that puts `generation_id`, `provider`, and `model` on the synthesized `gateway_stream_terminated` frame. - Teach `findGatewayError` to recognize gateway-synthesized frames, which key on `code` + `origin` rather than a `Gateway*` class name or a `*_error` type. Without this the new fields are silently dropped. - Extract `provider` and `model`, and read `generation_id` alongside `generationId` since the frame is snake_case. `provider` matters because a canonical slug like `zai/glm-5.2` is served by many upstreams, so the failing one is not inferable from the model id. - Always log `details` when non-empty. Recognized failures still omit the raw `error` to bypass the `[object Object]` inspector rendering; unrecognized ones keep it for the stack. - Add a `gateway-stream-terminated` catalog rule tagged `transient`. The frame carries no status code and is not marked retryable, so it fell through `classifyModelCallError` to `recoverable` and parked the turn on first contact with zero retries. `transient` earns it the standard three attempts with backoff — a transient upstream throttle should not kill a turn. The upstream error text is deliberately not consumed here. eve's logs reach public CI output, and provider error bodies echo request content, so that detail stays gateway-side until the redacting extractor on AIG-394 lands. `generationId` is the handle for looking it up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Signed-off-by: Walter Korman <shaper@vercel.com>

vercel Bot deployed to Preview – eve-docs-4759 August 1, 2026 04:43 View deployment

vercel Bot deployed to Preview – eve-docs August 1, 2026 04:43 View deployment

shaper force-pushed the walterkorman/gateway-stream-termination-diagnostics branch from 62d5dc6 to be4cb15 Compare August 1, 2026 05:07

vercel Bot deployed to Preview – eve-docs-4759 August 1, 2026 05:08 View deployment

vercel Bot deployed to Preview – eve-docs August 1, 2026 05:08 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat(eve): capture gateway correlation details on model-call failures - #1502

feat(eve): capture gateway correlation details on model-call failures#1502
shaper wants to merge 1 commit into
mainfrom
walterkorman/gateway-stream-termination-diagnostics

shaper commented Aug 1, 2026 •

edited

Loading

Uh oh!

vercel Bot commented Aug 1, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 1, 2026 •

edited

Loading

Package Details

`eve init` install details

Payload Size Graph

Top Function Payloads

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

Uh oh!

Conversation

shaper commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

The behavior change worth reviewing

What this deliberately does not capture

Testing

Uh oh!

vercel Bot commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Bundle + Package Summary: apps/fixtures/weather-agent

Delta vs main (6ad578a)

eve init install

Package Details

eve init install details

Payload Size Graph

Top Function Payloads

Build Timing: e2e/fixtures/agent-tools-sandbox

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

shaper commented Aug 1, 2026 •

edited

Loading

vercel Bot commented Aug 1, 2026 •

edited

Loading

github-actions Bot commented Aug 1, 2026 •

edited

Loading

Bundle + Package Summary: `apps/fixtures/weather-agent`

Delta vs `main (6ad578a)`

`eve init` install

`eve init` install details

Build Timing: `e2e/fixtures/agent-tools-sandbox`