Skip to content

Commit 15bc592

Browse files
author
luop
committed
fix(proxy-core): bind protocol-level sticky channel from response IDs
Three related bugs in protocol-level session stickiness shipped in spec session-stick-routing: P1 (binding timing): surfaces wrote sticky channels using continuation IDs taken from the *current request* (previous_response_id / tool_result.tool_use_id), but the next request queries with a key built from the *next request's* incoming ID — which is the current response's newly-minted ID, never the request-side one. Sticky never hit. Now bind from the response payload (response.id / tool_use.id) so the next request's lookup key matches. Three follow-up corrections from review: - The first cut of bindSurfaceStickyChannelFromResponse early-returned when the request-side key was a CLI-level fallback, on the mistaken belief that this would "steal CLI-level semantics". CLI-level keys and protocol-level keys live in disjoint keyspaces and never collide; the early-return broke the most common production path (CLI client carrying a session header on round 1 with no continuation ID). The guard is removed; the helper now binds whenever the response yields a usable continuation ID, regardless of the request-side key. - The four chatSurface bind sites in the Anthropic-downstream branch forwarded raw upstream payloads to the extractor. When the upstream was OpenAI Chat / Responses (cross-protocol fallback), the raw payload carried tool calls under choices[].message.tool_calls or output[].tool_calls — which the Anthropic extractor does not read. Surfaces now feed the bind helper a NormalizedFinalResponse instead, hitting extractor path 2 (top-level toolCalls[].id). proxyStream was extended to expose getTerminalNormalizedFinal() for streaming paths, and its OpenAI Chat aggregator is now always-on so claude downstream + OpenAI Chat upstream still captures tool_call.id. - The Anthropic-native SSE fast path in proxyStream (consumeSseEventBlock returning handled=true) short-circuited before applyOpenAiChatStreamEvent ran, so claude-downstream + Anthropic-upstream traffic — the most common case — never captured tool_use.id from content_block_start. The handled branch now also runs transformStreamEvent + the aggregator (discarding the produced lines because the consumer already forwarded the original frame). New regression suite proxyStream.test.ts exercises this path. P2 (toggle bypass): the protocol-level branch in buildSurfaceStickySessionKey skipped the proxyStickySessionEnabled guard, so disabling sticky still produced a non-empty key that reserved a session-scoped lease and could 503. Added the guard so protocol-level matches CLI-level semantics. P3 (failure clears protocol-level binding): clearSurfaceStickyChannel was called from 16 surface error paths, violating spec Requirement 6.4 / 8.4 ("failure does not clear; wait for TTL or next success to overwrite"). Added a proto-v1| prefix noop guard inside clearSurfaceStickyChannel — call sites untouched, blast radius zero. Implementation details: - New extractAnthropicMessagesContinuationIdsFromResponse() in transformers/anthropic/messages/sessionId.ts, mirroring the existing extractResponsesTerminalResponseId() in transformers/openai/responses/continuation.ts. - New bindSurfaceStickyChannelFromResponse() in proxy-core/surfaces/sharedSurface.ts that re-binds protocol-level keys with response-side IDs after success. Always attempts the bind when sticky is enabled and a usable continuation ID is extracted. - proxyStream.ts (transformers/openai/chat) exposes getTerminalNormalizedFinal(); chat aggregator state is always-on so cross-protocol upstreams (OpenAI Chat → Anthropic downstream) and the native Anthropic fast path both surface tool_call.id at a single capture point. - Wired into 5 bind call sites in openAiResponsesSurface.ts and 4 Anthropic-branch bind call sites in chatSurface.ts. count_tokens bind site explicitly does not bind from response (no tool_use.id in token-count payloads). - Existing bindSurfaceStickyChannel and 16 clearSurfaceStickyChannel call sites unchanged. Tests: - New regression suite sessionStick.bugCondition.test.ts (8 cases) proves the three bugs are fixed, including the round-1-CLI-key follow-up case for OpenAI Responses and Anthropic Messages. - New regression suite proxyStream.test.ts (4 cases) covers the Anthropic-native + cross-protocol + same-protocol + empty-stream shapes; the Anthropic-native case fails on the pre-fix handled branch (verified by reverting just that hunk). - sessionStick.integration.test.ts rewritten: scenarios A/B now bind from response payload; new scenarios G (P2 toggle), H (retry override), I (P3 failure preservation), J (cross-protocol NormalizedFinalResponse with toolCalls[]). - 12 new parametrized cases for the Anthropic response-side extractor (43 cases total in the file). - Architecture test extended: extractResponsesTerminalResponseId and the new Anthropic extractor are now in the protocol-pure boundary matrix; proxyChannelCoordinator and channelSelection asserted free of "proto-v1|" literals (no protocol awareness leak). Verification: - npm run build:server: clean - npm run repo:drift-check: 0 new violations - All session-stick-routing existing test suites green (236/236 across chat / anthropic / sticky surfaces) - Cross-protocol-image-forwarding suites green
1 parent 491b272 commit 15bc592

10 files changed

Lines changed: 2653 additions & 151 deletions

File tree

‎src/server/proxy-core/surfaces/chatSurface.ts‎

Lines changed: 110 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ import { maybeHandleWebSearchOnlySimulation } from '../webSearchSimulation.js';
5757
import {
5858
acquireSurfaceChannelLease,
5959
bindSurfaceStickyChannel,
60+
bindSurfaceStickyChannelFromResponse,
6061
buildSurfaceChannelBusyMessage,
6162
buildSurfaceStickySessionKey,
6263
clearSurfaceStickyChannel,
@@ -626,6 +627,15 @@ export async function handleChatSurfaceRequest(
626627
}
627628
},
628629
};
630+
// P1 fix (spec session-stick-routing-binding-timing-fix): accumulate
631+
// the latest object-shaped payload seen during streaming so that the
632+
// success-terminal binding can re-key the protocol-level sticky map
633+
// against the response-side `tool_use.id` (Anthropic Messages branch).
634+
// Both `streamSession.run` paths (looksLikeResponsesSseText fallback
635+
// and pure SSE) feed payloads here via `onParsedPayload`. The
636+
// `streamSession.consumeUpstreamFinalPayload` path holds `fallbackData`
637+
// directly and does not rely on this accumulator.
638+
let lastResponseSidePayload: unknown = null;
629639
const streamSession = openAiChatTransformer.proxyStream.createSession({
630640
downstreamFormat,
631641
modelName,
@@ -634,6 +644,7 @@ export async function handleChatSurfaceRequest(
634644
if (payload && typeof payload === 'object') {
635645
upstreamUsagePresent = upstreamUsagePresent || hasProxyUsagePayload(payload);
636646
parsedUsage = mergeProxyUsage(parsedUsage, parseProxyUsage(payload));
647+
lastResponseSidePayload = payload;
637648
}
638649
},
639650
writeLines,
@@ -701,6 +712,31 @@ export async function handleChatSurfaceRequest(
701712
stickySessionKey,
702713
selected,
703714
});
715+
// P1 fix (spec session-stick-routing-binding-timing-fix):
716+
// 仅在 Anthropic Messages 分支用响应侧 tool_use.id 重新绑定协议级 key。
717+
// OpenAI Chat Completions 分支不参与协议级 sticky（spec
718+
// session-stick-routing Requirement 10.4 显式 out of scope）。
719+
// Cross-protocol fix: 用 `streamSession.getTerminalNormalizedFinal()`
720+
// 拿到协议无关的 NormalizedFinalResponse（含 `toolCalls[].id`），
721+
// 而不是单帧 `lastResponseSidePayload` —— 因为 SSE 协议下 tool_call.id
722+
// 通常只在第一帧出现，单帧 payload 拿不到完整 id。
723+
if (downstreamFormat === 'claude') {
724+
const responsePayload = streamSession.getTerminalNormalizedFinal()
725+
?? lastResponseSidePayload;
726+
if (responsePayload) {
727+
bindSurfaceStickyChannelFromResponse({
728+
requestSideStickySessionKey: stickySessionKey,
729+
protocolHint: 'anthropic/messages',
730+
responsePayload,
731+
scope: {
732+
downstreamApiKeyId,
733+
downstreamPath,
734+
requestedModel,
735+
},
736+
selected,
737+
});
738+
}
739+
}
704740
return;
705741
}
706742
let fallbackData: unknown = null;
@@ -801,6 +837,31 @@ export async function handleChatSurfaceRequest(
801837
stickySessionKey,
802838
selected,
803839
});
840+
// P1 fix (spec session-stick-routing-binding-timing-fix):
841+
// 仅在 Anthropic Messages 分支用响应侧 tool_use.id 重新绑定协议级 key。
842+
// OpenAI Chat Completions 分支不参与协议级 sticky（spec
843+
// session-stick-routing Requirement 10.4 显式 out of scope）。
844+
// Cross-protocol fix: 用 `streamSession.getTerminalNormalizedFinal()`
845+
// 拿到协议无关的 NormalizedFinalResponse（含 `toolCalls[].id`），
846+
// 而不是单帧 `lastResponseSidePayload` —— 因为 SSE 协议下 tool_call.id
847+
// 通常只在第一帧出现，单帧 payload 拿不到完整 id。
848+
if (downstreamFormat === 'claude') {
849+
const responsePayload = streamSession.getTerminalNormalizedFinal()
850+
?? lastResponseSidePayload;
851+
if (responsePayload) {
852+
bindSurfaceStickyChannelFromResponse({
853+
requestSideStickySessionKey: stickySessionKey,
854+
protocolHint: 'anthropic/messages',
855+
responsePayload,
856+
scope: {
857+
downstreamApiKeyId,
858+
downstreamPath,
859+
requestedModel,
860+
},
861+
selected,
862+
});
863+
}
864+
}
804865
return;
805866
} else {
806867
const upstreamReader = getRuntimeResponseReader(upstream);
@@ -887,6 +948,30 @@ export async function handleChatSurfaceRequest(
887948
stickySessionKey,
888949
selected,
889950
});
951+
// P1 fix (spec session-stick-routing-binding-timing-fix):
952+
// 仅在 Anthropic Messages 分支用响应侧 tool_use.id 重新绑定协议级 key。
953+
// OpenAI Chat Completions 分支不参与协议级 sticky（spec
954+
// session-stick-routing Requirement 10.4 显式 out of scope）。
955+
// Cross-protocol fix: 用 `streamSession.getTerminalNormalizedFinal()`
956+
// 拿到协议无关的 NormalizedFinalResponse（含 `toolCalls[].id`），
957+
// 而不是单帧 `lastResponseSidePayload`。
958+
if (downstreamFormat === 'claude') {
959+
const responsePayload = streamSession.getTerminalNormalizedFinal()
960+
?? lastResponseSidePayload;
961+
if (responsePayload) {
962+
bindSurfaceStickyChannelFromResponse({
963+
requestSideStickySessionKey: stickySessionKey,
964+
protocolHint: 'anthropic/messages',
965+
responsePayload,
966+
scope: {
967+
downstreamApiKeyId,
968+
downstreamPath,
969+
requestedModel,
970+
},
971+
selected,
972+
});
973+
}
974+
}
890975
return;
891976
}
892977

@@ -985,6 +1070,27 @@ export async function handleChatSurfaceRequest(
9851070
stickySessionKey,
9861071
selected,
9871072
});
1073+
// P1 fix (spec session-stick-routing-binding-timing-fix):
1074+
// 仅在 Anthropic Messages 分支用响应侧 tool_use.id 重新绑定协议级 key。
1075+
// OpenAI Chat Completions 分支不参与协议级 sticky（spec
1076+
// session-stick-routing Requirement 10.4 显式 out of scope）。
1077+
// Cross-protocol fix: 用 `normalizedFinal`（NormalizedFinalResponse），
1078+
// 而不是 raw `upstreamData` —— upstream 协议形态可能是 OpenAI Chat /
1079+
// OpenAI Responses，extractor 只看 Anthropic native `content[]` 或顶层
1080+
// `toolCalls[]`，传 normalized 形态命中 path 2 fallback。
1081+
if (downstreamFormat === 'claude') {
1082+
bindSurfaceStickyChannelFromResponse({
1083+
requestSideStickySessionKey: stickySessionKey,
1084+
protocolHint: 'anthropic/messages',
1085+
responsePayload: normalizedFinal,
1086+
scope: {
1087+
downstreamApiKeyId,
1088+
downstreamPath,
1089+
requestedModel,
1090+
},
1091+
selected,
1092+
});
1093+
}
9881094

9891095
return reply.send(downstreamResponse);
9901096
} catch (err: any) {
@@ -1430,6 +1536,10 @@ export async function handleClaudeCountTokensSurfaceRequest(
14301536
stickySessionKey,
14311537
selected,
14321538
});
1539+
// 设计决策（spec session-stick-routing-binding-timing-fix design.md §3.2）：
1540+
// count_tokens 上游响应不产出新的 tool_use.id（只返回 token 计数），
1541+
// 因此本入口仅保留 CLI 级 sticky 回退（旧 bindSurfaceStickyChannel 调用），
1542+
// 不调用响应侧重新绑定 helper —— 即使调用也会因响应侧提取返回空数组而 noop。
14331543
await finalizeDebugSuccess(
14341544
upstream.status,
14351545
upstreamRequest.path,

‎src/server/proxy-core/surfaces/openAiResponsesSurface.ts‎

Lines changed: 91 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -74,6 +74,7 @@ import { shouldAbortSameSiteEndpointFallback } from '../../services/proxyRetryPo
7474
import {
7575
acquireSurfaceChannelLease,
7676
bindSurfaceStickyChannel,
77+
bindSurfaceStickyChannelFromResponse,
7778
buildSurfaceChannelBusyMessage,
7879
buildSurfaceStickySessionKey,
7980
clearSurfaceStickyChannel,
@@ -924,6 +925,16 @@ export async function handleOpenAiResponsesSurfaceRequest(
924925
promptTokensIncludeCache: null,
925926
};
926927
let upstreamUsagePresent = false;
928+
// P1 fix (spec session-stick-routing-binding-timing-fix): accumulate
929+
// the latest object-shaped payload seen during streaming so that the
930+
// success-terminal binding can re-key the protocol-level sticky map
931+
// against the response-side `response.id`. Both `streamSession.run`
932+
// paths (websocket-replaces-SSE fallback at site 1, and pure SSE at
933+
// site 4) feed payloads here via `onParsedPayload`. The
934+
// `streamSession.consumeUpstreamFinalPayload` path (site 2) and the
935+
// `websocketTransportRequest` collected-payload path (site 3) hold
936+
// their payload directly and do not rely on this accumulator.
937+
let lastResponseSidePayload: unknown = null;
927938
const writeLines = (lines: string[]) => {
928939
for (const line of lines) reply.raw.write(line);
929940
};
@@ -939,6 +950,7 @@ export async function handleOpenAiResponsesSurfaceRequest(
939950
if (codexSessionStoreKey) {
940951
rememberCodexSessionResponseId(codexSessionStoreKey, payload);
941952
}
953+
lastResponseSidePayload = payload;
942954
}
943955
},
944956
writeLines,
@@ -991,6 +1003,23 @@ export async function handleOpenAiResponsesSurfaceRequest(
9911003
stickySessionKey,
9921004
selected,
9931005
});
1006+
// P1 fix (spec session-stick-routing-binding-timing-fix):
1007+
// re-key the protocol-level sticky binding using the
1008+
// response-side `response.id`. The accumulator captured the
1009+
// latest object payload via `streamSession.onParsedPayload`,
1010+
// which includes the `response.completed` event payload at
1011+
// terminal success.
1012+
bindSurfaceStickyChannelFromResponse({
1013+
requestSideStickySessionKey: stickySessionKey,
1014+
protocolHint: 'openai/responses',
1015+
responsePayload: lastResponseSidePayload,
1016+
scope: {
1017+
downstreamApiKeyId,
1018+
downstreamPath,
1019+
requestedModel,
1020+
},
1021+
selected,
1022+
});
9941023
return;
9951024
}
9961025
let upstreamData: unknown = rawText;
@@ -1083,6 +1112,21 @@ export async function handleOpenAiResponsesSurfaceRequest(
10831112
stickySessionKey,
10841113
selected,
10851114
});
1115+
// P1 fix (spec session-stick-routing-binding-timing-fix):
1116+
// re-key the protocol-level sticky binding using the
1117+
// response-side `response.id` carried in the parsed
1118+
// upstream JSON payload.
1119+
bindSurfaceStickyChannelFromResponse({
1120+
requestSideStickySessionKey: stickySessionKey,
1121+
protocolHint: 'openai/responses',
1122+
responsePayload: upstreamData,
1123+
scope: {
1124+
downstreamApiKeyId,
1125+
downstreamPath,
1126+
requestedModel,
1127+
},
1128+
selected,
1129+
});
10861130
return;
10871131
}
10881132

@@ -1125,6 +1169,21 @@ export async function handleOpenAiResponsesSurfaceRequest(
11251169
stickySessionKey,
11261170
selected,
11271171
});
1172+
// P1 fix (spec session-stick-routing-binding-timing-fix):
1173+
// re-key the protocol-level sticky binding using the
1174+
// response-side `response.id` carried in the collected
1175+
// SSE final payload (websocket-replaces-SSE transport).
1176+
bindSurfaceStickyChannelFromResponse({
1177+
requestSideStickySessionKey: stickySessionKey,
1178+
protocolHint: 'openai/responses',
1179+
responsePayload: collectedPayload,
1180+
scope: {
1181+
downstreamApiKeyId,
1182+
downstreamPath,
1183+
requestedModel,
1184+
},
1185+
selected,
1186+
});
11281187
return;
11291188
} catch {
11301189
// Fall through to the generic stream session for response.failed/error terminals.
@@ -1239,6 +1298,23 @@ export async function handleOpenAiResponsesSurfaceRequest(
12391298
stickySessionKey,
12401299
selected,
12411300
});
1301+
// P1 fix (spec session-stick-routing-binding-timing-fix):
1302+
// re-key the protocol-level sticky binding using the
1303+
// response-side `response.id`. The accumulator captured the
1304+
// latest object payload via `streamSession.onParsedPayload`,
1305+
// which includes the `response.completed` event payload at
1306+
// terminal success.
1307+
bindSurfaceStickyChannelFromResponse({
1308+
requestSideStickySessionKey: stickySessionKey,
1309+
protocolHint: 'openai/responses',
1310+
responsePayload: lastResponseSidePayload,
1311+
scope: {
1312+
downstreamApiKeyId,
1313+
downstreamPath,
1314+
requestedModel,
1315+
},
1316+
selected,
1317+
});
12421318
return;
12431319
}
12441320

@@ -1357,6 +1433,21 @@ export async function handleOpenAiResponsesSurfaceRequest(
13571433
stickySessionKey,
13581434
selected,
13591435
});
1436+
// P1 fix (spec session-stick-routing-binding-timing-fix):
1437+
// re-key the protocol-level sticky binding using the
1438+
// response-side `response.id` carried in the parsed
1439+
// upstream JSON payload.
1440+
bindSurfaceStickyChannelFromResponse({
1441+
requestSideStickySessionKey: stickySessionKey,
1442+
protocolHint: 'openai/responses',
1443+
responsePayload: upstreamData,
1444+
scope: {
1445+
downstreamApiKeyId,
1446+
downstreamPath,
1447+
requestedModel,
1448+
},
1449+
selected,
1450+
});
13601451
return reply.send(downstreamData);
13611452
} catch (err: any) {
13621453
clearSurfaceStickyChannel({

0 commit comments

Comments
 (0)