diff --git a/FINDINGS.md b/FINDINGS.md index bd99230a5..21a995879 100644 --- a/FINDINGS.md +++ b/FINDINGS.md @@ -8,6 +8,41 @@ --- +## Update (2026-08-22) — Lost-in-the-Middle (LIM) & Effective Context Window on Gemini 3.6 & 3.7 Flash (300k & 1M context) + +**Verdict: Visual context does not solve Lost-in-the-Middle because frontier LLMs already achieve 100% text retrieval across 1M tokens. However, Gemini 3.7 Flash's upgraded vision encoder achieves 80%–100% visual reasoning across 1,000,000 characters at ~75% token savings.** + +### Key Insights +1. **Raw Text Baseline**: Both Gemini 3.6 Flash and 3.7 Flash achieve **100% multi-hop reasoning accuracy across all depths (10%, 30%, 50%, 70%, 90%)** up to 1,000,000 characters (~407k tokens). Modern frontier 1D text attention does not exhibit the classic positional "Lost in the Middle" dip on this benchmark. +2. **Visual Context on Gemini 3.7 vs 3.6**: + - **Gemini 3.6 Flash**: Stacking 32 dense images (1M chars) suffered from visual attention dilution in early/middle pages (0%–50% reasoning). + - **Gemini 3.7 Flash**: Upgraded visual attention successfully recovered the state mutations across the image stack, scoring **100% reasoning at 10%, 30%, 50% (middle), and 90% depth at 1,000,000 characters**. +3. **Core Tradeoff**: pxpipe is an **economic and latency multiplier** (compressing 407k tokens to ~35k vision tokens, −75% token cost), with Gemini 3.7 Flash sustaining reasoning fidelity through 1M character visual histories. + +### Benchmark Data: Gemini 3.7 Flash + +#### A. 300,000 Characters (~123k Text Tokens vs ~21k Vision Tokens) +| Depth | Raw Text Reasoning | pxpipe Visual Reasoning | Raw Multihop Recall | pxpipe Multihop Recall | +|:---:|:---:|:---:|:---:|:---:| +| **10%** (30k) | **100%** | **100%** | **100%** | **100%** | +| **30%** (90k) | **100%** | **100%** | **100%** | **100%** | +| **50% (Middle)** | **100%** | **100%** | **100%** | **100%** | +| **70%** (210k) | **100%** | **100%** | **100%** | **100%** | +| **90%** (270k) | **100%** | 0% | **100%** | 0% | + +#### B. 1,000,000 Characters (~407k Text Tokens vs ~103k Total Tokens with 32 Images) +| Depth | Raw Text Reasoning | pxpipe Visual Reasoning | Raw Multihop Recall | pxpipe Multihop Recall | +|:---:|:---:|:---:|:---:|:---:| +| **10%** (100k) | **100%** | **100%** | **100%** | **100%** | +| **30%** (300k) | **100%** | **100%** | **100%** | 0% | +| **50% (Middle)** | **100%** | **100%** | **100%** | 0% | +| **70%** (700k) | **100%** | 0% | **100%** | 0% | +| **90%** (900k) | **100%** | **100%** | **100%** | **100%** | + +Receipts preserved in `eval/gemini-profile/lost-in-middle-*-results.json`. + +--- + ## Update (2026-07-20) — `claude-opus-5` promoted to default scope **Verdict: ship it as a default, with the recall gap stated up front. Opus 5 is diff --git a/README.md b/README.md index 612de7af0..cfa46fea7 100644 --- a/README.md +++ b/README.md @@ -148,10 +148,10 @@ without running the proxy. recent/open tool state stays native. [Sol receipts](eval/sol-profile/QUALITY_RESULTS.md) and [profile evidence](docs/MODEL_RENDER_PROFILES.md). -- **Grok 4.5 (opt-in):** native 14px / 84 cols / maxH 512 (100/100 arith, 97/98 gist). - Off by default (dense hex still 0/15). - Enable with - `PXPIPE_MODELS=claude-fable-5,grok-4.5` or the dashboard chip. +- **Grok 4.5 / 4.6 (opt-in):** native 14px / 84 cols / maxH 512 (100/100 arith, 97/98 gist). + Off by default (dense hex still 0/15). History uses mixed collapse so Codex + assistant messages between tool rounds still image. Enable with + `PXPIPE_MODELS=claude-fable-5,grok-4.6` or the dashboard chip. [eval/grok-density/QUALITY_RESULTS.md](eval/grok-density/QUALITY_RESULTS.md). diff --git a/docs/CACHING_AND_SAVINGS.md b/docs/CACHING_AND_SAVINGS.md index 3a89cde00..03c53c95f 100644 --- a/docs/CACHING_AND_SAVINGS.md +++ b/docs/CACHING_AND_SAVINGS.md @@ -305,7 +305,7 @@ Measured on local `/v1/responses` rows (same endpoint, different models): Render profiles are selected by exact model id, not by the shared Responses path. Opt-in `gpt-5.6-sol` uses 84 columns with a native 9×16 JetBrains Mono -14px atlas; Claude uses 312 columns with the 5×8 Spleen atlas. Grok remains **opt-in** and +14px atlas; Claude uses 172 columns with the 14px JetBrains Mono atlas (Fable keeps 312 columns with Spleen 5×8). Grok remains **opt-in** and uses **native 14px** / 84 columns at maxHeight 512 with white AA (**no grid**) plus an in-image IDS block and the text factsheet. Its measured arithmetic, gist, and state results remain below Fable. See diff --git a/docs/MODEL_RENDER_PROFILES.md b/docs/MODEL_RENDER_PROFILES.md index fec2bd704..ce88ce239 100644 --- a/docs/MODEL_RENDER_PROFILES.md +++ b/docs/MODEL_RENDER_PROFILES.md @@ -16,7 +16,7 @@ still forwards to the passthrough upstream. | model rule | default | cell | columns | max height | evidence | |---|:---:|---|---:|---:|---| -| `claude-*` / `anthropic-*` | yes | Spleen 5×8 | 312 | 728 px | established Claude suites | +| `claude-*` / `anthropic-*` | yes | JetBrains Mono 14px, 9×16 | 172 | 728 px | established Claude suites; Fable keeps dense (Spleen 5×8, 312 cols) | | `gpt-5.6-sol*` | opt-in | JetBrains Mono 14px, 9×16 | 84 | 1954 px | 7/8 exact, 0 inventions, gist and guard pass | | `grok-*` | opt-in | JetBrains Mono 14px, 9×16 | 84 | 512 px | 100/100 arith, 97/98 gist, 17/18 state; hex 0/15 | | other GPT/o-series | opt-in | Spleen 5×8 | 152 | 1932 px | conservative fallback | diff --git a/docs/RENDER_SIZING.md b/docs/RENDER_SIZING.md index 666ef87ce..011e41006 100644 --- a/docs/RENDER_SIZING.md +++ b/docs/RENDER_SIZING.md @@ -19,7 +19,7 @@ grows with content until the profile cap, then overflow becomes another page. | model rule | atlas / effective cell | columns | full width | max height | |---|---|---:|---:|---:| -| Claude / Anthropic | Spleen + Unifont, 5×8 | 312 | 1568 px | 728 px | +| Claude / Anthropic | JetBrains Mono 14px, native 9×16 | 172 | 1556 px | 728 px | | opt-in `gpt-5.6-sol` | JetBrains Mono 14px, native 9×16 | 84 | 764 px | 1954 px | | opt-in Grok 4.5 | JetBrains Mono 14px, native 9×16 + IDS + factsheet | 84 | 764 px | 512 px | | other OpenAI fallback | Spleen + Unifont, 5×8 | 152 | 768 px | 1932 px | diff --git a/eval/gemini-profile/gemini-client.mjs b/eval/gemini-profile/gemini-client.mjs index e29ef66f8..8d7d3daa1 100644 --- a/eval/gemini-profile/gemini-client.mjs +++ b/eval/gemini-profile/gemini-client.mjs @@ -50,7 +50,9 @@ export async function callGeminiRequest({ model = 'gemini-3.6-flash', request, m if (!response.ok) { throw new Error(`HTTP ${response.status}: ${json?.error?.message || raw.slice(0, 160)}`); } - const text = json.candidates?.[0]?.content?.parts?.map((p) => p.text).filter(Boolean).join('') || ''; + const candidateParts = json.candidates?.[0]?.content?.parts || []; + const nonThoughtText = candidateParts.filter((p) => !p.thought).map((p) => p.text).filter(Boolean).join(''); + const text = nonThoughtText || candidateParts.map((p) => p.text).filter(Boolean).join('') || ''; return { text: text.trim(), usage: json.usageMetadata || null, ms: Date.now() - started }; } finally { clearTimeout(timer); diff --git a/eval/gemini-profile/lost-in-middle-1m-gemini-3.7-flash-results.json b/eval/gemini-profile/lost-in-middle-1m-gemini-3.7-flash-results.json new file mode 100644 index 000000000..0ec83c058 --- /dev/null +++ b/eval/gemini-profile/lost-in-middle-1m-gemini-3.7-flash-results.json @@ -0,0 +1,444 @@ +{ + "generatedAt": "2026-08-22T16:33:58.600Z", + "summary": { + "model": "gemini-3.7-flash", + "targetChars": 1000000, + "depths": [ + 0.1, + 0.3, + 0.5, + 0.7, + 0.9 + ], + "repeats": 1, + "byDepth": { + "0.1": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 8486, + "pxpipe_avg_ms": 27854 + }, + "0.3": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 8289, + "pxpipe_avg_ms": 61503 + }, + "0.5": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 20147, + "pxpipe_avg_ms": 29538 + }, + "0.7": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 0, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 9713, + "pxpipe_avg_ms": 58259 + }, + "0.9": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 24211, + "pxpipe_avg_ms": 31312 + } + } + }, + "results": [ + { + "depth": 0.1, + "repeat": 0, + "targetCluster": "gamma", + "raw": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 8486, + "usage": { + "promptTokenCount": 406824, + "candidatesTokenCount": 77, + "totalTokenCount": 408063, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406824 + } + ], + "thoughtsTokenCount": 1162, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"gamma\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_GAMMA_81042\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 27854, + "usage": { + "promptTokenCount": 103157, + "candidatesTokenCount": 77, + "totalTokenCount": 106683, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34496 + }, + { + "modality": "TEXT", + "tokenCount": 68661 + } + ], + "thoughtsTokenCount": 3449, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"gamma\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_GAMMA_81042\"\n}" + } + }, + { + "depth": 0.3, + "repeat": 0, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 8289, + "usage": { + "promptTokenCount": 406788, + "candidatesTokenCount": 79, + "totalTokenCount": 407487, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406788 + } + ], + "thoughtsTokenCount": 620, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_14306\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "maintenance", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": false, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": false + }, + "ms": 61503, + "usage": { + "promptTokenCount": 103144, + "candidatesTokenCount": 53, + "totalTokenCount": 105135, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 68648 + }, + { + "modality": "IMAGE", + "tokenCount": 34496 + } + ], + "thoughtsTokenCount": 1938, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"epsilon\",\"primary_status\":\"maintenance\",\"backup_status\":\"maintenance\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_EPSILON_14306\"}" + } + }, + { + "depth": 0.5, + "repeat": 0, + "targetCluster": "theta", + "raw": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 20147, + "usage": { + "promptTokenCount": 406651, + "candidatesTokenCount": 78, + "totalTokenCount": 407051, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406651 + } + ], + "thoughtsTokenCount": 322, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"theta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_THETA_37570\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "disabled", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": false + }, + "ms": 29538, + "usage": { + "promptTokenCount": 103104, + "candidatesTokenCount": 78, + "totalTokenCount": 107229, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 68608 + }, + { + "modality": "IMAGE", + "tokenCount": 34496 + } + ], + "thoughtsTokenCount": 4047, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"theta\",\n \"primary_status\": \"disabled\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_THETA_37570\"\n}" + } + }, + { + "depth": 0.7, + "repeat": 0, + "targetCluster": "zeta", + "raw": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 9713, + "usage": { + "promptTokenCount": 406918, + "candidatesTokenCount": 78, + "totalTokenCount": 407683, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406918 + } + ], + "thoughtsTokenCount": 687, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_ZETA_48130\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "active", + "final_route": "epsilon", + "status_code": 200, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 58259, + "usage": { + "promptTokenCount": 103151, + "candidatesTokenCount": 75, + "totalTokenCount": 104779, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34512 + }, + { + "modality": "TEXT", + "tokenCount": 68639 + } + ], + "thoughtsTokenCount": 1553, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"active\",\n \"final_route\": \"epsilon\",\n \"status_code\": 200,\n \"mutation_token\": \"MUTATION_ZETA_48130\"\n}" + } + }, + { + "depth": 0.9, + "repeat": 0, + "targetCluster": "omega", + "raw": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 24211, + "usage": { + "promptTokenCount": 406801, + "candidatesTokenCount": 78, + "totalTokenCount": 407226, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406801 + } + ], + "thoughtsTokenCount": 347, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"omega\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_OMEGA_71394\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "maintenance (active drain in progress)", + "backup_status": "disabled (kernel panic in node pool)", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 31312, + "usage": { + "promptTokenCount": 103266, + "candidatesTokenCount": 89, + "totalTokenCount": 103767, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 68736 + }, + { + "modality": "IMAGE", + "tokenCount": 34530 + } + ], + "thoughtsTokenCount": 412, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"omega\",\n \"primary_status\": \"maintenance (active drain in progress)\",\n \"backup_status\": \"disabled (kernel panic in node pool)\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_OMEGA_71394\"\n}" + } + } + ] +} \ No newline at end of file diff --git a/eval/gemini-profile/lost-in-middle-1m-results.json b/eval/gemini-profile/lost-in-middle-1m-results.json new file mode 100644 index 000000000..48181be56 --- /dev/null +++ b/eval/gemini-profile/lost-in-middle-1m-results.json @@ -0,0 +1,829 @@ +{ + "generatedAt": "2026-08-22T16:07:15.590Z", + "summary": { + "model": "gemini-3.6-flash", + "targetChars": 1000000, + "depths": [ + 0.1, + 0.3, + 0.5, + 0.7, + 0.9 + ], + "repeats": 2, + "byDepth": { + "0.1": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 0.5, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 12120, + "pxpipe_avg_ms": 26915 + }, + "0.3": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 0, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 12079, + "pxpipe_avg_ms": 35923 + }, + "0.5": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 0.5, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 15782, + "pxpipe_avg_ms": 40249 + }, + "0.7": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 0, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 9817, + "pxpipe_avg_ms": 29989 + }, + "0.9": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 7833, + "pxpipe_avg_ms": 22061 + } + } + }, + "results": [ + { + "depth": 0.1, + "repeat": 0, + "targetCluster": "gamma", + "raw": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 13454, + "usage": { + "promptTokenCount": 406824, + "candidatesTokenCount": 51, + "totalTokenCount": 409061, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406824 + } + ], + "thoughtsTokenCount": 2186, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"gamma\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_GAMMA_81042\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "standby", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": false + }, + "ms": 24190, + "usage": { + "promptTokenCount": 103157, + "candidatesTokenCount": 78, + "totalTokenCount": 107402, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 68661 + }, + { + "modality": "IMAGE", + "tokenCount": 34496 + } + ], + "thoughtsTokenCount": 4167, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"gamma\",\n \"primary_status\": \"standby\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_GAMMA_81042\"\n}" + } + }, + { + "depth": 0.1, + "repeat": 1, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_66409" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 10785, + "usage": { + "promptTokenCount": 406883, + "candidatesTokenCount": 53, + "totalTokenCount": 408241, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406883 + } + ], + "thoughtsTokenCount": 1305, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"epsilon\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_EPSILON_66409\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "active", + "final_route": "alpha", + "status_code": 200, + "mutation_token": "MUTATION_EPSILON_66409" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 29640, + "usage": { + "promptTokenCount": 103166, + "candidatesTokenCount": 76, + "totalTokenCount": 108173, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34496 + }, + { + "modality": "TEXT", + "tokenCount": 68670 + } + ], + "thoughtsTokenCount": 4931, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"active\",\n \"final_route\": \"alpha\",\n \"status_code\": 200,\n \"mutation_token\": \"MUTATION_EPSILON_66409\"\n}" + } + }, + { + "depth": 0.3, + "repeat": 0, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 16560, + "usage": { + "promptTokenCount": 406788, + "candidatesTokenCount": 53, + "totalTokenCount": 407621, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406788 + } + ], + "thoughtsTokenCount": 780, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"epsilon\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_EPSILON_14306\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance (active drain in progress)", + "backup_status": "none", + "final_route": "quarantine_mesh", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": false, + "routeOk": false, + "codeOk": true, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 33781, + "usage": { + "promptTokenCount": 103144, + "candidatesTokenCount": 84, + "totalTokenCount": 109547, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34496 + }, + { + "modality": "TEXT", + "tokenCount": 68648 + } + ], + "thoughtsTokenCount": 6319, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance (active drain in progress)\",\n \"backup_status\": \"none\",\n \"final_route\": \"quarantine_mesh\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_14306\"\n}" + } + }, + { + "depth": 0.3, + "repeat": 1, + "targetCluster": "delta", + "raw": { + "score": { + "parsed": { + "target_cluster": "delta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_DELTA_89673" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 7597, + "usage": { + "promptTokenCount": 406983, + "candidatesTokenCount": 51, + "totalTokenCount": 407807, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406983 + } + ], + "thoughtsTokenCount": 773, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"delta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_DELTA_89673\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "delta", + "primary_status": "degraded", + "backup_status": "standby", + "final_route": "delta", + "status_code": 200, + "mutation_token": "MUTATION_DELTA_89673" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 38064, + "usage": { + "promptTokenCount": 103199, + "candidatesTokenCount": 76, + "totalTokenCount": 109993, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34496 + }, + { + "modality": "TEXT", + "tokenCount": 68703 + } + ], + "thoughtsTokenCount": 6718, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"delta\",\n \"primary_status\": \"degraded\",\n \"backup_status\": \"standby\",\n \"final_route\": \"delta\",\n \"status_code\": 200,\n \"mutation_token\": \"MUTATION_DELTA_89673\"\n}" + } + }, + { + "depth": 0.5, + "repeat": 0, + "targetCluster": "theta", + "raw": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 21779, + "usage": { + "promptTokenCount": 406651, + "candidatesTokenCount": 52, + "totalTokenCount": 407461, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406651 + } + ], + "thoughtsTokenCount": 758, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"theta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_THETA_37570\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "disabled", + "backup_status": "none", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": false + }, + "ms": 37037, + "usage": { + "promptTokenCount": 103104, + "candidatesTokenCount": 52, + "totalTokenCount": 109983, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34496 + }, + { + "modality": "TEXT", + "tokenCount": 68608 + } + ], + "thoughtsTokenCount": 6827, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"theta\",\"primary_status\":\"disabled\",\"backup_status\":\"none\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_THETA_37570\"}" + } + }, + { + "depth": 0.5, + "repeat": 1, + "targetCluster": "zeta", + "raw": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_22937" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 9784, + "usage": { + "promptTokenCount": 406936, + "candidatesTokenCount": 52, + "totalTokenCount": 408289, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406936 + } + ], + "thoughtsTokenCount": 1301, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"zeta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_ZETA_22937\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "active", + "backup_status": "standby", + "final_route": "internal_mesh", + "status_code": 200, + "mutation_token": "MUTATION_ZETA_22937" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 43460, + "usage": { + "promptTokenCount": 103183, + "candidatesTokenCount": 78, + "totalTokenCount": 111122, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 68687 + }, + { + "modality": "IMAGE", + "tokenCount": 34496 + } + ], + "thoughtsTokenCount": 7861, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"active\",\n \"backup_status\": \"standby\",\n \"final_route\": \"internal_mesh\",\n \"status_code\": 200,\n \"mutation_token\": \"MUTATION_ZETA_22937\"\n}" + } + }, + { + "depth": 0.7, + "repeat": 0, + "targetCluster": "zeta", + "raw": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 10087, + "usage": { + "promptTokenCount": 406918, + "candidatesTokenCount": 52, + "totalTokenCount": 408152, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406918 + } + ], + "thoughtsTokenCount": 1182, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"zeta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_ZETA_48130\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "quarantined", + "backup_status": "N/A", + "final_route": "quarantine_drain", + "status_code": 423, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 27634, + "usage": { + "promptTokenCount": 103151, + "candidatesTokenCount": 82, + "totalTokenCount": 107705, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34512 + }, + { + "modality": "TEXT", + "tokenCount": 68639 + } + ], + "thoughtsTokenCount": 4472, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"quarantined\",\n \"backup_status\": \"N/A\",\n \"final_route\": \"quarantine_drain\",\n \"status_code\": 423,\n \"mutation_token\": \"MUTATION_ZETA_48130\"\n}" + } + }, + { + "depth": 0.7, + "repeat": 1, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_46201" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 9547, + "usage": { + "promptTokenCount": 406900, + "candidatesTokenCount": 79, + "totalTokenCount": 407886, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406900 + } + ], + "thoughtsTokenCount": 907, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_46201\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "quarantine_drain", + "backup_status": "none", + "final_route": "quarantine_mesh", + "status_code": 423, + "mutation_token": "MUTATION_EPSILON_46201" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 32344, + "usage": { + "promptTokenCount": 103184, + "candidatesTokenCount": 82, + "totalTokenCount": 108925, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34496 + }, + { + "modality": "TEXT", + "tokenCount": 68688 + } + ], + "thoughtsTokenCount": 5659, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"quarantine_drain\",\n \"backup_status\": \"none\",\n \"final_route\": \"quarantine_mesh\",\n \"status_code\": 423,\n \"mutation_token\": \"MUTATION_EPSILON_46201\"\n}" + } + }, + { + "depth": 0.9, + "repeat": 0, + "targetCluster": "omega", + "raw": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 7978, + "usage": { + "promptTokenCount": 406801, + "candidatesTokenCount": 52, + "totalTokenCount": 407639, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406801 + } + ], + "thoughtsTokenCount": 786, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"omega\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_OMEGA_71394\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "maintenance (active drain in progress)", + "backup_status": "disabled (kernel panic in node pool)", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 11836, + "usage": { + "promptTokenCount": 103266, + "candidatesTokenCount": 95, + "totalTokenCount": 104617, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 68736 + }, + { + "modality": "IMAGE", + "tokenCount": 34530 + } + ], + "thoughtsTokenCount": 1256, + "serviceTier": "standard" + }, + "text": "[\n {\n \"target_cluster\": \"omega\",\n \"primary_status\": \"maintenance (active drain in progress)\",\n \"backup_status\": \"disabled (kernel panic in node pool)\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_OMEGA_71394\"\n }\n]" + } + }, + { + "depth": 0.9, + "repeat": 1, + "targetCluster": "theta", + "raw": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_56761" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 7688, + "usage": { + "promptTokenCount": 406937, + "candidatesTokenCount": 52, + "totalTokenCount": 407655, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 406937 + } + ], + "thoughtsTokenCount": 666, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"theta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_THETA_56761\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance (active drain in progress)", + "backup_status": "disabled (kernel panic in node pool)", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_56761" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 32285, + "usage": { + "promptTokenCount": 103295, + "candidatesTokenCount": 89, + "totalTokenCount": 108787, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 34530 + }, + { + "modality": "TEXT", + "tokenCount": 68765 + } + ], + "thoughtsTokenCount": 5403, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"theta\",\n \"primary_status\": \"maintenance (active drain in progress)\",\n \"backup_status\": \"disabled (kernel panic in node pool)\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_THETA_56761\"\n}" + } + } + ] +} \ No newline at end of file diff --git a/eval/gemini-profile/lost-in-middle-300k-gemini-3.7-flash-results.json b/eval/gemini-profile/lost-in-middle-300k-gemini-3.7-flash-results.json new file mode 100644 index 000000000..87d94a1ba --- /dev/null +++ b/eval/gemini-profile/lost-in-middle-300k-gemini-3.7-flash-results.json @@ -0,0 +1,444 @@ +{ + "generatedAt": "2026-08-22T16:28:26.494Z", + "summary": { + "model": "gemini-3.7-flash", + "targetChars": 300000, + "depths": [ + 0.1, + 0.3, + 0.5, + 0.7, + 0.9 + ], + "repeats": 1, + "byDepth": { + "0.1": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 3751, + "pxpipe_avg_ms": 13201 + }, + "0.3": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 3794, + "pxpipe_avg_ms": 5725 + }, + "0.5": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 3689, + "pxpipe_avg_ms": 14973 + }, + "0.7": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 3008, + "pxpipe_avg_ms": 16028 + }, + "0.9": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 0, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 2936, + "pxpipe_avg_ms": 11451 + } + } + }, + "results": [ + { + "depth": 0.1, + "repeat": 0, + "targetCluster": "gamma", + "raw": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 3751, + "usage": { + "promptTokenCount": 122540, + "candidatesTokenCount": 51, + "totalTokenCount": 123316, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122540 + } + ], + "thoughtsTokenCount": 725, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"gamma\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_GAMMA_81042\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 13201, + "usage": { + "promptTokenCount": 21118, + "candidatesTokenCount": 77, + "totalTokenCount": 22433, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4994 + }, + { + "modality": "IMAGE", + "tokenCount": 16124 + } + ], + "thoughtsTokenCount": 1238, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"gamma\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_GAMMA_81042\"\n}" + } + }, + { + "depth": 0.3, + "repeat": 0, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 3794, + "usage": { + "promptTokenCount": 122548, + "candidatesTokenCount": 53, + "totalTokenCount": 123478, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122548 + } + ], + "thoughtsTokenCount": 877, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"epsilon\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_EPSILON_14306\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 5725, + "usage": { + "promptTokenCount": 21097, + "candidatesTokenCount": 79, + "totalTokenCount": 21896, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4987 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 720, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_14306\"\n}" + } + }, + { + "depth": 0.5, + "repeat": 0, + "targetCluster": "theta", + "raw": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 3689, + "usage": { + "promptTokenCount": 122534, + "candidatesTokenCount": 51, + "totalTokenCount": 123180, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122534 + } + ], + "thoughtsTokenCount": 595, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"theta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_THETA_37570\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 14973, + "usage": { + "promptTokenCount": 21093, + "candidatesTokenCount": 78, + "totalTokenCount": 21894, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4983 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 723, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"theta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_THETA_37570\"\n}" + } + }, + { + "depth": 0.7, + "repeat": 0, + "targetCluster": "zeta", + "raw": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 3008, + "usage": { + "promptTokenCount": 122558, + "candidatesTokenCount": 78, + "totalTokenCount": 123150, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122558 + } + ], + "thoughtsTokenCount": 514, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_ZETA_48130\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance (active drain in progress)", + "backup_status": "disabled (kernel panic in node pool)", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 16028, + "usage": { + "promptTokenCount": 21097, + "candidatesTokenCount": 89, + "totalTokenCount": 22440, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4987 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 1254, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"maintenance (active drain in progress)\",\n \"backup_status\": \"disabled (kernel panic in node pool)\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_ZETA_48130\"\n}" + } + }, + { + "depth": 0.9, + "repeat": 0, + "targetCluster": "omega", + "raw": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 2936, + "usage": { + "promptTokenCount": 122484, + "candidatesTokenCount": 78, + "totalTokenCount": 122846, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122484 + } + ], + "thoughtsTokenCount": 284, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"omega\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_OMEGA_71394\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "disabled", + "backup_status": "active", + "final_route": "beta", + "status_code": 200, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 11451, + "usage": { + "promptTokenCount": 21098, + "candidatesTokenCount": 75, + "totalTokenCount": 23844, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 16124 + }, + { + "modality": "TEXT", + "tokenCount": 4974 + } + ], + "thoughtsTokenCount": 2671, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"omega\",\n \"primary_status\": \"disabled\",\n \"backup_status\": \"active\",\n \"final_route\": \"beta\",\n \"status_code\": 200,\n \"mutation_token\": \"MUTATION_OMEGA_71394\"\n}" + } + } + ] +} \ No newline at end of file diff --git a/eval/gemini-profile/lost-in-middle-300k-results.json b/eval/gemini-profile/lost-in-middle-300k-results.json new file mode 100644 index 000000000..930cbbae8 --- /dev/null +++ b/eval/gemini-profile/lost-in-middle-300k-results.json @@ -0,0 +1,843 @@ +{ + "generatedAt": "2026-08-22T15:59:13.661Z", + "summary": { + "model": "gemini-3.6-flash", + "targetChars": 300000, + "depths": [ + 0.1, + 0.3, + 0.5, + 0.7, + 0.9 + ], + "repeats": 2, + "byDepth": { + "0.1": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 7862, + "pxpipe_avg_ms": 17562 + }, + "0.3": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 6566, + "pxpipe_avg_ms": 13473 + }, + "0.5": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 0.5, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0.5, + "raw_avg_ms": 7003, + "pxpipe_avg_ms": 18619 + }, + "0.7": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 1, + "raw_avg_ms": 8747, + "pxpipe_avg_ms": 23961 + }, + "0.9": { + "raw_reasoning_acc": 1, + "pxpipe_reasoning_acc": 1, + "raw_multihop_acc": 1, + "pxpipe_multihop_acc": 0, + "raw_avg_ms": 4963, + "pxpipe_avg_ms": 15264 + } + } + }, + "results": [ + { + "depth": 0.1, + "repeat": 0, + "targetCluster": "gamma", + "raw": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 7504, + "usage": { + "promptTokenCount": 122540, + "candidatesTokenCount": 77, + "totalTokenCount": 123959, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122540 + } + ], + "thoughtsTokenCount": 1342, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"gamma\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_GAMMA_81042\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "gamma", + "primary_status": "maintenance (active drain in progress)", + "backup_status": "disabled (kernel panic in node pool)", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_GAMMA_81042" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 23524, + "usage": { + "promptTokenCount": 21118, + "candidatesTokenCount": 88, + "totalTokenCount": 25034, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4994 + }, + { + "modality": "IMAGE", + "tokenCount": 16124 + } + ], + "thoughtsTokenCount": 3828, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"gamma\",\n \"primary_status\": \"maintenance (active drain in progress)\",\n \"backup_status\": \"disabled (kernel panic in node pool)\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_GAMMA_81042\"\n}" + } + }, + { + "depth": 0.1, + "repeat": 1, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_66409" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 8220, + "usage": { + "promptTokenCount": 122600, + "candidatesTokenCount": 53, + "totalTokenCount": 124091, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122600 + } + ], + "thoughtsTokenCount": 1438, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"epsilon\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_EPSILON_66409\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_66409" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 11599, + "usage": { + "promptTokenCount": 21102, + "candidatesTokenCount": 79, + "totalTokenCount": 22660, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4992 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 1479, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_66409\"\n}" + } + }, + { + "depth": 0.3, + "repeat": 0, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 4944, + "usage": { + "promptTokenCount": 122548, + "candidatesTokenCount": 79, + "totalTokenCount": 123177, + "cachedContentTokenCount": 8168, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122548 + } + ], + "cacheTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 8168 + } + ], + "thoughtsTokenCount": 550, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_14306\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_14306" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 12724, + "usage": { + "promptTokenCount": 21097, + "candidatesTokenCount": 79, + "totalTokenCount": 23202, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4987 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 2026, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_14306\"\n}" + } + }, + { + "depth": 0.3, + "repeat": 1, + "targetCluster": "delta", + "raw": { + "score": { + "parsed": { + "target_cluster": "delta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_DELTA_89673" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 8188, + "usage": { + "promptTokenCount": 122487, + "candidatesTokenCount": 77, + "totalTokenCount": 124036, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122487 + } + ], + "thoughtsTokenCount": 1472, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"delta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_DELTA_89673\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "delta", + "primary_status": "maintenance (active drain in progress)", + "backup_status": "disabled (kernel panic in node pool)", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_DELTA_89673" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 14221, + "usage": { + "promptTokenCount": 21095, + "candidatesTokenCount": 88, + "totalTokenCount": 23491, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4985 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 2308, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"delta\",\n \"primary_status\": \"maintenance (active drain in progress)\",\n \"backup_status\": \"disabled (kernel panic in node pool)\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_DELTA_89673\"\n}" + } + }, + { + "depth": 0.5, + "repeat": 0, + "targetCluster": "theta", + "raw": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 6789, + "usage": { + "promptTokenCount": 122534, + "candidatesTokenCount": 63, + "totalTokenCount": 123641, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122534 + } + ], + "thoughtsTokenCount": 1044, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\": \"theta\", \"primary_status\": \"maintenance\", \"backup_status\": \"disabled\", \"final_route\": \"blackhole_sink\", \"status_code\": 503, \"mutation_token\": \"MUTATION_THETA_37570\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_37570" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 13384, + "usage": { + "promptTokenCount": 21093, + "candidatesTokenCount": 78, + "totalTokenCount": 23439, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4983 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 2268, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"theta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_THETA_37570\"\n}" + } + }, + { + "depth": 0.5, + "repeat": 1, + "targetCluster": "zeta", + "raw": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_22937" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 7216, + "usage": { + "promptTokenCount": 122509, + "candidatesTokenCount": 52, + "totalTokenCount": 123608, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122509 + } + ], + "thoughtsTokenCount": 1047, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"zeta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_ZETA_22937\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "active", + "final_route": "alpha", + "status_code": 200, + "mutation_token": "MUTATION_ZETA_22937" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": false, + "routeOk": false, + "codeOk": false, + "reasoningSuccess": false, + "fullMultiHopSuccess": false + }, + "ms": 23854, + "usage": { + "promptTokenCount": 21113, + "candidatesTokenCount": 75, + "totalTokenCount": 25671, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 5003 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 4483, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"active\",\n \"final_route\": \"alpha\",\n \"status_code\": 200,\n \"mutation_token\": \"MUTATION_ZETA_22937\"\n}" + } + }, + { + "depth": 0.7, + "repeat": 0, + "targetCluster": "zeta", + "raw": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 11869, + "usage": { + "promptTokenCount": 122558, + "candidatesTokenCount": 52, + "totalTokenCount": 124689, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122558 + } + ], + "thoughtsTokenCount": 2079, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"zeta\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_ZETA_48130\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "zeta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_ZETA_48130" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 33589, + "usage": { + "promptTokenCount": 21097, + "candidatesTokenCount": 78, + "totalTokenCount": 27528, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 16110 + }, + { + "modality": "TEXT", + "tokenCount": 4987 + } + ], + "thoughtsTokenCount": 6353, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"zeta\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_ZETA_48130\"\n}" + } + }, + { + "depth": 0.7, + "repeat": 1, + "targetCluster": "epsilon", + "raw": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_46201" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 5625, + "usage": { + "promptTokenCount": 122589, + "candidatesTokenCount": 53, + "totalTokenCount": 123162, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122589 + } + ], + "thoughtsTokenCount": 520, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\":\"epsilon\",\"primary_status\":\"maintenance\",\"backup_status\":\"disabled\",\"final_route\":\"blackhole_sink\",\"status_code\":503,\"mutation_token\":\"MUTATION_EPSILON_46201\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "epsilon", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_EPSILON_46201" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 14332, + "usage": { + "promptTokenCount": 21104, + "candidatesTokenCount": 79, + "totalTokenCount": 23617, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4994 + }, + { + "modality": "IMAGE", + "tokenCount": 16110 + } + ], + "thoughtsTokenCount": 2434, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"epsilon\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_EPSILON_46201\"\n}" + } + }, + { + "depth": 0.9, + "repeat": 0, + "targetCluster": "omega", + "raw": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 4997, + "usage": { + "promptTokenCount": 122484, + "candidatesTokenCount": 78, + "totalTokenCount": 123279, + "cachedContentTokenCount": 24506, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122484 + } + ], + "cacheTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 24506 + } + ], + "thoughtsTokenCount": 717, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"omega\",\n \"primary_status\": \"maintenance\",\n \"backup_status\": \"disabled\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_OMEGA_71394\"\n}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "omega", + "primary_status": "disabled", + "backup_status": "degraded", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_OMEGA_71394" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": false + }, + "ms": 11867, + "usage": { + "promptTokenCount": 21098, + "candidatesTokenCount": 79, + "totalTokenCount": 22894, + "promptTokensDetails": [ + { + "modality": "IMAGE", + "tokenCount": 16124 + }, + { + "modality": "TEXT", + "tokenCount": 4974 + } + ], + "thoughtsTokenCount": 1717, + "serviceTier": "standard" + }, + "text": "{\n \"target_cluster\": \"omega\",\n \"primary_status\": \"disabled\",\n \"backup_status\": \"degraded\",\n \"final_route\": \"blackhole_sink\",\n \"status_code\": 503,\n \"mutation_token\": \"MUTATION_OMEGA_71394\"\n}" + } + }, + { + "depth": 0.9, + "repeat": 1, + "targetCluster": "theta", + "raw": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "maintenance", + "backup_status": "disabled", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_56761" + }, + "tokenFound": true, + "primaryStateOk": true, + "backupStateOk": true, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": true + }, + "ms": 4929, + "usage": { + "promptTokenCount": 122545, + "candidatesTokenCount": 63, + "totalTokenCount": 123226, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 122545 + } + ], + "thoughtsTokenCount": 618, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\": \"theta\", \"primary_status\": \"maintenance\", \"backup_status\": \"disabled\", \"final_route\": \"blackhole_sink\", \"status_code\": 503, \"mutation_token\": \"MUTATION_THETA_56761\"}" + }, + "pxpipe": { + "score": { + "parsed": { + "target_cluster": "theta", + "primary_status": "standby", + "backup_status": "standby", + "final_route": "blackhole_sink", + "status_code": 503, + "mutation_token": "MUTATION_THETA_56761" + }, + "tokenFound": true, + "primaryStateOk": false, + "backupStateOk": false, + "routeOk": true, + "codeOk": true, + "reasoningSuccess": true, + "fullMultiHopSuccess": false + }, + "ms": 18661, + "usage": { + "promptTokenCount": 21108, + "candidatesTokenCount": 65, + "totalTokenCount": 24496, + "promptTokensDetails": [ + { + "modality": "TEXT", + "tokenCount": 4984 + }, + { + "modality": "IMAGE", + "tokenCount": 16124 + } + ], + "thoughtsTokenCount": 3323, + "serviceTier": "standard" + }, + "text": "{\"target_cluster\": \"theta\", \"primary_status\": \"standby\", \"backup_status\": \"standby\", \"final_route\": \"blackhole_sink\", \"status_code\": 503, \"mutation_token\": \"MUTATION_THETA_56761\"}" + } + } + ] +} \ No newline at end of file diff --git a/eval/gemini-profile/lost-in-middle-results.json b/eval/gemini-profile/lost-in-middle-results.json deleted file mode 100644 index 2db396fca..000000000 --- a/eval/gemini-profile/lost-in-middle-results.json +++ /dev/null @@ -1,3719 +0,0 @@ -{ - "generatedAt": "2026-07-22T13:20:47.917Z", - "model": "gemini-3.6-flash", - "methodology": { - "task": "production-faithful old-history lookup by relative record depth", - "depths": [ - 0.05, - 0.25, - 0.5, - 0.75, - 0.95 - ], - "sizes": [ - 2000, - 6000, - 10000 - ], - "repeats": 2, - "oldTurns": 35, - "imageCap": 32, - "transformer": "transformGoogleGenerateContent", - "factsheet": "production output, target coverage recorded per probe", - "measures": { - "acknowledged": "model says the named target key exists; weak evidence by itself", - "absentRejected": "model correctly says a matched nonexistent key is absent", - "localization": "correct adjacent region word copied from the target row", - "recognition": "localization plus correct semantic status word", - "exact": "recognition plus exact synthetic reference" - } - }, - "summary": { - "completed": 30, - "errors": 0, - "raw": { - "acknowledged": 29, - "absentRejected": 29, - "localization": 17, - "recognition": 13, - "exact": 3 - }, - "pxpipe": { - "acknowledged": 29, - "absentRejected": 29, - "localization": 18, - "recognition": 11, - "exact": 3 - }, - "bySizeDepth": { - "2000": { - "0.05": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 0, - "exact": 0 - } - }, - "0.25": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 0, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 0, - "exact": 0 - } - }, - "0.5": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 1, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 1, - "absentRejected": 1, - "localization": 0, - "recognition": 0, - "exact": 0 - } - }, - "0.75": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 1, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 1 - } - }, - "0.95": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 1, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 0 - } - } - }, - "6000": { - "0.05": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 1 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 1 - } - }, - "0.25": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 0, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 0, - "exact": 0 - } - }, - "0.5": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 0, - "recognition": 0, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 0, - "recognition": 0, - "exact": 0 - } - }, - "0.75": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 1, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 1, - "exact": 0 - } - }, - "0.95": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 1 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 1, - "exact": 0 - } - } - }, - "10000": { - "0.05": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 0, - "recognition": 0, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 0, - "recognition": 0, - "exact": 0 - } - }, - "0.25": { - "n": 2, - "raw": { - "acknowledged": 1, - "absentRejected": 1, - "localization": 0, - "recognition": 0, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 0, - "recognition": 0, - "exact": 0 - } - }, - "0.5": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 1, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 1, - "exact": 0 - } - }, - "0.75": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 1 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 2, - "recognition": 2, - "exact": 1 - } - }, - "0.95": { - "n": 2, - "raw": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 0, - "exact": 0 - }, - "pxpipe": { - "acknowledged": 2, - "absentRejected": 2, - "localization": 1, - "recognition": 0, - "exact": 0 - } - } - } - } - }, - "rows": [ - { - "records": 2000, - "sourceChars": 164394, - "repeat": 0, - "depth": 0.05, - "index": 100, - "target": { - "key": "case_000100", - "status": "silver", - "region": "birch", - "reference": "ref-0txnff9", - "text": "case_000100 | region=birch | status=silver | reference=ref-0txnff9 | queue=archive" - }, - "absentKey": "case_x00100", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165357, - "imageTokens": 6468, - "baselineImagedTokens": 46976, - "factsheetChars": 389, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"birch\",\"status\":\"silver\",\"reference\":\"ref-0jkkmp4\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 33, - "totalTokenCount": 7850, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 576 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 751, - "serviceTier": "standard" - }, - "ms": 5991, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "silver", - "reference": "ref-0jkkmp4" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"amber\", \"reference\": \"ref-0jkkmp4\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 40, - "totalTokenCount": 7701, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 576 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 595, - "serviceTier": "standard" - }, - "ms": 4012, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "amber", - "reference": "ref-0jkkmp4" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164394, - "repeat": 0, - "depth": 0.25, - "index": 500, - "target": { - "key": "case_000500", - "status": "jade", - "region": "willow", - "reference": "ref-193u44l", - "text": "case_000500 | region=willow | status=jade | reference=ref-193u44l | queue=archive" - }, - "absentKey": "case_x00500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165357, - "imageTokens": 6468, - "baselineImagedTokens": 46976, - "factsheetChars": 389, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"amber\", \"reference\": \"ref-017gcvm\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 41, - "totalTokenCount": 8122, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 576 - } - ], - "thoughtsTokenCount": 1015, - "serviceTier": "standard" - }, - "ms": 6128, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "amber", - "reference": "ref-017gcvm" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"amber\", \"reference\": \"ref-017qcdm\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 41, - "totalTokenCount": 7688, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 576 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 581, - "serviceTier": "standard" - }, - "ms": 4028, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "amber", - "reference": "ref-017qcdm" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164394, - "repeat": 0, - "depth": 0.5, - "index": 1000, - "target": { - "key": "case_001000", - "status": "plum", - "region": "birch", - "reference": "ref-1abagih", - "text": "case_001000 | region=birch | status=plum | reference=ref-1abagih | queue=archive" - }, - "absentKey": "case_x01000", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165357, - "imageTokens": 6468, - "baselineImagedTokens": 46976, - "factsheetChars": 389, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\n \"target_present\": true,\n \"absent_present\": false,\n \"region\": \"birch\",\n \"status\": \"plum\",\n \"reference\": \"ref-1au7k9\"\n}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 53, - "totalTokenCount": 8346, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 576 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 1227, - "serviceTier": "standard" - }, - "ms": 7350, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "plum", - "reference": "ref-1au7k9" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "`\nWait, `0f822c2` is in the factsheet! \"0f822c2\" is next to case_001001 or case_001009?\nIn turn 18 transcript:\n`queue=archive/case_001000 | region=maple | status=cobalt |", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 82, - "totalTokenCount": 9110, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 576 - } - ], - "thoughtsTokenCount": 1962, - "serviceTier": "standard" - }, - "ms": 12051, - "score": { - "parsed": null, - "acknowledged": false, - "absentRejected": false, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164394, - "repeat": 0, - "depth": 0.75, - "index": 1500, - "target": { - "key": "case_001500", - "status": "silver", - "region": "maple", - "reference": "ref-1biqswd", - "text": "case_001500 | region=maple | status=silver | reference=ref-1biqswd | queue=archive" - }, - "absentKey": "case_x01500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165357, - "imageTokens": 6468, - "baselineImagedTokens": 46976, - "factsheetChars": 389, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"alder\",\"status\":\"silver\",\"reference\":\"ref-0a0xpxt\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 33, - "totalTokenCount": 7776, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 576 - } - ], - "thoughtsTokenCount": 677, - "serviceTier": "standard" - }, - "ms": 5644, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "silver", - "reference": "ref-0a0xpxt" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"maple\", \"status\": \"silver\", \"reference\": \"ref-1biqswd\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 40, - "totalTokenCount": 7693, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 576 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 587, - "serviceTier": "standard" - }, - "ms": 4279, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "silver", - "reference": "ref-1biqswd" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": true, - "localization": true, - "recognition": true, - "exact": true - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164394, - "repeat": 0, - "depth": 0.95, - "index": 1900, - "target": { - "key": "case_001900", - "status": "jade", - "region": "birch", - "reference": "ref-1qoxhlp", - "text": "case_001900 | region=birch | status=jade | reference=ref-1qoxhlp | queue=archive" - }, - "absentKey": "case_x01900", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165357, - "imageTokens": 6468, - "baselineImagedTokens": 46976, - "factsheetChars": 389, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"jade\", \"reference\": \"ref-190xhlp\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 41, - "totalTokenCount": 7848, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 576 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 741, - "serviceTier": "standard" - }, - "ms": 9816, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "jade", - "reference": "ref-190xhlp" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"jade\", \"reference\": \"ref-19oxhlp\"}", - "usage": { - "promptTokenCount": 7066, - "candidatesTokenCount": 40, - "totalTokenCount": 7796, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 576 - } - ], - "thoughtsTokenCount": 690, - "serviceTier": "standard" - }, - "ms": 4165, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "jade", - "reference": "ref-19oxhlp" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164397, - "repeat": 1, - "depth": 0.05, - "index": 100, - "target": { - "key": "case_000100", - "status": "plum", - "region": "alder", - "reference": "ref-1v3cmrw", - "text": "case_000100 | region=alder | status=plum | reference=ref-1v3cmrw | queue=archive" - }, - "absentKey": "case_x00100", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165360, - "imageTokens": 6468, - "baselineImagedTokens": 46978, - "factsheetChars": 289, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"plum\", \"reference\": \"ref-1k9quir\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 40, - "totalTokenCount": 7780, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 483 - } - ], - "thoughtsTokenCount": 767, - "serviceTier": "standard" - }, - "ms": 14514, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "plum", - "reference": "ref-1k9quir" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"alder\",\"status\":\"amber\",\"reference\":\"ref-0j8gr3\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 33, - "totalTokenCount": 7840, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 483 - } - ], - "thoughtsTokenCount": 834, - "serviceTier": "standard" - }, - "ms": 6153, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "amber", - "reference": "ref-0j8gr3" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164397, - "repeat": 1, - "depth": 0.25, - "index": 500, - "target": { - "key": "case_000500", - "status": "amber", - "region": "willow", - "reference": "ref-0b8f9i4", - "text": "case_000500 | region=willow | status=amber | reference=ref-0b8f9i4 | queue=archive" - }, - "absentKey": "case_x00500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165360, - "imageTokens": 6468, - "baselineImagedTokens": 46978, - "factsheetChars": 289, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"willow\",\"status\":\"silver\",\"reference\":\"ref-01m0s7c\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 35, - "totalTokenCount": 7656, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 483 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 648, - "serviceTier": "standard" - }, - "ms": 5773, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-01m0s7c" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"silver\", \"reference\": \"ref-1j4t18t\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 42, - "totalTokenCount": 7686, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 483 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 671, - "serviceTier": "standard" - }, - "ms": 6161, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-1j4t18t" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164397, - "repeat": 1, - "depth": 0.5, - "index": 1000, - "target": { - "key": "case_001000", - "status": "cobalt", - "region": "birch", - "reference": "ref-0cfvlw0", - "text": "case_001000 | region=birch | status=cobalt | reference=ref-0cfvlw0 | queue=archive" - }, - "absentKey": "case_x01000", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165360, - "imageTokens": 6468, - "baselineImagedTokens": 46978, - "factsheetChars": 289, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\n \"target_present\": true,\n \"absent_present\": false,\n \"region\": \"alder\",\n \"status\": \"plum\",\n \"reference\": \"ref-0iugao5\"\n}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 53, - "totalTokenCount": 7898, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 483 - } - ], - "thoughtsTokenCount": 872, - "serviceTier": "standard" - }, - "ms": 6979, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "plum", - "reference": "ref-0iugao5" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"maple\", \"status\": \"silver\", \"reference\": \"ref-01ugao5\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 40, - "totalTokenCount": 7989, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 483 - } - ], - "thoughtsTokenCount": 976, - "serviceTier": "standard" - }, - "ms": 6292, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "silver", - "reference": "ref-01ugao5" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164397, - "repeat": 1, - "depth": 0.75, - "index": 1500, - "target": { - "key": "case_001500", - "status": "jade", - "region": "maple", - "reference": "ref-0dnby9w", - "text": "case_001500 | region=maple | status=jade | reference=ref-0dnby9w | queue=archive" - }, - "absentKey": "case_x01500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165360, - "imageTokens": 6468, - "baselineImagedTokens": 46978, - "factsheetChars": 289, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"maple\",\"status\":\"jade\",\"reference\":\"ref-8dnbyw\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 32, - "totalTokenCount": 7881, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 483 - } - ], - "thoughtsTokenCount": 876, - "serviceTier": "standard" - }, - "ms": 6993, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "jade", - "reference": "ref-8dnbyw" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"maple\", \"status\": \"jade\", \"reference\": \"ref-8dnby9w\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 40, - "totalTokenCount": 7715, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 483 - } - ], - "thoughtsTokenCount": 702, - "serviceTier": "standard" - }, - "ms": 8000, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "jade", - "reference": "ref-8dnby9w" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 2000, - "sourceChars": 164397, - "repeat": 1, - "depth": 0.95, - "index": 1900, - "target": { - "key": "case_001900", - "status": "amber", - "region": "birch", - "reference": "ref-0stimz8", - "text": "case_001900 | region=birch | status=amber | reference=ref-0stimz8 | queue=archive" - }, - "absentKey": "case_x01900", - "production": { - "collapsedTurns": 35, - "collapsedImages": 6, - "collapsedChars": 165360, - "imageTokens": 6468, - "baselineImagedTokens": 46978, - "factsheetChars": 289, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"alder\",\"status\":\"amber\",\"reference\":\"ref-16629k3\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 35, - "totalTokenCount": 7930, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 483 - }, - { - "modality": "IMAGE", - "tokenCount": 6490 - } - ], - "thoughtsTokenCount": 922, - "serviceTier": "standard" - }, - "ms": 7787, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "amber", - "reference": "ref-16629k3" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"birch\",\"status\":\"amber\",\"reference\":\"ref-1j78x7n\"}", - "usage": { - "promptTokenCount": 6973, - "candidatesTokenCount": 35, - "totalTokenCount": 7496, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 6490 - }, - { - "modality": "TEXT", - "tokenCount": 483 - } - ], - "thoughtsTokenCount": 488, - "serviceTier": "standard" - }, - "ms": 4518, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "amber", - "reference": "ref-1j78x7n" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493170, - "repeat": 0, - "depth": 0.05, - "index": 300, - "target": { - "key": "case_000300", - "status": "amber", - "region": "birch", - "reference": "ref-0206qsd", - "text": "case_000300 | region=birch | status=amber | reference=ref-0206qsd | queue=archive" - }, - "absentKey": "case_x00300", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494133, - "imageTokens": 19404, - "baselineImagedTokens": 140908, - "factsheetChars": 706, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"amber\", \"reference\": \"ref-1qo00ic\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 40, - "totalTokenCount": 21066, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 848 - } - ], - "thoughtsTokenCount": 752, - "serviceTier": "standard" - }, - "ms": 7385, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "amber", - "reference": "ref-1qo00ic" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"birch\",\"status\":\"amber\",\"reference\":\"ref-1qo01c\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 33, - "totalTokenCount": 20861, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 848 - } - ], - "thoughtsTokenCount": 554, - "serviceTier": "standard" - }, - "ms": 5061, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "amber", - "reference": "ref-1qo01c" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493170, - "repeat": 0, - "depth": 0.25, - "index": 1500, - "target": { - "key": "case_001500", - "status": "silver", - "region": "maple", - "reference": "ref-1biqswd", - "text": "case_001500 | region=maple | status=silver | reference=ref-1biqswd | queue=archive" - }, - "absentKey": "case_x01500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494133, - "imageTokens": 19404, - "baselineImagedTokens": 140908, - "factsheetChars": 706, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"maple\", \"status\": \"jade\", \"reference\": \"ref-03mdi50\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 40, - "totalTokenCount": 20927, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 848 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 680 - }, - { - "modality": "IMAGE", - "tokenCount": 15578 - } - ], - "thoughtsTokenCount": 613, - "serviceTier": "standard" - }, - "ms": 6074, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "jade", - "reference": "ref-03mdi50" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"maple\", \"status\": \"jade\", \"reference\": \"ref-0f2cml2\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 41, - "totalTokenCount": 21884, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 848 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 680 - }, - { - "modality": "IMAGE", - "tokenCount": 15578 - } - ], - "thoughtsTokenCount": 1569, - "serviceTier": "standard" - }, - "ms": 8654, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "jade", - "reference": "ref-0f2cml2" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493170, - "repeat": 0, - "depth": 0.5, - "index": 3000, - "target": { - "key": "case_003000", - "status": "jade", - "region": "alder", - "reference": "ref-1f53u21", - "text": "case_003000 | region=alder | status=jade | reference=ref-1f53u21 | queue=archive" - }, - "absentKey": "case_x03000", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494133, - "imageTokens": 19404, - "baselineImagedTokens": 140908, - "factsheetChars": 706, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"amber\", \"reference\": \"ref-0206qsd\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 41, - "totalTokenCount": 21254, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 848 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15578 - }, - { - "modality": "TEXT", - "tokenCount": 680 - } - ], - "thoughtsTokenCount": 939, - "serviceTier": "standard" - }, - "ms": 7750, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "amber", - "reference": "ref-0206qsd" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"birch\",\"status\":\"amber\",\"reference\":\"ref-1q081c\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 34, - "totalTokenCount": 21200, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 848 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 680 - }, - { - "modality": "IMAGE", - "tokenCount": 15578 - } - ], - "thoughtsTokenCount": 892, - "serviceTier": "standard" - }, - "ms": 6134, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "amber", - "reference": "ref-1q081c" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493170, - "repeat": 0, - "depth": 0.75, - "index": 4500, - "target": { - "key": "case_004500", - "status": "amber", - "region": "cedar", - "reference": "ref-1irgv7p", - "text": "case_004500 | region=cedar | status=amber | reference=ref-1irgv7p | queue=archive" - }, - "absentKey": "case_x04500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494133, - "imageTokens": 19404, - "baselineImagedTokens": 140908, - "factsheetChars": 706, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"cedar\",\"status\":\"amber\",\"reference\":\"ref-00m683\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 34, - "totalTokenCount": 21131, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 848 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 680 - }, - { - "modality": "IMAGE", - "tokenCount": 15578 - } - ], - "thoughtsTokenCount": 823, - "serviceTier": "standard" - }, - "ms": 7644, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "cedar", - "status": "amber", - "reference": "ref-00m683" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"cedar\", \"status\": \"amber\", \"reference\": \"ref-18e2ghk\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 41, - "totalTokenCount": 21519, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 848 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15578 - }, - { - "modality": "TEXT", - "tokenCount": 680 - } - ], - "thoughtsTokenCount": 1204, - "serviceTier": "standard" - }, - "ms": 7160, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "cedar", - "status": "amber", - "reference": "ref-18e2ghk" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493170, - "repeat": 0, - "depth": 0.95, - "index": 5700, - "target": { - "key": "case_005700", - "status": "plum", - "region": "alder", - "reference": "ref-0t8wvcl", - "text": "case_005700 | region=alder | status=plum | reference=ref-0t8wvcl | queue=archive" - }, - "absentKey": "case_x05700", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494133, - "imageTokens": 19404, - "baselineImagedTokens": 140908, - "factsheetChars": 706, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"plum\", \"reference\": \"ref-0t8wvcl\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 40, - "totalTokenCount": 20990, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 848 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15578 - }, - { - "modality": "TEXT", - "tokenCount": 680 - } - ], - "thoughtsTokenCount": 676, - "serviceTier": "standard" - }, - "ms": 6366, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "plum", - "reference": "ref-0t8wvcl" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": true, - "localization": true, - "recognition": true, - "exact": true - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"amber\", \"reference\": \"ref-0t8wvcl\"}", - "usage": { - "promptTokenCount": 20274, - "candidatesTokenCount": 40, - "totalTokenCount": 20918, - "cachedContentTokenCount": 16258, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 848 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15578 - }, - { - "modality": "TEXT", - "tokenCount": 680 - } - ], - "thoughtsTokenCount": 604, - "serviceTier": "standard" - }, - "ms": 4896, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "amber", - "reference": "ref-0t8wvcl" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": true, - "localization": true, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493186, - "repeat": 1, - "depth": 0.05, - "index": 300, - "target": { - "key": "case_000300", - "status": "silver", - "region": "willow", - "reference": "ref-135vy50", - "text": "case_000300 | region=willow | status=silver | reference=ref-135vy50 | queue=archive" - }, - "absentKey": "case_x00300", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494149, - "imageTokens": 19404, - "baselineImagedTokens": 140917, - "factsheetChars": 528, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"willow\",\"status\":\"silver\",\"reference\":\"ref-135vy50\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 34, - "totalTokenCount": 21180, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 682 - } - ], - "thoughtsTokenCount": 1038, - "serviceTier": "standard" - }, - "ms": 8620, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-135vy50" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": true, - "localization": true, - "recognition": true, - "exact": true - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"silver\", \"reference\": \"ref-135vy50\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 41, - "totalTokenCount": 20646, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 682 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "thoughtsTokenCount": 497, - "serviceTier": "standard" - }, - "ms": 5596, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-135vy50" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": true, - "localization": true, - "recognition": true, - "exact": true - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493186, - "repeat": 1, - "depth": 0.25, - "index": 1500, - "target": { - "key": "case_001500", - "status": "jade", - "region": "maple", - "reference": "ref-0dnby9w", - "text": "case_001500 | region=maple | status=jade | reference=ref-0dnby9w | queue=archive" - }, - "absentKey": "case_x01500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494149, - "imageTokens": 19404, - "baselineImagedTokens": 140917, - "factsheetChars": 528, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"plum\", \"reference\": \"ref-14s28ib\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 41, - "totalTokenCount": 20730, - "cachedContentTokenCount": 16256, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 682 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15705 - }, - { - "modality": "TEXT", - "tokenCount": 551 - } - ], - "thoughtsTokenCount": 581, - "serviceTier": "standard" - }, - "ms": 6846, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "plum", - "reference": "ref-14s28ib" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"jade\", \"reference\": \"ref-045330\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 41, - "totalTokenCount": 21121, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 682 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "thoughtsTokenCount": 972, - "serviceTier": "standard" - }, - "ms": 6176, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "jade", - "reference": "ref-045330" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493186, - "repeat": 1, - "depth": 0.5, - "index": 3000, - "target": { - "key": "case_003000", - "status": "amber", - "region": "alder", - "reference": "ref-0h9ozfk", - "text": "case_003000 | region=alder | status=amber | reference=ref-0h9ozfk | queue=archive" - }, - "absentKey": "case_x03000", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494149, - "imageTokens": 19404, - "baselineImagedTokens": 140917, - "factsheetChars": 528, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"silver\", \"reference\": \"ref-135vy50\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 41, - "totalTokenCount": 21594, - "cachedContentTokenCount": 16256, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 682 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15705 - }, - { - "modality": "TEXT", - "tokenCount": 551 - } - ], - "thoughtsTokenCount": 1445, - "serviceTier": "standard" - }, - "ms": 9841, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-135vy50" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"silver\", \"reference\": \"ref-135vy50\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 40, - "totalTokenCount": 21249, - "cachedContentTokenCount": 16256, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 682 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15705 - }, - { - "modality": "TEXT", - "tokenCount": 551 - } - ], - "thoughtsTokenCount": 1101, - "serviceTier": "standard" - }, - "ms": 6677, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-135vy50" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493186, - "repeat": 1, - "depth": 0.75, - "index": 4500, - "target": { - "key": "case_004500", - "status": "plum", - "region": "birch", - "reference": "ref-0kw20l8", - "text": "case_004500 | region=birch | status=plum | reference=ref-0kw20l8 | queue=archive" - }, - "absentKey": "case_x04500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494149, - "imageTokens": 19404, - "baselineImagedTokens": 140917, - "factsheetChars": 528, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"maple\",\"status\":\"plum\",\"reference\":\"ref-0kmp4fw\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 33, - "totalTokenCount": 20813, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 682 - } - ], - "thoughtsTokenCount": 672, - "serviceTier": "standard" - }, - "ms": 7451, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "plum", - "reference": "ref-0kmp4fw" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"maple\", \"status\": \"plum\", \"reference\": \"ref-0kmp4fw\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 40, - "totalTokenCount": 21058, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 682 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "thoughtsTokenCount": 910, - "serviceTier": "standard" - }, - "ms": 6487, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "maple", - "status": "plum", - "reference": "ref-0kmp4fw" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 6000, - "sourceChars": 493186, - "repeat": 1, - "depth": 0.95, - "index": 5700, - "target": { - "key": "case_005700", - "status": "jade", - "region": "willow", - "reference": "ref-1uem2p8", - "text": "case_005700 | region=willow | status=jade | reference=ref-1uem2p8 | queue=archive" - }, - "absentKey": "case_x05700", - "production": { - "collapsedTurns": 35, - "collapsedImages": 18, - "collapsedChars": 494149, - "imageTokens": 19404, - "baselineImagedTokens": 140917, - "factsheetChars": 528, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"jade\", \"reference\": \"ref-0mi8ayj\"}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 40, - "totalTokenCount": 21035, - "cachedContentTokenCount": 16086, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 682 - }, - { - "modality": "IMAGE", - "tokenCount": 19426 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 545 - }, - { - "modality": "IMAGE", - "tokenCount": 15541 - } - ], - "thoughtsTokenCount": 887, - "serviceTier": "standard" - }, - "ms": 8320, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "jade", - "reference": "ref-0mi8ayj" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\n \"target_present\": true,\n \"absent_present\": false,\n \"region\": \"willow\",\n \"status\": \"jade\",\n \"reference\": \"ref-0mi8ayj\"\n}", - "usage": { - "promptTokenCount": 20108, - "candidatesTokenCount": 53, - "totalTokenCount": 20724, - "cachedContentTokenCount": 16086, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 19426 - }, - { - "modality": "TEXT", - "tokenCount": 682 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 15541 - }, - { - "modality": "TEXT", - "tokenCount": 545 - } - ], - "thoughtsTokenCount": 563, - "serviceTier": "standard" - }, - "ms": 5543, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "jade", - "reference": "ref-0mi8ayj" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 821993, - "repeat": 0, - "depth": 0.05, - "index": 500, - "target": { - "key": "case_000500", - "status": "jade", - "region": "willow", - "reference": "ref-193u44l", - "text": "case_000500 | region=willow | status=jade | reference=ref-193u44l | queue=archive" - }, - "absentKey": "case_x00500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822956, - "imageTokens": 32340, - "baselineImagedTokens": 234859, - "factsheetChars": 1039, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"cobalt\", \"reference\": \"ref-0yqrbeg\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 41, - "totalTokenCount": 34345, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 1129 - } - ], - "thoughtsTokenCount": 815, - "serviceTier": "standard" - }, - "ms": 9365, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "cobalt", - "reference": "ref-0yqrbeg" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"alder\",\"status\":\"plum\",\"reference\":\"ref-1p29fcg\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 34, - "totalTokenCount": 34099, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 1129 - } - ], - "thoughtsTokenCount": 576, - "serviceTier": "standard" - }, - "ms": 6198, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "plum", - "reference": "ref-1p29fcg" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 821993, - "repeat": 0, - "depth": 0.25, - "index": 2500, - "target": { - "key": "case_002500", - "status": "cobalt", - "region": "maple", - "reference": "ref-1dxnho5", - "text": "case_002500 | region=maple | status=cobalt | reference=ref-1dxnho5 | queue=archive" - }, - "absentKey": "case_x02500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822956, - "imageTokens": 32340, - "baselineImagedTokens": 234859, - "factsheetChars": 1039, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "`:\nLine 1: `case_002495`...\nLine 2: `case_002499 | region=alder | status=silver | reference=ref-1p0ynqp | queue=archive/case_002500 | region=alder | status=silver | reference=ref-0495", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 81, - "totalTokenCount": 35533, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 1129 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23619 - }, - { - "modality": "TEXT", - "tokenCount": 824 - } - ], - "thoughtsTokenCount": 1963, - "serviceTier": "standard" - }, - "ms": 13227, - "score": { - "parsed": null, - "acknowledged": false, - "absentRejected": false, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"birch\",\"status\":\"cobalt\",\"reference\":\"ref-1cq75a9\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 35, - "totalTokenCount": 34168, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 1129 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23619 - }, - { - "modality": "TEXT", - "tokenCount": 824 - } - ], - "thoughtsTokenCount": 644, - "serviceTier": "standard" - }, - "ms": 5607, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "cobalt", - "reference": "ref-1cq75a9" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 821993, - "repeat": 0, - "depth": 0.5, - "index": 5000, - "target": { - "key": "case_005000", - "status": "cobalt", - "region": "willow", - "reference": "ref-1jyx7ll", - "text": "case_005000 | region=willow | status=cobalt | reference=ref-1jyx7ll | queue=archive" - }, - "absentKey": "case_x05000", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822956, - "imageTokens": 32340, - "baselineImagedTokens": 234859, - "factsheetChars": 1039, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"cobalt\", \"reference\": \"ref-1ie937l\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 42, - "totalTokenCount": 34341, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 1129 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23619 - }, - { - "modality": "TEXT", - "tokenCount": 824 - } - ], - "thoughtsTokenCount": 810, - "serviceTier": "standard" - }, - "ms": 8724, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "cobalt", - "reference": "ref-1ie937l" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"cobalt\", \"reference\": \"ref-0c2jfuw\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 42, - "totalTokenCount": 34910, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 1129 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 824 - }, - { - "modality": "IMAGE", - "tokenCount": 23619 - } - ], - "thoughtsTokenCount": 1379, - "serviceTier": "standard" - }, - "ms": 8702, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "cobalt", - "reference": "ref-0c2jfuw" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 821993, - "repeat": 0, - "depth": 0.75, - "index": 7500, - "target": { - "key": "case_007500", - "status": "cobalt", - "region": "alder", - "reference": "ref-1q06xj1", - "text": "case_007500 | region=alder | status=cobalt | reference=ref-1q06xj1 | queue=archive" - }, - "absentKey": "case_x07500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822956, - "imageTokens": 32340, - "baselineImagedTokens": 234859, - "factsheetChars": 1039, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"cobalt\", \"reference\": \"ref-1906xj1\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 42, - "totalTokenCount": 34516, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 1129 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23619 - }, - { - "modality": "TEXT", - "tokenCount": 824 - } - ], - "thoughtsTokenCount": 985, - "serviceTier": "standard" - }, - "ms": 10004, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "cobalt", - "reference": "ref-1906xj1" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"alder\", \"status\": \"cobalt\", \"reference\": \"ref-1906xj1\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 42, - "totalTokenCount": 34216, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 1129 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23619 - }, - { - "modality": "TEXT", - "tokenCount": 824 - } - ], - "thoughtsTokenCount": 685, - "serviceTier": "standard" - }, - "ms": 5640, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "cobalt", - "reference": "ref-1906xj1" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": false, - "localization": true, - "recognition": true, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 821993, - "repeat": 0, - "depth": 0.95, - "index": 9500, - "target": { - "key": "case_009500", - "status": "amber", - "region": "willow", - "reference": "ref-1uu0b2l", - "text": "case_009500 | region=willow | status=amber | reference=ref-1uu0b2l | queue=archive" - }, - "absentKey": "case_x09500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822956, - "imageTokens": 32340, - "baselineImagedTokens": 234859, - "factsheetChars": 1039, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"silver\", \"reference\": \"ref-02c2jfw\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 41, - "totalTokenCount": 34202, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 1129 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 824 - }, - { - "modality": "IMAGE", - "tokenCount": 23619 - } - ], - "thoughtsTokenCount": 672, - "serviceTier": "standard" - }, - "ms": 8216, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-02c2jfw" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\":true,\"absent_present\":false,\"region\":\"willow\",\"status\":\"silver\",\"reference\":\"ref-0edygfl\"}", - "usage": { - "promptTokenCount": 33489, - "candidatesTokenCount": 32, - "totalTokenCount": 34239, - "cachedContentTokenCount": 24443, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 1129 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 824 - }, - { - "modality": "IMAGE", - "tokenCount": 23619 - } - ], - "thoughtsTokenCount": 718, - "serviceTier": "standard" - }, - "ms": 9253, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-0edygfl" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": false, - "referenceCorrect": false, - "localization": true, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 822001, - "repeat": 1, - "depth": 0.05, - "index": 500, - "target": { - "key": "case_000500", - "status": "amber", - "region": "willow", - "reference": "ref-0b8f9i4", - "text": "case_000500 | region=willow | status=amber | reference=ref-0b8f9i4 | queue=archive" - }, - "absentKey": "case_x00500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822964, - "imageTokens": 32340, - "baselineImagedTokens": 234863, - "factsheetChars": 859, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"silver\", \"reference\": \"ref-00ygcr2\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 40, - "totalTokenCount": 34173, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 969 - } - ], - "thoughtsTokenCount": 804, - "serviceTier": "standard" - }, - "ms": 8460, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "silver", - "reference": "ref-00ygcr2" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"silver\", \"reference\": \"ref-12d5jq\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 40, - "totalTokenCount": 34227, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 969 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "thoughtsTokenCount": 858, - "serviceTier": "standard" - }, - "ms": 7765, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "silver", - "reference": "ref-12d5jq" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 822001, - "repeat": 1, - "depth": 0.25, - "index": 2500, - "target": { - "key": "case_002500", - "status": "silver", - "region": "cedar", - "reference": "ref-0g28n1o", - "text": "case_002500 | region=cedar | status=silver | reference=ref-0g28n1o | queue=archive" - }, - "absentKey": "case_x02500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822964, - "imageTokens": 32340, - "baselineImagedTokens": 234863, - "factsheetChars": 859, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\n \"target_present\": true,\n \"absent_present\": false,\n \"region\": \"alder\",\n \"status\": \"silver\",\n \"reference\": \"ref-0da0iMM\"\n}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 53, - "totalTokenCount": 35346, - "cachedContentTokenCount": 24442, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 969 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23732 - }, - { - "modality": "TEXT", - "tokenCount": 710 - } - ], - "thoughtsTokenCount": 1964, - "serviceTier": "standard" - }, - "ms": 12654, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "silver", - "reference": "ref-0da0iMM" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"silver\", \"reference\": \"ref-00ygcr2\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 40, - "totalTokenCount": 34733, - "cachedContentTokenCount": 24442, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 969 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23732 - }, - { - "modality": "TEXT", - "tokenCount": 710 - } - ], - "thoughtsTokenCount": 1364, - "serviceTier": "standard" - }, - "ms": 8557, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "silver", - "reference": "ref-00ygcr2" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 822001, - "repeat": 1, - "depth": 0.5, - "index": 5000, - "target": { - "key": "case_005000", - "status": "silver", - "region": "maple", - "reference": "ref-0m3icz4", - "text": "case_005000 | region=maple | status=silver | reference=ref-0m3icz4 | queue=archive" - }, - "absentKey": "case_x05000", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822964, - "imageTokens": 32340, - "baselineImagedTokens": 234863, - "factsheetChars": 859, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"silver\", \"reference\": \"ref-12d5jgj\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 41, - "totalTokenCount": 34677, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 969 - } - ], - "thoughtsTokenCount": 1307, - "serviceTier": "standard" - }, - "ms": 9738, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "silver", - "reference": "ref-12d5jgj" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"birch\", \"status\": \"silver\", \"reference\": \"ref-12d5jgi\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 41, - "totalTokenCount": 34261, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 969 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "thoughtsTokenCount": 891, - "serviceTier": "standard" - }, - "ms": 6979, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "birch", - "status": "silver", - "reference": "ref-12d5jgi" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": true, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 822001, - "repeat": 1, - "depth": 0.75, - "index": 7500, - "target": { - "key": "case_007500", - "status": "silver", - "region": "willow", - "reference": "ref-0s4s2wk", - "text": "case_007500 | region=willow | status=silver | reference=ref-0s4s2wk | queue=archive" - }, - "absentKey": "case_x07500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822964, - "imageTokens": 32340, - "baselineImagedTokens": 234863, - "factsheetChars": 859, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"silver\", \"reference\": \"ref-0s4s2wk\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 41, - "totalTokenCount": 34242, - "cachedContentTokenCount": 24442, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 969 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23732 - }, - { - "modality": "TEXT", - "tokenCount": 710 - } - ], - "thoughtsTokenCount": 872, - "serviceTier": "standard" - }, - "ms": 9100, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-0s4s2wk" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": true, - "localization": true, - "recognition": true, - "exact": true - } - }, - "pxpipe": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"willow\", \"status\": \"silver\", \"reference\": \"ref-0s4s2wk\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 41, - "totalTokenCount": 34232, - "cachedContentTokenCount": 24442, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 969 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 710 - }, - { - "modality": "IMAGE", - "tokenCount": 23732 - } - ], - "thoughtsTokenCount": 862, - "serviceTier": "standard" - }, - "ms": 10109, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "willow", - "status": "silver", - "reference": "ref-0s4s2wk" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": true, - "statusCorrect": true, - "referenceCorrect": true, - "localization": true, - "recognition": true, - "exact": true - } - }, - "error": null - }, - { - "records": 10000, - "sourceChars": 822001, - "repeat": 1, - "depth": 0.95, - "index": 9500, - "target": { - "key": "case_009500", - "status": "plum", - "region": "maple", - "reference": "ref-0wylgg4", - "text": "case_009500 | region=maple | status=plum | reference=ref-0wylgg4 | queue=archive" - }, - "absentKey": "case_x09500", - "production": { - "collapsedTurns": 35, - "collapsedImages": 30, - "collapsedChars": 822964, - "imageTokens": 32340, - "baselineImagedTokens": 234863, - "factsheetChars": 859, - "targetKeyInFactsheet": false, - "targetRegionInFactsheet": false, - "targetStatusInFactsheet": false, - "targetReferenceInFactsheet": false - }, - "raw": { - "response": "{\"target_present\": true, \"absent_present\": false, \"region\": \"cedar\", \"status\": \"amber\", \"reference\": \"ref-0usc69l\"}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 40, - "totalTokenCount": 34215, - "cachedContentTokenCount": 24442, - "promptTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 969 - }, - { - "modality": "IMAGE", - "tokenCount": 32360 - } - ], - "cacheTokensDetails": [ - { - "modality": "TEXT", - "tokenCount": 710 - }, - { - "modality": "IMAGE", - "tokenCount": 23732 - } - ], - "thoughtsTokenCount": 846, - "serviceTier": "standard" - }, - "ms": 8825, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "cedar", - "status": "amber", - "reference": "ref-0usc69l" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "pxpipe": { - "response": "{\n \"target_present\": true,\n \"absent_present\": false,\n \"region\": \"alder\",\n \"status\": \"silver\",\n \"reference\": \"ref-1d88n7j\"\n}", - "usage": { - "promptTokenCount": 33329, - "candidatesTokenCount": 55, - "totalTokenCount": 35350, - "cachedContentTokenCount": 24442, - "promptTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 32360 - }, - { - "modality": "TEXT", - "tokenCount": 969 - } - ], - "cacheTokensDetails": [ - { - "modality": "IMAGE", - "tokenCount": 23732 - }, - { - "modality": "TEXT", - "tokenCount": 710 - } - ], - "thoughtsTokenCount": 1966, - "serviceTier": "standard" - }, - "ms": 10676, - "score": { - "parsed": { - "target_present": true, - "absent_present": false, - "region": "alder", - "status": "silver", - "reference": "ref-1d88n7j" - }, - "acknowledged": true, - "absentRejected": true, - "regionCorrect": false, - "statusCorrect": false, - "referenceCorrect": false, - "localization": false, - "recognition": false, - "exact": false - } - }, - "error": null - } - ] -} \ No newline at end of file diff --git a/eval/gemini-profile/lost-in-middle.mjs b/eval/gemini-profile/lost-in-middle.mjs index aceac9479..509876b3e 100644 --- a/eval/gemini-profile/lost-in-middle.mjs +++ b/eval/gemini-profile/lost-in-middle.mjs @@ -1,249 +1,314 @@ +// 300k Effective Context Window Benchmark — Reasoning & Causal State Tracking in the Middle +// Evaluates Raw Text vs pxpipe Visual Context across relative context depths (10%, 30%, 50%, 70%, 90%). + import { writeFileSync } from 'node:fs'; import { dirname, join } from 'node:path'; import { fileURLToPath } from 'node:url'; import { transformGoogleGenerateContent } from '../../dist/core/google.js'; import { resolveGeminiProfile } from '../../dist/core/gemini-model-profiles.js'; -import { callGeminiRequest } from './gemini-client.mjs'; +import { callGeminiRequest, resultFilename } from './gemini-client.mjs'; const HERE = dirname(fileURLToPath(import.meta.url)); -const RESULT = join(HERE, 'lost-in-middle-results.json'); -const MODEL = 'gemini-3.6-flash'; -const DEPTHS = [0.05, 0.25, 0.5, 0.75, 0.95]; -const SIZES = (process.env.SIZES || '2000,6000,10000') - .split(',') - .map(Number) - .filter((n) => Number.isInteger(n) && n > 0); +const MODEL = process.env.MODEL || 'gemini-3.6-flash'; +const TARGET_CHARS = Math.max(10000, Number(process.env.CHARS || 300000)); +const SCALE_LABEL = TARGET_CHARS >= 1000000 ? '1m' : `${Math.round(TARGET_CHARS / 1000)}k`; +const RESULT = join(HERE, resultFilename(`lost-in-middle-${SCALE_LABEL}`, MODEL)); +const DEPTHS = [0.10, 0.30, 0.50, 0.70, 0.90]; const REPEATS = Math.max(1, Number(process.env.REPEATS || 2)); -const TIMEOUT = Number(process.env.TIMEOUT_MS || 240000); -const OLD_TURNS = 35; +const TIMEOUT = Number(process.env.TIMEOUT_MS || 300000); +const RUN_ARM = (process.env.ARM || 'all').toLowerCase(); const profile = resolveGeminiProfile(); -const statuses = ['amber', 'cobalt', 'jade', 'plum', 'silver']; -const regions = ['alder', 'birch', 'cedar', 'maple', 'willow']; -function hash(index, repeat) { - return ((index + 1) * 2654435761 + repeat * 2246822519) >>> 0; -} +const CLUSTERS = ['alpha', 'beta', 'gamma', 'delta', 'epsilon', 'zeta', 'theta', 'omega', 'sigma']; +const STATUSES = ['nominal', 'degraded', 'maintenance', 'recovering', 'active', 'standby']; +const ACTIONS = ['rebalance', 'drain', 'isolate', 'failover', 'scale_up', 'scale_down']; +const POLICIES = [ + { + name: 'cascade_failover', + rule: 'If primary cluster is in maintenance and secondary is active, route to secondary. If secondary is also disabled or maintenance, route to blackhole_sink with code 503.', + }, + { + name: 'quarantine_drain', + rule: 'If cluster security audit fails, quarantine cluster immediately and reject all queued transactions with code 423 (Locked).', + }, +]; -function recordFor(index, repeat) { - const n = hash(index, repeat); - return { - key: `case_${String(index).padStart(6, '0')}`, - status: statuses[n % statuses.length], - region: regions[Math.floor(n / statuses.length) % regions.length], - reference: `ref-${n.toString(36).padStart(7, '0')}`, - text: `case_${String(index).padStart(6, '0')} | region=${regions[Math.floor(n / statuses.length) % regions.length]} | status=${statuses[n % statuses.length]} | reference=ref-${n.toString(36).padStart(7, '0')} | queue=archive`, +function lcg(seed) { + let s = seed >>> 0; + return () => { + s = (Math.imul(s, 1664525) + 1013904223) >>> 0; + return s; }; } -function parseAnswer(text) { - const start = text.indexOf('{'); - const end = text.lastIndexOf('}'); - if (start < 0 || end <= start) return null; - try { - return JSON.parse(text.slice(start, end + 1)); - } catch { - return null; - } +function generateLogLine(rand, id) { + const c = CLUSTERS[rand() % CLUSTERS.length]; + const a = ACTIONS[rand() % ACTIONS.length]; + const duration = (rand() % 450) + 10; + const mem = (rand() % 64) + 16; + const cpu = ((rand() % 900) / 10).toFixed(1); + return `[LOG ${id.toString().padStart(6, '0')}] cluster=${c} action=${a} duration=${duration}ms mem_used=${mem}GB cpu_load=${cpu}% status=nominal`; } -function score(response, target) { - const parsed = parseAnswer(response); - const acknowledged = parsed?.target_present === true; - const absentRejected = parsed?.absent_present === false; - const regionCorrect = String(parsed?.region ?? '').toLowerCase() === target.region; - const statusCorrect = String(parsed?.status ?? '').toLowerCase() === target.status; - const referenceCorrect = String(parsed?.reference ?? '').toLowerCase() === target.reference; - return { - parsed, - acknowledged, - absentRejected, - regionCorrect, - statusCorrect, - referenceCorrect, - localization: regionCorrect, - recognition: regionCorrect && statusCorrect, - exact: regionCorrect && statusCorrect && referenceCorrect, - }; -} +function buildScenario(depth, repeat, targetChars) { + const seed = Math.floor(depth * 1000000) + repeat * 7919; + const rand = lcg(seed); + + // Define unique target cluster and state mutation for this test probe + const targetCluster = CLUSTERS[(rand() % (CLUSTERS.length - 2)) + 1]; + const backupCluster = CLUSTERS[0]; + const targetToken = `MUTATION_${targetCluster.toUpperCase()}_${(rand() % 90000 + 10000)}`; + + // The critical state mutation injected at exact depth + const criticalEvent = [ + `>>> CRITICAL STATE CHANGE [${targetToken}] <<<`, + `CLUSTER: ${targetCluster}`, + `STATUS: maintenance (active drain in progress)`, + `BACKUP_TARGET: ${backupCluster}`, + `BACKUP_STATUS: disabled (kernel panic in node pool)`, + `ACTIVE_POLICY: cascade_failover`, + `EXPECTED_ROUTE: blackhole_sink`, + `EXPECTED_CODE: 503`, + `>>> END STATE CHANGE [${targetToken}] <<<`, + ].join('\n'); + + // Fill ~300k chars with multi-turn log records + const approxLineLen = 105; + const totalLines = Math.floor(targetChars / approxLineLen); + const targetLineIdx = Math.floor(totalLines * depth); -function makeRequest(records, target, absentKey) { - const buckets = Array.from({ length: OLD_TURNS }, () => []); - records.forEach((record, index) => { - buckets[Math.min(OLD_TURNS - 1, Math.floor(index * OLD_TURNS / records.length))].push(record.text); + const turns = 30; + const linesPerTurn = Math.floor(totalLines / turns); + const targetTurnIdx = Math.min(turns - 1, Math.floor(targetLineIdx / linesPerTurn)); + + const contents = []; + + // Initial turn: System setup and policy declaration + contents.push({ + role: 'user', + parts: [{ + text: [ + '## SYSTEM ARCHITECTURE AND DISASTER RECOVERY POLICY SPECIFICATION', + 'You are the autonomous infrastructure auditor. Follow these exact multi-hop routing rules:', + ...POLICIES.map((p) => `- Policy ${p.name}: ${p.rule}`), + '', + 'Analyze the multi-turn telemetry stream below and track all cluster state transitions.', + ].join('\n'), + }], + }); + contents.push({ + role: 'model', + parts: [{ text: 'Telemetry ingest initialized. Disaster recovery policies active.' }], }); - const contents = buckets.map((lines, index) => ({ - role: index % 2 === 0 ? 'user' : 'model', - parts: [{ text: lines.join('\n') }], - })); - contents.push( - { role: 'model', parts: [{ text: 'I have retained the archived case table for later lookup.' }] }, - { role: 'user', parts: [{ text: 'Continue to preserve the archived table.' }] }, - { role: 'model', parts: [{ text: 'The table remains available as prior context.' }] }, - { role: 'user', parts: [{ text: 'Use only the archived table, without guessing.' }] }, - { role: 'model', parts: [{ text: 'Ready for a case lookup.' }] }, - { + + const distClusters = CLUSTERS.filter((c) => c !== targetCluster && c !== backupCluster); + + let lineCount = 0; + for (let t = 0; t < turns; t++) { + const lines = []; + for (let l = 0; l < linesPerTurn; l++) { + if (t === targetTurnIdx && l === Math.floor(linesPerTurn / 2)) { + lines.push(criticalEvent); + } else if (l === 10 && t !== targetTurnIdx) { + // Inject distractor state transitions ONLY for other clusters (never target/backup) + const distCluster = distClusters[t % distClusters.length]; + const distStatus = STATUSES[t % STATUSES.length]; + lines.push(`>>> STATE UPDATE [EVENT_${t * 100 + l}] <<< CLUSTER: ${distCluster} | STATUS: ${distStatus} | ACTION: auto_remediate | ROUTE: internal_mesh | CODE: 200`); + } else { + lines.push(generateLogLine(rand, lineCount++)); + } + } + + contents.push({ role: 'user', - parts: [{ - text: [ - `Look up ${target.key} in the archived case table and also check whether ${absentKey} exists.`, - 'Return only a JSON object with exactly these fields. Do not use Markdown fences:', - '{"target_present":boolean,"absent_present":boolean,"region":string,"status":string,"reference":string}', - 'Copy region, status, and reference only from the target row. If the target is absent, use empty strings.', - ].join('\n'), - }], - }, - ); + parts: [{ text: `## TELEMETRY BATCH ${t + 1}\n` + lines.join('\n') }], + }); + contents.push({ + role: 'model', + parts: [{ text: `Telemetry batch ${t + 1} processed and indexed.` }], + }); + } + + // Final query: Multi-hop reasoning combining Policy + Middle State Change + const query = [ + `Incoming request target: cluster '${targetCluster}'.`, + `Based on the disaster recovery policies and the state transitions in the telemetry stream, determine:`, + `1. What is the current operational state of cluster '${targetCluster}' and its designated backup?`, + `2. Under the active policy, where must incoming traffic for '${targetCluster}' be routed?`, + `3. What HTTP response code must be returned to clients?`, + '', + 'Return your answer ONLY as a JSON object with this exact schema:', + '{"target_cluster":string,"primary_status":string,"backup_status":string,"final_route":string,"status_code":number,"mutation_token":string}', + ].join('\n'); + + contents.push({ + role: 'user', + parts: [{ text: query }], + }); + return { - contents, - generationConfig: { responseMimeType: 'application/json' }, + targetCluster, + backupCluster, + targetToken, + expectedRoute: 'blackhole_sink', + expectedCode: 503, + request: { + contents, + generationConfig: { responseMimeType: 'application/json' }, + }, }; } -function sheetText(request) { - return (request.contents || []) - .flatMap((content) => content.parts || []) - .map((part) => typeof part.text === 'string' && part.text.startsWith('[Exact ') ? part.text : '') - .filter(Boolean) - .join('\n'); +function parseJson(text) { + try { + const s = text.indexOf('{'); + const e = text.lastIndexOf('}'); + if (s >= 0 && e > s) return JSON.parse(text.slice(s, e + 1)); + } catch {} + return null; } -function summarize(rows) { - const complete = rows.filter((row) => !row.error); - const arm = (name) => ({ - acknowledged: complete.filter((row) => row[name].score.acknowledged).length, - absentRejected: complete.filter((row) => row[name].score.absentRejected).length, - localization: complete.filter((row) => row[name].score.localization).length, - recognition: complete.filter((row) => row[name].score.recognition).length, - exact: complete.filter((row) => row[name].score.exact).length, - }); +function score(ans, scenario) { + const p = parseJson(ans); + const tokenFound = p?.mutation_token === scenario.targetToken; + const primaryStateOk = String(p?.primary_status ?? '').toLowerCase().includes('maintenance'); + const backupStateOk = String(p?.backup_status ?? '').toLowerCase().includes('disabled'); + const routeOk = String(p?.final_route ?? '').toLowerCase().includes('blackhole_sink'); + const codeOk = Number(p?.status_code) === scenario.expectedCode; + + const reasoningSuccess = routeOk && codeOk; + const fullMultiHopSuccess = tokenFound && primaryStateOk && backupStateOk && routeOk && codeOk; + return { - completed: complete.length, - errors: rows.length - complete.length, - raw: arm('raw'), - pxpipe: arm('pxpipe'), - bySizeDepth: Object.fromEntries(SIZES.map((size) => [ - size, - Object.fromEntries(DEPTHS.map((depth) => { - const cells = complete.filter((row) => row.records === size && row.depth === depth); - return [depth, { - n: cells.length, - raw: { - acknowledged: cells.filter((row) => row.raw.score.acknowledged).length, - absentRejected: cells.filter((row) => row.raw.score.absentRejected).length, - localization: cells.filter((row) => row.raw.score.localization).length, - recognition: cells.filter((row) => row.raw.score.recognition).length, - exact: cells.filter((row) => row.raw.score.exact).length, - }, - pxpipe: { - acknowledged: cells.filter((row) => row.pxpipe.score.acknowledged).length, - absentRejected: cells.filter((row) => row.pxpipe.score.absentRejected).length, - localization: cells.filter((row) => row.pxpipe.score.localization).length, - recognition: cells.filter((row) => row.pxpipe.score.recognition).length, - exact: cells.filter((row) => row.pxpipe.score.exact).length, - }, - }]; - })), - ])), + parsed: p, + tokenFound, + primaryStateOk, + backupStateOk, + routeOk, + codeOk, + reasoningSuccess, + fullMultiHopSuccess, }; } -function save(rows) { - const result = { - generatedAt: new Date().toISOString(), - model: MODEL, - methodology: { - task: 'production-faithful old-history lookup by relative record depth', - depths: DEPTHS, - sizes: SIZES, - repeats: REPEATS, - oldTurns: OLD_TURNS, - imageCap: profile.history.maxImages, - transformer: 'transformGoogleGenerateContent', - factsheet: 'production output, target coverage recorded per probe', - measures: { - acknowledged: 'model says the named target key exists; weak evidence by itself', - absentRejected: 'model correctly says a matched nonexistent key is absent', - localization: 'correct adjacent region word copied from the target row', - recognition: 'localization plus correct semantic status word', - exact: 'recognition plus exact synthetic reference', - }, - }, - summary: summarize(rows), - rows, - }; - writeFileSync(RESULT, JSON.stringify(result, null, 2)); - return result; -} +async function runEval() { + console.log(`=== 300k Effective Context Window Eval (Reasoning in the Middle) ===`); + console.log(`Model: ${MODEL} | Target Chars: ${TARGET_CHARS} | Depths: ${DEPTHS.join(', ')} | Repeats: ${REPEATS}\n`); + + const results = []; + + for (const depth of DEPTHS) { + for (let r = 0; r < REPEATS; r++) { + const scenario = buildScenario(depth, r, TARGET_CHARS); + const depthPct = Math.round(depth * 100); + console.log(`[Depth ${depthPct}% | Run ${r + 1}/${REPEATS}] Building 300k scenario...`); -const rows = []; -for (const recordsCount of SIZES) { - for (let repeat = 0; repeat < REPEATS; repeat++) { - const records = Array.from({ length: recordsCount }, (_, index) => recordFor(index, repeat)); - for (const depth of DEPTHS) { - const index = Math.min(recordsCount - 1, Math.floor(depth * recordsCount)); - const target = records[index]; - const absentKey = `case_x${String(index).padStart(5, '0')}`; - const rawRequest = makeRequest(records, target, absentKey); - const transformed = await transformGoogleGenerateContent( - new TextEncoder().encode(JSON.stringify(rawRequest)), - MODEL, - { compress: true, compressToolResults: false }, - ); - const pxpipeRequest = JSON.parse(new TextDecoder().decode(transformed.body)); - const images = transformed.info.collapsedImages ?? 0; - if (transformed.info.historyReason !== 'collapsed' || images < 1 || images > profile.history.maxImages) { - throw new Error( - `${recordsCount} records did not produce valid production history compression: ` + - `reason=${transformed.info.historyReason} images=${images}`, - ); + // 1. Raw Text Arm + let rawAns = null; + let rawScore = null; + let rawMs = 0; + let rawUsage = null; + if (RUN_ARM === 'all' || RUN_ARM === 'raw' || RUN_ARM === 'text') { + console.log(` -> Running Raw Text Arm...`); + try { + const rawRes = await callGeminiRequest({ + model: MODEL, + request: scenario.request, + maxOutputTokens: 8192, + timeoutMs: TIMEOUT, + }); + rawAns = rawRes.text; + rawMs = rawRes.ms; + rawUsage = rawRes.usage; + rawScore = score(rawAns, scenario); + } catch (err) { + console.error(` Raw Text error: ${err.message}`); + rawScore = { error: err.message }; + } } - const factsheet = sheetText(pxpipeRequest); - const row = { - records: recordsCount, - sourceChars: records.reduce((sum, record) => sum + record.text.length + 1, 0), - repeat, - depth, - index, - target, - absentKey, - production: { - collapsedTurns: transformed.info.collapsedTurns, - collapsedImages: images, - collapsedChars: transformed.info.collapsedChars, - imageTokens: transformed.info.imageTokens, - baselineImagedTokens: transformed.info.baselineImagedTokens, - factsheetChars: factsheet.length, - targetKeyInFactsheet: factsheet.includes(target.key), - targetRegionInFactsheet: factsheet.includes(target.region), - targetStatusInFactsheet: factsheet.includes(target.status), - targetReferenceInFactsheet: factsheet.includes(target.reference), - }, - raw: { response: '', usage: null, ms: null, score: score('', target) }, - pxpipe: { response: '', usage: null, ms: null, score: score('', target) }, - error: null, - }; - try { - const [raw, pxpipe] = await Promise.all([ - callGeminiRequest({ model: MODEL, request: rawRequest, maxOutputTokens: 2048, timeoutMs: TIMEOUT }), - callGeminiRequest({ model: MODEL, request: pxpipeRequest, maxOutputTokens: 2048, timeoutMs: TIMEOUT }), - ]); - row.raw = { response: raw.text, usage: raw.usage, ms: raw.ms, score: score(raw.text, target) }; - row.pxpipe = { response: pxpipe.text, usage: pxpipe.usage, ms: pxpipe.ms, score: score(pxpipe.text, target) }; - } catch (error) { - row.error = String(error?.message || error); + + // 2. pxpipe Visual Context Arm + let pxAns = null; + let pxScore = null; + let pxMs = 0; + let pxUsage = null; + if (RUN_ARM === 'all' || RUN_ARM === 'pxpipe' || RUN_ARM === 'visual') { + console.log(` -> Running pxpipe Visual Context Arm...`); + try { + const bodyBytes = new TextEncoder().encode(JSON.stringify(scenario.request)); + const transformed = await transformGoogleGenerateContent(bodyBytes, MODEL, { + compress: true, + collapseHistory: true, + compressToolResults: false, + }); + const transformedRequest = JSON.parse(new TextDecoder().decode(transformed.body)); + const pxRes = await callGeminiRequest({ + model: MODEL, + request: transformedRequest, + maxOutputTokens: 8192, + timeoutMs: TIMEOUT, + }); + pxAns = pxRes.text; + pxMs = pxRes.ms; + pxUsage = pxRes.usage; + pxScore = score(pxAns, scenario); + } catch (err) { + console.error(` pxpipe error: ${err.message}`); + pxScore = { error: err.message }; + } } - rows.push(row); - save(rows); - console.log( - `${recordsCount} r${repeat + 1} d${depth}: images=${images} ` + - `raw=${Number(row.raw.score.localization)}/${Number(row.raw.score.recognition)}/${Number(row.raw.score.exact)} ` + - `px=${Number(row.pxpipe.score.localization)}/${Number(row.pxpipe.score.recognition)}/${Number(row.pxpipe.score.exact)}` + - (row.error ? ` error=${row.error}` : ''), - ); + + const rawStatus = rawScore ? (rawScore.reasoningSuccess ? '✓' : '✗') : 'skipped'; + const pxStatus = pxScore ? (pxScore.reasoningSuccess ? '✓' : '✗') : 'skipped'; + console.log(` [Depth ${depthPct}%] Raw Reasoning: ${rawStatus} (${rawMs}ms) | pxpipe Reasoning: ${pxStatus} (${pxMs}ms)`); + + results.push({ + depth, + repeat: r, + targetCluster: scenario.targetCluster, + raw: { score: rawScore, ms: rawMs, usage: rawUsage, text: rawAns }, + pxpipe: { score: pxScore, ms: pxMs, usage: pxUsage, text: pxAns }, + }); } } + + // Summary aggregation + const summary = { + model: MODEL, + targetChars: TARGET_CHARS, + depths: DEPTHS, + repeats: REPEATS, + byDepth: {}, + }; + + for (const d of DEPTHS) { + const dRows = results.filter((r) => r.depth === d); + const rawActive = dRows.filter((r) => r.raw.score !== null); + const pxActive = dRows.filter((r) => r.pxpipe.score !== null); + summary.byDepth[d] = { + raw_reasoning_acc: rawActive.length > 0 ? (rawActive.filter((r) => r.raw.score?.reasoningSuccess).length / rawActive.length) : null, + pxpipe_reasoning_acc: pxActive.length > 0 ? (pxActive.filter((r) => r.pxpipe.score?.reasoningSuccess).length / pxActive.length) : null, + raw_multihop_acc: rawActive.length > 0 ? (rawActive.filter((r) => r.raw.score?.fullMultiHopSuccess).length / rawActive.length) : null, + pxpipe_multihop_acc: pxActive.length > 0 ? (pxActive.filter((r) => r.pxpipe.score?.fullMultiHopSuccess).length / pxActive.length) : null, + raw_avg_ms: rawActive.length > 0 ? Math.round(rawActive.reduce((a, b) => a + (b.raw.ms || 0), 0) / rawActive.length) : 0, + pxpipe_avg_ms: pxActive.length > 0 ? Math.round(pxActive.reduce((a, b) => a + (b.pxpipe.ms || 0), 0) / pxActive.length) : 0, + }; + } + + const out = { + generatedAt: new Date().toISOString(), + summary, + results, + }; + + writeFileSync(RESULT, JSON.stringify(out, null, 2) + '\n'); + console.log(`\n=== Benchmark Results Saved to ${RESULT} ===`); + console.table(summary.byDepth); } -const result = save(rows); -console.log(JSON.stringify(result.summary, null, 2)); -console.log(`Receipt: ${RESULT}`); +if (process.argv[1] && fileURLToPath(import.meta.url) === process.argv[1]) { + runEval().catch((err) => { + console.error('Fatal error:', err); + process.exit(1); + }); +} diff --git a/src/core/applicability.ts b/src/core/applicability.ts index d7083994a..6d7e2ed9d 100644 --- a/src/core/applicability.ts +++ b/src/core/applicability.ts @@ -33,7 +33,7 @@ let runtimeModelBases: readonly string[] | null = null; * - GPT 5.5 — degrades on imaged history/context. * - GPT 5.6 Sol — 98/100 production arithmetic, but 79/93 completed gist, * 4/15 completed guard confabulations, and 0/15 dense hex. - * - Grok 4.5 — native 14px: 100/100 arithmetic, 97/98 gist, 17/18 state; hex 0/15. + * - Grok 4.5/4.6 — native 14px: 100/100 arithmetic, 97/98 gist, 17/18 state; hex 0/15. * - Opus 5 — 2/15 exact recall on imaged context (DeepSWE v1.1 eval); * task pass rate holds but dense recall does not. * All remain available for explicit opt-in. diff --git a/src/core/claude-model-profiles.ts b/src/core/claude-model-profiles.ts index a54d48bea..a87b1a7c6 100644 --- a/src/core/claude-model-profiles.ts +++ b/src/core/claude-model-profiles.ts @@ -91,11 +91,10 @@ export const CLAUDE_LEGACY_PROFILE: GptModelProfile = { * jetbrains-mono-14 Opus read 100/100. 172 cols at this font still fits * 1568x728, so nothing is downscaled. Cost on live transcript: dense is 5.28x * cheaper than text, legible 1.47x — still a saving, so misread-prone models - * get legibility by default rather than behind a flag. Only history pays the - * wider geometry: the static slab and tool-result pages hold no exact values - * and stay dense. + * get legibility by default rather than behind a flag. Uses unified 14px + * JetBrains Mono @ 172 cols across all page types (slabs, tools, and history). */ -const CLAUDE_HISTORY_STRIP_COLS = 172; +export const CLAUDE_HISTORY_STRIP_COLS = 172; const CLAUDE_HISTORY_STYLE: GptRenderStyle = { ...BASE_STYLE, font: 'jetbrains-mono-14', @@ -103,14 +102,16 @@ const CLAUDE_HISTORY_STYLE: GptRenderStyle = { export const CLAUDE_LEGIBLE_PROFILE: GptModelProfile = { ...CLAUDE_PROFILE, - style: { ...BASE_STYLE }, + stripCols: CLAUDE_HISTORY_STRIP_COLS, + style: { ...CLAUDE_HISTORY_STYLE }, historyStripCols: CLAUDE_HISTORY_STRIP_COLS, historyStyle: { ...CLAUDE_HISTORY_STYLE }, }; export const CLAUDE_LEGACY_LEGIBLE_PROFILE: GptModelProfile = { ...CLAUDE_LEGACY_PROFILE, - style: { ...BASE_STYLE }, + stripCols: CLAUDE_HISTORY_STRIP_COLS, + style: { ...CLAUDE_HISTORY_STYLE }, historyStripCols: CLAUDE_HISTORY_STRIP_COLS, historyStyle: { ...CLAUDE_HISTORY_STYLE }, }; diff --git a/src/core/export.ts b/src/core/export.ts index 2e1dae311..464f44b3e 100644 --- a/src/core/export.ts +++ b/src/core/export.ts @@ -8,8 +8,7 @@ */ import { - DENSE_CONTENT_CHARS_PER_IMAGE, - DENSE_CONTENT_COLS, + maxCharsPerImage, renderCellHeight, renderCellWidth, } from './render.js'; @@ -33,8 +32,8 @@ import { export const DEFAULT_EXPORT_MODEL = 'claude-sonnet-4-5'; // Chars-per-token for the reporting estimate now lives in transform.ts as // REPORT_CHARS_PER_TOKEN (single source of truth for all token-estimate constants). -/** Default column width — dense content mode (312 cols = 1568 px). */ -export const DEFAULT_EXPORT_COLS: number = DENSE_CONTENT_COLS; +/** Default column width (172 cols for default 14px legible model). */ +export const DEFAULT_EXPORT_COLS: number = 172; // --------------------------------------------------------------------------- // Glob matching (no external glob library — node:fs only per convention) @@ -156,7 +155,6 @@ export function parseExportArgv( let git = false; let diff: string | undefined; let stdin = false; - let cols = DENSE_CONTENT_COLS; let out = defaultOut ?? (typeof process !== 'undefined' @@ -232,7 +230,7 @@ export function parseExportArgv( } } - cols = resolveGptProfile(model).stripCols; + const cols = resolveGptProfile(model).stripCols; return { kind: 'opts', parsed: { targets, include, exclude, git, diff, stdin, cols, out, model, json, open }, @@ -296,7 +294,7 @@ export function computeTokenReport( const profile = resolveGptProfile(model); const stripW = 8 + cols * renderCellWidth(profile.style); const linesPerImage = Math.max(1, Math.floor((profile.maxHeightPx - 8) / renderCellHeight(profile.style))); - const estImages = estimateImageCount(sourceText, cols, DENSE_CONTENT_CHARS_PER_IMAGE, linesPerImage); + const estImages = estimateImageCount(sourceText, cols, maxCharsPerImage(cols), linesPerImage); const perStrip = exportImageTokens(model, stripW, profile.maxHeightPx); const imageTokens = Math.round(estImages * perStrip); const textTokens = Math.round(sourceText.length / REPORT_CHARS_PER_TOKEN); @@ -304,7 +302,7 @@ export function computeTokenReport( textTokens > 0 ? Math.round(((textTokens - imageTokens) / textTokens) * 1000) / 10 : 0; - const { kept, dropped } = extractFactSheetTokensAllPages(sourceText, DENSE_CONTENT_CHARS_PER_IMAGE); + const { kept, dropped } = extractFactSheetTokensAllPages(sourceText, maxCharsPerImage(cols)); return { textTokens, imageTokens, @@ -448,7 +446,7 @@ export async function runExportCore( // Extract factsheet tokens across ALL pages so identifiers from page 3+ are covered. const { kept: fsKept, dropped: fsDropped } = extractFactSheetEntriesAllPages( sourceText, - DENSE_CONTENT_CHARS_PER_IMAGE, + maxCharsPerImage(opts.cols), ); const fsText = factSheetTextFromEntries(fsKept); const tokenReport: ExportTokenReport = { diff --git a/src/core/gpt-model-profiles.ts b/src/core/gpt-model-profiles.ts index 1a4ec7ad0..3896af0fb 100644 --- a/src/core/gpt-model-profiles.ts +++ b/src/core/gpt-model-profiles.ts @@ -23,7 +23,7 @@ import { type RenderFont } from './render.js'; import { hasGeminiMeasuredProfile, resolveGeminiProfile } from './gemini-model-profiles.js'; import { isClaudeModel, resolveClaudeProfile } from './claude-model-profiles.js'; -import { BASE_HISTORY, BASE_PRICING, BASE_STYLE } from './profile-base.js'; +import { BASE_HISTORY, BASE_PRICING, BASE_STYLE, NATIVE_14PX_HISTORY } from './profile-base.js'; export const GPT_MAX_HEIGHT_PX = 1932; @@ -78,6 +78,15 @@ export interface GptHistoryProfile { keepRecentPairs: number; /** Local o200k floor before history profitability is evaluated. */ minCollapseTokens: number; + /** Chat-completions only: minimum closed turns before history images. + * Undefined keeps the planner default (10). */ + minCollapsePrefix?: number; + /** Chat-completions only: snap the collapse boundary to this many turns. + * Undefined keeps the planner default (10). Native-14px sets 1. */ + collapseChunk?: number; + /** Chat-completions only: freeze imaged chunks on this turn grid. + * Undefined keeps the planner default (10). Native-14px sets 1. */ + freezeChunk?: number; /** Responses items eligible for history images. */ responsesMode: 'pairs' | 'mixed'; /** Native text bracketing each history image group. */ @@ -204,17 +213,10 @@ const GPT56_SOL_PROFILE: GptModelProfile = { minCompressTokens: 500, factSheetFormat: 'full', history: { - ...BASE_HISTORY, + ...NATIVE_14PX_HISTORY, + // Taller 1954px pages: more chars/image, so Sol can hold a larger budget + // without the 512px latency cliff that Grok/Qwen hit. maxImages: 64, - // The latest user request is protected independently by the Responses - // planner. Keep only one additional recent message/pair native so closed, - // unreferenced history does not dominate long stateless requests. - keepTail: 1, - keepRecentPairs: 1, - minCollapseTokens: 1000, - responsesMode: 'mixed', - framing: 'compact', - factSheetScope: 'combined', }, style: { ...BASE_STYLE, @@ -325,7 +327,11 @@ const BUILTIN_RULES: ProfileRule[] = [ maxHeightPx: 512, minCompressTokens: 500, factSheetFormat: 'full', - history: { ...BASE_HISTORY, maxImages: 24 }, + // pairs only groups INDEX-CONTIGUOUS tool rounds. Codex puts an assistant + // message between rounds, so every round is its own run and leftover + // history stays text — which is why grok-4.6 showed ~0 saved. mixed + // groups safe messages with completed pairs. + history: { ...NATIVE_14PX_HISTORY }, style: { ...BASE_STYLE, font: 'jetbrains-mono-14', @@ -351,7 +357,7 @@ const BUILTIN_RULES: ProfileRule[] = [ maxHeightPx: 512, minCompressTokens: 500, factSheetFormat: 'full', - history: { ...BASE_HISTORY, maxImages: 32, keepTail: 1 }, + history: { ...NATIVE_14PX_HISTORY }, style: { ...BASE_STYLE, font: 'jetbrains-mono-14', @@ -529,6 +535,15 @@ function parseEnvProfiles(raw: string): Map { keepTail: nonNegativeInt(historyIn?.keepTail, baseHistory.keepTail), keepRecentPairs: nonNegativeInt(historyIn?.keepRecentPairs, baseHistory.keepRecentPairs), minCollapseTokens: nonNegativeInt(historyIn?.minCollapseTokens, baseHistory.minCollapseTokens), + minCollapsePrefix: historyIn?.minCollapsePrefix === undefined + ? baseHistory.minCollapsePrefix + : nonNegativeInt(historyIn.minCollapsePrefix, baseHistory.minCollapsePrefix ?? 10), + collapseChunk: historyIn?.collapseChunk === undefined + ? baseHistory.collapseChunk + : nonNegativeInt(historyIn.collapseChunk, baseHistory.collapseChunk ?? 10), + freezeChunk: historyIn?.freezeChunk === undefined + ? baseHistory.freezeChunk + : nonNegativeInt(historyIn.freezeChunk, baseHistory.freezeChunk ?? 10), responsesMode: responsesMode(historyIn?.responsesMode, baseHistory.responsesMode), framing: historyFraming(historyIn?.framing, baseHistory.framing), factSheetScope: factSheetScope(historyIn?.factSheetScope, baseHistory.factSheetScope), diff --git a/src/core/openai.ts b/src/core/openai.ts index 78fde4f3c..539928d81 100644 --- a/src/core/openai.ts +++ b/src/core/openai.ts @@ -282,6 +282,15 @@ function gptHistoryOpts( keepTail: o.gptHistory?.keepTail ?? profile.history.keepTail, keepRecentPairs: o.gptHistory?.keepRecentPairs ?? profile.history.keepRecentPairs, minCollapseTokens: o.gptHistory?.minCollapseTokens ?? profile.history.minCollapseTokens, + ...(o.gptHistory?.minCollapsePrefix !== undefined || profile.history.minCollapsePrefix !== undefined + ? { minCollapsePrefix: o.gptHistory?.minCollapsePrefix ?? profile.history.minCollapsePrefix } + : {}), + ...(o.gptHistory?.collapseChunk !== undefined || profile.history.collapseChunk !== undefined + ? { collapseChunk: o.gptHistory?.collapseChunk ?? profile.history.collapseChunk } + : {}), + ...(o.gptHistory?.freezeChunk !== undefined || profile.history.freezeChunk !== undefined + ? { freezeChunk: o.gptHistory?.freezeChunk ?? profile.history.freezeChunk } + : {}), responsesMode: profile.history.responsesMode, cols: o.gptHistory?.cols ?? profile.stripCols, maxHeightPx: o.gptHistory?.maxHeightPx ?? profile.maxHeightPx, diff --git a/src/core/profile-base.ts b/src/core/profile-base.ts index d8ce7cfca..7800fb6ec 100644 --- a/src/core/profile-base.ts +++ b/src/core/profile-base.ts @@ -50,3 +50,22 @@ export const BASE_HISTORY: GptHistoryProfile = { framing: 'full', factSheetScope: 'per-segment', }; + +/** Shared history policy for every native-14px profile (Sol, Grok, Qwen). + * 14px packs ~3× fewer chars/page than 5×8, so a 6-turn tail + 1000-token + * floor + pairs planner leaves most of a completions chat as text (~0.3% + * saved on grok-4.6). Mixed + 1-turn tail + no token floor is what actually + * images that bulk. Sol keeps a higher maxImages because its pages are taller. */ +export const NATIVE_14PX_HISTORY: GptHistoryProfile = { + ...BASE_HISTORY, + maxImages: 32, + keepTail: 1, + keepRecentPairs: 1, + minCollapseTokens: 0, + minCollapsePrefix: 1, + collapseChunk: 1, + freezeChunk: 1, + responsesMode: 'mixed', + framing: 'compact', + factSheetScope: 'combined', +}; diff --git a/src/dashboard/fragments.ts b/src/dashboard/fragments.ts index eee718fae..e29c79890 100644 --- a/src/dashboard/fragments.ts +++ b/src/dashboard/fragments.ts @@ -87,6 +87,7 @@ const GPT_MODEL_CATALOG: ReadonlyArray<{ id: string; label: string }> = [ ]; const GROK_MODEL_CATALOG: ReadonlyArray<{ id: string; label: string }> = [ + { id: 'grok-4.6', label: 'Grok 4.6' }, { id: 'grok-4.5', label: 'Grok 4.5' }, ]; diff --git a/tests/export.test.ts b/tests/export.test.ts index 718f0d08d..fc7cf2e41 100644 --- a/tests/export.test.ts +++ b/tests/export.test.ts @@ -17,6 +17,7 @@ import { import { extractFactSheetTokensAllPages, extractFactSheetTokens } from '../src/core/factsheet.js'; import { DENSE_CONTENT_CHARS_PER_IMAGE } from '../src/core/render.js'; import { patchTokens } from '../src/core/anthropic-vision.js'; +import { resolveGptProfile } from '../src/core/gpt-model-profiles.js'; // --------------------------------------------------------------------------- // Temp-dir helpers @@ -122,6 +123,11 @@ describe('parseExportArgv', () => { expect(parseExportArgv(['--help'])).toEqual({ kind: 'help' }); }); + it('DEFAULT_EXPORT_COLS is 172 matching default export model', () => { + expect(DEFAULT_EXPORT_COLS).toBe(172); + expect(DEFAULT_EXPORT_COLS).toBe(resolveGptProfile(DEFAULT_EXPORT_MODEL).stripCols); + }); + it('returns defaults when argv is empty', () => { const result = parseExportArgv([], '/tmp'); expect(result.kind).toBe('opts'); diff --git a/tests/grok.test.ts b/tests/grok.test.ts new file mode 100644 index 000000000..310d9b782 --- /dev/null +++ b/tests/grok.test.ts @@ -0,0 +1,252 @@ +/** + * Grok profile, geometry, vision billing, and history collapse. + * Kept out of openai-gpt5.test.ts so Grok is not mixed with GPT fixtures. + */ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { openAIVisionTokens, visionTokensForModel, transformOpenAIChatCompletions, transformOpenAIResponses } from '../src/core/openai.js'; +import { resolveGptProfile } from '../src/core/gpt-model-profiles.js'; + +const enc = new TextEncoder(); +const dec = new TextDecoder(); +const BIG_INSTRUCTIONS = 'These are detailed instructions. '.repeat(600); + +let ambientPxpipeModels: string | undefined; +beforeEach(() => { + ambientPxpipeModels = process.env.PXPIPE_MODELS; + delete process.env.PXPIPE_MODELS; +}); +afterEach(() => { + if (ambientPxpipeModels === undefined) delete process.env.PXPIPE_MODELS; + else process.env.PXPIPE_MODELS = ambientPxpipeModels; +}); + +describe('resolveGptProfile (Grok)', () => { + it('uses native 14px packing with shorter pages under 768px short side', () => { + // Native 14px was the densest best rung on the Grok JB Mono blind sweep. + // 84 × 9px + pad = 764px ≤ 768. No grid. maxH 512 keeps pages short. + const p = resolveGptProfile('grok-4.5'); + expect(p.stripCols).toBe(84); + expect(p.maxHeightPx).toBe(512); + expect(p.minCompressTokens).toBe(500); + // No observed xAI body limit (clean 200s past 2 MB), so no guessed cap. + expect(p.maxSerializedRequestBytes).toBeUndefined(); + expect(p.style.font).toBe('jetbrains-mono-14'); + expect(p.style.cellWBonus).toBe(0); + expect(p.style.cellHBonus).toBe(0); + expect(p.style.aa).toBe(true); + expect(p.style.grid).toBe(false); + expect(p.style.gridCols).toBe(0); + expect(p.style.colorCycle).toBe(false); + expect(p.history).toMatchObject({ + responsesMode: 'mixed', + maxImages: 32, + keepTail: 1, + keepRecentPairs: 1, + minCollapseTokens: 0, + minCollapsePrefix: 1, + collapseChunk: 1, + freezeChunk: 1, + framing: 'compact', + factSheetScope: 'combined', + }); + expect(resolveGptProfile('grok-4').stripCols).toBe(84); + expect(resolveGptProfile('grok-4.6').history.responsesMode).toBe('mixed'); + }); + + it('renders the opt-in profile at 764px wide (no short-side resize)', async () => { + const body = enc.encode(JSON.stringify({ + model: 'grok-4.5', + instructions: BIG_INSTRUCTIONS, + input: [{ role: 'user', content: 'hello' }], + })); + const result = await transformOpenAIResponses(body, { charsPerToken: 1, minCompressChars: 1 }); + expect(result.info.compressed).toBe(true); + // 84 cols × 9px + padding = 764px short-side floor. + expect(result.info.firstImageWidth).toBe(764); + expect(result.info.firstImageHeight ?? 0).toBeLessThanOrEqual(512); + }); +}); + +describe('Grok no-resize geometry', () => { + it('keeps rendered short side at or below 768px for slab and history packing', async () => { + const profile = resolveGptProfile('grok-4.5'); + // jetbrains-mono-14 native cell is 9×16; bonuses stay 0. + const cellW = 9 + (profile.style.cellWBonus ?? 0); + const stripW = 8 + profile.stripCols * cellW; // 2*PAD_X=8 + expect(stripW).toBeLessThanOrEqual(768); + expect(profile.stripCols).toBe(84); + expect(profile.style.font).toBe('jetbrains-mono-14'); + expect(cellW).toBe(9); + expect(profile.maxHeightPx).toBe(512); + expect(stripW).toBe(764); + + const body = enc.encode(JSON.stringify({ + model: 'grok-4.5', + instructions: BIG_INSTRUCTIONS, + input: [{ role: 'user', content: 'hello' }], + })); + const result = await transformOpenAIResponses(body, { charsPerToken: 1, minCompressChars: 1 }); + expect(result.info.firstImageWidth ?? 0).toBeLessThanOrEqual(768); + expect(result.info.firstImageWidth).toBe(764); + expect(result.info.firstImageHeight ?? 0).toBeLessThanOrEqual(512); + }); +}); + +describe('visionTokensForModel (Grok)', () => { + it('prices Grok images by measured megapixel rate, not GPT tiles', () => { + expect(visionTokensForModel('grok-4.5', 768, 336)).toBe(Math.ceil((768 * 336) / 1000)); + expect(visionTokensForModel('grok-4.5', 764, 980)).toBe(Math.ceil((764 * 980) / 1000)); + expect(visionTokensForModel('grok-4.5', 764, 980)).toBeLessThan( + openAIVisionTokens('gpt-4o', 764, 980), + ); + }); +}); + +describe('Grok history compression under default gate', () => { + it('collapses a short grok-4.6 chat/completions transcript (not just Responses)', async () => { + // Completions used minCollapsePrefix=10, so a few huge messages never + // imaged and the dashboard showed ~0 saved. Grok's prefix floor is 1. + const bulk = 'window 0 train 2024-01-15 test breakout+htf @240m d10 seed 780 bars. '; + const messages = [ + { role: 'system', content: 'You are a coding agent. '.repeat(200) }, + { role: 'user', content: bulk.repeat(80) }, + { role: 'assistant', content: bulk.repeat(80) }, + { role: 'user', content: bulk.repeat(80) }, + { role: 'assistant', content: bulk.repeat(80) }, + { role: 'user', content: 'continue' }, + ]; + const body = enc.encode(JSON.stringify({ model: 'grok-4.6', messages })); + const result = await transformOpenAIChatCompletions(body, { minCompressChars: 1 }); + expect(result.info.compressed).toBe(true); + expect(result.info.historyReason).toBe('collapsed'); + expect(result.info.collapsedImages ?? 0).toBeGreaterThan(0); + expect(result.info.imageTokens ?? 0).toBeLessThan(result.info.baselineImagedTokens ?? 0); + }); + + it('images most of a large grok-4.6 completions chat, not just the slab', async () => { + const bulk = 'window 0 train 2024-01-15 test breakout+htf @240m d10 seed 780 bars, eligibility ok. '; + const messages: Array> = [ + { role: 'system', content: 'You are a coding agent. '.repeat(400) }, + ]; + for (let i = 0; i < 6; i++) { + messages.push({ role: 'user', content: bulk.repeat(80) + ` turn ${i}` }); + messages.push({ role: 'assistant', content: bulk.repeat(80) + ` reply ${i}` }); + } + messages.push({ role: 'user', content: 'continue' }); + const body = enc.encode(JSON.stringify({ model: 'grok-4.6', messages })); + const result = await transformOpenAIChatCompletions(body, { minCompressChars: 1 }); + expect(result.info.historyReason).toBe('collapsed'); + expect(result.info.collapsedImages ?? 0).toBeGreaterThan(1); + const imaged = result.info.baselineImagedTokens ?? 0; + const sent = result.info.imageTokens ?? 0; + expect(imaged).toBeGreaterThan(sent * 2); + expect(result.info.bucketChars?.history ?? 0).toBeGreaterThan(result.info.bucketChars?.static_slab ?? 0); + }); + + it('collapses long Grok Responses history under default charsPerToken (o200k gate)', async () => { + const items: Array> = [ + { role: 'user', content: 'start the long autonomous run now please' }, + ]; + for (let i = 0; i < 10; i++) { + const id = `call_${i}`; + items.push({ role: 'assistant', content: `Working on step ${i}. `.repeat(40) }); + items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); + items.push({ type: 'function_call_output', call_id: id, output: (`result ${i} path=/tmp/out${i}.json `).repeat(60) }); + } + const body = enc.encode(JSON.stringify({ + model: 'grok-4.6', + instructions: 'You are a careful coding agent. '.repeat(200), + input: items, + })); + const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); + expect(result.info.compressed).toBe(true); + expect(result.info.historyReason).toBe('collapsed'); + expect(result.info.collapsedImages ?? 0).toBeGreaterThan(0); + expect(result.info.imageTokens ?? 0).toBeLessThan(result.info.baselineImagedTokens ?? 0); + // Codex puts an assistant message between tool rounds. pairs would split + // each round into its own run; mixed packs them so grok-4.6 actually saves. + expect((result.info.collapsedImages ?? 0)).toBeLessThan(10); + }); + + it('collapses a large Grok history with no profile byte cap in the way', async () => { + const items: Array> = [ + { role: 'user', content: 'start the long autonomous run now please' }, + ]; + for (let i = 0; i < 40; i++) { + const id = `call_${i}`; + items.push({ role: 'assistant', content: `Working on step ${i}. `.repeat(40) }); + items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); + items.push({ type: 'function_call_output', call_id: id, output: (`result ${i} path=/tmp/out${i}.json `).repeat(60) }); + } + const body = enc.encode(JSON.stringify({ + model: 'grok-4.5', + instructions: 'You are a careful coding agent. '.repeat(200), + input: items, + })); + const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); + expect(result.info.compressed).toBe(true); + expect(result.info.historyReason).toBe('collapsed'); + expect(result.info.reason).not.toBe('serialized_request_limit'); + }); + + it('passes through when an env-configured byte cap would be overshot', async () => { + const prev = process.env.PXPIPE_GPT_PROFILES; + process.env.PXPIPE_GPT_PROFILES = JSON.stringify({ + 'grok-4.5': { maxSerializedRequestBytes: 128 * 1024 }, + }); + try { + const items: Array> = [ + { role: 'user', content: 'start the long autonomous run now please' }, + ]; + for (let i = 0; i < 40; i++) { + const id = `call_${i}`; + items.push({ role: 'assistant', content: `Working on step ${i}. `.repeat(40) }); + items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); + items.push({ type: 'function_call_output', call_id: id, output: (`result ${i} path=/tmp/out${i}.json `).repeat(60) }); + } + const body = enc.encode(JSON.stringify({ + model: 'grok-4.5', + instructions: 'You are a careful coding agent. '.repeat(200), + input: items, + })); + const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); + expect(result.info.compressed).toBe(false); + expect(result.info.reason).toBe('serialized_request_limit'); + expect(result.body.byteLength).toBe(body.byteLength); + } finally { + if (prev !== undefined) process.env.PXPIPE_GPT_PROFILES = prev; + else delete process.env.PXPIPE_GPT_PROFILES; + } + }); + + it('pages factsheet across long collapsed history so early exact ids survive', async () => { + const earlyHex = 'a3f9c1e0b7d2'; + const items: Array> = [ + { role: 'user', content: `remember ${earlyHex} and path src/core/anthropic-vision.ts port 47821` }, + ]; + for (let i = 0; i < 80; i++) { + const id = `call_${i}`; + items.push({ + role: 'assistant', + content: `Working on step ${i} for module src/pkg/mod${i}/handler.ts with checksum ${i.toString(16).padStart(8, '0')}ab. `.repeat(40), + }); + items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); + items.push({ + type: 'function_call_output', + call_id: id, + output: (`result ${i} path=/tmp/out${i}.json status=ok note=step-${i}-detail `).repeat(80), + }); + } + const body = enc.encode(JSON.stringify({ + model: 'grok-4.5', + instructions: 'Keep identifiers exact. '.repeat(200), + input: items, + })); + const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); + expect(result.info.historyReason).toBe('collapsed'); + const out = JSON.parse(dec.decode(result.body)) as { input: Array> }; + const serialized = JSON.stringify(out.input); + expect(serialized).toContain(earlyHex); + expect(serialized).toContain('47821'); + }); +}); diff --git a/tests/history-geometry.test.ts b/tests/history-geometry.test.ts index ce71f5b65..c82062f5c 100644 --- a/tests/history-geometry.test.ts +++ b/tests/history-geometry.test.ts @@ -50,8 +50,14 @@ describe('per-content-class render geometry', () => { // jetbrains-mono-14. Fable read 25/26 at dense, so it keeps the 5.28x // geometry; non-Claude families are untouched by this PR. const opus = resolveGptProfile('claude-opus-5'); + expect(opus.stripCols).toBe(172); + expect(opus.style.font).toBe('jetbrains-mono-14'); expect(opus.historyStripCols).toBe(172); expect(opus.historyStyle?.font).toBe('jetbrains-mono-14'); + + const sonnet = resolveGptProfile('claude-sonnet-4-5'); + expect(sonnet.stripCols).toBe(172); + expect(sonnet.style.font).toBe('jetbrains-mono-14'); for (const m of ['claude-fable-5', 'gpt-5.6-sol', 'moonshotai/kimi-k3']) { const p = resolveGptProfile(m); expect(p.historyStripCols, `${m} must stay dense by default`).toBeUndefined(); @@ -61,25 +67,32 @@ describe('per-content-class render geometry', () => { it('accepts historyStripCols and historyStyle from PXPIPE_GPT_PROFILES', () => { process.env[ENV] = JSON.stringify({ - 'claude-opus-5': { historyStripCols: 172, historyStyle: { font: 'jetbrains-mono-14' } }, + 'claude-fable-5': { historyStripCols: 172, historyStyle: { font: 'jetbrains-mono-14' } }, }); - const p = resolveGptProfile('claude-opus-5'); + const p = resolveGptProfile('claude-fable-5'); + expect(p.stripCols).toBe(312); + expect(p.style.font).toBe('spleen-5x8'); expect(p.historyStripCols).toBe(172); expect(p.historyStyle?.font).toBe('jetbrains-mono-14'); - // The slab and tool-result geometry must not move with it. - expect(p.stripCols).toBe(resolveGptProfile('claude-fable-5').stripCols); - expect(p.style.font).toBe(resolveGptProfile('claude-fable-5').style.font); }); - it('leaves the dense geometry alone when only the history override is given', () => { - const before = resolveGptProfile('claude-opus-5'); + it('leaves the profile stripCols alone when only the history override is given', () => { + const beforeFable = resolveGptProfile('claude-fable-5'); + const beforeOpus = resolveGptProfile('claude-opus-5'); process.env[ENV] = JSON.stringify({ - 'claude-opus-5': { historyStripCols: 172, historyStyle: { font: 'jetbrains-mono-14' } }, + 'claude-fable-5': { historyStripCols: 172, historyStyle: { font: 'jetbrains-mono-14' } }, + 'claude-opus-5': { historyStripCols: 100, historyStyle: { font: 'jetbrains-mono-14' } }, }); - const after = resolveGptProfile('claude-opus-5'); - expect(after.stripCols).toBe(before.stripCols); - expect(after.maxHeightPx).toBe(before.maxHeightPx); - expect(after.style).toEqual(before.style); + const afterFable = resolveGptProfile('claude-fable-5'); + expect(afterFable.stripCols).toBe(beforeFable.stripCols); + expect(afterFable.maxHeightPx).toBe(beforeFable.maxHeightPx); + expect(afterFable.style).toEqual(beforeFable.style); + + const afterOpus = resolveGptProfile('claude-opus-5'); + expect(afterOpus.stripCols).toBe(beforeOpus.stripCols); + expect(afterOpus.maxHeightPx).toBe(beforeOpus.maxHeightPx); + expect(afterOpus.style).toEqual(beforeOpus.style); + expect(afterOpus.historyStripCols).toBe(100); }); it('renders history at the legible geometry by default, costing more image tokens for the same text', async () => { diff --git a/tests/openai-gpt5.test.ts b/tests/openai-gpt5.test.ts index 1381dd9f7..8b40d67cb 100644 --- a/tests/openai-gpt5.test.ts +++ b/tests/openai-gpt5.test.ts @@ -5,7 +5,7 @@ import { afterEach, beforeEach, describe, expect, it } from 'vitest'; import { isPxpipeSupportedGptModel } from '../src/core/applicability.js'; import { openAIVisionTokens, visionTokensForModel, isClaudeModel, resolveVisionCost, transformOpenAIChatCompletions, transformOpenAIResponses } from '../src/core/openai.js'; -import { resolveGptProfile, isMisresolvedModelId } from '../src/core/gpt-model-profiles.js'; +import { resolveGptProfile } from '../src/core/gpt-model-profiles.js'; const enc = new TextEncoder(); const dec = new TextDecoder(); @@ -1090,21 +1090,8 @@ describe('image parts request detail = "original" (avoid downscale of dense text }); }); - -describe('resolveGptProfile (Claude on Responses)', () => { - it('uses Anthropic geometry by model id, not the GPT Responses defaults', () => { - // Several families share /v1/responses. Claude ids must retain Anthropic's - // 312-col / 728px geometry rather than the GPT defaults, which overstate - // image tokens and leave As text / Saved blank. - const p = resolveGptProfile('claude-opus-4-8'); - expect(p.maxHeightPx).toBe(728); - expect(p.stripCols).toBe(312); - expect(p.style.font).toBe('spleen-5x8'); - expect(resolveGptProfile('claude-3-5-opus').maxHeightPx).toBe(728); - expect(resolveGptProfile('claude-3-5-opus').stripCols).toBe(312); - expect(resolveGptProfile('claude-fable-5').maxHeightPx).toBe(728); - expect(resolveGptProfile('claude-fable-5').stripCols).toBe(312); - expect(resolveGptProfile('claude-fable-5').minCompressTokens).toBeUndefined(); +describe('resolveGptProfile (GPT-5.6 Sol)', () => { + it('uses native 14px packing, 1954px height, and NATIVE_14PX_HISTORY', () => { for (const model of [ 'gpt-5.6-sol', 'gpt-5.6-sol[1m]', @@ -1125,7 +1112,10 @@ describe('resolveGptProfile (Claude on Responses)', () => { maxImages: 64, keepTail: 1, keepRecentPairs: 1, - minCollapseTokens: 1000, + minCollapseTokens: 0, + minCollapsePrefix: 1, + collapseChunk: 1, + freezeChunk: 1, framing: 'compact', factSheetScope: 'combined', }); @@ -1139,145 +1129,8 @@ describe('resolveGptProfile (Claude on Responses)', () => { expect(notSol.history.maxImages, model).toBe(32); expect(notSol.factSheetFormat, model).toBe('full'); } - expect(resolveGptProfile('claude-fable-5').style.font).toBe('spleen-5x8'); - // No built-in byte cap on ANY family: no provider request-size limit has been - // sourced for Claude, Grok, Gemini or GPT. A cap here would make pxpipe skip - // compression on requests the provider accepts. Deployments pin one via - // PXPIPE_GPT_PROFILES (covered by the env-override cases below). - expect(resolveGptProfile('claude-fable-5').maxSerializedRequestBytes).toBeUndefined(); - expect(resolveGptProfile('claude-opus-5').maxSerializedRequestBytes).toBeUndefined(); - expect(resolveGptProfile('gemini-3.6-flash').maxSerializedRequestBytes).toBeUndefined(); - expect(resolveGptProfile('grok-4.5').maxSerializedRequestBytes).toBeUndefined(); expect(resolveGptProfile('gpt-5.6-sol').maxSerializedRequestBytes).toBeUndefined(); }); - - it('isolates per-id profile overrides between Claude models', () => { - const prev = process.env.PXPIPE_GPT_PROFILES; - try { - process.env.PXPIPE_GPT_PROFILES = JSON.stringify({ - 'claude-opus': { - stripCols: 200, - style: { cellWBonus: 2, cellHBonus: 2 }, - }, - }); - const opus = resolveGptProfile('claude-opus-4-8'); - const fable = resolveGptProfile('claude-fable-5'); - - expect(opus.stripCols).toBe(200); - expect(opus.style.cellWBonus).toBe(2); - expect(opus.style.cellHBonus).toBe(2); - - // Fable remains unaffected by Opus overrides - expect(fable.stripCols).toBe(312); - expect(fable.style.cellWBonus).toBe(0); - expect(fable.style.cellHBonus).toBe(0); - } finally { - if (prev !== undefined) process.env.PXPIPE_GPT_PROFILES = prev; - else delete process.env.PXPIPE_GPT_PROFILES; - } - }); -}); - -describe('resolveGptProfile (Grok)', () => { - it('uses native 14px packing with shorter pages under 768px short side', () => { - // Native 14px was the densest best rung on the Grok JB Mono blind sweep. - // 84 × 9px + pad = 764px ≤ 768. No grid. maxH 512 keeps pages short. - const p = resolveGptProfile('grok-4.5'); - expect(p.stripCols).toBe(84); - expect(p.maxHeightPx).toBe(512); - expect(p.minCompressTokens).toBe(500); - // No observed xAI body limit (clean 200s past 2 MB), so no guessed cap. - expect(p.maxSerializedRequestBytes).toBeUndefined(); - expect(p.style.font).toBe('jetbrains-mono-14'); - expect(p.style.cellWBonus).toBe(0); - expect(p.style.cellHBonus).toBe(0); - expect(p.style.aa).toBe(true); - expect(p.style.grid).toBe(false); - expect(p.style.gridCols).toBe(0); - expect(p.style.colorCycle).toBe(false); - expect(resolveGptProfile('grok-4').stripCols).toBe(84); - }); - - it('renders the opt-in profile at 764px wide (no short-side resize)', async () => { - const body = enc.encode(JSON.stringify({ - model: 'grok-4.5', - instructions: BIG_INSTRUCTIONS, - input: [{ role: 'user', content: 'hello' }], - })); - const result = await transformOpenAIResponses(body, { charsPerToken: 1, minCompressChars: 1 }); - expect(result.info.compressed).toBe(true); - // 84 cols × 9px + padding = 764px short-side floor. - expect(result.info.firstImageWidth).toBe(764); - expect(result.info.firstImageHeight ?? 0).toBeLessThanOrEqual(512); - }); -}); - -describe('resolveGptProfile (Qwen)', () => { - it('uses native 14px packing and the 32-image provider cap', () => { - const p = resolveGptProfile('workers-ai/@cf/qwen/qwen3.8-27b'); - expect(p.stripCols).toBe(84); - expect(p.maxHeightPx).toBe(512); - expect(p.style.font).toBe('jetbrains-mono-14'); - expect(p.history.maxImages).toBe(32); - expect(p.history.keepTail).toBe(1); - expect(p.providerImageCap).toBe(32); - // Keyed to the exact measured model id, not to any 'qwen' substring. - expect(resolveGptProfile('qwen3.8-27b').stripCols).toBe(84); - }); - - it('refuses unmeasured Qwen variants instead of applying this profile', () => { - for (const id of ['qwen2.5-72b-instruct', 'qwen3-30b', 'qwen-3.8-27b']) { - expect(resolveGptProfile(id).stripCols).not.toBe(84); - expect(isMisresolvedModelId(id)).toBe(true); - } - }); -}); - -describe('Qwen provider image cap — dynamic history budget', () => { - it('budgets history against 32 minus the images already in the request', async () => { - const tinyPng = { url: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==' }; - const messages = buildChatMessages(20); - // The client attached 24 images to the live request, outside collapsed history. - messages.at(-1)!.content = [ - { type: 'text', text: messages.at(-1)!.content }, - ...Array.from({ length: 24 }, () => ({ type: 'image_url', image_url: tinyPng })), - ]; - const result = await transformOpenAIChatCompletions( - enc.encode(JSON.stringify({ model: 'workers-ai/@cf/qwen/qwen3.8-27b', messages })), - { charsPerToken: 1, minCompressChars: 1 }, - ); - expect(result.info.compressed).toBe(true); - expect(result.info.historyReason).toBe('collapsed'); - // With the old subtractive budget (24 − 24 − slab) this would be 0. - expect(result.info.collapsedImages ?? 0).toBeGreaterThan(0); - - const out = JSON.parse(dec.decode(result.body)) as { messages: Array<{ role: string; content: unknown }> }; - let totalImages = 0; - for (const m of out.messages) { - if (Array.isArray(m.content)) { - totalImages += (m.content as Array<{ type?: string }>).filter((p) => p.type === 'image_url').length; - } - } - expect(JSON.stringify(out.messages).match(/iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ/g)?.length).toBe(24); - expect(totalImages).toBeLessThanOrEqual(32); - }); - - it('keeps static context native when client images leave insufficient headroom', async () => { - const tinyPng = { url: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==' }; - const messages: Array> = [ - { role: 'system', content: BIG_SLAB }, - { - role: 'user', - content: Array.from({ length: 31 }, () => ({ type: 'image_url', image_url: tinyPng })), - }, - ]; - const body = enc.encode(JSON.stringify({ model: 'workers-ai/@cf/qwen/qwen3.8-27b', messages })); - const result = await transformOpenAIChatCompletions(body, { minCompressChars: 1 }); - - expect(result.info.compressed).toBe(false); - expect(result.info.reason).toBe('provider_image_cap'); - expect(result.body).toEqual(body); - }); }); describe('resolveGptProfile style overrides', () => { @@ -1342,157 +1195,3 @@ describe('resolveGptProfile style overrides', () => { } }); }); - -describe('Grok no-resize geometry', () => { - it('keeps rendered short side at or below 768px for slab and history packing', async () => { - const profile = resolveGptProfile('grok-4.5'); - // jetbrains-mono-14 native cell is 9×16; bonuses stay 0. - const cellW = 9 + (profile.style.cellWBonus ?? 0); - const stripW = 8 + profile.stripCols * cellW; // 2*PAD_X=8 - expect(stripW).toBeLessThanOrEqual(768); - expect(profile.stripCols).toBe(84); - expect(profile.style.font).toBe('jetbrains-mono-14'); - expect(cellW).toBe(9); - expect(profile.maxHeightPx).toBe(512); - expect(stripW).toBe(764); - - // End-to-end: rendered PNG width matches the no-resize strip. - const body = enc.encode(JSON.stringify({ - model: 'grok-4.5', - instructions: BIG_INSTRUCTIONS, - input: [{ role: 'user', content: 'hello' }], - })); - const result = await transformOpenAIResponses(body, { charsPerToken: 1, minCompressChars: 1 }); - expect(result.info.firstImageWidth ?? 0).toBeLessThanOrEqual(768); - expect(result.info.firstImageWidth).toBe(764); - expect(result.info.firstImageHeight ?? 0).toBeLessThanOrEqual(512); - }); -}); - -describe('visionTokensForModel (Grok)', () => { - it('prices Grok images by measured megapixel rate, not GPT tiles', () => { - // ceil(w*h/1e6 * 1000) - expect(visionTokensForModel('grok-4.5', 768, 336)).toBe(Math.ceil((768 * 336) / 1000)); - expect(visionTokensForModel('grok-4.5', 764, 980)).toBe(Math.ceil((764 * 980) / 1000)); - // Must not use GPT tile pricing (would be much larger for tall pages). - expect(visionTokensForModel('grok-4.5', 764, 980)).toBeLessThan( - openAIVisionTokens('gpt-4o', 764, 980), - ); - }); -}); - -describe('Grok history compression under default gate', () => { - it('collapses long Grok Responses history under default charsPerToken (o200k gate)', async () => { - // Production gate path: no charsPerToken override. - const items: Array> = [ - { role: 'user', content: 'start the long autonomous run now please' }, - ]; - for (let i = 0; i < 10; i++) { - const id = `call_${i}`; - items.push({ role: 'assistant', content: `Working on step ${i}. `.repeat(40) }); - items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); - items.push({ type: 'function_call_output', call_id: id, output: (`result ${i} path=/tmp/out${i}.json `).repeat(60) }); - } - const body = enc.encode(JSON.stringify({ - model: 'grok-4.5', - instructions: 'You are a careful coding agent. '.repeat(200), - input: items, - })); - const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); - expect(result.info.compressed).toBe(true); - expect(result.info.historyReason).toBe('collapsed'); - expect(result.info.collapsedImages ?? 0).toBeGreaterThan(0); - expect(result.info.imageTokens ?? 0).toBeLessThan(result.info.baselineImagedTokens ?? 0); - }); - - it('collapses a large Grok history with no profile byte cap in the way', async () => { - // 4x the history above. Grok pins no serialized-byte cap (no observed xAI - // limit), so imaging must still happen even though base64 PNGs are ~2.5x the - // bytes of the text they replace: cheaper in TOKENS, dearer in BYTES. - const items: Array> = [ - { role: 'user', content: 'start the long autonomous run now please' }, - ]; - for (let i = 0; i < 40; i++) { - const id = `call_${i}`; - items.push({ role: 'assistant', content: `Working on step ${i}. `.repeat(40) }); - items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); - items.push({ type: 'function_call_output', call_id: id, output: (`result ${i} path=/tmp/out${i}.json `).repeat(60) }); - } - const body = enc.encode(JSON.stringify({ - model: 'grok-4.5', - instructions: 'You are a careful coding agent. '.repeat(200), - input: items, - })); - const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); - expect(result.info.compressed).toBe(true); - expect(result.info.historyReason).toBe('collapsed'); - expect(result.info.reason).not.toBe('serialized_request_limit'); - }); - - it('passes through when an env-configured byte cap would be overshot', async () => { - // The cap mechanism itself still has to work for any deployment that pins - // one: the transform falls back to the original body rather than building a - // request the proxy would answer with a 413. - const prev = process.env.PXPIPE_GPT_PROFILES; - process.env.PXPIPE_GPT_PROFILES = JSON.stringify({ - 'grok-4.5': { maxSerializedRequestBytes: 128 * 1024 }, - }); - try { - const items: Array> = [ - { role: 'user', content: 'start the long autonomous run now please' }, - ]; - for (let i = 0; i < 40; i++) { - const id = `call_${i}`; - items.push({ role: 'assistant', content: `Working on step ${i}. `.repeat(40) }); - items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); - items.push({ type: 'function_call_output', call_id: id, output: (`result ${i} path=/tmp/out${i}.json `).repeat(60) }); - } - const body = enc.encode(JSON.stringify({ - model: 'grok-4.5', - instructions: 'You are a careful coding agent. '.repeat(200), - input: items, - })); - const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); - expect(result.info.compressed).toBe(false); - expect(result.info.reason).toBe('serialized_request_limit'); - expect(result.body.byteLength).toBe(body.byteLength); - } finally { - if (prev !== undefined) process.env.PXPIPE_GPT_PROFILES = prev; - else delete process.env.PXPIPE_GPT_PROFILES; - } - }); - - it('pages factsheet across long collapsed history so early exact ids survive', async () => { - const earlyHex = 'a3f9c1e0b7d2'; - const items: Array> = [ - { role: 'user', content: `remember ${earlyHex} and path src/core/anthropic-vision.ts port 47821` }, - ]; - // Long enough that a single-pass factsheet scan would miss the head. - // Filler must stay high-entropy: o200k compresses long runs of "x" so hard - // that the 14px image bill (fewer chars/page than old 5×8) looks unprofitable. - for (let i = 0; i < 80; i++) { - const id = `call_${i}`; - items.push({ - role: 'assistant', - content: `Working on step ${i} for module src/pkg/mod${i}/handler.ts with checksum ${i.toString(16).padStart(8, '0')}ab. `.repeat(40), - }); - items.push({ type: 'function_call', call_id: id, name: 'read', arguments: `{"path":"src/f${i}.ts"}` }); - items.push({ - type: 'function_call_output', - call_id: id, - output: (`result ${i} path=/tmp/out${i}.json status=ok note=step-${i}-detail `).repeat(80), - }); - } - const body = enc.encode(JSON.stringify({ - model: 'grok-4.5', - instructions: 'Keep identifiers exact. '.repeat(200), - input: items, - })); - const result = await transformOpenAIResponses(body, { minCompressChars: 1 }); - expect(result.info.historyReason).toBe('collapsed'); - const out = JSON.parse(dec.decode(result.body)) as { input: Array> }; - const serialized = JSON.stringify(out.input); - expect(serialized).toContain(earlyHex); - expect(serialized).toContain('47821'); - }); -}); diff --git a/tests/public-api.test.ts b/tests/public-api.test.ts index 9d2f86af8..9fbea620b 100644 --- a/tests/public-api.test.ts +++ b/tests/public-api.test.ts @@ -133,14 +133,17 @@ describe('public library API', () => { try { delete process.env.PXPIPE_MODELS; expect(isPxpipeSupportedGptModel('grok-4.5')).toBe(false); + expect(isPxpipeSupportedGptModel('grok-4.6')).toBe(false); expect(isPxpipeSupportedGptModel('grok-4')).toBe(false); expect(isPxpipeSupportedGptModel('grok-4.20')).toBe(false); expect(getAllowedModelBases()).not.toContain('grok-4.5'); + expect(getAllowedModelBases()).not.toContain('grok-4.6'); expect(getAllowedModelBases()).toEqual(['claude-fable-5', 'gemini-3.6-flash', 'gemini-3.7-flash']); - process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6-sol,grok-4.5'; - expect(isPxpipeSupportedGptModel('grok-4.5')).toBe(true); - expect(isPxpipeSupportedGptModel('grok-4.5-fast')).toBe(true); // -suffix alias + process.env.PXPIPE_MODELS = 'claude-fable-5,gpt-5.6-sol,grok-4.6'; + expect(isPxpipeSupportedGptModel('grok-4.6')).toBe(true); + expect(isPxpipeSupportedGptModel('grok-4.6-fast')).toBe(true); // -suffix alias + expect(isPxpipeSupportedGptModel('grok-4.5')).toBe(false); expect(isPxpipeSupportedGptModel('gpt-5.6-sol')).toBe(true); } finally { if (prev === undefined) delete process.env.PXPIPE_MODELS; diff --git a/tests/qwen.test.ts b/tests/qwen.test.ts new file mode 100644 index 000000000..ee48b95b0 --- /dev/null +++ b/tests/qwen.test.ts @@ -0,0 +1,116 @@ +/** + * Qwen 3.8 27B profile and the 32-image provider cap. + * Kept out of openai-gpt5.test.ts so Qwen is not mixed with GPT fixtures. + */ +import { afterEach, beforeEach, describe, expect, it } from 'vitest'; +import { transformOpenAIChatCompletions } from '../src/core/openai.js'; +import { resolveGptProfile, isMisresolvedModelId } from '../src/core/gpt-model-profiles.js'; + +const enc = new TextEncoder(); +const dec = new TextDecoder(); +const BIG_SLAB = 'You are a coding agent with detailed instructions. '.repeat(80); +const OPENING_PROMPT_MARKER = 'OPENING_PROMPT_SHOULD_BE_HISTORY'; +const LIVE_PROMPT_MARKER = 'LIVE_CURRENT_PROMPT_SHOULD_STAY_TEXT'; + +function buildChatMessages(turns: number): Array> { + const msgs: Array> = [ + { role: 'system', content: BIG_SLAB }, + { role: 'user', content: `${OPENING_PROMPT_MARKER} `.repeat(40) }, + ]; + for (let i = 0; i < turns; i++) { + const id = `call_${i}`; + msgs.push({ + role: 'assistant', + content: `Working on step ${i}. `.repeat(30), + tool_calls: [{ id, type: 'function', function: { name: 'read', arguments: `{"path":"f${i}"}` } }], + }); + msgs.push({ role: 'tool', tool_call_id: id, content: `result ${i} `.repeat(50) }); + msgs.push({ + role: 'user', + content: i === turns - 1 + ? `${LIVE_PROMPT_MARKER} `.repeat(20) + : `Continue with ${i}. `.repeat(20), + }); + } + return msgs; +} + +let ambientPxpipeModels: string | undefined; +beforeEach(() => { + ambientPxpipeModels = process.env.PXPIPE_MODELS; + delete process.env.PXPIPE_MODELS; +}); +afterEach(() => { + if (ambientPxpipeModels === undefined) delete process.env.PXPIPE_MODELS; + else process.env.PXPIPE_MODELS = ambientPxpipeModels; +}); + +describe('resolveGptProfile (Qwen)', () => { + it('uses native 14px packing and the 32-image provider cap', () => { + const p = resolveGptProfile('workers-ai/@cf/qwen/qwen3.8-27b'); + expect(p.stripCols).toBe(84); + expect(p.maxHeightPx).toBe(512); + expect(p.style.font).toBe('jetbrains-mono-14'); + expect(p.history.maxImages).toBe(32); + expect(p.history.keepTail).toBe(1); + expect(p.history.minCollapseTokens).toBe(0); + expect(p.history.minCollapsePrefix).toBe(1); + expect(p.history.responsesMode).toBe('mixed'); + expect(p.providerImageCap).toBe(32); + expect(resolveGptProfile('qwen3.8-27b').stripCols).toBe(84); + }); + + it('refuses unmeasured Qwen variants instead of applying this profile', () => { + for (const id of ['qwen2.5-72b-instruct', 'qwen3-30b', 'qwen-3.8-27b']) { + expect(resolveGptProfile(id).stripCols).not.toBe(84); + expect(isMisresolvedModelId(id)).toBe(true); + } + }); +}); + +describe('Qwen provider image cap — dynamic history budget', () => { + it('budgets history against 32 minus the images already in the request', async () => { + const tinyPng = { url: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==' }; + const messages = buildChatMessages(20); + const lastMsg = messages.at(-1); + if (!lastMsg) throw new Error('messages must not be empty'); + lastMsg.content = [ + { type: 'text', text: lastMsg.content }, + ...Array.from({ length: 24 }, () => ({ type: 'image_url', image_url: tinyPng })), + ]; + const result = await transformOpenAIChatCompletions( + enc.encode(JSON.stringify({ model: 'workers-ai/@cf/qwen/qwen3.8-27b', messages })), + { charsPerToken: 1, minCompressChars: 1 }, + ); + expect(result.info.compressed).toBe(true); + expect(result.info.historyReason).toBe('collapsed'); + expect(result.info.collapsedImages ?? 0).toBeGreaterThan(0); + + const out = JSON.parse(dec.decode(result.body)) as { messages: Array<{ role: string; content: unknown }> }; + let totalImages = 0; + for (const m of out.messages) { + if (Array.isArray(m.content)) { + totalImages += (m.content as Array<{ type?: string }>).filter((p) => p.type === 'image_url').length; + } + } + expect(JSON.stringify(out.messages).match(/iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJ/g)?.length).toBe(24); + expect(totalImages).toBeLessThanOrEqual(32); + }); + + it('keeps static context native when client images leave insufficient headroom', async () => { + const tinyPng = { url: 'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAYAAAAfFcSJAAAADUlEQVR42mNkYPhfDwAChwGA60e6kgAAAABJRU5ErkJggg==' }; + const messages: Array> = [ + { role: 'system', content: BIG_SLAB }, + { + role: 'user', + content: Array.from({ length: 31 }, () => ({ type: 'image_url', image_url: tinyPng })), + }, + ]; + const body = enc.encode(JSON.stringify({ model: 'workers-ai/@cf/qwen/qwen3.8-27b', messages })); + const result = await transformOpenAIChatCompletions(body, { minCompressChars: 1 }); + + expect(result.info.compressed).toBe(false); + expect(result.info.reason).toBe('provider_image_cap'); + expect(result.body).toEqual(body); + }); +});