Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
35 changes: 35 additions & 0 deletions FINDINGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,41 @@

---

## Update (2026-08-22) — Lost-in-the-Middle (LIM) & Effective Context Window on Gemini 3.6 & 3.7 Flash (300k & 1M context)

**Verdict: Visual context does not solve Lost-in-the-Middle because frontier LLMs already achieve 100% text retrieval across 1M tokens. However, Gemini 3.7 Flash's upgraded vision encoder achieves 80%–100% visual reasoning across 1,000,000 characters at ~75% token savings.**

### Key Insights
1. **Raw Text Baseline**: Both Gemini 3.6 Flash and 3.7 Flash achieve **100% multi-hop reasoning accuracy across all depths (10%, 30%, 50%, 70%, 90%)** up to 1,000,000 characters (~407k tokens). Modern frontier 1D text attention does not exhibit the classic positional "Lost in the Middle" dip on this benchmark.
2. **Visual Context on Gemini 3.7 vs 3.6**:
- **Gemini 3.6 Flash**: Stacking 32 dense images (1M chars) suffered from visual attention dilution in early/middle pages (0%–50% reasoning).
- **Gemini 3.7 Flash**: Upgraded visual attention successfully recovered the state mutations across the image stack, scoring **100% reasoning at 10%, 30%, 50% (middle), and 90% depth at 1,000,000 characters**.
3. **Core Tradeoff**: pxpipe is an **economic and latency multiplier** (compressing 407k tokens to ~35k vision tokens, −75% token cost), with Gemini 3.7 Flash sustaining reasoning fidelity through 1M character visual histories.

### Benchmark Data: Gemini 3.7 Flash

#### A. 300,000 Characters (~123k Text Tokens vs ~21k Vision Tokens)
| Depth | Raw Text Reasoning | pxpipe Visual Reasoning | Raw Multihop Recall | pxpipe Multihop Recall |
|:---:|:---:|:---:|:---:|:---:|
| **10%** (30k) | **100%** | **100%** | **100%** | **100%** |
| **30%** (90k) | **100%** | **100%** | **100%** | **100%** |
| **50% (Middle)** | **100%** | **100%** | **100%** | **100%** |
| **70%** (210k) | **100%** | **100%** | **100%** | **100%** |
| **90%** (270k) | **100%** | 0% | **100%** | 0% |

#### B. 1,000,000 Characters (~407k Text Tokens vs ~103k Total Tokens with 32 Images)
| Depth | Raw Text Reasoning | pxpipe Visual Reasoning | Raw Multihop Recall | pxpipe Multihop Recall |
|:---:|:---:|:---:|:---:|:---:|
| **10%** (100k) | **100%** | **100%** | **100%** | **100%** |
| **30%** (300k) | **100%** | **100%** | **100%** | 0% |
| **50% (Middle)** | **100%** | **100%** | **100%** | 0% |
| **70%** (700k) | **100%** | 0% | **100%** | 0% |
| **90%** (900k) | **100%** | **100%** | **100%** | **100%** |

Receipts preserved in `eval/gemini-profile/lost-in-middle-*-results.json`.

---

## Update (2026-07-20) — `claude-opus-5` promoted to default scope

**Verdict: ship it as a default, with the recall gap stated up front. Opus 5 is
Expand Down
8 changes: 4 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -148,10 +148,10 @@ without running the proxy.
recent/open tool state stays native.
[Sol receipts](eval/sol-profile/QUALITY_RESULTS.md) and
[profile evidence](docs/MODEL_RENDER_PROFILES.md).
- **Grok 4.5 (opt-in):** native 14px / 84 cols / maxH 512 (100/100 arith, 97/98 gist).
Off by default (dense hex still 0/15).
Enable with
`PXPIPE_MODELS=claude-fable-5,grok-4.5` or the dashboard chip.
- **Grok 4.5 / 4.6 (opt-in):** native 14px / 84 cols / maxH 512 (100/100 arith, 97/98 gist).
Off by default (dense hex still 0/15). History uses mixed collapse so Codex
assistant messages between tool rounds still image. Enable with
`PXPIPE_MODELS=claude-fable-5,grok-4.6` or the dashboard chip.
[eval/grok-density/QUALITY_RESULTS.md](eval/grok-density/QUALITY_RESULTS.md).

</details>
Expand Down
2 changes: 1 addition & 1 deletion docs/CACHING_AND_SAVINGS.md
Original file line number Diff line number Diff line change
Expand Up @@ -305,7 +305,7 @@ Measured on local `/v1/responses` rows (same endpoint, different models):

Render profiles are selected by exact model id, not by the shared Responses
path. Opt-in `gpt-5.6-sol` uses 84 columns with a native 9×16 JetBrains Mono
14px atlas; Claude uses 312 columns with the 5×8 Spleen atlas. Grok remains **opt-in** and
14px atlas; Claude uses 172 columns with the 14px JetBrains Mono atlas (Fable keeps 312 columns with Spleen 5×8). Grok remains **opt-in** and
uses **native 14px** / 84 columns at maxHeight 512 with white AA (**no grid**) plus an
in-image IDS block and the text factsheet. Its measured arithmetic, gist, and
state results remain below Fable. See
Expand Down
2 changes: 1 addition & 1 deletion docs/MODEL_RENDER_PROFILES.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ still forwards to the passthrough upstream.

| model rule | default | cell | columns | max height | evidence |
|---|:---:|---|---:|---:|---|
| `claude-*` / `anthropic-*` | yes | Spleen 5×8 | 312 | 728 px | established Claude suites |
| `claude-*` / `anthropic-*` | yes | JetBrains Mono 14px, 9×16 | 172 | 728 px | established Claude suites; Fable keeps dense (Spleen 5×8, 312 cols) |
| `gpt-5.6-sol*` | opt-in | JetBrains Mono 14px, 9×16 | 84 | 1954 px | 7/8 exact, 0 inventions, gist and guard pass |
| `grok-*` | opt-in | JetBrains Mono 14px, 9×16 | 84 | 512 px | 100/100 arith, 97/98 gist, 17/18 state; hex 0/15 |
| other GPT/o-series | opt-in | Spleen 5×8 | 152 | 1932 px | conservative fallback |
Expand Down
2 changes: 1 addition & 1 deletion docs/RENDER_SIZING.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ grows with content until the profile cap, then overflow becomes another page.

| model rule | atlas / effective cell | columns | full width | max height |
|---|---|---:|---:|---:|
| Claude / Anthropic | Spleen + Unifont, 5×8 | 312 | 1568 px | 728 px |
| Claude / Anthropic | JetBrains Mono 14px, native 9×16 | 172 | 1556 px | 728 px |
| opt-in `gpt-5.6-sol` | JetBrains Mono 14px, native 9×16 | 84 | 764 px | 1954 px |
| opt-in Grok 4.5 | JetBrains Mono 14px, native 9×16 + IDS + factsheet | 84 | 764 px | 512 px |
| other OpenAI fallback | Spleen + Unifont, 5×8 | 152 | 768 px | 1932 px |
Expand Down
4 changes: 3 additions & 1 deletion eval/gemini-profile/gemini-client.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,9 @@ export async function callGeminiRequest({ model = 'gemini-3.6-flash', request, m
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${json?.error?.message || raw.slice(0, 160)}`);
}
const text = json.candidates?.[0]?.content?.parts?.map((p) => p.text).filter(Boolean).join('') || '';
const candidateParts = json.candidates?.[0]?.content?.parts || [];
const nonThoughtText = candidateParts.filter((p) => !p.thought).map((p) => p.text).filter(Boolean).join('');
const text = nonThoughtText || candidateParts.map((p) => p.text).filter(Boolean).join('') || '';
return { text: text.trim(), usage: json.usageMetadata || null, ms: Date.now() - started };
} finally {
clearTimeout(timer);
Expand Down
Loading
Loading