Skip to content

fix(eval): refuse a context window wider than the retrieval depth #201

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrsibe merged 1 commit into from
Sep 30, 2026
Merged

fix(eval): refuse a context window wider than the retrieval depth #201

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
72 changes: 22 additions & 50 deletions docs/eval/sweep-v1.6.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@
"noResultRate": 0,
"meanContextChars": 2078.9333333333334,
"chunkCount": 19,
"latencyP95Ms": 88.15
"latencyP95Ms": 13.28
},
{
"strategy": "dense",
Expand All @@ -27,21 +27,7 @@
"noResultRate": 0,
"meanContextChars": 3326.8333333333335,
"chunkCount": 19,
"latencyP95Ms": 89.14
},
{
"strategy": "dense",
"candidateK": 5,
"contextK": 8,
"recallAt5": 1,
"ndcgAt10": 0.94375,
"mapAt10": 0.922222,
"contextPrecision": 0.213333,
"contextRecall": 1,
"noResultRate": 0,
"meanContextChars": 3326.8333333333335,
"chunkCount": 19,
"latencyP95Ms": 98.4
"latencyP95Ms": 13.14
},
{
"strategy": "dense",
Expand All @@ -55,7 +41,7 @@
"noResultRate": 0,
"meanContextChars": 2078.9333333333334,
"chunkCount": 19,
"latencyP95Ms": 107.2
"latencyP95Ms": 13.97
},
{
"strategy": "dense",
Expand All @@ -69,7 +55,7 @@
"noResultRate": 0,
"meanContextChars": 3326.8333333333335,
"chunkCount": 19,
"latencyP95Ms": 77.76
"latencyP95Ms": 14.81
},
{
"strategy": "dense",
Expand All @@ -83,7 +69,7 @@
"noResultRate": 0,
"meanContextChars": 5311.633333333333,
"chunkCount": 19,
"latencyP95Ms": 91.92
"latencyP95Ms": 15.14
},
{
"strategy": "dense",
Expand All @@ -97,7 +83,7 @@
"noResultRate": 0,
"meanContextChars": 2078.9333333333334,
"chunkCount": 19,
"latencyP95Ms": 94.88
"latencyP95Ms": 16.19
},
{
"strategy": "dense",
Expand All @@ -111,7 +97,7 @@
"noResultRate": 0,
"meanContextChars": 3326.8333333333335,
"chunkCount": 19,
"latencyP95Ms": 85.49
"latencyP95Ms": 18.66
},
{
"strategy": "dense",
Expand All @@ -125,7 +111,7 @@
"noResultRate": 0,
"meanContextChars": 5311.633333333333,
"chunkCount": 19,
"latencyP95Ms": 84.23
"latencyP95Ms": 14.61
},
{
"strategy": "dense",
Expand All @@ -139,7 +125,7 @@
"noResultRate": 0,
"meanContextChars": 2078.9333333333334,
"chunkCount": 19,
"latencyP95Ms": 91.85
"latencyP95Ms": 17.65
},
{
"strategy": "dense",
Expand All @@ -153,7 +139,7 @@
"noResultRate": 0,
"meanContextChars": 3326.8333333333335,
"chunkCount": 19,
"latencyP95Ms": 86.76
"latencyP95Ms": 19.74
},
{
"strategy": "dense",
Expand All @@ -167,7 +153,7 @@
"noResultRate": 0,
"meanContextChars": 5311.633333333333,
"chunkCount": 19,
"latencyP95Ms": 89.21
"latencyP95Ms": 17.64
},
{
"strategy": "hybrid",
Expand All @@ -181,7 +167,7 @@
"noResultRate": 0,
"meanContextChars": 2044.1333333333334,
"chunkCount": 19,
"latencyP95Ms": 81.29
"latencyP95Ms": 13.37
},
{
"strategy": "hybrid",
Expand All @@ -195,21 +181,7 @@
"noResultRate": 0,
"meanContextChars": 3361.6,
"chunkCount": 19,
"latencyP95Ms": 79.52
},
{
"strategy": "hybrid",
"candidateK": 5,
"contextK": 8,
"recallAt5": 1,
"ndcgAt10": 0.956053,
"mapAt10": 0.938889,
"contextPrecision": 0.213333,
"contextRecall": 1,
"noResultRate": 0,
"meanContextChars": 3361.6,
"chunkCount": 19,
"latencyP95Ms": 85.98
"latencyP95Ms": 14.7
},
{
"strategy": "hybrid",
Expand All @@ -223,7 +195,7 @@
"noResultRate": 0,
"meanContextChars": 2062.733333333333,
"chunkCount": 19,
"latencyP95Ms": 86.15
"latencyP95Ms": 22.31
},
{
"strategy": "hybrid",
Expand All @@ -237,7 +209,7 @@
"noResultRate": 0,
"meanContextChars": 3525.633333333333,
"chunkCount": 19,
"latencyP95Ms": 92.74
"latencyP95Ms": 18.66
},
{
"strategy": "hybrid",
Expand All @@ -251,7 +223,7 @@
"noResultRate": 0,
"meanContextChars": 5509.8,
"chunkCount": 19,
"latencyP95Ms": 89.97
"latencyP95Ms": 15.84
},
{
"strategy": "hybrid",
Expand All @@ -265,7 +237,7 @@
"noResultRate": 0,
"meanContextChars": 2056.9333333333334,
"chunkCount": 19,
"latencyP95Ms": 59.56
"latencyP95Ms": 20.78
},
{
"strategy": "hybrid",
Expand All @@ -279,7 +251,7 @@
"noResultRate": 0,
"meanContextChars": 3445.866666666667,
"chunkCount": 19,
"latencyP95Ms": 77.17
"latencyP95Ms": 23.35
},
{
"strategy": "hybrid",
Expand All @@ -293,7 +265,7 @@
"noResultRate": 0,
"meanContextChars": 5572.6,
"chunkCount": 19,
"latencyP95Ms": 91.41
"latencyP95Ms": 22.32
},
{
"strategy": "hybrid",
Expand All @@ -307,7 +279,7 @@
"noResultRate": 0,
"meanContextChars": 2056.9333333333334,
"chunkCount": 19,
"latencyP95Ms": 89.62
"latencyP95Ms": 23.47
},
{
"strategy": "hybrid",
Expand All @@ -321,7 +293,7 @@
"noResultRate": 0,
"meanContextChars": 3445.866666666667,
"chunkCount": 19,
"latencyP95Ms": 66.45
"latencyP95Ms": 21.71
},
{
"strategy": "hybrid",
Expand All @@ -335,7 +307,7 @@
"noResultRate": 0,
"meanContextChars": 5572.6,
"chunkCount": 19,
"latencyP95Ms": 76.65
"latencyP95Ms": 21.68
}
]
}
63 changes: 38 additions & 25 deletions docs/eval/sweep-v1.6.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,35 +5,48 @@ Generated by `node scripts/eval-sweep.mjs`. Numbers are harness output; do not e
## What was measured

The real harness, the same corpus, chunking held fixed, over
dense / hybrid × candidateK {5, 10, 20, 40} × contextK {3, 5, 8} — 24 runs.
dense / hybrid × candidateK {5, 10, 20, 40} × contextK {3, 5, 8} — 22 runs.
Each row differs from its neighbour in one parameter.

2 further cell(s) were **skipped** because `contextK > candidateK`; see below.

| Strategy | candidateK | contextK | Recall@5 | nDCG@10 | MAP@10 | Context P | Context R | No-result | Context chars | Index | p95 |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| dense | 5 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 88.15 ms |
| dense | 5 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 89.14 ms |
| dense | 5 | 8 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 98.40 ms |
| dense | 10 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 107.20 ms |
| dense | 10 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 77.76 ms |
| dense | 10 | 8 | 1.0000 | 0.9437 | 0.9222 | 0.1333 | 1.0000 | 0.0000 | 5312 | 19 | 91.92 ms |
| dense | 20 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 94.88 ms |
| dense | 20 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 85.49 ms |
| dense | 20 | 8 | 1.0000 | 0.9437 | 0.9222 | 0.1333 | 1.0000 | 0.0000 | 5312 | 19 | 84.23 ms |
| dense | 40 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 91.85 ms |
| dense | 40 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 86.76 ms |
| dense | 40 | 8 | 1.0000 | 0.9437 | 0.9222 | 0.1333 | 1.0000 | 0.0000 | 5312 | 19 | 89.21 ms |
| hybrid | 5 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2044 | 19 | 81.29 ms |
| hybrid | 5 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3362 | 19 | 79.52 ms |
| hybrid | 5 | 8 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3362 | 19 | 85.98 ms |
| hybrid | 10 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2063 | 19 | 86.15 ms |
| hybrid | 10 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3526 | 19 | 92.74 ms |
| hybrid | 10 | 8 | 1.0000 | 0.9561 | 0.9389 | 0.1333 | 1.0000 | 0.0000 | 5510 | 19 | 89.97 ms |
| hybrid | 20 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2057 | 19 | 59.56 ms |
| hybrid | 20 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3446 | 19 | 77.17 ms |
| hybrid | 20 | 8 | 1.0000 | 0.9561 | 0.9389 | 0.1333 | 1.0000 | 0.0000 | 5573 | 19 | 91.41 ms |
| hybrid | 40 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2057 | 19 | 89.62 ms |
| hybrid | 40 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3446 | 19 | 66.45 ms |
| hybrid | 40 | 8 | 1.0000 | 0.9561 | 0.9389 | 0.1333 | 1.0000 | 0.0000 | 5573 | 19 | 76.65 ms |
| dense | 5 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 13.28 ms |
| dense | 5 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 13.14 ms |
| dense | 10 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 13.97 ms |
| dense | 10 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 14.81 ms |
| dense | 10 | 8 | 1.0000 | 0.9437 | 0.9222 | 0.1333 | 1.0000 | 0.0000 | 5312 | 19 | 15.14 ms |
| dense | 20 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 16.19 ms |
| dense | 20 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 18.66 ms |
| dense | 20 | 8 | 1.0000 | 0.9437 | 0.9222 | 0.1333 | 1.0000 | 0.0000 | 5312 | 19 | 14.61 ms |
| dense | 40 | 3 | 1.0000 | 0.9437 | 0.9222 | 0.3556 | 1.0000 | 0.0000 | 2079 | 19 | 17.65 ms |
| dense | 40 | 5 | 1.0000 | 0.9437 | 0.9222 | 0.2133 | 1.0000 | 0.0000 | 3327 | 19 | 19.74 ms |
| dense | 40 | 8 | 1.0000 | 0.9437 | 0.9222 | 0.1333 | 1.0000 | 0.0000 | 5312 | 19 | 17.64 ms |
| hybrid | 5 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2044 | 19 | 13.37 ms |
| hybrid | 5 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3362 | 19 | 14.70 ms |
| hybrid | 10 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2063 | 19 | 22.31 ms |
| hybrid | 10 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3526 | 19 | 18.66 ms |
| hybrid | 10 | 8 | 1.0000 | 0.9561 | 0.9389 | 0.1333 | 1.0000 | 0.0000 | 5510 | 19 | 15.84 ms |
| hybrid | 20 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2057 | 19 | 20.78 ms |
| hybrid | 20 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3446 | 19 | 23.35 ms |
| hybrid | 20 | 8 | 1.0000 | 0.9561 | 0.9389 | 0.1333 | 1.0000 | 0.0000 | 5573 | 19 | 22.32 ms |
| hybrid | 40 | 3 | 1.0000 | 0.9561 | 0.9389 | 0.3556 | 1.0000 | 0.0000 | 2057 | 19 | 23.47 ms |
| hybrid | 40 | 5 | 1.0000 | 0.9561 | 0.9389 | 0.2133 | 1.0000 | 0.0000 | 3446 | 19 | 21.71 ms |
| hybrid | 40 | 8 | 1.0000 | 0.9561 | 0.9389 | 0.1333 | 1.0000 | 0.0000 | 5573 | 19 | 21.68 ms |


## Skipped cells

`contextK > candidateK` cannot be filled: the harness fetches `candidateK`
passages, so a wider window would contain fewer passages than it claims. These
2 cell(s) are excluded rather than reported as equal to a narrower one:

- `dense` candidateK=5, contextK=8
- `hybrid` candidateK=5, contextK=8

The harness refuses the same combination at the flag level, so a typo fails loudly.


## How to read it

Expand Down
7 changes: 7 additions & 0 deletions eval/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,13 @@ writes one dashboard with quality, context precision/recall, prompt size, index
and latency side by side. Its grid maximum is labelled as **not** a recommendation:
selecting on the same questions is how a benchmark becomes a lookup table.

`contextK > candidateK` is not a cell in that grid. The harness fetches `candidateK`
passages, so a wider window can never be filled; the sweep skips those combinations
and names them in the report, and the harness refuses the same combination from the
command line. Before this was enforced, `contextK=8` at `candidateK=5` was reported as
identical to `contextK=5` — not because 8 assessed the same as 5, but because
passages 6–8 did not exist.

`eval:prepare` downloads the pinned `multilingual-e5-small` revision into the app's model
cache and verifies it. `eval` never touches the network: if the model is missing it stops
with
Expand Down
Loading
Loading