Skip to content

feat(eval): hit rate, MAP, and per-query-type metrics #195

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrsibe merged 1 commit into from
Sep 30, 2026
Merged

feat(eval): hit rate, MAP, and per-query-type metrics #195

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
104 changes: 104 additions & 0 deletions docs/eval/baseline-v1.6.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,12 +26,87 @@
"recallAt10": 1,
"mrr": 0.927778,
"ndcgAt10": 0.94375,
"hitRateAt5": 1,
"mapAt10": 0.922222,
"evidencePrecisionAt5": 0.213333
},
"byType": [
{
"type": "cross-lingual",
"questions": 1,
"metrics": {
"recallAt1": 0,
"recallAt5": 1,
"recallAt10": 1,
"mrr": 0.5,
"ndcgAt10": 0.63093,
"hitRateAt5": 1,
"mapAt10": 0.5,
"evidencePrecisionAt5": 0.2
}
},
{
"type": "exact",
"questions": 6,
"metrics": {
"recallAt1": 1,
"recallAt5": 1,
"recallAt10": 1,
"mrr": 1,
"ndcgAt10": 1,
"hitRateAt5": 1,
"mapAt10": 1,
"evidencePrecisionAt5": 0.2
}
},
{
"type": "multi-hop",
"questions": 2,
"metrics": {
"recallAt1": 0.5,
"recallAt5": 1,
"recallAt10": 1,
"mrr": 1,
"ndcgAt10": 0.95986,
"hitRateAt5": 1,
"mapAt10": 0.916667,
"evidencePrecisionAt5": 0.4
}
},
{
"type": "semantic",
"questions": 18,
"metrics": {
"recallAt1": 0.833333,
"recallAt5": 1,
"recallAt10": 1,
"mrr": 0.907407,
"ndcgAt10": 0.931214,
"hitRateAt5": 1,
"mapAt10": 0.907407,
"evidencePrecisionAt5": 0.2
}
},
{
"type": "zh",
"questions": 3,
"metrics": {
"recallAt1": 1,
"recallAt5": 1,
"recallAt10": 1,
"mrr": 1,
"ndcgAt10": 1,
"hitRateAt5": 1,
"mapAt10": 1,
"evidencePrecisionAt5": 0.2
}
}
],
"perQuestion": [
{
"id": "q001",
"question": "Why is bedload harder to measure than suspended sediment?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -62,6 +137,7 @@
{
"id": "q002",
"question": "How many replicate samples are collected at each river station?",
"type": "exact",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -92,6 +168,7 @@
{
"id": "q003",
"question": "What is the central trade-off in lithium-ion cell design?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -122,6 +199,7 @@
{
"id": "q004",
"question": "Why do nickel-rich battery packs need more aggressive thermal management?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -152,6 +230,7 @@
{
"id": "q005",
"question": "What happens once the separator in a battery cell melts?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -182,6 +261,7 @@
{
"id": "q006",
"question": "At what temperature do honeybees begin to forage?",
"type": "exact",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -212,6 +292,7 @@
{
"id": "q007",
"question": "What does a late frost damage during full bloom?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -242,6 +323,7 @@
{
"id": "q008",
"question": "Why is a continuous tree canopy more effective at cooling than isolated trees?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -272,6 +354,7 @@
{
"id": "q009",
"question": "Why are trees with aggressive surface roots unsuitable for narrow verges?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -302,6 +385,7 @@
{
"id": "q010",
"question": "At what temperature is lactic acid fermentation fastest?",
"type": "exact",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -332,6 +416,7 @@
{
"id": "q011",
"question": "Is the salt percentage in fermentation based on vegetable weight or water weight?",
"type": "exact",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -362,6 +447,7 @@
{
"id": "q012",
"question": "Why must tidal turbines be sited in places with very fast currents?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -392,6 +478,7 @@
{
"id": "q013",
"question": "What is the main environmental concern for tidal energy installations?",
"type": "semantic",
"firstRelevantRank": 3,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -422,6 +509,7 @@
{
"id": "q014",
"question": "In lake monitoring, how is the sampling depth actually recorded?",
"type": "exact",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -452,6 +540,7 @@
{
"id": "q015",
"question": "Why does deep-water oxygen fall while a lake remains stratified?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -482,6 +571,7 @@
{
"id": "q016",
"question": "How do supercapacitors hold their charge?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -512,6 +602,7 @@
{
"id": "q017",
"question": "Why can solitary bees pollinate a bloom week that is too cold for honeybee hives?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -542,6 +633,7 @@
{
"id": "q018",
"question": "Why is one continuous planted roof layer better than several isolated planted beds?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -572,6 +664,7 @@
{
"id": "q019",
"question": "Where do acetic acid bacteria sit in a vinegar culture?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -602,6 +695,7 @@
{
"id": "q020",
"question": "What happens if a vinegar culture is sealed airtight?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -632,6 +726,7 @@
{
"id": "q021",
"question": "Why is wave energy harder to schedule ahead than tidal energy?",
"type": "semantic",
"firstRelevantRank": 2,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -662,6 +757,7 @@
{
"id": "q022",
"question": "Where does siting for wave energy devices concentrate, and where does it not?",
"type": "exact",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -692,6 +788,7 @@
{
"id": "q023",
"question": "How does the river sampling protocol differ from the lake sampling protocol?",
"type": "multi-hop",
"firstRelevantRank": 1,
"relevantCount": 2,
"retrievedCount": 19,
Expand Down Expand Up @@ -724,6 +821,7 @@
{
"id": "q024",
"question": "A street canopy and a green roof are both said to cool; what surface does each one shade?",
"type": "multi-hop",
"firstRelevantRank": 1,
"relevantCount": 2,
"retrievedCount": 19,
Expand Down Expand Up @@ -756,6 +854,7 @@
{
"id": "q025",
"question": "Which preservation method depends on keeping air away from the food?",
"type": "semantic",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -786,6 +885,7 @@
{
"id": "q026",
"question": "Why can one cold morning cost a grower the whole crop even when colonies are brought in?",
"type": "semantic",
"firstRelevantRank": 2,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -816,6 +916,7 @@
{
"id": "q027",
"question": "绿茶应该怎样保存才能减缓氧化？",
"type": "zh",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -846,6 +947,7 @@
{
"id": "q028",
"question": "茶叶储存的相对湿度上限是多少？",
"type": "zh",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -876,6 +978,7 @@
{
"id": "q029",
"question": "为什么冷冻保存的茶叶取出后不能立刻打开包装？",
"type": "zh",
"firstRelevantRank": 1,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down Expand Up @@ -906,6 +1009,7 @@
{
"id": "q030",
"question": "为什么潮汐能比风能和太阳能更容易提前安排发电？",
"type": "cross-lingual",
"firstRelevantRank": 2,
"relevantCount": 1,
"retrievedCount": 19,
Expand Down
20 changes: 18 additions & 2 deletions docs/eval/baseline-v1.6.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,10 +24,26 @@ Generated by `npm run eval`. The numbers below are harness output — do not edi
| Recall@10 | 1.0000 |
| MRR | 0.9278 |
| nDCG@10 | 0.9437 |
| Hit rate@5 | 1.0000 |
| MAP@10 | 0.9222 |
| Evidence precision@5 | 0.2133 |

Timing is informational only and is **not** frozen: indexing 1582 ms, query
p50 11.35 ms, p95 18.82 ms on the
### By query type

A single average hides a change that helps one kind of question and hurts another.
The type comes from `type` in `questions.jsonl`; untagged questions report as
`untagged` rather than disappearing.

| Type | Questions | Recall@5 | nDCG@10 | Hit rate@5 | MAP@10 |
| --- | --- | --- | --- | --- | --- |
| cross-lingual | 1 | 1.0000 | 0.6309 | 1.0000 | 0.5000 |
| exact | 6 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| multi-hop | 2 | 1.0000 | 0.9599 | 1.0000 | 0.9167 |
| semantic | 18 | 1.0000 | 0.9312 | 1.0000 | 0.9074 |
| zh | 3 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |

Timing is informational only and is **not** frozen: indexing 1545 ms, query
p50 11.04 ms, p95 14.14 ms on the
machine that produced this file. Timing and index size depend on hardware and on the
corpus, so they must never be the reason two runs differ.

Expand Down
14 changes: 14 additions & 0 deletions eval/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@ dataset needs a new question.
{
"id": "q001",
"question": "Why is bedload harder to measure than suspended sediment?",
"type": "semantic",
"relevant": [
{
"document": "river-monitoring.md",
Expand All @@ -78,6 +79,10 @@ Ground truth uses **corpus identity, never database identity**:
- `page` is `null` for unpaginated sources.
- `quote` is an optional excerpt. The runner fails if the referenced block no longer
contains it, so a parser change cannot silently move the ground truth.
- `type` is an optional query class; the report groups every metric by it. Values in use:
`exact` (number/name/detail), `semantic` (why/how), `multi-hop` (two or more blocks),
`cross-lingual` (question language differs from the source), `zh` (Chinese over a
Chinese source). Untagged questions report as `untagged`.

Runtime `documentId`s are random and `blockId`s embed them, so neither may appear here.
This is what lets #78 change chunking without invalidating the dataset: the ground truth
Expand Down Expand Up @@ -115,11 +120,20 @@ from unanswerable queries.
- **Recall@1/5/10** — share of ground-truth blocks covered by the first k passages.
- **MRR** — reciprocal rank of the first relevant passage.
- **nDCG@10** — binary-gain discounted cumulative gain.
- **Hit rate@5** — share of questions with at least one relevant passage in the first 5.
Deliberately blunt: it says the answer was *reachable*, where Recall@5 says the material
was *complete*. A two-passage question that finds one scores 1.0 and 0.5 respectively.
- **MAP@10** — mean average precision. The one metric here that combines ranking position
with coverage, so pulling a second relevant passage from rank 9 to rank 2 moves it.
- **Evidence precision@5** — of the first 5 retrieved passages, the share that cover a
ground-truth block. This is **retrieval precision, not answer citation recall**: the
harness runs no model and produces no answer. Answer-level citation correctness is
covered by the resolver (#70); a model-driven answer eval would be a separate
deliverable.
- **By query type** — the same metrics per `type` in `questions.jsonl` (`exact`,
`semantic`, `multi-hop`, `cross-lingual`, `zh`). A single average hides a change that
helps one kind of question and hurts another; the current baseline already shows this,
with `cross-lingual` at nDCG 0.63 against 0.93–1.00 elsewhere.
- **Latency p50/p95** — informational only. Timing is **not** frozen, and the committed
JSON excludes it so two runs diff cleanly.

Expand Down
Loading
Loading