Skip to content

feat(eval): nine cross-lingual questions, and the corpus stops being saturated #203

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mrsibe merged 1 commit into from
Sep 30, 2026
Merged

feat(eval): nine cross-lingual questions, and the corpus stops being saturated #203

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
298 changes: 281 additions & 17 deletions docs/eval/baseline-v1.6.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,34 +17,34 @@
"threshold": 0.5,
"corpus": "eval/corpus",
"documents": 13,
"questions": 36,
"questions": 44,
"chunkCount": 19
},
"metrics": {
"recallAt1": 0.833333,
"recallAt5": 1,
"recallAt1": 0.657895,
"recallAt5": 0.921053,
"recallAt10": 1,
"mrr": 0.927778,
"ndcgAt10": 0.94375,
"hitRateAt5": 1,
"mapAt10": 0.922222,
"contextPrecision": 0.355556,
"contextRecall": 1
"mrr": 0.800909,
"ndcgAt10": 0.84764,
"hitRateAt5": 0.921053,
"mapAt10": 0.796523,
"contextPrecision": 0.324561,
"contextRecall": 0.921053
},
"byType": [
{
"type": "cross-lingual",
"questions": 1,
"questions": 9,
"metrics": {
"recallAt1": 0,
"recallAt5": 1,
"recallAt5": 0.666667,
"recallAt10": 1,
"mrr": 0.5,
"ndcgAt10": 0.63093,
"hitRateAt5": 1,
"mapAt10": 0.5,
"contextPrecision": 0.333333,
"contextRecall": 1
"mrr": 0.344577,
"ndcgAt10": 0.503192,
"hitRateAt5": 0.666667,
"mapAt10": 0.344577,
"contextPrecision": 0.222222,
"contextRecall": 0.666667
}
},
{
Expand Down Expand Up @@ -1294,6 +1294,270 @@
[],
[]
]
},
{
"id": "q037",
"question": "推移质为什么比悬移质更难测？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 8,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 2084,
"matchesByRank": [
[],
[],
[],
[],
[],
[],
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
},
{
"id": "q038",
"question": "河流监测中，每个站点要采几份平行样？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 7,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 1480,
"matchesByRank": [
[],
[],
[],
[],
[],
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
},
{
"id": "q039",
"question": "设计锂离子电芯时最核心的取舍是什么？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 2,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 1994,
"matchesByRank": [
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
},
{
"id": "q040",
"question": "富镍电池为什么对散热要求更高？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 2,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 1994,
"matchesByRank": [
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
},
{
"id": "q041",
"question": "电芯隔膜一旦熔化会导致什么后果？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 3,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 1449,
"matchesByRank": [
[],
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
},
{
"id": "q042",
"question": "蜜蜂大致从什么温度开始出巢觅食？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 3,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 1321,
"matchesByRank": [
[],
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
},
{
"id": "q043",
"question": "花期遭遇晚霜，主要受损的是什么？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 2,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 956,
"matchesByRank": [
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
},
{
"id": "q044",
"question": "为什么成片的树冠比孤立的树降温效果更好？",
"type": "cross-lingual",
"answerable": true,
"firstRelevantRank": 6,
"relevantCount": 1,
"retrievedCount": 19,
"contextChars": 1447,
"matchesByRank": [
[],
[],
[],
[],
[],
[
0
],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[],
[]
]
}
]
}
24 changes: 12 additions & 12 deletions docs/eval/baseline-v1.6.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,22 +12,22 @@ Generated by `npm run eval`. The numbers below are harness output — do not edi
| Ranks | `candidateK=20, threshold=0.5` |
| Context width | `contextK=3` |
| Corpus | `eval/corpus` (13 documents) |
| Split | `all` (36 questions, 30 answerable) |
| Split | `all` (44 questions, 38 answerable) |
| Index size | 19 chunks |

## Metrics

| Metric | Value |
| --- | --- |
| Recall@1 | 0.8333 |
| Recall@5 | 1.0000 |
| Recall@1 | 0.6579 |
| Recall@5 | 0.9211 |
| Recall@10 | 1.0000 |
| MRR | 0.9278 |
| nDCG@10 | 0.9437 |
| Hit rate@5 | 1.0000 |
| MAP@10 | 0.9222 |
| Context precision@3 | 0.3556 |
| Context recall@3 | 1.0000 |
| MRR | 0.8009 |
| nDCG@10 | 0.8476 |
| Hit rate@5 | 0.9211 |
| MAP@10 | 0.7965 |
| Context precision@3 | 0.3246 |
| Context recall@3 | 0.9211 |

### By query type

Expand All @@ -37,7 +37,7 @@ The type comes from `type` in `questions.jsonl`; untagged questions report as

| Type | Questions | Recall@5 | nDCG@10 | Hit rate@5 | MAP@10 |
| --- | --- | --- | --- | --- | --- |
| cross-lingual | 1 | 1.0000 | 0.6309 | 1.0000 | 0.5000 |
| cross-lingual | 9 | 0.6667 | 0.5032 | 0.6667 | 0.3446 |
| exact | 6 | 1.0000 | 1.0000 | 1.0000 | 1.0000 |
| multi-hop | 2 | 1.0000 | 0.9599 | 1.0000 | 0.9167 |
| semantic | 18 | 1.0000 | 0.9312 | 1.0000 | 0.9074 |
Expand All @@ -57,8 +57,8 @@ was pulled in anyway. This is the row a threshold decision should move.
| Returned no results | 0.0000 (0/6) |
| Mean passages retrieved | 19.00 |

Timing is informational only and is **not** frozen: indexing 1496 ms, query
p50 11.42 ms, p95 17.31 ms on the
Timing is informational only and is **not** frozen: indexing 1552 ms, query
p50 11.82 ms, p95 14.45 ms on the
machine that produced this file. Timing and index size depend on hardware and on the
corpus, so they must never be the reason two runs differ.

Expand Down
Loading
Loading