diff --git a/README.ko.md b/README.ko.md index 7aa6517..384c085 100644 --- a/README.ko.md +++ b/README.ko.md @@ -302,8 +302,11 @@ batch_results = estimator.score_batch( ) ``` -이 모듈은 scorer를 학습하지 않고, reranking Strategy를 추가하지 않으며, async -inference를 수행하지 않습니다. 병렬 batch scoring은 같은 encoder/scorer instance를 +이 모듈은 scorer를 학습하지 않고 async inference를 수행하지 않습니다. estimator는 +스코어링 utility이며, 이를 소비하는 실험적 reranker가 +`AnswerConfidenceRerankStrategy`(`ranksmith.strategies`)입니다 — 아래 +"Answer Confidence Reranking (experimental)" 참고. 병렬 batch scoring은 같은 +encoder/scorer instance를 worker thread들이 공유하므로 thread-safe backend에서만 `max_workers>1`을 사용해야 합니다. 첫 worker error에서 pending work를 취소하지만, 이미 시작된 Python thread는 background에서 완료될 수 있습니다. @@ -341,6 +344,36 @@ result = train_confidence_scorer( print(result.export_path) ``` +### Answer Confidence Reranking (experimental) + +`AnswerConfidenceRerankStrategy`는 학습된 `answer_confidence` estimator를 +CBDR 방식의 reranker로 만듭니다: 문서마다 모델이 그 문맥으로 질문에 답하고(문서당 +LLM 1회), 그 답변이 맞을 로컬 structural confidence로 문서를 정렬합니다. 한 질문 +안에서 이 정렬은 confidence 변화 기준 정렬과 동일합니다. + +```python +from ranksmith import AnswerConfidenceRerankStrategy, AzureOpenAIReranker +from ranksmith.confidence import StructuralConfidenceEstimator + +estimator = StructuralConfidenceEstimator.from_artifact( + "artifacts/answer_confidence.joblib" +) +reranker = AzureOpenAIReranker( + api_key="...", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="gpt-4o-mini", + strategy=AnswerConfidenceRerankStrategy(estimator=estimator), +) +``` + +> **실험적 — 기본 선택 아님.** 학습된 `answer_confidence` artifact(정답이 있는 QA +> 데이터)가 필요하고 문서당 LLM answer 1회 비용이 듭니다. 스펙에 기록된 소규모 +> 자체 평가(held-out SQuAD 15개, 리포 밖 환경에서 실행 — 증거 artifact는 커밋되어 +> 있지 않음)에서는 순수 `ListwiseStrategy`에 4배 비용으로 졌습니다. 아직 기존 +> 전략을 이기는 세팅을 입증하지 못했고, 가능성 있는 영역(listwise window 초과 +> 후보)은 미측정입니다. 수치와 한계는 `docs/specs/spec_confidence_aware_reranking.md`, +> 표준 벤치마크 절차는 `docs/benchmarks/answer_confidence_askubuntu.md` 참고. + ## 실전 가이드 (Examples) 실행 가능한 예제는 `examples/` 폴더에 있습니다. @@ -358,8 +391,9 @@ print(result.export_path) 이 레포에는 [Claude Code](https://code.claude.com/docs) 플러그인 `ranksmith-advisor`가 포함되어 있습니다. 사용 사례에 맞는 reranking strategy 선택을 돕고 CI로 검증된 동작 스니펫을 제시하며, 제안 코드가 라이브러리의 실제 -계약을 따르도록 ranksmith 전용 guardrail을 적용합니다(미구현 provider 호출, -`algorithm` 문자열 확장, `confidence`를 reranker로 취급하는 패턴을 차단). +계약을 따르도록 ranksmith 전용 guardrail을 적용합니다(Azure만 번들 provider이고, +`confidence` estimator는 스코어링 utility, `AnswerConfidenceRerankStrategy`는 +그 위에 얹은 실험적 reranker). Claude Code에서 사용: @@ -390,7 +424,7 @@ top-5만 출력할 수 있습니다. Live LLM 호출에는 Azure OpenAI deployme 중간 단계에서 실패할 수 있으므로 정확한 provider-call telemetry는 아닙니다. 커밋된 근거 artifact는 다음과 같습니다. -- [`benchmark-results/live/askubuntu-bm25-top20-default-live.v3.merged.json`](https://github.com/pko89403/ranksmith/blob/main/benchmark-results/live/askubuntu-bm25-top20-default-live.v3.merged.json) +- [`benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json`](https://github.com/pko89403/ranksmith/blob/main/benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json) - [`benchmark-results/pyserini/askubuntu-bm25-top20.trec`](https://github.com/pko89403/ranksmith/blob/main/benchmark-results/pyserini/askubuntu-bm25-top20.trec) | Method | NDCG@5 | MRR@5 | Recall@5 | Valid rows | Invalid rate | Nominal LLM calls/query | LLM row attempts/query incl. retries | @@ -402,13 +436,25 @@ top-5만 출력할 수 있습니다. Live LLM 호출에는 Azure OpenAI deployme | `tourrank_r2` | 0.4236 | 0.5725 | 0.3601 | 361/361 | 0.000 | 8 | 1.03 | | `setwise_hs_s10` | 0.3653 | 0.5059 | 0.3005 | 361/361 | 0.000 | 12 | 1.00 | | `prp_sliding_p1` | 0.4065 | 0.5818 | 0.3277 | 361/361 | 0.000 | 38 | 1.00 | +| `answer_confidence` *(scorer: SQuAD v1.1, 도메인 밖)* | 0.1722 | 0.2862 | 0.1435 | 361/361 | 0.000 | 20 | 1.00 | `tourrank_r2`는 NDCG@5와 Recall@5가 가장 높았고, `prp_sliding_p1`은 MRR@5가 가장 높았습니다. `single_call_listwise@20`은 one-shot listwise baseline입니다. `rankgpt_sw_w5`는 이 top-20 설정의 실제 sliding-window listwise baseline입니다. `acurank_k5_b1`은 AcuRank uncertainty boundary를 `@5` 평가 cutoff와 맞춘 설정입니다. `setwise_hs_s10`은 20개 후보에서 평가 대상 top-5만 추출하는 실용적인 -Setwise Heapsort 설정입니다. +Setwise Heapsort 설정입니다. `answer_confidence`는 SQuAD v1.1로 학습한 +scorer를 사용하며(AskUbuntu 기준 도메인 밖, [러닝북](docs/benchmarks/answer_confidence_askubuntu.md) +참고) 이 벤치마크에서는 BM25 baseline보다 낮은 점수를 기록했습니다. 이기도록 +튜닝하지 않고 측정된 그대로 보고합니다. + +왜 이기게 튜닝하지 않는가: 스코어러를 이 벤치마크 분포에 맞추면 알고리즘의 +일반적인 품질이 아니라 AskUbuntu에 대한 과적합을 측정하게 됩니다. +`scripts/train_answer_confidence.py`가 test roc_auc < 0.6이면 cherry-pick된 +체크포인트를 통과시키지 않고 fast-fail하는 것도 같은 이유입니다. 더 강한 +domain-in 스코어러를 만드는 건 정당한 후속 작업이지만([스펙](docs/specs/spec_confidence_aware_reranking.md)의 +"남은 작업" 참고), 그건 더 나은 artifact를 학습해서 같은 커맨드를 다시 +돌리는 것이지 보고 방식을 바꾸는 게 아닙니다. 재시도 후에도 `single_call_listwise@20` 2개 row와 `acurank_k5_b1` 5개 row가 invalid로 남았습니다. 이 row는 보정하지 않고 invalid rate에 반영했습니다. diff --git a/README.md b/README.md index 6abbf86..2a4ed6b 100644 --- a/README.md +++ b/README.md @@ -306,8 +306,11 @@ batch_results = estimator.score_batch( ) ``` -This module does not train a scorer, does not add a reranking Strategy, and -does not perform async inference. Parallel batch scoring shares the same +This module does not train a scorer and does not perform async inference. The +estimator is a scoring utility; `AnswerConfidenceRerankStrategy` +(`ranksmith.strategies`) is the experimental reranker that consumes it — see +[Answer Confidence Reranking](#answer-confidence-reranking-experimental). +Parallel batch scoring shares the same encoder and scorer instances across worker threads, so use `max_workers>1` only with thread-safe backends. It cancels pending work on the first worker error, but Python threads that have already started may finish in the background. @@ -345,6 +348,41 @@ result = train_confidence_scorer( print(result.export_path) ``` +### Answer Confidence Reranking (experimental) + +`AnswerConfidenceRerankStrategy` turns a trained `answer_confidence` estimator +into a reranker in the CBDR spirit: for each candidate the model answers the +query from that document (one LLM call per document), and the document is scored +by the local structural confidence that the answer is correct. Documents are +ordered by that confidence, descending — which within a single query equals +ranking by confidence change. + +```python +from ranksmith import AnswerConfidenceRerankStrategy, AzureOpenAIReranker +from ranksmith.confidence import StructuralConfidenceEstimator + +estimator = StructuralConfidenceEstimator.from_artifact( + "artifacts/answer_confidence.joblib" +) +reranker = AzureOpenAIReranker( + api_key="...", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="gpt-4o-mini", + strategy=AnswerConfidenceRerankStrategy(estimator=estimator), +) +``` + +> **Experimental — not a default choice.** It needs a trained +> `answer_confidence` artifact (QA data with gold answers) and costs one LLM +> answer call per document. In the spec's small self-reported eval (15 +> held-out SQuAD queries, run outside this repo — no evidence artifact is +> committed) it lost to a plain `ListwiseStrategy` at four times the LLM +> cost. No setting has yet shown it beating an existing strategy; its +> plausible niche (candidate sets larger than the listwise window) is +> unmeasured. Numbers and caveats live in +> `docs/specs/spec_confidence_aware_reranking.md`; the standard-benchmark +> procedure is `docs/benchmarks/answer_confidence_askubuntu.md`. + ## Examples Runnable examples live in the `examples/` directory. @@ -363,7 +401,8 @@ This repo ships a [Claude Code](https://code.claude.com/docs) plugin, `ranksmith-advisor`, that helps you choose a reranking strategy for your use case and returns working, CI-verified snippets. It encodes ranksmith-specific guardrails, so the suggested code follows the library's real contracts (Azure -is the only bundled provider, and `confidence` is not a reranker). +is the only bundled provider; the `confidence` estimator is a scoring utility, +and `AnswerConfidenceRerankStrategy` is an experimental reranker built on it). Use it from Claude Code: @@ -394,7 +433,7 @@ attempts. Row attempts are useful for retry accounting, but they are not exact provider-call telemetry for multi-call methods that can fail partway through an algorithm run. The committed evidence artifacts are: -- [`benchmark-results/live/askubuntu-bm25-top20-default-live.v3.merged.json`](https://github.com/pko89403/ranksmith/blob/main/benchmark-results/live/askubuntu-bm25-top20-default-live.v3.merged.json) +- [`benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json`](https://github.com/pko89403/ranksmith/blob/main/benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json) - [`benchmark-results/pyserini/askubuntu-bm25-top20.trec`](https://github.com/pko89403/ranksmith/blob/main/benchmark-results/pyserini/askubuntu-bm25-top20.trec) | Method | NDCG@5 | MRR@5 | Recall@5 | Valid rows | Invalid rate | Nominal LLM calls/query | LLM row attempts/query incl. retries | @@ -406,13 +445,26 @@ algorithm run. The committed evidence artifacts are: | `tourrank_r2` | 0.4236 | 0.5725 | 0.3601 | 361/361 | 0.000 | 8 | 1.03 | | `setwise_hs_s10` | 0.3653 | 0.5059 | 0.3005 | 361/361 | 0.000 | 12 | 1.00 | | `prp_sliding_p1` | 0.4065 | 0.5818 | 0.3277 | 361/361 | 0.000 | 38 | 1.00 | +| `answer_confidence` *(scorer: SQuAD v1.1, out-of-domain)* | 0.1722 | 0.2862 | 0.1435 | 361/361 | 0.000 | 20 | 1.00 | `tourrank_r2` had the best NDCG@5 and Recall@5, while `prp_sliding_p1` had the best MRR@5. `single_call_listwise@20` is the one-shot listwise baseline. `rankgpt_sw_w5` is the true sliding-window listwise baseline for this top-20 setup. `acurank_k5_b1` aligns AcuRank's uncertainty boundary with the `@5` evaluation cutoff. `setwise_hs_s10` is a practical Setwise Heapsort setting -that extracts only the evaluated top-5 from 20 candidates. +that extracts only the evaluated top-5 from 20 candidates. `answer_confidence` +uses a scorer trained on SQuAD v1.1 (out-of-domain for AskUbuntu, see +[the runbook](docs/benchmarks/answer_confidence_askubuntu.md)) and scores +below the BM25 baseline here; it is reported as measured, not tuned to win. + +Why not tune it to win: fitting the scorer to this benchmark's distribution +would measure overfitting to AskUbuntu, not the algorithm's general quality — +the same reason `scripts/train_answer_confidence.py` fast-fails below +`roc_auc 0.6` instead of letting a cherry-picked checkpoint through. A +stronger in-domain scorer is a legitimate follow-up (see "남은 작업" in +[the spec](docs/specs/spec_confidence_aware_reranking.md)), but that means +training a better artifact and re-running this exact command, not adjusting +the report. After retries, 2 `single_call_listwise@20` rows and 5 `acurank_k5_b1` rows remained invalid. They are included in the invalid-rate accounting instead of diff --git a/benchmark-results/askubuntu-bm25-top20-answer-confidence.checkpoint.jsonl b/benchmark-results/askubuntu-bm25-top20-answer-confidence.checkpoint.jsonl new file mode 100644 index 0000000..6ac5d77 --- /dev/null +++ b/benchmark-results/askubuntu-bm25-top20-answer-confidence.checkpoint.jsonl @@ -0,0 +1,361 @@ +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query0", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.8687949224876582, "recall@5": 0.3333333333333333}, "query_id": "test_query0", "ranked_ids": ["apositive_test_query0_00010", "apositive_test_query0_00000", "apositive_test_query0_00006", "apositive_test_query0_00003", "negative_test_query359_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query1", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query1", "ranked_ids": ["negative_test_query245_00006", "apositive_test_query348_00003", "negative_test_query120_00006", "negative_test_query61_00002", "negative_test_query27_00017"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query10", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.4776237035032179, "recall@5": 0.6666666666666666}, "query_id": "test_query10", "ranked_ids": ["negative_test_query10_00010", "apositive_test_query10_00001", "negative_test_query104_00011", "negative_test_query10_00000", "apositive_test_query10_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query100", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query100", "ranked_ids": ["negative_test_query100_00010", "negative_test_query290_00015", "negative_test_query100_00009", "apositive_test_query181_00000", "negative_test_query100_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query101", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query101", "ranked_ids": ["apositive_test_query196_00000", "apositive_test_query336_00001", "negative_test_query61_00002", "negative_test_query173_00013", "negative_test_query241_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query102", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query102", "ranked_ids": ["negative_test_query209_00006", "negative_test_query108_00009", "negative_test_query219_00004", "negative_test_query131_00009", "negative_test_query205_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query103", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.14285714285714285}, "query_id": "test_query103", "ranked_ids": ["negative_test_query103_00005", "apositive_test_query103_00005", "negative_test_query185_00003", "negative_test_query307_00004", "negative_test_query307_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query104", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query104", "ranked_ids": ["negative_test_query104_00010", "negative_test_query104_00016", "negative_test_query48_00010", "negative_test_query104_00004", "apositive_test_query48_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query105", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query105", "ranked_ids": ["negative_test_query313_00013", "negative_test_query105_00005", "negative_test_query267_00011", "negative_test_query119_00005", "negative_test_query168_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query106", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query106", "ranked_ids": ["negative_test_query326_00008", "apositive_test_query177_00008", "negative_test_query147_00001", "negative_test_query162_00007", "apositive_test_query42_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query107", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query107", "ranked_ids": ["negative_test_query146_00000", "negative_test_query296_00001", "negative_test_query31_00013", "apositive_test_query142_00002", "negative_test_query105_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query108", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.125}, "query_id": "test_query108", "ranked_ids": ["negative_test_query313_00012", "negative_test_query334_00013", "apositive_test_query108_00005", "negative_test_query57_00009", "negative_test_query311_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query109", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query109", "ranked_ids": ["negative_test_query109_00003", "negative_test_query6_00006", "negative_test_query109_00000", "negative_test_query51_00004", "negative_test_query109_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query11", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.125}, "query_id": "test_query11", "ranked_ids": ["apositive_test_query11_00001", "negative_test_query226_00010", "negative_test_query309_00002", "negative_test_query358_00005", "negative_test_query11_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query110", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query110", "ranked_ids": ["apositive_test_query11_00002", "negative_test_query110_00014", "apositive_test_query330_00001", "negative_test_query17_00008", "negative_test_query110_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query111", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query111", "ranked_ids": ["negative_test_query243_00001", "negative_test_query277_00010", "negative_test_query111_00018", "negative_test_query292_00010", "negative_test_query277_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query112", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query112", "ranked_ids": ["apositive_test_query286_00005", "negative_test_query112_00013", "negative_test_query80_00007", "negative_test_query112_00015", "apositive_test_query106_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query113", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.16666666666666666}, "query_id": "test_query113", "ranked_ids": ["negative_test_query38_00003", "negative_test_query113_00003", "apositive_test_query113_00001", "negative_test_query105_00000", "apositive_test_query186_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query114", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.14285714285714285}, "query_id": "test_query114", "ranked_ids": ["negative_test_query262_00007", "apositive_test_query114_00004", "negative_test_query80_00008", "negative_test_query106_00009", "negative_test_query114_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query115", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.36005461457723403, "recall@5": 0.3333333333333333}, "query_id": "test_query115", "ranked_ids": ["negative_test_query205_00004", "apositive_test_query115_00000", "apositive_test_query121_00001", "apositive_test_query115_00002", "negative_test_query139_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query116", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.38685280723454163, "recall@5": 0.5}, "query_id": "test_query116", "ranked_ids": ["negative_test_query116_00000", "apositive_test_query116_00000", "negative_test_query59_00007", "negative_test_query116_00007", "apositive_test_query49_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query117", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5087403079104241, "recall@5": 0.25}, "query_id": "test_query117", "ranked_ids": ["apositive_test_query117_00003", "negative_test_query63_00001", "apositive_test_query117_00007", "negative_test_query117_00003", "negative_test_query175_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query118", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query118", "ranked_ids": ["apositive_test_query340_00002", "negative_test_query88_00007", "apositive_test_query174_00003", "apositive_test_query174_00005", "apositive_test_query174_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query119", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query119", "ranked_ids": ["negative_test_query119_00005", "apositive_test_query79_00004", "negative_test_query157_00007", "negative_test_query359_00005", "negative_test_query152_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query12", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query12", "ranked_ids": ["negative_test_query234_00014", "negative_test_query12_00003", "negative_test_query12_00014", "negative_test_query12_00007", "negative_test_query12_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query120", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query120", "ranked_ids": ["negative_test_query120_00008", "apositive_test_query352_00000", "negative_test_query303_00018", "negative_test_query120_00002", "negative_test_query120_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query121", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.48522855511632257, "recall@5": 0.3333333333333333}, "query_id": "test_query121", "ranked_ids": ["apositive_test_query121_00001", "negative_test_query121_00012", "negative_test_query121_00007", "apositive_test_query121_00000", "negative_test_query115_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query122", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.38685280723454163, "recall@5": 1.0}, "query_id": "test_query122", "ranked_ids": ["negative_test_query64_00017", "negative_test_query86_00001", "apositive_test_query173_00001", "negative_test_query134_00007", "apositive_test_query122_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query123", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.2}, "query_id": "test_query123", "ranked_ids": ["apositive_test_query123_00002", "negative_test_query123_00002", "negative_test_query123_00008", "negative_test_query123_00006", "negative_test_query269_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query124", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query124", "ranked_ids": ["negative_test_query124_00003", "negative_test_query42_00006", "apositive_test_query13_00004", "negative_test_query93_00010", "apositive_test_query162_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query125", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query125", "ranked_ids": ["negative_test_query301_00012", "apositive_test_query286_00009", "negative_test_query125_00013", "negative_test_query125_00014", "negative_test_query140_00017"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query126", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query126", "ranked_ids": ["negative_test_query131_00009", "apositive_test_query330_00001", "apositive_test_query5_00002", "apositive_test_query151_00003", "negative_test_query126_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query127", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.639945385422766, "recall@5": 0.17647058823529413}, "query_id": "test_query127", "ranked_ids": ["apositive_test_query127_00009", "negative_test_query267_00009", "apositive_test_query127_00012", "apositive_test_query94_00005", "apositive_test_query127_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query128", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.125}, "query_id": "test_query128", "ranked_ids": ["apositive_test_query128_00001", "negative_test_query75_00012", "apositive_test_query216_00002", "negative_test_query128_00011", "apositive_test_query21_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query129", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.125}, "query_id": "test_query129", "ranked_ids": ["negative_test_query63_00006", "apositive_test_query249_00003", "negative_test_query16_00007", "negative_test_query187_00012", "apositive_test_query129_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query13", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.5296347172140421, "recall@5": 0.23076923076923078}, "query_id": "test_query13", "ranked_ids": ["negative_test_query13_00004", "apositive_test_query13_00010", "apositive_test_query13_00008", "apositive_test_query13_00012", "negative_test_query13_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query130", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query130", "ranked_ids": ["negative_test_query90_00014", "negative_test_query130_00001", "negative_test_query289_00004", "negative_test_query198_00003", "negative_test_query284_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query131", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3903800499921017, "recall@5": 0.25}, "query_id": "test_query131", "ranked_ids": ["apositive_test_query131_00001", "negative_test_query131_00002", "negative_test_query313_00008", "negative_test_query168_00013", "negative_test_query184_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query132", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query132", "ranked_ids": ["apositive_test_query3_00004", "negative_test_query132_00012", "negative_test_query132_00018", "negative_test_query81_00005", "negative_test_query304_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query133", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query133", "ranked_ids": ["apositive_test_query129_00002", "negative_test_query12_00014", "apositive_test_query129_00007", "apositive_test_query174_00011", "negative_test_query206_00017"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query134", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query134", "ranked_ids": ["negative_test_query313_00012", "negative_test_query334_00013", "negative_test_query108_00003", "apositive_test_query238_00001", "apositive_test_query276_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query135", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query135", "ranked_ids": ["negative_test_query135_00001", "negative_test_query297_00017", "negative_test_query135_00007", "negative_test_query135_00008", "negative_test_query135_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query136", "metrics": {"mrr@5": 1.0, "ndcg@5": 1.0, "recall@5": 0.3333333333333333}, "query_id": "test_query136", "ranked_ids": ["apositive_test_query136_00002", "apositive_test_query136_00011", "apositive_test_query136_00000", "apositive_test_query136_00010", "apositive_test_query136_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query137", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.4144299250118475, "recall@5": 0.5}, "query_id": "test_query137", "ranked_ids": ["negative_test_query149_00013", "apositive_test_query137_00002", "negative_test_query74_00001", "apositive_test_query137_00000", "negative_test_query262_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query138", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query138", "ranked_ids": ["negative_test_query138_00015", "apositive_test_query244_00005", "negative_test_query138_00005", "negative_test_query138_00014", "negative_test_query138_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query139", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.46927872602275644, "recall@5": 0.3333333333333333}, "query_id": "test_query139", "ranked_ids": ["apositive_test_query139_00001", "negative_test_query262_00002", "negative_test_query198_00011", "negative_test_query101_00006", "apositive_test_query354_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query14", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query14", "ranked_ids": ["apositive_test_query64_00000", "apositive_test_query26_00006", "apositive_test_query296_00005", "negative_test_query14_00001", "negative_test_query179_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query140", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query140", "ranked_ids": ["negative_test_query140_00009", "negative_test_query215_00000", "negative_test_query257_00003", "negative_test_query140_00003", "negative_test_query140_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query141", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query141", "ranked_ids": ["negative_test_query245_00008", "negative_test_query245_00016", "negative_test_query308_00005", "negative_test_query57_00002", "apositive_test_query288_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query142", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.14285714285714285}, "query_id": "test_query142", "ranked_ids": ["negative_test_query200_00014", "negative_test_query337_00016", "negative_test_query142_00007", "apositive_test_query142_00002", "negative_test_query105_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query143", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query143", "ranked_ids": ["negative_test_query90_00014", "negative_test_query143_00004", "negative_test_query45_00009", "negative_test_query143_00011", "negative_test_query143_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query144", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query144", "ranked_ids": ["negative_test_query166_00001", "negative_test_query144_00012", "negative_test_query144_00006", "negative_test_query144_00001", "negative_test_query144_00017"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query145", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query145", "ranked_ids": ["negative_test_query211_00013", "negative_test_query230_00002", "negative_test_query53_00012", "negative_test_query112_00007", "negative_test_query187_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query146", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.0625}, "query_id": "test_query146", "ranked_ids": ["apositive_test_query146_00004", "apositive_test_query314_00009", "negative_test_query146_00002", "apositive_test_query314_00001", "negative_test_query146_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query147", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query147", "ranked_ids": ["negative_test_query17_00009", "negative_test_query147_00013", "negative_test_query147_00008", "negative_test_query309_00014", "negative_test_query200_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query148", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query148", "ranked_ids": ["negative_test_query210_00006", "negative_test_query321_00013", "apositive_test_query234_00003", "apositive_test_query304_00002", "negative_test_query148_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query149", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query149", "ranked_ids": ["negative_test_query149_00001", "negative_test_query149_00013", "apositive_test_query145_00001", "negative_test_query225_00006", "negative_test_query149_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query15", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query15", "ranked_ids": ["negative_test_query15_00014", "negative_test_query15_00007", "negative_test_query15_00004", "negative_test_query325_00011", "negative_test_query15_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query150", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query150", "ranked_ids": ["negative_test_query301_00007", "negative_test_query346_00008", "negative_test_query139_00005", "negative_test_query254_00003", "negative_test_query350_00018"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query151", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.24630238874073, "recall@5": 0.25}, "query_id": "test_query151", "ranked_ids": ["negative_test_query190_00014", "apositive_test_query151_00003", "negative_test_query126_00006", "negative_test_query151_00004", "negative_test_query293_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query152", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query152", "ranked_ids": ["negative_test_query349_00001", "negative_test_query262_00010", "negative_test_query342_00005", "negative_test_query237_00011", "negative_test_query349_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query153", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5531464700081437, "recall@5": 0.125}, "query_id": "test_query153", "ranked_ids": ["apositive_test_query153_00001", "apositive_test_query153_00003", "negative_test_query278_00006", "negative_test_query112_00007", "negative_test_query112_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query154", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query154", "ranked_ids": ["negative_test_query245_00006", "apositive_test_query33_00008", "apositive_test_query20_00003", "apositive_test_query358_00008", "negative_test_query68_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query155", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.16666666666666666}, "query_id": "test_query155", "ranked_ids": ["negative_test_query155_00006", "negative_test_query155_00005", "negative_test_query99_00001", "apositive_test_query155_00005", "negative_test_query155_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query156", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.43067655807339306, "recall@5": 1.0}, "query_id": "test_query156", "ranked_ids": ["negative_test_query49_00005", "apositive_test_query210_00001", "negative_test_query283_00009", "apositive_test_query156_00000", "negative_test_query156_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query157", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query157", "ranked_ids": ["negative_test_query271_00003", "apositive_test_query237_00004", "apositive_test_query152_00004", "negative_test_query237_00000", "negative_test_query152_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query158", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.48522855511632257, "recall@5": 0.2}, "query_id": "test_query158", "ranked_ids": ["apositive_test_query158_00000", "negative_test_query26_00005", "negative_test_query26_00010", "apositive_test_query158_00007", "negative_test_query102_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query159", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query159", "ranked_ids": ["negative_test_query264_00013", "apositive_test_query38_00003", "negative_test_query159_00016", "apositive_test_query212_00003", "negative_test_query122_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query16", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query16", "ranked_ids": ["negative_test_query16_00009", "negative_test_query308_00004", "negative_test_query16_00011", "negative_test_query16_00008", "negative_test_query16_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query160", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.07142857142857142}, "query_id": "test_query160", "ranked_ids": ["apositive_test_query64_00000", "negative_test_query271_00010", "negative_test_query304_00007", "negative_test_query330_00005", "apositive_test_query160_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query161", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.1111111111111111}, "query_id": "test_query161", "ranked_ids": ["apositive_test_query3_00006", "negative_test_query260_00004", "apositive_test_query314_00004", "apositive_test_query161_00008", "apositive_test_query301_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query162", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.08333333333333333}, "query_id": "test_query162", "ranked_ids": ["apositive_test_query194_00009", "negative_test_query162_00001", "negative_test_query343_00012", "negative_test_query177_00002", "apositive_test_query162_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query163", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query163", "ranked_ids": ["negative_test_query58_00015", "negative_test_query197_00002", "negative_test_query265_00015", "negative_test_query163_00011", "negative_test_query202_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query164", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query164", "ranked_ids": ["apositive_test_query146_00014", "negative_test_query164_00015", "negative_test_query245_00005", "negative_test_query321_00011", "negative_test_query100_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query165", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query165", "ranked_ids": ["negative_test_query327_00016", "negative_test_query165_00007", "negative_test_query86_00006", "negative_test_query264_00016", "negative_test_query165_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query166", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.16666666666666666}, "query_id": "test_query166", "ranked_ids": ["apositive_test_query224_00000", "apositive_test_query166_00002", "negative_test_query166_00001", "negative_test_query165_00015", "negative_test_query166_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query167", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query167", "ranked_ids": ["negative_test_query278_00010", "apositive_test_query288_00005", "negative_test_query141_00010", "negative_test_query167_00008", "negative_test_query167_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query168", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5585075862632192, "recall@5": 0.5}, "query_id": "test_query168", "ranked_ids": ["apositive_test_query168_00000", "negative_test_query168_00007", "negative_test_query260_00011", "apositive_test_query168_00001", "negative_test_query168_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query169", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query169", "ranked_ids": ["negative_test_query10_00013", "negative_test_query72_00013", "negative_test_query49_00017", "apositive_test_query330_00001", "apositive_test_query69_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query17", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query17", "ranked_ids": ["negative_test_query17_00016", "negative_test_query17_00005", "negative_test_query17_00014", "negative_test_query164_00006", "negative_test_query17_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query170", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.1111111111111111}, "query_id": "test_query170", "ranked_ids": ["negative_test_query170_00001", "negative_test_query227_00010", "negative_test_query227_00002", "apositive_test_query170_00000", "apositive_test_query297_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query171", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query171", "ranked_ids": ["apositive_test_query224_00000", "negative_test_query277_00007", "negative_test_query346_00005", "negative_test_query24_00007", "negative_test_query171_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query172", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query172", "ranked_ids": ["apositive_test_query124_00002", "negative_test_query349_00001", "negative_test_query173_00006", "negative_test_query8_00000", "negative_test_query350_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query173", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.38356636737133565, "recall@5": 0.3333333333333333}, "query_id": "test_query173", "ranked_ids": ["negative_test_query64_00017", "apositive_test_query173_00004", "apositive_test_query173_00001", "negative_test_query173_00006", "negative_test_query222_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query174", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.7860137352654724, "recall@5": 0.23529411764705882}, "query_id": "test_query174", "ranked_ids": ["apositive_test_query174_00003", "negative_test_query19_00012", "apositive_test_query174_00005", "apositive_test_query174_00007", "apositive_test_query174_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query175", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.16666666666666666}, "query_id": "test_query175", "ranked_ids": ["apositive_test_query175_00000", "negative_test_query204_00000", "apositive_test_query50_00015", "negative_test_query175_00013", "negative_test_query175_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query176", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query176", "ranked_ids": ["negative_test_query144_00018", "apositive_test_query92_00002", "negative_test_query24_00014", "negative_test_query195_00006", "negative_test_query284_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query177", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.1}, "query_id": "test_query177", "ranked_ids": ["negative_test_query124_00001", "apositive_test_query177_00008", "negative_test_query147_00001", "negative_test_query162_00007", "negative_test_query177_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query178", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query178", "ranked_ids": ["apositive_test_query60_00011", "negative_test_query31_00004", "negative_test_query274_00001", "negative_test_query321_00017", "negative_test_query178_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query179", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query179", "ranked_ids": ["apositive_test_query316_00001", "negative_test_query191_00003", "negative_test_query297_00000", "negative_test_query206_00004", "negative_test_query241_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query18", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query18", "ranked_ids": ["negative_test_query88_00002", "negative_test_query215_00000", "negative_test_query257_00003", "negative_test_query140_00003", "negative_test_query18_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query180", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query180", "ranked_ids": ["negative_test_query246_00003", "negative_test_query207_00007", "negative_test_query180_00002", "negative_test_query180_00007", "apositive_test_query262_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query181", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query181", "ranked_ids": ["negative_test_query181_00005", "negative_test_query181_00010", "negative_test_query348_00007", "apositive_test_query119_00011", "negative_test_query181_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query182", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query182", "ranked_ids": ["negative_test_query182_00011", "negative_test_query182_00018", "negative_test_query342_00004", "negative_test_query237_00006", "negative_test_query182_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query183", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query183", "ranked_ids": ["negative_test_query183_00008", "negative_test_query7_00009", "apositive_test_query7_00001", "negative_test_query183_00010", "negative_test_query7_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query184", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query184", "ranked_ids": ["apositive_test_query33_00004", "negative_test_query156_00001", "negative_test_query216_00008", "negative_test_query216_00002", "apositive_test_query316_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query185", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query185", "ranked_ids": ["negative_test_query350_00007", "negative_test_query185_00002", "apositive_test_query307_00008", "apositive_test_query103_00000", "negative_test_query48_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query186", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query186", "ranked_ids": ["apositive_test_query79_00005", "negative_test_query105_00001", "negative_test_query29_00003", "negative_test_query313_00005", "apositive_test_query50_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query187", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query187", "ranked_ids": ["negative_test_query245_00006", "apositive_test_query33_00008", "negative_test_query13_00005", "negative_test_query245_00015", "apositive_test_query358_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query188", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.2}, "query_id": "test_query188", "ranked_ids": ["negative_test_query112_00017", "negative_test_query20_00006", "apositive_test_query20_00009", "negative_test_query344_00007", "apositive_test_query188_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query189", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.2}, "query_id": "test_query189", "ranked_ids": ["negative_test_query189_00004", "apositive_test_query189_00002", "negative_test_query189_00002", "negative_test_query55_00003", "negative_test_query140_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query19", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query19", "ranked_ids": ["negative_test_query106_00007", "negative_test_query292_00004", "apositive_test_query174_00007", "negative_test_query229_00002", "negative_test_query141_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query190", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query190", "ranked_ids": ["apositive_test_query234_00001", "negative_test_query190_00013", "negative_test_query233_00008", "negative_test_query265_00004", "negative_test_query205_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query191", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5413996682199069, "recall@5": 0.5}, "query_id": "test_query191", "ranked_ids": ["apositive_test_query191_00000", "negative_test_query191_00003", "negative_test_query274_00002", "negative_test_query241_00007", "apositive_test_query191_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query192", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5087403079104241, "recall@5": 0.4}, "query_id": "test_query192", "ranked_ids": ["apositive_test_query192_00002", "negative_test_query192_00002", "apositive_test_query192_00004", "negative_test_query120_00007", "negative_test_query350_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query193", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.36005461457723403, "recall@5": 0.15384615384615385}, "query_id": "test_query193", "ranked_ids": ["negative_test_query193_00006", "apositive_test_query193_00001", "apositive_test_query99_00003", "apositive_test_query193_00009", "negative_test_query146_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query194", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.27727342735504823, "recall@5": 0.14285714285714285}, "query_id": "test_query194", "ranked_ids": ["negative_test_query262_00007", "negative_test_query177_00007", "negative_test_query95_00016", "apositive_test_query194_00001", "apositive_test_query194_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query195", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.16666666666666666}, "query_id": "test_query195", "ranked_ids": ["apositive_test_query195_00002", "apositive_test_query251_00000", "negative_test_query109_00008", "negative_test_query199_00003", "negative_test_query195_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query196", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.2}, "query_id": "test_query196", "ranked_ids": ["apositive_test_query32_00004", "apositive_test_query196_00003", "apositive_test_query32_00011", "negative_test_query139_00011", "negative_test_query173_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query197", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.46927872602275644, "recall@5": 0.3333333333333333}, "query_id": "test_query197", "ranked_ids": ["apositive_test_query197_00000", "apositive_test_query189_00001", "negative_test_query163_00001", "apositive_test_query34_00000", "negative_test_query163_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query198", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query198", "ranked_ids": ["negative_test_query293_00011", "negative_test_query198_00004", "negative_test_query239_00007", "negative_test_query293_00010", "negative_test_query198_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query199", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query199", "ranked_ids": ["negative_test_query350_00007", "negative_test_query199_00006", "negative_test_query185_00002", "negative_test_query199_00002", "negative_test_query221_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query2", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query2", "ranked_ids": ["negative_test_query44_00008", "negative_test_query2_00012", "negative_test_query197_00004", "negative_test_query348_00005", "negative_test_query104_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query20", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.38356636737133565, "recall@5": 0.18181818181818182}, "query_id": "test_query20", "ranked_ids": ["apositive_test_query33_00008", "apositive_test_query20_00009", "apositive_test_query20_00003", "negative_test_query66_00002", "apositive_test_query66_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query200", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query200", "ranked_ids": ["negative_test_query200_00015", "negative_test_query38_00005", "negative_test_query264_00007", "negative_test_query264_00006", "negative_test_query200_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query201", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query201", "ranked_ids": ["negative_test_query71_00002", "negative_test_query0_00004", "negative_test_query47_00013", "negative_test_query209_00008", "apositive_test_query30_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query202", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query202", "ranked_ids": ["negative_test_query235_00001", "apositive_test_query61_00003", "apositive_test_query206_00000", "negative_test_query202_00004", "apositive_test_query309_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query203", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query203", "ranked_ids": ["negative_test_query130_00001", "apositive_test_query30_00005", "apositive_test_query284_00003", "negative_test_query35_00004", "negative_test_query203_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query204", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.36005461457723403, "recall@5": 0.11764705882352941}, "query_id": "test_query204", "ranked_ids": ["negative_test_query204_00000", "apositive_test_query204_00000", "apositive_test_query353_00002", "apositive_test_query204_00008", "negative_test_query353_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query205", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query205", "ranked_ids": ["negative_test_query31_00015", "negative_test_query205_00007", "negative_test_query115_00000", "apositive_test_query256_00002", "negative_test_query205_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query206", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query206", "ranked_ids": ["negative_test_query64_00016", "negative_test_query206_00001", "negative_test_query206_00018", "negative_test_query206_00003", "negative_test_query299_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query207", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.16666666666666666}, "query_id": "test_query207", "ranked_ids": ["apositive_test_query207_00005", "negative_test_query207_00007", "negative_test_query207_00000", "negative_test_query207_00003", "negative_test_query207_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query208", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query208", "ranked_ids": ["negative_test_query128_00001", "negative_test_query73_00008", "apositive_test_query128_00002", "negative_test_query128_00003", "negative_test_query208_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query209", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query209", "ranked_ids": ["negative_test_query313_00012", "apositive_test_query119_00008", "negative_test_query102_00016", "apositive_test_query158_00004", "negative_test_query26_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query21", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query21", "ranked_ids": ["negative_test_query21_00009", "negative_test_query330_00000", "negative_test_query234_00006", "apositive_test_query128_00003", "apositive_test_query128_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query210", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query210", "ranked_ids": ["negative_test_query288_00006", "negative_test_query210_00006", "negative_test_query321_00013", "negative_test_query156_00015", "negative_test_query202_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query211", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query211", "ranked_ids": ["negative_test_query159_00006", "negative_test_query211_00012", "apositive_test_query101_00000", "negative_test_query211_00000", "negative_test_query211_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query212", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query212", "ranked_ids": ["apositive_test_query326_00000", "apositive_test_query98_00000", "apositive_test_query11_00007", "negative_test_query282_00010", "negative_test_query344_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query213", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.14285714285714285}, "query_id": "test_query213", "ranked_ids": ["apositive_test_query213_00002", "negative_test_query138_00016", "negative_test_query224_00015", "negative_test_query213_00004", "negative_test_query213_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query214", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.5, "recall@5": 1.0}, "query_id": "test_query214", "ranked_ids": ["negative_test_query214_00015", "apositive_test_query234_00001", "apositive_test_query214_00000", "negative_test_query214_00010", "apositive_test_query160_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query215", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.5, "recall@5": 1.0}, "query_id": "test_query215", "ranked_ids": ["negative_test_query215_00000", "negative_test_query257_00003", "apositive_test_query215_00000", "negative_test_query257_00001", "negative_test_query215_00018"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query216", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.2}, "query_id": "test_query216", "ranked_ids": ["apositive_test_query33_00004", "negative_test_query216_00008", "apositive_test_query216_00002", "negative_test_query216_00009", "negative_test_query246_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query217", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.125}, "query_id": "test_query217", "ranked_ids": ["negative_test_query28_00002", "negative_test_query230_00002", "negative_test_query53_00012", "apositive_test_query217_00000", "negative_test_query185_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query218", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.14285714285714285}, "query_id": "test_query218", "ranked_ids": ["apositive_test_query218_00004", "negative_test_query218_00003", "apositive_test_query60_00012", "negative_test_query218_00000", "negative_test_query274_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query219", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query219", "ranked_ids": ["negative_test_query165_00010", "negative_test_query308_00006", "negative_test_query24_00007", "negative_test_query187_00015", "negative_test_query219_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query22", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query22", "ranked_ids": ["negative_test_query22_00000", "negative_test_query22_00012", "negative_test_query22_00007", "negative_test_query22_00008", "negative_test_query66_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query220", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.16666666666666666}, "query_id": "test_query220", "ranked_ids": ["negative_test_query188_00007", "apositive_test_query20_00009", "negative_test_query344_00007", "apositive_test_query220_00001", "apositive_test_query188_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query221", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query221", "ranked_ids": ["apositive_test_query174_00006", "negative_test_query278_00004", "negative_test_query277_00010", "negative_test_query184_00001", "negative_test_query171_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query222", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query222", "ranked_ids": ["negative_test_query222_00014", "apositive_test_query289_00002", "apositive_test_query275_00008", "negative_test_query275_00004", "apositive_test_query275_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query223", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query223", "ranked_ids": ["negative_test_query51_00003", "apositive_test_query259_00008", "negative_test_query323_00005", "negative_test_query47_00007", "negative_test_query359_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query224", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3903800499921017, "recall@5": 0.25}, "query_id": "test_query224", "ranked_ids": ["apositive_test_query224_00000", "negative_test_query224_00011", "negative_test_query98_00018", "negative_test_query224_00014", "apositive_test_query32_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query225", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.48522855511632257, "recall@5": 0.18181818181818182}, "query_id": "test_query225", "ranked_ids": ["apositive_test_query225_00000", "apositive_test_query145_00001", "negative_test_query225_00006", "apositive_test_query225_00003", "negative_test_query149_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query226", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.34519134224686937, "recall@5": 0.3333333333333333}, "query_id": "test_query226", "ranked_ids": ["negative_test_query226_00009", "apositive_test_query226_00002", "negative_test_query226_00010", "negative_test_query226_00007", "apositive_test_query226_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query227", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.16812753627111746, "recall@5": 0.25}, "query_id": "test_query227", "ranked_ids": ["negative_test_query227_00006", "apositive_test_query348_00004", "negative_test_query321_00004", "apositive_test_query227_00001", "apositive_test_query70_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query228", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.38685280723454163, "recall@5": 0.5}, "query_id": "test_query228", "ranked_ids": ["apositive_test_query322_00003", "apositive_test_query228_00000", "negative_test_query73_00016", "negative_test_query165_00011", "apositive_test_query254_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query229", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.1}, "query_id": "test_query229", "ranked_ids": ["apositive_test_query146_00004", "negative_test_query355_00001", "apositive_test_query229_00000", "negative_test_query94_00010", "negative_test_query44_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query23", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query23", "ranked_ids": ["apositive_test_query225_00000", "negative_test_query356_00008", "apositive_test_query146_00015", "negative_test_query292_00013", "apositive_test_query54_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query230", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query230", "ranked_ids": ["negative_test_query99_00003", "apositive_test_query3_00006", "negative_test_query230_00010", "apositive_test_query3_00000", "negative_test_query3_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query231", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query231", "ranked_ids": ["negative_test_query201_00011", "negative_test_query221_00006", "negative_test_query345_00011", "negative_test_query306_00006", "apositive_test_query201_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query232", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query232", "ranked_ids": ["negative_test_query232_00013", "negative_test_query232_00014", "negative_test_query232_00011", "negative_test_query232_00016", "negative_test_query232_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query233", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.18154179253735267, "recall@5": 0.3333333333333333}, "query_id": "test_query233", "ranked_ids": ["negative_test_query130_00001", "negative_test_query233_00006", "negative_test_query233_00001", "negative_test_query233_00005", "apositive_test_query233_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query234", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.24630238874073, "recall@5": 0.25}, "query_id": "test_query234", "ranked_ids": ["negative_test_query234_00010", "apositive_test_query234_00003", "apositive_test_query304_00002", "negative_test_query234_00011", "negative_test_query214_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query235", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3903800499921017, "recall@5": 0.25}, "query_id": "test_query235", "ranked_ids": ["apositive_test_query235_00003", "apositive_test_query32_00004", "negative_test_query235_00013", "negative_test_query235_00011", "negative_test_query235_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query236", "metrics": {"mrr@5": 1.0, "ndcg@5": 1.0, "recall@5": 1.0}, "query_id": "test_query236", "ranked_ids": ["apositive_test_query236_00000", "negative_test_query236_00002", "negative_test_query236_00005", "negative_test_query236_00000", "negative_test_query236_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query237", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.2}, "query_id": "test_query237", "ranked_ids": ["apositive_test_query237_00001", "negative_test_query237_00005", "negative_test_query237_00006", "apositive_test_query152_00004", "negative_test_query237_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query238", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query238", "ranked_ids": ["negative_test_query238_00011", "negative_test_query13_00003", "negative_test_query111_00013", "negative_test_query142_00011", "negative_test_query102_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query239", "metrics": {"mrr@5": 1.0, "ndcg@5": 1.0, "recall@5": 1.0}, "query_id": "test_query239", "ranked_ids": ["apositive_test_query239_00000", "negative_test_query239_00007", "negative_test_query239_00003", "negative_test_query167_00011", "negative_test_query198_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query24", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.43067655807339306, "recall@5": 1.0}, "query_id": "test_query24", "ranked_ids": ["apositive_test_query106_00002", "negative_test_query24_00002", "negative_test_query24_00004", "apositive_test_query24_00000", "negative_test_query24_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query240", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query240", "ranked_ids": ["negative_test_query240_00010", "negative_test_query240_00016", "negative_test_query240_00018", "negative_test_query297_00005", "negative_test_query27_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query241", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query241", "ranked_ids": ["negative_test_query206_00004", "negative_test_query241_00006", "negative_test_query173_00013", "negative_test_query241_00005", "negative_test_query191_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query242", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.34519134224686937, "recall@5": 0.13333333333333333}, "query_id": "test_query242", "ranked_ids": ["apositive_test_query247_00005", "apositive_test_query242_00001", "negative_test_query291_00003", "negative_test_query291_00005", "apositive_test_query242_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query243", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query243", "ranked_ids": ["apositive_test_query173_00004", "apositive_test_query160_00003", "negative_test_query243_00000", "negative_test_query205_00007", "negative_test_query243_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query244", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.1111111111111111}, "query_id": "test_query244", "ranked_ids": ["negative_test_query244_00010", "negative_test_query244_00009", "apositive_test_query244_00006", "negative_test_query244_00008", "negative_test_query244_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query245", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query245", "ranked_ids": ["negative_test_query293_00003", "negative_test_query202_00004", "negative_test_query135_00013", "negative_test_query122_00002", "negative_test_query159_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query246", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query246", "ranked_ids": ["negative_test_query6_00006", "negative_test_query252_00003", "negative_test_query134_00001", "negative_test_query218_00004", "negative_test_query58_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query247", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.16666666666666666}, "query_id": "test_query247", "ranked_ids": ["apositive_test_query247_00005", "apositive_test_query242_00001", "negative_test_query291_00003", "negative_test_query291_00005", "negative_test_query186_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query248", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.34519134224686937, "recall@5": 0.25}, "query_id": "test_query248", "ranked_ids": ["negative_test_query248_00001", "apositive_test_query248_00003", "negative_test_query248_00006", "negative_test_query205_00017", "apositive_test_query248_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query249", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.06666666666666667}, "query_id": "test_query249", "ranked_ids": ["negative_test_query249_00001", "negative_test_query325_00015", "negative_test_query301_00013", "negative_test_query326_00005", "apositive_test_query249_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query25", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.30078518014914984, "recall@5": 0.18181818181818182}, "query_id": "test_query25", "ranked_ids": ["negative_test_query328_00000", "negative_test_query45_00014", "apositive_test_query25_00008", "negative_test_query25_00001", "apositive_test_query25_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query250", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query250", "ranked_ids": ["negative_test_query101_00008", "negative_test_query61_00007", "negative_test_query192_00009", "negative_test_query57_00009", "negative_test_query64_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query251", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query251", "ranked_ids": ["negative_test_query234_00006", "negative_test_query251_00003", "negative_test_query143_00001", "negative_test_query56_00005", "negative_test_query170_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query252", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query252", "ranked_ids": ["apositive_test_query110_00001", "negative_test_query252_00009", "negative_test_query44_00011", "negative_test_query321_00004", "negative_test_query252_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query253", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query253", "ranked_ids": ["negative_test_query124_00001", "apositive_test_query162_00008", "negative_test_query177_00000", "apositive_test_query74_00002", "apositive_test_query13_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query254", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.5296347172140421, "recall@5": 0.2727272727272727}, "query_id": "test_query254", "ranked_ids": ["negative_test_query21_00009", "apositive_test_query254_00003", "apositive_test_query254_00005", "apositive_test_query254_00010", "apositive_test_query333_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query255", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.6309297535714575, "recall@5": 1.0}, "query_id": "test_query255", "ranked_ids": ["negative_test_query246_00003", "apositive_test_query255_00000", "negative_test_query278_00000", "negative_test_query255_00006", "negative_test_query326_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query256", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.36331756126716835, "recall@5": 0.5}, "query_id": "test_query256", "ranked_ids": ["negative_test_query183_00008", "negative_test_query7_00009", "apositive_test_query256_00002", "apositive_test_query256_00003", "apositive_test_query216_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query257", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.30078518014914984, "recall@5": 0.3333333333333333}, "query_id": "test_query257", "ranked_ids": ["negative_test_query215_00000", "negative_test_query257_00003", "apositive_test_query257_00003", "apositive_test_query286_00014", "apositive_test_query257_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query258", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query258", "ranked_ids": ["negative_test_query320_00002", "negative_test_query38_00002", "negative_test_query162_00004", "negative_test_query173_00010", "negative_test_query167_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query259", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.08333333333333333}, "query_id": "test_query259", "ranked_ids": ["apositive_test_query259_00008", "negative_test_query30_00009", "negative_test_query213_00004", "negative_test_query259_00006", "negative_test_query47_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query26", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.125}, "query_id": "test_query26", "ranked_ids": ["apositive_test_query26_00004", "negative_test_query26_00005", "negative_test_query26_00008", "negative_test_query119_00006", "negative_test_query158_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query260", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query260", "ranked_ids": ["negative_test_query6_00009", "negative_test_query6_00012", "apositive_test_query6_00000", "negative_test_query252_00004", "negative_test_query260_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query261", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.2}, "query_id": "test_query261", "ranked_ids": ["apositive_test_query261_00000", "negative_test_query261_00009", "negative_test_query261_00007", "negative_test_query261_00005", "negative_test_query118_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query262", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query262", "ranked_ids": ["negative_test_query326_00008", "apositive_test_query177_00008", "negative_test_query147_00001", "negative_test_query162_00007", "apositive_test_query42_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query263", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.16666666666666666}, "query_id": "test_query263", "ranked_ids": ["apositive_test_query263_00002", "apositive_test_query242_00011", "apositive_test_query291_00010", "negative_test_query263_00005", "negative_test_query307_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query264", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query264", "ranked_ids": ["negative_test_query264_00004", "negative_test_query264_00011", "negative_test_query264_00007", "apositive_test_query38_00003", "negative_test_query264_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query265", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query265", "ranked_ids": ["negative_test_query265_00002", "apositive_test_query234_00001", "negative_test_query165_00007", "negative_test_query165_00016", "negative_test_query190_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query266", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query266", "ranked_ids": ["negative_test_query266_00000", "negative_test_query266_00009", "negative_test_query266_00007", "negative_test_query266_00001", "negative_test_query141_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query267", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query267", "ranked_ids": ["apositive_test_query204_00000", "apositive_test_query353_00002", "negative_test_query30_00000", "apositive_test_query250_00000", "negative_test_query186_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query268", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.2}, "query_id": "test_query268", "ranked_ids": ["negative_test_query195_00009", "negative_test_query335_00002", "apositive_test_query268_00004", "negative_test_query268_00012", "negative_test_query268_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query269", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.16812753627111746, "recall@5": 0.25}, "query_id": "test_query269", "ranked_ids": ["negative_test_query269_00009", "negative_test_query329_00004", "apositive_test_query296_00005", "apositive_test_query269_00000", "negative_test_query329_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query27", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query27", "ranked_ids": ["apositive_test_query127_00010", "negative_test_query27_00009", "negative_test_query76_00009", "negative_test_query27_00014", "negative_test_query130_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query270", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.18154179253735267, "recall@5": 0.3333333333333333}, "query_id": "test_query270", "ranked_ids": ["negative_test_query270_00006", "negative_test_query270_00010", "negative_test_query270_00016", "negative_test_query270_00009", "apositive_test_query270_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query271", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.2640681225725909, "recall@5": 0.5}, "query_id": "test_query271", "ranked_ids": ["negative_test_query271_00007", "negative_test_query227_00009", "negative_test_query120_00003", "apositive_test_query271_00001", "apositive_test_query10_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query272", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query272", "ranked_ids": ["negative_test_query272_00017", "negative_test_query332_00013", "apositive_test_query251_00000", "negative_test_query328_00000", "negative_test_query272_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query273", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query273", "ranked_ids": ["negative_test_query273_00009", "negative_test_query273_00006", "apositive_test_query18_00001", "negative_test_query273_00008", "apositive_test_query18_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query274", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.15101961822780524, "recall@5": 0.25}, "query_id": "test_query274", "ranked_ids": ["negative_test_query274_00012", "negative_test_query218_00011", "negative_test_query34_00014", "negative_test_query352_00012", "apositive_test_query274_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query275", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.08333333333333333}, "query_id": "test_query275", "ranked_ids": ["apositive_test_query110_00001", "negative_test_query275_00007", "apositive_test_query93_00003", "negative_test_query149_00016", "apositive_test_query275_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query276", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.07692307692307693}, "query_id": "test_query276", "ranked_ids": ["negative_test_query146_00000", "apositive_test_query153_00006", "apositive_test_query146_00015", "negative_test_query251_00012", "apositive_test_query276_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query277", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.16812753627111746, "recall@5": 0.25}, "query_id": "test_query277", "ranked_ids": ["negative_test_query277_00009", "apositive_test_query359_00001", "negative_test_query305_00003", "apositive_test_query277_00002", "negative_test_query345_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query278", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query278", "ranked_ids": ["negative_test_query140_00006", "negative_test_query177_00001", "negative_test_query15_00007", "negative_test_query278_00009", "negative_test_query260_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query279", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5531464700081437, "recall@5": 0.4}, "query_id": "test_query279", "ranked_ids": ["apositive_test_query279_00004", "apositive_test_query279_00003", "apositive_test_query3_00000", "apositive_test_query76_00002", "negative_test_query279_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query28", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query28", "ranked_ids": ["negative_test_query28_00002", "apositive_test_query217_00000", "apositive_test_query217_00003", "negative_test_query185_00003", "negative_test_query307_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query280", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.16666666666666666}, "query_id": "test_query280", "ranked_ids": ["apositive_test_query280_00005", "negative_test_query280_00012", "negative_test_query9_00012", "negative_test_query280_00010", "negative_test_query106_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query281", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5531464700081437, "recall@5": 0.10526315789473684}, "query_id": "test_query281", "ranked_ids": ["apositive_test_query281_00004", "apositive_test_query281_00011", "apositive_test_query273_00000", "apositive_test_query273_00008", "negative_test_query273_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query282", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.6131471927654584, "recall@5": 0.5}, "query_id": "test_query282", "ranked_ids": ["apositive_test_query282_00000", "negative_test_query282_00003", "negative_test_query282_00006", "negative_test_query282_00004", "negative_test_query282_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query283", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query283", "ranked_ids": ["negative_test_query270_00006", "negative_test_query93_00009", "negative_test_query138_00005", "negative_test_query283_00014", "negative_test_query93_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query284", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.4703652827859579, "recall@5": 0.4}, "query_id": "test_query284", "ranked_ids": ["apositive_test_query284_00000", "negative_test_query284_00001", "apositive_test_query163_00003", "negative_test_query14_00014", "apositive_test_query284_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query285", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.30078518014914984, "recall@5": 0.25}, "query_id": "test_query285", "ranked_ids": ["negative_test_query285_00002", "negative_test_query86_00001", "apositive_test_query285_00001", "negative_test_query304_00003", "apositive_test_query285_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query286", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.4703652827859579, "recall@5": 0.13333333333333333}, "query_id": "test_query286", "ranked_ids": ["apositive_test_query286_00014", "apositive_test_query257_00000", "negative_test_query139_00008", "negative_test_query52_00001", "apositive_test_query286_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query287", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query287", "ranked_ids": ["apositive_test_query153_00003", "apositive_test_query133_00000", "apositive_test_query133_00004", "negative_test_query301_00010", "apositive_test_query133_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query288", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query288", "ranked_ids": ["negative_test_query288_00006", "apositive_test_query210_00001", "negative_test_query210_00005", "negative_test_query148_00014", "negative_test_query199_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query289", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.36005461457723403, "recall@5": 0.4}, "query_id": "test_query289", "ranked_ids": ["negative_test_query289_00004", "apositive_test_query289_00002", "negative_test_query113_00009", "apositive_test_query289_00000", "negative_test_query7_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query29", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.125}, "query_id": "test_query29", "ranked_ids": ["negative_test_query292_00015", "apositive_test_query29_00003", "negative_test_query213_00012", "negative_test_query29_00005", "negative_test_query29_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query290", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.15101961822780524, "recall@5": 0.25}, "query_id": "test_query290", "ranked_ids": ["negative_test_query290_00005", "negative_test_query98_00003", "negative_test_query342_00005", "negative_test_query290_00000", "apositive_test_query290_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query291", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query291", "ranked_ids": ["apositive_test_query247_00005", "apositive_test_query242_00001", "negative_test_query291_00003", "negative_test_query291_00005", "negative_test_query291_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query292", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.2960819109658652, "recall@5": 0.3333333333333333}, "query_id": "test_query292", "ranked_ids": ["negative_test_query292_00015", "apositive_test_query292_00001", "negative_test_query292_00000", "negative_test_query61_00011", "negative_test_query292_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query293", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.23463936301137822, "recall@5": 0.3333333333333333}, "query_id": "test_query293", "ranked_ids": ["negative_test_query293_00011", "negative_test_query293_00015", "apositive_test_query293_00002", "negative_test_query293_00010", "negative_test_query293_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query294", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query294", "ranked_ids": ["negative_test_query294_00007", "negative_test_query294_00001", "negative_test_query294_00004", "negative_test_query294_00012", "negative_test_query294_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query295", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query295", "ranked_ids": ["negative_test_query292_00015", "negative_test_query295_00012", "negative_test_query57_00009", "negative_test_query295_00001", "negative_test_query295_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query296", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.34519134224686937, "recall@5": 0.3333333333333333}, "query_id": "test_query296", "ranked_ids": ["negative_test_query329_00011", "apositive_test_query296_00004", "negative_test_query296_00001", "negative_test_query0_00004", "apositive_test_query296_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query297", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query297", "ranked_ids": ["negative_test_query297_00004", "negative_test_query297_00007", "apositive_test_query139_00001", "negative_test_query297_00012", "negative_test_query191_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query298", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.38364913903790027, "recall@5": 0.6666666666666666}, "query_id": "test_query298", "ranked_ids": ["negative_test_query182_00011", "negative_test_query193_00006", "negative_test_query182_00010", "apositive_test_query298_00000", "apositive_test_query298_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query299", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query299", "ranked_ids": ["negative_test_query288_00000", "negative_test_query58_00005", "negative_test_query89_00008", "apositive_test_query265_00002", "apositive_test_query227_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query3", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.09090909090909091}, "query_id": "test_query3", "ranked_ids": ["negative_test_query3_00007", "negative_test_query230_00010", "apositive_test_query3_00000", "negative_test_query3_00000", "negative_test_query3_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query30", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.14285714285714285}, "query_id": "test_query30", "ranked_ids": ["negative_test_query231_00008", "negative_test_query71_00010", "negative_test_query182_00014", "apositive_test_query30_00005", "negative_test_query231_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query300", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query300", "ranked_ids": ["negative_test_query300_00011", "negative_test_query300_00005", "negative_test_query300_00009", "negative_test_query141_00001", "negative_test_query300_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query301", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.15101961822780524, "recall@5": 0.25}, "query_id": "test_query301", "ranked_ids": ["apositive_test_query86_00000", "negative_test_query301_00007", "negative_test_query2_00008", "negative_test_query301_00013", "apositive_test_query301_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query302", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.14285714285714285}, "query_id": "test_query302", "ranked_ids": ["negative_test_query308_00005", "apositive_test_query302_00004", "negative_test_query308_00002", "negative_test_query302_00010", "apositive_test_query308_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query303", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.43067655807339306, "recall@5": 1.0}, "query_id": "test_query303", "ranked_ids": ["negative_test_query243_00000", "negative_test_query303_00017", "negative_test_query35_00003", "apositive_test_query303_00000", "negative_test_query173_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query304", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query304", "ranked_ids": ["apositive_test_query285_00001", "negative_test_query304_00003", "negative_test_query234_00010", "apositive_test_query285_00005", "apositive_test_query234_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query305", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query305", "ranked_ids": ["negative_test_query51_00003", "negative_test_query0_00004", "negative_test_query224_00015", "negative_test_query164_00011", "negative_test_query47_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query306", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query306", "ranked_ids": ["negative_test_query209_00008", "negative_test_query182_00014", "apositive_test_query289_00003", "apositive_test_query119_00006", "negative_test_query204_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query307", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query307", "ranked_ids": ["negative_test_query307_00006", "negative_test_query103_00012", "negative_test_query48_00018", "negative_test_query163_00010", "apositive_test_query279_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query308", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.27727342735504823, "recall@5": 0.2}, "query_id": "test_query308", "ranked_ids": ["apositive_test_query310_00008", "apositive_test_query324_00000", "negative_test_query324_00014", "apositive_test_query308_00002", "apositive_test_query308_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query309", "metrics": {"mrr@5": 1.0, "ndcg@5": 1.0, "recall@5": 1.0}, "query_id": "test_query309", "ranked_ids": ["apositive_test_query309_00000", "negative_test_query309_00003", "apositive_test_query358_00003", "apositive_test_query76_00002", "apositive_test_query161_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query31", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query31", "ranked_ids": ["negative_test_query38_00003", "apositive_test_query60_00011", "negative_test_query31_00004", "negative_test_query31_00007", "negative_test_query31_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query310", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.4912596920895758, "recall@5": 0.3}, "query_id": "test_query310", "ranked_ids": ["negative_test_query179_00013", "apositive_test_query310_00008", "negative_test_query74_00002", "apositive_test_query310_00002", "apositive_test_query310_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query311", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query311", "ranked_ids": ["negative_test_query330_00009", "negative_test_query288_00004", "negative_test_query330_00011", "negative_test_query240_00014", "apositive_test_query330_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query312", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query312", "ranked_ids": ["negative_test_query242_00001", "apositive_test_query237_00001", "apositive_test_query220_00005", "apositive_test_query98_00000", "negative_test_query294_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query313", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query313", "ranked_ids": ["negative_test_query313_00012", "negative_test_query334_00013", "negative_test_query336_00001", "negative_test_query119_00005", "negative_test_query334_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query314", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.7860137352654724, "recall@5": 0.23529411764705882}, "query_id": "test_query314", "ranked_ids": ["apositive_test_query314_00009", "negative_test_query146_00002", "apositive_test_query314_00001", "apositive_test_query314_00011", "apositive_test_query314_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query315", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.20210734650054757, "recall@5": 0.3333333333333333}, "query_id": "test_query315", "ranked_ids": ["apositive_test_query319_00005", "apositive_test_query82_00005", "negative_test_query315_00009", "apositive_test_query315_00002", "negative_test_query333_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query316", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.6131471927654584, "recall@5": 0.5}, "query_id": "test_query316", "ranked_ids": ["apositive_test_query316_00001", "negative_test_query301_00005", "negative_test_query316_00000", "negative_test_query316_00002", "negative_test_query316_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query317", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query317", "ranked_ids": ["negative_test_query317_00010", "negative_test_query331_00009", "negative_test_query55_00014", "negative_test_query317_00017", "negative_test_query317_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query318", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5413996682199069, "recall@5": 0.5}, "query_id": "test_query318", "ranked_ids": ["apositive_test_query318_00002", "negative_test_query318_00002", "negative_test_query318_00004", "negative_test_query73_00004", "apositive_test_query318_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query319", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.30078518014914984, "recall@5": 0.2222222222222222}, "query_id": "test_query319", "ranked_ids": ["negative_test_query319_00003", "negative_test_query82_00003", "apositive_test_query319_00004", "apositive_test_query82_00004", "apositive_test_query319_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query32", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query32", "ranked_ids": ["apositive_test_query196_00000", "apositive_test_query336_00001", "negative_test_query211_00016", "negative_test_query55_00011", "negative_test_query55_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query320", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query320", "ranked_ids": ["negative_test_query258_00012", "negative_test_query53_00014", "negative_test_query193_00006", "negative_test_query320_00005", "negative_test_query131_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query321", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.38685280723454163, "recall@5": 1.0}, "query_id": "test_query321", "ranked_ids": ["negative_test_query321_00002", "negative_test_query321_00010", "negative_test_query136_00004", "negative_test_query210_00016", "apositive_test_query321_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query322", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.3156484524795145, "recall@5": 0.2222222222222222}, "query_id": "test_query322", "ranked_ids": ["negative_test_query73_00008", "negative_test_query318_00002", "apositive_test_query322_00003", "apositive_test_query322_00005", "negative_test_query322_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query323", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.14285714285714285}, "query_id": "test_query323", "ranked_ids": ["negative_test_query255_00002", "negative_test_query323_00006", "negative_test_query209_00004", "apositive_test_query323_00006", "negative_test_query75_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query324", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.2960819109658652, "recall@5": 0.3333333333333333}, "query_id": "test_query324", "ranked_ids": ["negative_test_query218_00003", "apositive_test_query324_00000", "negative_test_query324_00014", "apositive_test_query308_00002", "negative_test_query324_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query325", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query325", "ranked_ids": ["negative_test_query293_00015", "apositive_test_query293_00002", "negative_test_query325_00007", "negative_test_query15_00016", "negative_test_query15_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query326", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.36005461457723403, "recall@5": 0.2222222222222222}, "query_id": "test_query326", "ranked_ids": ["negative_test_query265_00007", "apositive_test_query326_00000", "negative_test_query178_00012", "apositive_test_query326_00001", "negative_test_query17_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query327", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query327", "ranked_ids": ["negative_test_query271_00007", "negative_test_query327_00000", "negative_test_query227_00009", "negative_test_query327_00014", "negative_test_query327_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query328", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.1111111111111111}, "query_id": "test_query328", "ranked_ids": ["apositive_test_query40_00003", "apositive_test_query328_00004", "negative_test_query328_00003", "negative_test_query222_00016", "negative_test_query328_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query329", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query329", "ranked_ids": ["negative_test_query329_00006", "negative_test_query329_00011", "apositive_test_query7_00001", "negative_test_query35_00000", "negative_test_query329_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query33", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.1}, "query_id": "test_query33", "ranked_ids": ["negative_test_query154_00006", "apositive_test_query33_00004", "apositive_test_query154_00007", "negative_test_query154_00004", "apositive_test_query284_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query330", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query330", "ranked_ids": ["negative_test_query330_00001", "negative_test_query288_00006", "negative_test_query330_00009", "negative_test_query330_00000", "negative_test_query330_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query331", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.4776237035032179, "recall@5": 0.6666666666666666}, "query_id": "test_query331", "ranked_ids": ["negative_test_query214_00015", "apositive_test_query331_00002", "negative_test_query331_00008", "negative_test_query303_00001", "apositive_test_query331_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query332", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.2}, "query_id": "test_query332", "ranked_ids": ["apositive_test_query332_00001", "negative_test_query202_00002", "negative_test_query332_00002", "apositive_test_query348_00008", "negative_test_query115_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query333", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.16666666666666666}, "query_id": "test_query333", "ranked_ids": ["apositive_test_query333_00000", "apositive_test_query254_00005", "negative_test_query73_00008", "negative_test_query136_00003", "negative_test_query128_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query334", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query334", "ranked_ids": ["negative_test_query209_00003", "negative_test_query111_00017", "negative_test_query213_00009", "negative_test_query334_00010", "negative_test_query170_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query335", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.2}, "query_id": "test_query335", "ranked_ids": ["negative_test_query335_00009", "apositive_test_query335_00002", "negative_test_query210_00006", "negative_test_query321_00013", "negative_test_query335_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query336", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query336", "ranked_ids": ["apositive_test_query50_00012", "apositive_test_query38_00005", "apositive_test_query113_00005", "negative_test_query336_00015", "negative_test_query247_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query337", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query337", "ranked_ids": ["negative_test_query200_00015", "apositive_test_query38_00003", "apositive_test_query38_00000", "negative_test_query267_00001", "negative_test_query337_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query338", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.06666666666666667}, "query_id": "test_query338", "ranked_ids": ["negative_test_query119_00003", "apositive_test_query338_00001", "negative_test_query81_00004", "apositive_test_query114_00005", "negative_test_query81_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query339", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.23719771276929622, "recall@5": 0.5}, "query_id": "test_query339", "ranked_ids": ["apositive_test_query96_00002", "negative_test_query339_00005", "negative_test_query339_00007", "negative_test_query96_00004", "apositive_test_query339_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query34", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query34", "ranked_ids": ["apositive_test_query352_00000", "negative_test_query34_00014", "negative_test_query352_00012", "negative_test_query34_00008", "negative_test_query352_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query340", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3391602052736161, "recall@5": 0.2}, "query_id": "test_query340", "ranked_ids": ["apositive_test_query340_00002", "negative_test_query88_00007", "negative_test_query340_00001", "negative_test_query340_00006", "negative_test_query88_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query341", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query341", "ranked_ids": ["negative_test_query341_00005", "negative_test_query96_00007", "negative_test_query341_00009", "negative_test_query101_00003", "negative_test_query341_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query342", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query342", "ranked_ids": ["negative_test_query342_00016", "negative_test_query68_00005", "negative_test_query325_00015", "negative_test_query136_00003", "negative_test_query324_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query343", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query343", "ranked_ids": ["negative_test_query307_00006", "negative_test_query227_00010", "apositive_test_query279_00003", "apositive_test_query307_00008", "negative_test_query225_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query344", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query344", "ranked_ids": ["negative_test_query282_00004", "negative_test_query282_00001", "negative_test_query344_00002", "negative_test_query344_00004", "negative_test_query344_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query345", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query345", "ranked_ids": ["negative_test_query90_00003", "negative_test_query9_00002", "negative_test_query130_00004", "negative_test_query345_00011", "negative_test_query345_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query346", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query346", "ranked_ids": ["negative_test_query19_00012", "negative_test_query346_00008", "negative_test_query346_00010", "negative_test_query15_00008", "negative_test_query346_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query347", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.18154179253735267, "recall@5": 0.3333333333333333}, "query_id": "test_query347", "ranked_ids": ["negative_test_query347_00006", "negative_test_query251_00007", "negative_test_query347_00015", "negative_test_query347_00002", "apositive_test_query347_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query348", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.4703652827859579, "recall@5": 0.2}, "query_id": "test_query348", "ranked_ids": ["apositive_test_query348_00004", "negative_test_query297_00017", "apositive_test_query115_00002", "negative_test_query351_00010", "apositive_test_query348_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query349", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query349", "ranked_ids": ["apositive_test_query137_00002", "apositive_test_query42_00004", "negative_test_query74_00001", "apositive_test_query253_00008", "apositive_test_query74_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query35", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.3903800499921017, "recall@5": 0.25}, "query_id": "test_query35", "ranked_ids": ["apositive_test_query35_00003", "negative_test_query262_00002", "negative_test_query198_00011", "negative_test_query101_00006", "negative_test_query205_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query350", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query350", "ranked_ids": ["negative_test_query120_00007", "negative_test_query350_00013", "negative_test_query63_00000", "negative_test_query192_00005", "apositive_test_query33_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query351", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query351", "ranked_ids": ["apositive_test_query218_00004", "negative_test_query8_00003", "negative_test_query53_00011", "negative_test_query156_00006", "apositive_test_query60_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query352", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.4703652827859579, "recall@5": 0.4}, "query_id": "test_query352", "ranked_ids": ["apositive_test_query352_00000", "negative_test_query34_00014", "negative_test_query34_00008", "negative_test_query352_00010", "apositive_test_query352_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query353", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.06666666666666667}, "query_id": "test_query353", "ranked_ids": ["apositive_test_query204_00012", "negative_test_query353_00002", "apositive_test_query204_00000", "apositive_test_query353_00002", "apositive_test_query204_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query354", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.6843515475204855, "recall@5": 0.375}, "query_id": "test_query354", "ranked_ids": ["apositive_test_query354_00006", "apositive_test_query354_00004", "negative_test_query354_00004", "negative_test_query309_00013", "apositive_test_query354_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query355", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.1111111111111111}, "query_id": "test_query355", "ranked_ids": ["apositive_test_query100_00001", "negative_test_query355_00003", "negative_test_query264_00003", "apositive_test_query355_00005", "negative_test_query355_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query356", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.1111111111111111}, "query_id": "test_query356", "ranked_ids": ["apositive_test_query30_00005", "apositive_test_query356_00004", "negative_test_query253_00001", "negative_test_query15_00006", "negative_test_query102_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query357", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query357", "ranked_ids": ["apositive_test_query119_00002", "apositive_test_query221_00000", "apositive_test_query32_00002", "negative_test_query196_00001", "apositive_test_query119_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query358", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.3156484524795145, "recall@5": 0.2}, "query_id": "test_query358", "ranked_ids": ["apositive_test_query309_00000", "apositive_test_query161_00000", "apositive_test_query358_00000", "apositive_test_query358_00003", "apositive_test_query11_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query359", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query359", "ranked_ids": ["apositive_test_query0_00010", "apositive_test_query0_00000", "apositive_test_query0_00006", "apositive_test_query0_00003", "negative_test_query359_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query36", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query36", "ranked_ids": ["apositive_test_query286_00005", "negative_test_query112_00013", "negative_test_query80_00007", "negative_test_query112_00015", "apositive_test_query106_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query360", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.6131471927654584, "recall@5": 0.5}, "query_id": "test_query360", "ranked_ids": ["apositive_test_query360_00000", "negative_test_query360_00009", "negative_test_query189_00013", "apositive_test_query152_00002", "negative_test_query240_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query37", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query37", "ranked_ids": ["negative_test_query57_00009", "negative_test_query20_00003", "apositive_test_query177_00000", "apositive_test_query158_00007", "negative_test_query152_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query38", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.5087403079104241, "recall@5": 0.25}, "query_id": "test_query38", "ranked_ids": ["apositive_test_query38_00002", "apositive_test_query247_00005", "apositive_test_query38_00007", "negative_test_query267_00016", "negative_test_query267_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query39", "metrics": {"mrr@5": 0.2, "ndcg@5": 0.13120507751234178, "recall@5": 0.14285714285714285}, "query_id": "test_query39", "ranked_ids": ["negative_test_query232_00007", "negative_test_query39_00003", "negative_test_query232_00017", "negative_test_query331_00010", "apositive_test_query39_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query4", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.34519134224686937, "recall@5": 0.14285714285714285}, "query_id": "test_query4", "ranked_ids": ["negative_test_query266_00009", "apositive_test_query4_00010", "negative_test_query4_00002", "apositive_test_query174_00000", "apositive_test_query4_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query40", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query40", "ranked_ids": ["negative_test_query40_00012", "negative_test_query40_00010", "negative_test_query40_00006", "negative_test_query40_00005", "negative_test_query214_00015"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query41", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query41", "ranked_ids": ["negative_test_query282_00003", "negative_test_query41_00000", "negative_test_query98_00000", "negative_test_query41_00011", "negative_test_query41_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query42", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.36005461457723403, "recall@5": 0.25}, "query_id": "test_query42", "ranked_ids": ["negative_test_query250_00010", "apositive_test_query42_00003", "negative_test_query42_00000", "apositive_test_query42_00004", "apositive_test_query162_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query43", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.125}, "query_id": "test_query43", "ranked_ids": ["apositive_test_query146_00004", "apositive_test_query314_00009", "negative_test_query146_00002", "apositive_test_query43_00000", "negative_test_query43_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query44", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query44", "ranked_ids": ["negative_test_query293_00015", "negative_test_query21_00009", "negative_test_query330_00009", "negative_test_query141_00007", "negative_test_query190_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query45", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query45", "ranked_ids": ["negative_test_query143_00004", "negative_test_query45_00009", "negative_test_query45_00003", "negative_test_query143_00011", "negative_test_query342_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query46", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query46", "ranked_ids": ["apositive_test_query79_00005", "negative_test_query213_00012", "negative_test_query108_00003", "negative_test_query168_00003", "negative_test_query79_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query47", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query47", "ranked_ids": ["apositive_test_query331_00002", "negative_test_query47_00013", "negative_test_query325_00007", "negative_test_query144_00018", "negative_test_query47_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query48", "metrics": {"mrr@5": 1.0, "ndcg@5": 1.0, "recall@5": 1.0}, "query_id": "test_query48", "ranked_ids": ["apositive_test_query48_00000", "negative_test_query103_00012", "negative_test_query48_00018", "apositive_test_query307_00002", "negative_test_query48_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query49", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query49", "ranked_ids": ["negative_test_query342_00005", "negative_test_query52_00011", "negative_test_query15_00008", "negative_test_query156_00010", "apositive_test_query204_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query5", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query5", "ranked_ids": ["negative_test_query5_00004", "negative_test_query5_00007", "negative_test_query275_00001", "negative_test_query5_00014", "negative_test_query5_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query50", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.7227265726449519, "recall@5": 0.17647058823529413}, "query_id": "test_query50", "ranked_ids": ["apositive_test_query50_00006", "apositive_test_query50_00012", "apositive_test_query50_00015", "apositive_test_query186_00005", "apositive_test_query131_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query51", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.1}, "query_id": "test_query51", "ranked_ids": ["negative_test_query51_00000", "negative_test_query97_00001", "apositive_test_query51_00006", "negative_test_query51_00004", "negative_test_query209_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query52", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query52", "ranked_ids": ["negative_test_query215_00000", "negative_test_query257_00003", "apositive_test_query257_00003", "apositive_test_query286_00014", "apositive_test_query257_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query53", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query53", "ranked_ids": ["negative_test_query258_00018", "negative_test_query132_00015", "negative_test_query150_00014", "negative_test_query258_00013", "negative_test_query169_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query54", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query54", "ranked_ids": ["negative_test_query44_00007", "negative_test_query54_00006", "apositive_test_query86_00000", "apositive_test_query335_00002", "apositive_test_query51_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query55", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query55", "ranked_ids": ["negative_test_query55_00010", "negative_test_query55_00003", "negative_test_query55_00012", "negative_test_query55_00011", "negative_test_query234_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query56", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query56", "ranked_ids": ["negative_test_query46_00014", "negative_test_query56_00006", "apositive_test_query46_00001", "apositive_test_query238_00001", "negative_test_query56_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query57", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.1111111111111111}, "query_id": "test_query57", "ranked_ids": ["negative_test_query313_00012", "negative_test_query334_00013", "apositive_test_query57_00008", "apositive_test_query203_00000", "negative_test_query334_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query58", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query58", "ranked_ids": ["apositive_test_query311_00004", "negative_test_query308_00004", "negative_test_query58_00015", "negative_test_query27_00017", "negative_test_query240_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query59", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query59", "ranked_ids": ["negative_test_query116_00000", "negative_test_query59_00011", "negative_test_query59_00014", "negative_test_query59_00015", "negative_test_query202_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query6", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.23463936301137822, "recall@5": 0.3333333333333333}, "query_id": "test_query6", "ranked_ids": ["negative_test_query179_00007", "negative_test_query6_00012", "apositive_test_query6_00000", "negative_test_query252_00004", "apositive_test_query109_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query60", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.48522855511632257, "recall@5": 0.125}, "query_id": "test_query60", "ranked_ids": ["apositive_test_query60_00011", "negative_test_query31_00004", "apositive_test_query218_00004", "apositive_test_query60_00006", "apositive_test_query285_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query61", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.19519002499605084, "recall@5": 0.25}, "query_id": "test_query61", "ranked_ids": ["apositive_test_query173_00004", "negative_test_query327_00016", "apositive_test_query61_00003", "negative_test_query61_00002", "negative_test_query129_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query62", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.2640681225725909, "recall@5": 0.5}, "query_id": "test_query62", "ranked_ids": ["negative_test_query62_00003", "negative_test_query62_00002", "negative_test_query62_00000", "apositive_test_query62_00001", "negative_test_query219_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query63", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.4703652827859579, "recall@5": 0.2222222222222222}, "query_id": "test_query63", "ranked_ids": ["apositive_test_query63_00006", "negative_test_query321_00002", "negative_test_query297_00006", "negative_test_query63_00003", "apositive_test_query63_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query64", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query64", "ranked_ids": ["negative_test_query120_00000", "apositive_test_query160_00003", "negative_test_query149_00001", "negative_test_query120_00003", "negative_test_query64_00016"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query65", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query65", "ranked_ids": ["negative_test_query195_00009", "negative_test_query335_00002", "negative_test_query65_00000", "negative_test_query268_00012", "apositive_test_query356_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query66", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.34519134224686937, "recall@5": 0.3333333333333333}, "query_id": "test_query66", "ranked_ids": ["negative_test_query9_00012", "apositive_test_query66_00004", "apositive_test_query20_00005", "negative_test_query188_00006", "apositive_test_query66_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query67", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query67", "ranked_ids": ["apositive_test_query286_00004", "apositive_test_query286_00010", "negative_test_query123_00008", "negative_test_query67_00007", "negative_test_query260_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query68", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.2960819109658652, "recall@5": 0.3333333333333333}, "query_id": "test_query68", "ranked_ids": ["negative_test_query330_00009", "apositive_test_query68_00000", "negative_test_query288_00004", "negative_test_query330_00011", "negative_test_query68_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query69", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.639945385422766, "recall@5": 0.15789473684210525}, "query_id": "test_query69", "ranked_ids": ["apositive_test_query69_00010", "apositive_test_query257_00003", "apositive_test_query69_00005", "negative_test_query18_00010", "apositive_test_query69_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query7", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query7", "ranked_ids": ["apositive_test_query159_00001", "negative_test_query264_00013", "apositive_test_query38_00003", "negative_test_query159_00010", "negative_test_query7_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query70", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.24630238874073, "recall@5": 0.25}, "query_id": "test_query70", "ranked_ids": ["negative_test_query227_00006", "apositive_test_query70_00001", "negative_test_query227_00010", "apositive_test_query227_00000", "negative_test_query348_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query71", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query71", "ranked_ids": ["negative_test_query71_00002", "negative_test_query71_00015", "apositive_test_query30_00005", "negative_test_query71_00009", "negative_test_query71_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query72", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query72", "ranked_ids": ["negative_test_query17_00009", "negative_test_query72_00015", "negative_test_query72_00006", "negative_test_query72_00003", "negative_test_query72_00007"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query73", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query73", "ranked_ids": ["negative_test_query73_00013", "apositive_test_query333_00000", "negative_test_query73_00008", "negative_test_query233_00015", "negative_test_query128_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query74", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.16812753627111746, "recall@5": 0.25}, "query_id": "test_query74", "ranked_ids": ["apositive_test_query60_00006", "negative_test_query74_00001", "apositive_test_query113_00001", "apositive_test_query74_00002", "negative_test_query74_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query75", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.14285714285714285}, "query_id": "test_query75", "ranked_ids": ["negative_test_query99_00006", "apositive_test_query75_00001", "negative_test_query209_00004", "negative_test_query67_00017", "negative_test_query75_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query76", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.36005461457723403, "recall@5": 0.2857142857142857}, "query_id": "test_query76", "ranked_ids": ["negative_test_query76_00009", "apositive_test_query76_00000", "negative_test_query147_00007", "apositive_test_query76_00002", "negative_test_query181_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query77", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query77", "ranked_ids": ["negative_test_query329_00015", "negative_test_query77_00013", "negative_test_query342_00007", "negative_test_query98_00014", "negative_test_query301_00003"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query78", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query78", "ranked_ids": ["negative_test_query101_00008", "negative_test_query61_00007", "negative_test_query214_00010", "apositive_test_query38_00003", "apositive_test_query326_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query79", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.16666666666666666}, "query_id": "test_query79", "ranked_ids": ["negative_test_query38_00007", "negative_test_query313_00013", "apositive_test_query79_00004", "apositive_test_query295_00001", "negative_test_query157_00000"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query8", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query8", "ranked_ids": ["apositive_test_query192_00002", "negative_test_query329_00005", "negative_test_query8_00000", "negative_test_query8_00014", "negative_test_query96_00010"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query80", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query80", "ranked_ids": ["negative_test_query289_00005", "apositive_test_query179_00003", "negative_test_query287_00005", "negative_test_query80_00009", "negative_test_query80_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query81", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.34519134224686937, "recall@5": 0.2222222222222222}, "query_id": "test_query81", "ranked_ids": ["negative_test_query81_00005", "apositive_test_query81_00003", "negative_test_query81_00004", "negative_test_query145_00004", "apositive_test_query81_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query82", "metrics": {"mrr@5": 0.25, "ndcg@5": 0.14606834984270645, "recall@5": 0.1111111111111111}, "query_id": "test_query82", "ranked_ids": ["negative_test_query319_00003", "negative_test_query82_00003", "apositive_test_query319_00004", "apositive_test_query82_00004", "apositive_test_query319_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query83", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query83", "ranked_ids": ["apositive_test_query146_00014", "apositive_test_query168_00000", "negative_test_query279_00012", "negative_test_query168_00007", "negative_test_query287_00004"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query84", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query84", "ranked_ids": ["negative_test_query84_00013", "negative_test_query268_00009", "negative_test_query84_00006", "negative_test_query84_00005", "negative_test_query84_00017"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query85", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.4144299250118475, "recall@5": 0.5}, "query_id": "test_query85", "ranked_ids": ["apositive_test_query286_00004", "apositive_test_query85_00002", "apositive_test_query81_00008", "apositive_test_query85_00000", "negative_test_query85_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query86", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.24630238874073, "recall@5": 0.25}, "query_id": "test_query86", "ranked_ids": ["negative_test_query86_00001", "apositive_test_query86_00000", "negative_test_query86_00003", "negative_test_query86_00000", "negative_test_query86_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query87", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.6131471927654584, "recall@5": 0.5}, "query_id": "test_query87", "ranked_ids": ["apositive_test_query87_00001", "negative_test_query87_00002", "negative_test_query182_00017", "negative_test_query200_00011", "negative_test_query315_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query88", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.6131471927654584, "recall@5": 0.5}, "query_id": "test_query88", "ranked_ids": ["apositive_test_query88_00001", "negative_test_query88_00002", "negative_test_query88_00010", "negative_test_query88_00012", "negative_test_query105_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query89", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query89", "ranked_ids": ["negative_test_query89_00005", "negative_test_query68_00004", "negative_test_query233_00005", "apositive_test_query255_00000", "negative_test_query89_00008"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query9", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query9", "ranked_ids": ["negative_test_query9_00018", "negative_test_query9_00013", "negative_test_query9_00005", "negative_test_query9_00016", "negative_test_query9_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query90", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query90", "ranked_ids": ["negative_test_query227_00006", "negative_test_query90_00012", "negative_test_query227_00001", "negative_test_query127_00001", "negative_test_query90_00002"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query91", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query91", "ranked_ids": ["negative_test_query358_00004", "apositive_test_query342_00001", "apositive_test_query11_00006", "negative_test_query279_00011", "negative_test_query303_00011"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query92", "metrics": {"mrr@5": 0.3333333333333333, "ndcg@5": 0.16958010263680806, "recall@5": 0.16666666666666666}, "query_id": "test_query92", "ranked_ids": ["apositive_test_query100_00001", "negative_test_query122_00018", "apositive_test_query92_00001", "apositive_test_query99_00003", "negative_test_query283_00009"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query93", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.14285714285714285}, "query_id": "test_query93", "ranked_ids": ["apositive_test_query243_00006", "apositive_test_query93_00003", "apositive_test_query335_00003", "negative_test_query108_00003", "negative_test_query280_00012"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query94", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.5147714448836774, "recall@5": 0.375}, "query_id": "test_query94", "ranked_ids": ["apositive_test_query127_00010", "apositive_test_query94_00002", "apositive_test_query94_00006", "negative_test_query94_00002", "apositive_test_query94_00005"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query95", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query95", "ranked_ids": ["negative_test_query244_00010", "apositive_test_query292_00001", "negative_test_query95_00008", "apositive_test_query177_00000", "negative_test_query95_00006"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query96", "metrics": {"mrr@5": 1.0, "ndcg@5": 0.46927872602275644, "recall@5": 0.3333333333333333}, "query_id": "test_query96", "ranked_ids": ["apositive_test_query96_00002", "negative_test_query339_00005", "negative_test_query339_00007", "negative_test_query96_00004", "apositive_test_query339_00001"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query97", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query97", "ranked_ids": ["negative_test_query51_00009", "negative_test_query113_00009", "negative_test_query113_00006", "negative_test_query209_00008", "negative_test_query16_00014"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query98", "metrics": {"mrr@5": 0.0, "ndcg@5": 0.0, "recall@5": 0.0}, "query_id": "test_query98", "ranked_ids": ["negative_test_query98_00018", "negative_test_query98_00011", "negative_test_query98_00005", "apositive_test_query14_00002", "negative_test_query23_00013"], "valid": true} +{"algorithm": "answer_confidence", "fixture_id": "askubuntu-bm25-test-test_query99", "metrics": {"mrr@5": 0.5, "ndcg@5": 0.21398626473452756, "recall@5": 0.1111111111111111}, "query_id": "test_query99", "ranked_ids": ["apositive_test_query193_00010", "apositive_test_query99_00001", "apositive_test_query333_00003", "apositive_test_query253_00003", "apositive_test_query153_00012"], "valid": true} diff --git a/benchmark-results/askubuntu-bm25-top20-answer-confidence.json b/benchmark-results/askubuntu-bm25-top20-answer-confidence.json new file mode 100644 index 0000000..3ca6d1c --- /dev/null +++ b/benchmark-results/askubuntu-bm25-top20-answer-confidence.json @@ -0,0 +1,6550 @@ +{ + "aggregate": [ + { + "algorithm": "answer_confidence", + "case_count": 361, + "invalid_rate": 0.0, + "metrics": { + "mrr@5": 0.28619575253924284, + "ndcg@5": 0.17217008703277661, + "recall@5": 0.1434797614811208 + }, + "valid_count": 361 + } + ], + "algorithm": [ + "answer_confidence" + ], + "benchmark_type": "reranking_with_first_stage_candidates", + "cache_dir": "/Users/skiiwoo/Documents/ranksmith/.benchmark-cache/askubuntu-bm25", + "call_estimates": { + "answer_confidence": 7220 + }, + "candidate_count": 20, + "candidate_strategy": "candidate_file", + "candidates": "benchmark-results/pyserini/askubuntu-bm25-top20.trec", + "case_count": 361, + "checkpoint_output": "benchmark-results/askubuntu-bm25-top20-answer-confidence.checkpoint.jsonl", + "dataset": "benchmark-cache", + "dataset_name": "askubuntu-bm25", + "fixture": null, + "method_settings": { + "answer_confidence": { + "candidate_count": 20, + "stride": 10, + "top_k_early_stop": false, + "window_size": 20 + } + }, + "passes": 10, + "per_query": [ + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00006", + "apositive_test_query0_00003", + "negative_test_query359_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query1", + "ranked_ids": [ + "negative_test_query245_00006", + "apositive_test_query348_00003", + "negative_test_query120_00006", + "negative_test_query61_00002", + "negative_test_query27_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4776237035032179, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query10", + "ranked_ids": [ + "negative_test_query10_00010", + "apositive_test_query10_00001", + "negative_test_query104_00011", + "negative_test_query10_00000", + "apositive_test_query10_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query100", + "ranked_ids": [ + "negative_test_query100_00010", + "negative_test_query290_00015", + "negative_test_query100_00009", + "apositive_test_query181_00000", + "negative_test_query100_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query196_00000", + "apositive_test_query336_00001", + "negative_test_query61_00002", + "negative_test_query173_00013", + "negative_test_query241_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query209_00006", + "negative_test_query108_00009", + "negative_test_query219_00004", + "negative_test_query131_00009", + "negative_test_query205_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "apositive_test_query103_00005", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query307_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "negative_test_query104_00010", + "negative_test_query104_00016", + "negative_test_query48_00010", + "negative_test_query104_00004", + "apositive_test_query48_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query105", + "ranked_ids": [ + "negative_test_query313_00013", + "negative_test_query105_00005", + "negative_test_query267_00011", + "negative_test_query119_00005", + "negative_test_query168_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query106", + "ranked_ids": [ + "negative_test_query326_00008", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "apositive_test_query42_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "negative_test_query146_00000", + "negative_test_query296_00001", + "negative_test_query31_00013", + "apositive_test_query142_00002", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query108", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query108_00005", + "negative_test_query57_00009", + "negative_test_query311_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query109", + "ranked_ids": [ + "negative_test_query109_00003", + "negative_test_query6_00006", + "negative_test_query109_00000", + "negative_test_query51_00004", + "negative_test_query109_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00001", + "negative_test_query226_00010", + "negative_test_query309_00002", + "negative_test_query358_00005", + "negative_test_query11_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "apositive_test_query11_00002", + "negative_test_query110_00014", + "apositive_test_query330_00001", + "negative_test_query17_00008", + "negative_test_query110_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query243_00001", + "negative_test_query277_00010", + "negative_test_query111_00018", + "negative_test_query292_00010", + "negative_test_query277_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "apositive_test_query286_00005", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query112_00015", + "apositive_test_query106_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query113", + "ranked_ids": [ + "negative_test_query38_00003", + "negative_test_query113_00003", + "apositive_test_query113_00001", + "negative_test_query105_00000", + "apositive_test_query186_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query114", + "ranked_ids": [ + "negative_test_query262_00007", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "negative_test_query106_00009", + "negative_test_query114_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query115", + "ranked_ids": [ + "negative_test_query205_00004", + "apositive_test_query115_00000", + "apositive_test_query121_00001", + "apositive_test_query115_00002", + "negative_test_query139_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "negative_test_query116_00000", + "apositive_test_query116_00000", + "negative_test_query59_00007", + "negative_test_query116_00007", + "apositive_test_query49_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00003", + "negative_test_query63_00001", + "apositive_test_query117_00007", + "negative_test_query117_00003", + "negative_test_query175_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "apositive_test_query340_00002", + "negative_test_query88_00007", + "apositive_test_query174_00003", + "apositive_test_query174_00005", + "apositive_test_query174_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query119", + "ranked_ids": [ + "negative_test_query119_00005", + "apositive_test_query79_00004", + "negative_test_query157_00007", + "negative_test_query359_00005", + "negative_test_query152_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query234_00014", + "negative_test_query12_00003", + "negative_test_query12_00014", + "negative_test_query12_00007", + "negative_test_query12_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00008", + "apositive_test_query352_00000", + "negative_test_query303_00018", + "negative_test_query120_00002", + "negative_test_query120_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00001", + "negative_test_query121_00012", + "negative_test_query121_00007", + "apositive_test_query121_00000", + "negative_test_query115_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query64_00017", + "negative_test_query86_00001", + "apositive_test_query173_00001", + "negative_test_query134_00007", + "apositive_test_query122_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query123", + "ranked_ids": [ + "apositive_test_query123_00002", + "negative_test_query123_00002", + "negative_test_query123_00008", + "negative_test_query123_00006", + "negative_test_query269_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "negative_test_query124_00003", + "negative_test_query42_00006", + "apositive_test_query13_00004", + "negative_test_query93_00010", + "apositive_test_query162_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query301_00012", + "apositive_test_query286_00009", + "negative_test_query125_00013", + "negative_test_query125_00014", + "negative_test_query140_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query126", + "ranked_ids": [ + "negative_test_query131_00009", + "apositive_test_query330_00001", + "apositive_test_query5_00002", + "apositive_test_query151_00003", + "negative_test_query126_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00009", + "negative_test_query267_00009", + "apositive_test_query127_00012", + "apositive_test_query94_00005", + "apositive_test_query127_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query128", + "ranked_ids": [ + "apositive_test_query128_00001", + "negative_test_query75_00012", + "apositive_test_query216_00002", + "negative_test_query128_00011", + "apositive_test_query21_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.125 + }, + "query_id": "test_query129", + "ranked_ids": [ + "negative_test_query63_00006", + "apositive_test_query249_00003", + "negative_test_query16_00007", + "negative_test_query187_00012", + "apositive_test_query129_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query13", + "ranked_ids": [ + "negative_test_query13_00004", + "apositive_test_query13_00010", + "apositive_test_query13_00008", + "apositive_test_query13_00012", + "negative_test_query13_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query130", + "ranked_ids": [ + "negative_test_query90_00014", + "negative_test_query130_00001", + "negative_test_query289_00004", + "negative_test_query198_00003", + "negative_test_query284_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query131", + "ranked_ids": [ + "apositive_test_query131_00001", + "negative_test_query131_00002", + "negative_test_query313_00008", + "negative_test_query168_00013", + "negative_test_query184_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "apositive_test_query3_00004", + "negative_test_query132_00012", + "negative_test_query132_00018", + "negative_test_query81_00005", + "negative_test_query304_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query129_00002", + "negative_test_query12_00014", + "apositive_test_query129_00007", + "apositive_test_query174_00011", + "negative_test_query206_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query134", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query108_00003", + "apositive_test_query238_00001", + "apositive_test_query276_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00001", + "negative_test_query297_00017", + "negative_test_query135_00007", + "negative_test_query135_00008", + "negative_test_query135_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query136", + "ranked_ids": [ + "apositive_test_query136_00002", + "apositive_test_query136_00011", + "apositive_test_query136_00000", + "apositive_test_query136_00010", + "apositive_test_query136_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query137", + "ranked_ids": [ + "negative_test_query149_00013", + "apositive_test_query137_00002", + "negative_test_query74_00001", + "apositive_test_query137_00000", + "negative_test_query262_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query138", + "ranked_ids": [ + "negative_test_query138_00015", + "apositive_test_query244_00005", + "negative_test_query138_00005", + "negative_test_query138_00014", + "negative_test_query138_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query139_00001", + "negative_test_query262_00002", + "negative_test_query198_00011", + "negative_test_query101_00006", + "apositive_test_query354_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query14", + "ranked_ids": [ + "apositive_test_query64_00000", + "apositive_test_query26_00006", + "apositive_test_query296_00005", + "negative_test_query14_00001", + "negative_test_query179_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00009", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00003", + "negative_test_query140_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query141", + "ranked_ids": [ + "negative_test_query245_00008", + "negative_test_query245_00016", + "negative_test_query308_00005", + "negative_test_query57_00002", + "apositive_test_query288_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query142", + "ranked_ids": [ + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query142_00007", + "apositive_test_query142_00002", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query143", + "ranked_ids": [ + "negative_test_query90_00014", + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query143_00011", + "negative_test_query143_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query166_00001", + "negative_test_query144_00012", + "negative_test_query144_00006", + "negative_test_query144_00001", + "negative_test_query144_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query145", + "ranked_ids": [ + "negative_test_query211_00013", + "negative_test_query230_00002", + "negative_test_query53_00012", + "negative_test_query112_00007", + "negative_test_query187_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.0625 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query146_00004", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "negative_test_query146_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query147", + "ranked_ids": [ + "negative_test_query17_00009", + "negative_test_query147_00013", + "negative_test_query147_00008", + "negative_test_query309_00014", + "negative_test_query200_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query210_00006", + "negative_test_query321_00013", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query148_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "negative_test_query149_00001", + "negative_test_query149_00013", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query149_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00014", + "negative_test_query15_00007", + "negative_test_query15_00004", + "negative_test_query325_00011", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query150", + "ranked_ids": [ + "negative_test_query301_00007", + "negative_test_query346_00008", + "negative_test_query139_00005", + "negative_test_query254_00003", + "negative_test_query350_00018" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query190_00014", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "negative_test_query151_00004", + "negative_test_query293_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query349_00001", + "negative_test_query262_00010", + "negative_test_query342_00005", + "negative_test_query237_00011", + "negative_test_query349_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.125 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00001", + "apositive_test_query153_00003", + "negative_test_query278_00006", + "negative_test_query112_00007", + "negative_test_query112_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query154", + "ranked_ids": [ + "negative_test_query245_00006", + "apositive_test_query33_00008", + "apositive_test_query20_00003", + "apositive_test_query358_00008", + "negative_test_query68_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00006", + "negative_test_query155_00005", + "negative_test_query99_00001", + "apositive_test_query155_00005", + "negative_test_query155_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "negative_test_query49_00005", + "apositive_test_query210_00001", + "negative_test_query283_00009", + "apositive_test_query156_00000", + "negative_test_query156_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "negative_test_query271_00003", + "apositive_test_query237_00004", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query152_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2 + }, + "query_id": "test_query158", + "ranked_ids": [ + "apositive_test_query158_00000", + "negative_test_query26_00005", + "negative_test_query26_00010", + "apositive_test_query158_00007", + "negative_test_query102_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query159", + "ranked_ids": [ + "negative_test_query264_00013", + "apositive_test_query38_00003", + "negative_test_query159_00016", + "apositive_test_query212_00003", + "negative_test_query122_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query16", + "ranked_ids": [ + "negative_test_query16_00009", + "negative_test_query308_00004", + "negative_test_query16_00011", + "negative_test_query16_00008", + "negative_test_query16_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query160", + "ranked_ids": [ + "apositive_test_query64_00000", + "negative_test_query271_00010", + "negative_test_query304_00007", + "negative_test_query330_00005", + "apositive_test_query160_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query161", + "ranked_ids": [ + "apositive_test_query3_00006", + "negative_test_query260_00004", + "apositive_test_query314_00004", + "apositive_test_query161_00008", + "apositive_test_query301_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.08333333333333333 + }, + "query_id": "test_query162", + "ranked_ids": [ + "apositive_test_query194_00009", + "negative_test_query162_00001", + "negative_test_query343_00012", + "negative_test_query177_00002", + "apositive_test_query162_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query58_00015", + "negative_test_query197_00002", + "negative_test_query265_00015", + "negative_test_query163_00011", + "negative_test_query202_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query146_00014", + "negative_test_query164_00015", + "negative_test_query245_00005", + "negative_test_query321_00011", + "negative_test_query100_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query327_00016", + "negative_test_query165_00007", + "negative_test_query86_00006", + "negative_test_query264_00016", + "negative_test_query165_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query166", + "ranked_ids": [ + "apositive_test_query224_00000", + "apositive_test_query166_00002", + "negative_test_query166_00001", + "negative_test_query165_00015", + "negative_test_query166_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query278_00010", + "apositive_test_query288_00005", + "negative_test_query141_00010", + "negative_test_query167_00008", + "negative_test_query167_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query168_00000", + "negative_test_query168_00007", + "negative_test_query260_00011", + "apositive_test_query168_00001", + "negative_test_query168_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "negative_test_query10_00013", + "negative_test_query72_00013", + "negative_test_query49_00017", + "apositive_test_query330_00001", + "apositive_test_query69_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00016", + "negative_test_query17_00005", + "negative_test_query17_00014", + "negative_test_query164_00006", + "negative_test_query17_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query170", + "ranked_ids": [ + "negative_test_query170_00001", + "negative_test_query227_00010", + "negative_test_query227_00002", + "apositive_test_query170_00000", + "apositive_test_query297_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query171", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query277_00007", + "negative_test_query346_00005", + "negative_test_query24_00007", + "negative_test_query171_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query172", + "ranked_ids": [ + "apositive_test_query124_00002", + "negative_test_query349_00001", + "negative_test_query173_00006", + "negative_test_query8_00000", + "negative_test_query350_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query64_00017", + "apositive_test_query173_00004", + "apositive_test_query173_00001", + "negative_test_query173_00006", + "negative_test_query222_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00003", + "negative_test_query19_00012", + "apositive_test_query174_00005", + "apositive_test_query174_00007", + "apositive_test_query174_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00000", + "negative_test_query204_00000", + "apositive_test_query50_00015", + "negative_test_query175_00013", + "negative_test_query175_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query144_00018", + "apositive_test_query92_00002", + "negative_test_query24_00014", + "negative_test_query195_00006", + "negative_test_query284_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query124_00001", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query177_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query178", + "ranked_ids": [ + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query274_00001", + "negative_test_query321_00017", + "negative_test_query178_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "apositive_test_query316_00001", + "negative_test_query191_00003", + "negative_test_query297_00000", + "negative_test_query206_00004", + "negative_test_query241_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query18", + "ranked_ids": [ + "negative_test_query88_00002", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00003", + "negative_test_query18_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query180", + "ranked_ids": [ + "negative_test_query246_00003", + "negative_test_query207_00007", + "negative_test_query180_00002", + "negative_test_query180_00007", + "apositive_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00005", + "negative_test_query181_00010", + "negative_test_query348_00007", + "apositive_test_query119_00011", + "negative_test_query181_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "negative_test_query182_00011", + "negative_test_query182_00018", + "negative_test_query342_00004", + "negative_test_query237_00006", + "negative_test_query182_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00008", + "negative_test_query7_00009", + "apositive_test_query7_00001", + "negative_test_query183_00010", + "negative_test_query7_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "apositive_test_query33_00004", + "negative_test_query156_00001", + "negative_test_query216_00008", + "negative_test_query216_00002", + "apositive_test_query316_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query185", + "ranked_ids": [ + "negative_test_query350_00007", + "negative_test_query185_00002", + "apositive_test_query307_00008", + "apositive_test_query103_00000", + "negative_test_query48_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "apositive_test_query79_00005", + "negative_test_query105_00001", + "negative_test_query29_00003", + "negative_test_query313_00005", + "apositive_test_query50_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query245_00006", + "apositive_test_query33_00008", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query358_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "negative_test_query112_00017", + "negative_test_query20_00006", + "apositive_test_query20_00009", + "negative_test_query344_00007", + "apositive_test_query188_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00004", + "apositive_test_query189_00002", + "negative_test_query189_00002", + "negative_test_query55_00003", + "negative_test_query140_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query19", + "ranked_ids": [ + "negative_test_query106_00007", + "negative_test_query292_00004", + "apositive_test_query174_00007", + "negative_test_query229_00002", + "negative_test_query141_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query234_00001", + "negative_test_query190_00013", + "negative_test_query233_00008", + "negative_test_query265_00004", + "negative_test_query205_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "negative_test_query191_00003", + "negative_test_query274_00002", + "negative_test_query241_00007", + "apositive_test_query191_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00002", + "negative_test_query192_00002", + "apositive_test_query192_00004", + "negative_test_query120_00007", + "negative_test_query350_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query193", + "ranked_ids": [ + "negative_test_query193_00006", + "apositive_test_query193_00001", + "apositive_test_query99_00003", + "apositive_test_query193_00009", + "negative_test_query146_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query262_00007", + "negative_test_query177_00007", + "negative_test_query95_00016", + "apositive_test_query194_00001", + "apositive_test_query194_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query195", + "ranked_ids": [ + "apositive_test_query195_00002", + "apositive_test_query251_00000", + "negative_test_query109_00008", + "negative_test_query199_00003", + "negative_test_query195_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query32_00004", + "apositive_test_query196_00003", + "apositive_test_query32_00011", + "negative_test_query139_00011", + "negative_test_query173_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query197", + "ranked_ids": [ + "apositive_test_query197_00000", + "apositive_test_query189_00001", + "negative_test_query163_00001", + "apositive_test_query34_00000", + "negative_test_query163_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query293_00011", + "negative_test_query198_00004", + "negative_test_query239_00007", + "negative_test_query293_00010", + "negative_test_query198_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query350_00007", + "negative_test_query199_00006", + "negative_test_query185_00002", + "negative_test_query199_00002", + "negative_test_query221_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query44_00008", + "negative_test_query2_00012", + "negative_test_query197_00004", + "negative_test_query348_00005", + "negative_test_query104_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query20_00009", + "apositive_test_query20_00003", + "negative_test_query66_00002", + "apositive_test_query66_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query200_00015", + "negative_test_query38_00005", + "negative_test_query264_00007", + "negative_test_query264_00006", + "negative_test_query200_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query201", + "ranked_ids": [ + "negative_test_query71_00002", + "negative_test_query0_00004", + "negative_test_query47_00013", + "negative_test_query209_00008", + "apositive_test_query30_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query235_00001", + "apositive_test_query61_00003", + "apositive_test_query206_00000", + "negative_test_query202_00004", + "apositive_test_query309_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query203", + "ranked_ids": [ + "negative_test_query130_00001", + "apositive_test_query30_00005", + "apositive_test_query284_00003", + "negative_test_query35_00004", + "negative_test_query203_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.11764705882352941 + }, + "query_id": "test_query204", + "ranked_ids": [ + "negative_test_query204_00000", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00008", + "negative_test_query353_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query31_00015", + "negative_test_query205_00007", + "negative_test_query115_00000", + "apositive_test_query256_00002", + "negative_test_query205_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query64_00016", + "negative_test_query206_00001", + "negative_test_query206_00018", + "negative_test_query206_00003", + "negative_test_query299_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00005", + "negative_test_query207_00007", + "negative_test_query207_00000", + "negative_test_query207_00003", + "negative_test_query207_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "negative_test_query128_00001", + "negative_test_query73_00008", + "apositive_test_query128_00002", + "negative_test_query128_00003", + "negative_test_query208_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query209", + "ranked_ids": [ + "negative_test_query313_00012", + "apositive_test_query119_00008", + "negative_test_query102_00016", + "apositive_test_query158_00004", + "negative_test_query26_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "negative_test_query21_00009", + "negative_test_query330_00000", + "negative_test_query234_00006", + "apositive_test_query128_00003", + "apositive_test_query128_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query288_00006", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query156_00015", + "negative_test_query202_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query211", + "ranked_ids": [ + "negative_test_query159_00006", + "negative_test_query211_00012", + "apositive_test_query101_00000", + "negative_test_query211_00000", + "negative_test_query211_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query212", + "ranked_ids": [ + "apositive_test_query326_00000", + "apositive_test_query98_00000", + "apositive_test_query11_00007", + "negative_test_query282_00010", + "negative_test_query344_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00002", + "negative_test_query138_00016", + "negative_test_query224_00015", + "negative_test_query213_00004", + "negative_test_query213_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "negative_test_query214_00015", + "apositive_test_query234_00001", + "apositive_test_query214_00000", + "negative_test_query214_00010", + "apositive_test_query160_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query215_00000", + "negative_test_query257_00001", + "negative_test_query215_00018" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query33_00004", + "negative_test_query216_00008", + "apositive_test_query216_00002", + "negative_test_query216_00009", + "negative_test_query246_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "negative_test_query28_00002", + "negative_test_query230_00002", + "negative_test_query53_00012", + "apositive_test_query217_00000", + "negative_test_query185_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query218", + "ranked_ids": [ + "apositive_test_query218_00004", + "negative_test_query218_00003", + "apositive_test_query60_00012", + "negative_test_query218_00000", + "negative_test_query274_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "negative_test_query165_00010", + "negative_test_query308_00006", + "negative_test_query24_00007", + "negative_test_query187_00015", + "negative_test_query219_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "negative_test_query22_00000", + "negative_test_query22_00012", + "negative_test_query22_00007", + "negative_test_query22_00008", + "negative_test_query66_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query220", + "ranked_ids": [ + "negative_test_query188_00007", + "apositive_test_query20_00009", + "negative_test_query344_00007", + "apositive_test_query220_00001", + "apositive_test_query188_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query174_00006", + "negative_test_query278_00004", + "negative_test_query277_00010", + "negative_test_query184_00001", + "negative_test_query171_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query222", + "ranked_ids": [ + "negative_test_query222_00014", + "apositive_test_query289_00002", + "apositive_test_query275_00008", + "negative_test_query275_00004", + "apositive_test_query275_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query51_00003", + "apositive_test_query259_00008", + "negative_test_query323_00005", + "negative_test_query47_00007", + "negative_test_query359_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query224", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query224_00011", + "negative_test_query98_00018", + "negative_test_query224_00014", + "apositive_test_query32_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00000", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "apositive_test_query225_00003", + "negative_test_query149_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query226", + "ranked_ids": [ + "negative_test_query226_00009", + "apositive_test_query226_00002", + "negative_test_query226_00010", + "negative_test_query226_00007", + "apositive_test_query226_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00006", + "apositive_test_query348_00004", + "negative_test_query321_00004", + "apositive_test_query227_00001", + "apositive_test_query70_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query228", + "ranked_ids": [ + "apositive_test_query322_00003", + "apositive_test_query228_00000", + "negative_test_query73_00016", + "negative_test_query165_00011", + "apositive_test_query254_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query146_00004", + "negative_test_query355_00001", + "apositive_test_query229_00000", + "negative_test_query94_00010", + "negative_test_query44_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query23", + "ranked_ids": [ + "apositive_test_query225_00000", + "negative_test_query356_00008", + "apositive_test_query146_00015", + "negative_test_query292_00013", + "apositive_test_query54_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query99_00003", + "apositive_test_query3_00006", + "negative_test_query230_00010", + "apositive_test_query3_00000", + "negative_test_query3_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query231", + "ranked_ids": [ + "negative_test_query201_00011", + "negative_test_query221_00006", + "negative_test_query345_00011", + "negative_test_query306_00006", + "apositive_test_query201_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "negative_test_query232_00013", + "negative_test_query232_00014", + "negative_test_query232_00011", + "negative_test_query232_00016", + "negative_test_query232_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query233", + "ranked_ids": [ + "negative_test_query130_00001", + "negative_test_query233_00006", + "negative_test_query233_00001", + "negative_test_query233_00005", + "apositive_test_query233_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00010", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query234_00011", + "negative_test_query214_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00003", + "apositive_test_query32_00004", + "negative_test_query235_00013", + "negative_test_query235_00011", + "negative_test_query235_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "negative_test_query236_00002", + "negative_test_query236_00005", + "negative_test_query236_00000", + "negative_test_query236_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "apositive_test_query237_00001", + "negative_test_query237_00005", + "negative_test_query237_00006", + "apositive_test_query152_00004", + "negative_test_query237_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "negative_test_query238_00011", + "negative_test_query13_00003", + "negative_test_query111_00013", + "negative_test_query142_00011", + "negative_test_query102_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "apositive_test_query239_00000", + "negative_test_query239_00007", + "negative_test_query239_00003", + "negative_test_query167_00011", + "negative_test_query198_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "apositive_test_query106_00002", + "negative_test_query24_00002", + "negative_test_query24_00004", + "apositive_test_query24_00000", + "negative_test_query24_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "negative_test_query240_00010", + "negative_test_query240_00016", + "negative_test_query240_00018", + "negative_test_query297_00005", + "negative_test_query27_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query173_00013", + "negative_test_query241_00005", + "negative_test_query191_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query247_00005", + "apositive_test_query242_00001", + "negative_test_query291_00003", + "negative_test_query291_00005", + "apositive_test_query242_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query173_00004", + "apositive_test_query160_00003", + "negative_test_query243_00000", + "negative_test_query205_00007", + "negative_test_query243_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query244", + "ranked_ids": [ + "negative_test_query244_00010", + "negative_test_query244_00009", + "apositive_test_query244_00006", + "negative_test_query244_00008", + "negative_test_query244_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query293_00003", + "negative_test_query202_00004", + "negative_test_query135_00013", + "negative_test_query122_00002", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query6_00006", + "negative_test_query252_00003", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "apositive_test_query247_00005", + "apositive_test_query242_00001", + "negative_test_query291_00003", + "negative_test_query291_00005", + "negative_test_query186_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.25 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00001", + "apositive_test_query248_00003", + "negative_test_query248_00006", + "negative_test_query205_00017", + "apositive_test_query248_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query249", + "ranked_ids": [ + "negative_test_query249_00001", + "negative_test_query325_00015", + "negative_test_query301_00013", + "negative_test_query326_00005", + "apositive_test_query249_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query25", + "ranked_ids": [ + "negative_test_query328_00000", + "negative_test_query45_00014", + "apositive_test_query25_00008", + "negative_test_query25_00001", + "apositive_test_query25_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query250", + "ranked_ids": [ + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query192_00009", + "negative_test_query57_00009", + "negative_test_query64_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query251", + "ranked_ids": [ + "negative_test_query234_00006", + "negative_test_query251_00003", + "negative_test_query143_00001", + "negative_test_query56_00005", + "negative_test_query170_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query252", + "ranked_ids": [ + "apositive_test_query110_00001", + "negative_test_query252_00009", + "negative_test_query44_00011", + "negative_test_query321_00004", + "negative_test_query252_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query253", + "ranked_ids": [ + "negative_test_query124_00001", + "apositive_test_query162_00008", + "negative_test_query177_00000", + "apositive_test_query74_00002", + "apositive_test_query13_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query254", + "ranked_ids": [ + "negative_test_query21_00009", + "apositive_test_query254_00003", + "apositive_test_query254_00005", + "apositive_test_query254_00010", + "apositive_test_query333_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "negative_test_query246_00003", + "apositive_test_query255_00000", + "negative_test_query278_00000", + "negative_test_query255_00006", + "negative_test_query326_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.36331756126716835, + "recall@5": 0.5 + }, + "query_id": "test_query256", + "ranked_ids": [ + "negative_test_query183_00008", + "negative_test_query7_00009", + "apositive_test_query256_00002", + "apositive_test_query256_00003", + "apositive_test_query216_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query257", + "ranked_ids": [ + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00003", + "apositive_test_query286_00014", + "apositive_test_query257_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "negative_test_query320_00002", + "negative_test_query38_00002", + "negative_test_query162_00004", + "negative_test_query173_00010", + "negative_test_query167_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.08333333333333333 + }, + "query_id": "test_query259", + "ranked_ids": [ + "apositive_test_query259_00008", + "negative_test_query30_00009", + "negative_test_query213_00004", + "negative_test_query259_00006", + "negative_test_query47_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "apositive_test_query26_00004", + "negative_test_query26_00005", + "negative_test_query26_00008", + "negative_test_query119_00006", + "negative_test_query158_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "negative_test_query6_00009", + "negative_test_query6_00012", + "apositive_test_query6_00000", + "negative_test_query252_00004", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query261", + "ranked_ids": [ + "apositive_test_query261_00000", + "negative_test_query261_00009", + "negative_test_query261_00007", + "negative_test_query261_00005", + "negative_test_query118_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query262", + "ranked_ids": [ + "negative_test_query326_00008", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "apositive_test_query42_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query263", + "ranked_ids": [ + "apositive_test_query263_00002", + "apositive_test_query242_00011", + "apositive_test_query291_00010", + "negative_test_query263_00005", + "negative_test_query307_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "negative_test_query264_00004", + "negative_test_query264_00011", + "negative_test_query264_00007", + "apositive_test_query38_00003", + "negative_test_query264_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query265", + "ranked_ids": [ + "negative_test_query265_00002", + "apositive_test_query234_00001", + "negative_test_query165_00007", + "negative_test_query165_00016", + "negative_test_query190_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "negative_test_query266_00000", + "negative_test_query266_00009", + "negative_test_query266_00007", + "negative_test_query266_00001", + "negative_test_query141_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "negative_test_query30_00000", + "apositive_test_query250_00000", + "negative_test_query186_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query268", + "ranked_ids": [ + "negative_test_query195_00009", + "negative_test_query335_00002", + "apositive_test_query268_00004", + "negative_test_query268_00012", + "negative_test_query268_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query269", + "ranked_ids": [ + "negative_test_query269_00009", + "negative_test_query329_00004", + "apositive_test_query296_00005", + "apositive_test_query269_00000", + "negative_test_query329_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "apositive_test_query127_00010", + "negative_test_query27_00009", + "negative_test_query76_00009", + "negative_test_query27_00014", + "negative_test_query130_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query270", + "ranked_ids": [ + "negative_test_query270_00006", + "negative_test_query270_00010", + "negative_test_query270_00016", + "negative_test_query270_00009", + "apositive_test_query270_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query271", + "ranked_ids": [ + "negative_test_query271_00007", + "negative_test_query227_00009", + "negative_test_query120_00003", + "apositive_test_query271_00001", + "apositive_test_query10_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00017", + "negative_test_query332_00013", + "apositive_test_query251_00000", + "negative_test_query328_00000", + "negative_test_query272_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00009", + "negative_test_query273_00006", + "apositive_test_query18_00001", + "negative_test_query273_00008", + "apositive_test_query18_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query274", + "ranked_ids": [ + "negative_test_query274_00012", + "negative_test_query218_00011", + "negative_test_query34_00014", + "negative_test_query352_00012", + "apositive_test_query274_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.08333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query110_00001", + "negative_test_query275_00007", + "apositive_test_query93_00003", + "negative_test_query149_00016", + "apositive_test_query275_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query146_00000", + "apositive_test_query153_00006", + "apositive_test_query146_00015", + "negative_test_query251_00012", + "apositive_test_query276_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query277", + "ranked_ids": [ + "negative_test_query277_00009", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "apositive_test_query277_00002", + "negative_test_query345_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query140_00006", + "negative_test_query177_00001", + "negative_test_query15_00007", + "negative_test_query278_00009", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query279_00004", + "apositive_test_query279_00003", + "apositive_test_query3_00000", + "apositive_test_query76_00002", + "negative_test_query279_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query28", + "ranked_ids": [ + "negative_test_query28_00002", + "apositive_test_query217_00000", + "apositive_test_query217_00003", + "negative_test_query185_00003", + "negative_test_query307_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "negative_test_query280_00012", + "negative_test_query9_00012", + "negative_test_query280_00010", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.10526315789473684 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query281_00004", + "apositive_test_query281_00011", + "apositive_test_query273_00000", + "apositive_test_query273_00008", + "negative_test_query273_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query282", + "ranked_ids": [ + "apositive_test_query282_00000", + "negative_test_query282_00003", + "negative_test_query282_00006", + "negative_test_query282_00004", + "negative_test_query282_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query270_00006", + "negative_test_query93_00009", + "negative_test_query138_00005", + "negative_test_query283_00014", + "negative_test_query93_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.4 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "negative_test_query284_00001", + "apositive_test_query163_00003", + "negative_test_query14_00014", + "apositive_test_query284_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.25 + }, + "query_id": "test_query285", + "ranked_ids": [ + "negative_test_query285_00002", + "negative_test_query86_00001", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "apositive_test_query285_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query286", + "ranked_ids": [ + "apositive_test_query286_00014", + "apositive_test_query257_00000", + "negative_test_query139_00008", + "negative_test_query52_00001", + "apositive_test_query286_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query153_00003", + "apositive_test_query133_00000", + "apositive_test_query133_00004", + "negative_test_query301_00010", + "apositive_test_query133_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query288", + "ranked_ids": [ + "negative_test_query288_00006", + "apositive_test_query210_00001", + "negative_test_query210_00005", + "negative_test_query148_00014", + "negative_test_query199_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00004", + "apositive_test_query289_00002", + "negative_test_query113_00009", + "apositive_test_query289_00000", + "negative_test_query7_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query29", + "ranked_ids": [ + "negative_test_query292_00015", + "apositive_test_query29_00003", + "negative_test_query213_00012", + "negative_test_query29_00005", + "negative_test_query29_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query290", + "ranked_ids": [ + "negative_test_query290_00005", + "negative_test_query98_00003", + "negative_test_query342_00005", + "negative_test_query290_00000", + "apositive_test_query290_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query247_00005", + "apositive_test_query242_00001", + "negative_test_query291_00003", + "negative_test_query291_00005", + "negative_test_query291_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query292", + "ranked_ids": [ + "negative_test_query292_00015", + "apositive_test_query292_00001", + "negative_test_query292_00000", + "negative_test_query61_00011", + "negative_test_query292_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query293", + "ranked_ids": [ + "negative_test_query293_00011", + "negative_test_query293_00015", + "apositive_test_query293_00002", + "negative_test_query293_00010", + "negative_test_query293_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00007", + "negative_test_query294_00001", + "negative_test_query294_00004", + "negative_test_query294_00012", + "negative_test_query294_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query295", + "ranked_ids": [ + "negative_test_query292_00015", + "negative_test_query295_00012", + "negative_test_query57_00009", + "negative_test_query295_00001", + "negative_test_query295_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query296", + "ranked_ids": [ + "negative_test_query329_00011", + "apositive_test_query296_00004", + "negative_test_query296_00001", + "negative_test_query0_00004", + "apositive_test_query296_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query297_00004", + "negative_test_query297_00007", + "apositive_test_query139_00001", + "negative_test_query297_00012", + "negative_test_query191_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.38364913903790027, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "negative_test_query182_00011", + "negative_test_query193_00006", + "negative_test_query182_00010", + "apositive_test_query298_00000", + "apositive_test_query298_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query288_00000", + "negative_test_query58_00005", + "negative_test_query89_00008", + "apositive_test_query265_00002", + "apositive_test_query227_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query3", + "ranked_ids": [ + "negative_test_query3_00007", + "negative_test_query230_00010", + "apositive_test_query3_00000", + "negative_test_query3_00000", + "negative_test_query3_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query30", + "ranked_ids": [ + "negative_test_query231_00008", + "negative_test_query71_00010", + "negative_test_query182_00014", + "apositive_test_query30_00005", + "negative_test_query231_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query300_00011", + "negative_test_query300_00005", + "negative_test_query300_00009", + "negative_test_query141_00001", + "negative_test_query300_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query301", + "ranked_ids": [ + "apositive_test_query86_00000", + "negative_test_query301_00007", + "negative_test_query2_00008", + "negative_test_query301_00013", + "apositive_test_query301_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query302", + "ranked_ids": [ + "negative_test_query308_00005", + "apositive_test_query302_00004", + "negative_test_query308_00002", + "negative_test_query302_00010", + "apositive_test_query308_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query243_00000", + "negative_test_query303_00017", + "negative_test_query35_00003", + "apositive_test_query303_00000", + "negative_test_query173_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query285_00001", + "negative_test_query304_00003", + "negative_test_query234_00010", + "apositive_test_query285_00005", + "apositive_test_query234_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query305", + "ranked_ids": [ + "negative_test_query51_00003", + "negative_test_query0_00004", + "negative_test_query224_00015", + "negative_test_query164_00011", + "negative_test_query47_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query306", + "ranked_ids": [ + "negative_test_query209_00008", + "negative_test_query182_00014", + "apositive_test_query289_00003", + "apositive_test_query119_00006", + "negative_test_query204_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "negative_test_query307_00006", + "negative_test_query103_00012", + "negative_test_query48_00018", + "negative_test_query163_00010", + "apositive_test_query279_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.2 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query310_00008", + "apositive_test_query324_00000", + "negative_test_query324_00014", + "apositive_test_query308_00002", + "apositive_test_query308_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "apositive_test_query309_00000", + "negative_test_query309_00003", + "apositive_test_query358_00003", + "apositive_test_query76_00002", + "apositive_test_query161_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query38_00003", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query31_00007", + "negative_test_query31_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.3 + }, + "query_id": "test_query310", + "ranked_ids": [ + "negative_test_query179_00013", + "apositive_test_query310_00008", + "negative_test_query74_00002", + "apositive_test_query310_00002", + "apositive_test_query310_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query330_00009", + "negative_test_query288_00004", + "negative_test_query330_00011", + "negative_test_query240_00014", + "apositive_test_query330_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "negative_test_query242_00001", + "apositive_test_query237_00001", + "apositive_test_query220_00005", + "apositive_test_query98_00000", + "negative_test_query294_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query336_00001", + "negative_test_query119_00005", + "negative_test_query334_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query314_00011", + "apositive_test_query314_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query319_00005", + "apositive_test_query82_00005", + "negative_test_query315_00009", + "apositive_test_query315_00002", + "negative_test_query333_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query316", + "ranked_ids": [ + "apositive_test_query316_00001", + "negative_test_query301_00005", + "negative_test_query316_00000", + "negative_test_query316_00002", + "negative_test_query316_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00010", + "negative_test_query331_00009", + "negative_test_query55_00014", + "negative_test_query317_00017", + "negative_test_query317_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00002", + "negative_test_query318_00002", + "negative_test_query318_00004", + "negative_test_query73_00004", + "apositive_test_query318_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query196_00000", + "apositive_test_query336_00001", + "negative_test_query211_00016", + "negative_test_query55_00011", + "negative_test_query55_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query320", + "ranked_ids": [ + "negative_test_query258_00012", + "negative_test_query53_00014", + "negative_test_query193_00006", + "negative_test_query320_00005", + "negative_test_query131_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "negative_test_query321_00002", + "negative_test_query321_00010", + "negative_test_query136_00004", + "negative_test_query210_00016", + "apositive_test_query321_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query322", + "ranked_ids": [ + "negative_test_query73_00008", + "negative_test_query318_00002", + "apositive_test_query322_00003", + "apositive_test_query322_00005", + "negative_test_query322_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query323", + "ranked_ids": [ + "negative_test_query255_00002", + "negative_test_query323_00006", + "negative_test_query209_00004", + "apositive_test_query323_00006", + "negative_test_query75_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query324", + "ranked_ids": [ + "negative_test_query218_00003", + "apositive_test_query324_00000", + "negative_test_query324_00014", + "apositive_test_query308_00002", + "negative_test_query324_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query293_00015", + "apositive_test_query293_00002", + "negative_test_query325_00007", + "negative_test_query15_00016", + "negative_test_query15_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query326", + "ranked_ids": [ + "negative_test_query265_00007", + "apositive_test_query326_00000", + "negative_test_query178_00012", + "apositive_test_query326_00001", + "negative_test_query17_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query327", + "ranked_ids": [ + "negative_test_query271_00007", + "negative_test_query327_00000", + "negative_test_query227_00009", + "negative_test_query327_00014", + "negative_test_query327_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query328", + "ranked_ids": [ + "apositive_test_query40_00003", + "apositive_test_query328_00004", + "negative_test_query328_00003", + "negative_test_query222_00016", + "negative_test_query328_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query329", + "ranked_ids": [ + "negative_test_query329_00006", + "negative_test_query329_00011", + "apositive_test_query7_00001", + "negative_test_query35_00000", + "negative_test_query329_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00006", + "apositive_test_query33_00004", + "apositive_test_query154_00007", + "negative_test_query154_00004", + "apositive_test_query284_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query330", + "ranked_ids": [ + "negative_test_query330_00001", + "negative_test_query288_00006", + "negative_test_query330_00009", + "negative_test_query330_00000", + "negative_test_query330_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4776237035032179, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query331", + "ranked_ids": [ + "negative_test_query214_00015", + "apositive_test_query331_00002", + "negative_test_query331_00008", + "negative_test_query303_00001", + "apositive_test_query331_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00001", + "negative_test_query202_00002", + "negative_test_query332_00002", + "apositive_test_query348_00008", + "negative_test_query115_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00000", + "apositive_test_query254_00005", + "negative_test_query73_00008", + "negative_test_query136_00003", + "negative_test_query128_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query209_00003", + "negative_test_query111_00017", + "negative_test_query213_00009", + "negative_test_query334_00010", + "negative_test_query170_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query335", + "ranked_ids": [ + "negative_test_query335_00009", + "apositive_test_query335_00002", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query335_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "apositive_test_query50_00012", + "apositive_test_query38_00005", + "apositive_test_query113_00005", + "negative_test_query336_00015", + "negative_test_query247_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00015", + "apositive_test_query38_00003", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query337_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query338", + "ranked_ids": [ + "negative_test_query119_00003", + "apositive_test_query338_00001", + "negative_test_query81_00004", + "apositive_test_query114_00005", + "negative_test_query81_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004", + "apositive_test_query339_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00000", + "negative_test_query34_00014", + "negative_test_query352_00012", + "negative_test_query34_00008", + "negative_test_query352_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00002", + "negative_test_query88_00007", + "negative_test_query340_00001", + "negative_test_query340_00006", + "negative_test_query88_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query341_00005", + "negative_test_query96_00007", + "negative_test_query341_00009", + "negative_test_query101_00003", + "negative_test_query341_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query342_00016", + "negative_test_query68_00005", + "negative_test_query325_00015", + "negative_test_query136_00003", + "negative_test_query324_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query343", + "ranked_ids": [ + "negative_test_query307_00006", + "negative_test_query227_00010", + "apositive_test_query279_00003", + "apositive_test_query307_00008", + "negative_test_query225_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query282_00004", + "negative_test_query282_00001", + "negative_test_query344_00002", + "negative_test_query344_00004", + "negative_test_query344_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query90_00003", + "negative_test_query9_00002", + "negative_test_query130_00004", + "negative_test_query345_00011", + "negative_test_query345_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query19_00012", + "negative_test_query346_00008", + "negative_test_query346_00010", + "negative_test_query15_00008", + "negative_test_query346_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query347", + "ranked_ids": [ + "negative_test_query347_00006", + "negative_test_query251_00007", + "negative_test_query347_00015", + "negative_test_query347_00002", + "apositive_test_query347_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query348_00004", + "negative_test_query297_00017", + "apositive_test_query115_00002", + "negative_test_query351_00010", + "apositive_test_query348_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "apositive_test_query137_00002", + "apositive_test_query42_00004", + "negative_test_query74_00001", + "apositive_test_query253_00008", + "apositive_test_query74_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query35_00003", + "negative_test_query262_00002", + "negative_test_query198_00011", + "negative_test_query101_00006", + "negative_test_query205_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query63_00000", + "negative_test_query192_00005", + "apositive_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query218_00004", + "negative_test_query8_00003", + "negative_test_query53_00011", + "negative_test_query156_00006", + "apositive_test_query60_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.4 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query352_00000", + "negative_test_query34_00014", + "negative_test_query34_00008", + "negative_test_query352_00010", + "apositive_test_query352_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query204_00012", + "negative_test_query353_00002", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.375 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00006", + "apositive_test_query354_00004", + "negative_test_query354_00004", + "negative_test_query309_00013", + "apositive_test_query354_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query100_00001", + "negative_test_query355_00003", + "negative_test_query264_00003", + "apositive_test_query355_00005", + "negative_test_query355_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query30_00005", + "apositive_test_query356_00004", + "negative_test_query253_00001", + "negative_test_query15_00006", + "negative_test_query102_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00002", + "apositive_test_query221_00000", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query119_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2 + }, + "query_id": "test_query358", + "ranked_ids": [ + "apositive_test_query309_00000", + "apositive_test_query161_00000", + "apositive_test_query358_00000", + "apositive_test_query358_00003", + "apositive_test_query11_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00006", + "apositive_test_query0_00003", + "negative_test_query359_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query36", + "ranked_ids": [ + "apositive_test_query286_00005", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query112_00015", + "apositive_test_query106_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00000", + "negative_test_query360_00009", + "negative_test_query189_00013", + "apositive_test_query152_00002", + "negative_test_query240_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query57_00009", + "negative_test_query20_00003", + "apositive_test_query177_00000", + "apositive_test_query158_00007", + "negative_test_query152_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query38", + "ranked_ids": [ + "apositive_test_query38_00002", + "apositive_test_query247_00005", + "apositive_test_query38_00007", + "negative_test_query267_00016", + "negative_test_query267_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query39", + "ranked_ids": [ + "negative_test_query232_00007", + "negative_test_query39_00003", + "negative_test_query232_00017", + "negative_test_query331_00010", + "apositive_test_query39_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query4", + "ranked_ids": [ + "negative_test_query266_00009", + "apositive_test_query4_00010", + "negative_test_query4_00002", + "apositive_test_query174_00000", + "apositive_test_query4_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00012", + "negative_test_query40_00010", + "negative_test_query40_00006", + "negative_test_query40_00005", + "negative_test_query214_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query41", + "ranked_ids": [ + "negative_test_query282_00003", + "negative_test_query41_00000", + "negative_test_query98_00000", + "negative_test_query41_00011", + "negative_test_query41_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.25 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query250_00010", + "apositive_test_query42_00003", + "negative_test_query42_00000", + "apositive_test_query42_00004", + "apositive_test_query162_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query146_00004", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query43_00000", + "negative_test_query43_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query44", + "ranked_ids": [ + "negative_test_query293_00015", + "negative_test_query21_00009", + "negative_test_query330_00009", + "negative_test_query141_00007", + "negative_test_query190_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query45_00003", + "negative_test_query143_00011", + "negative_test_query342_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "apositive_test_query79_00005", + "negative_test_query213_00012", + "negative_test_query108_00003", + "negative_test_query168_00003", + "negative_test_query79_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query331_00002", + "negative_test_query47_00013", + "negative_test_query325_00007", + "negative_test_query144_00018", + "negative_test_query47_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "apositive_test_query48_00000", + "negative_test_query103_00012", + "negative_test_query48_00018", + "apositive_test_query307_00002", + "negative_test_query48_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query49", + "ranked_ids": [ + "negative_test_query342_00005", + "negative_test_query52_00011", + "negative_test_query15_00008", + "negative_test_query156_00010", + "apositive_test_query204_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query5", + "ranked_ids": [ + "negative_test_query5_00004", + "negative_test_query5_00007", + "negative_test_query275_00001", + "negative_test_query5_00014", + "negative_test_query5_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00006", + "apositive_test_query50_00012", + "apositive_test_query50_00015", + "apositive_test_query186_00005", + "apositive_test_query131_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query51", + "ranked_ids": [ + "negative_test_query51_00000", + "negative_test_query97_00001", + "apositive_test_query51_00006", + "negative_test_query51_00004", + "negative_test_query209_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00003", + "apositive_test_query286_00014", + "apositive_test_query257_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00018", + "negative_test_query132_00015", + "negative_test_query150_00014", + "negative_test_query258_00013", + "negative_test_query169_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query44_00007", + "negative_test_query54_00006", + "apositive_test_query86_00000", + "apositive_test_query335_00002", + "apositive_test_query51_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query55", + "ranked_ids": [ + "negative_test_query55_00010", + "negative_test_query55_00003", + "negative_test_query55_00012", + "negative_test_query55_00011", + "negative_test_query234_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query56", + "ranked_ids": [ + "negative_test_query46_00014", + "negative_test_query56_00006", + "apositive_test_query46_00001", + "apositive_test_query238_00001", + "negative_test_query56_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query57", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query57_00008", + "apositive_test_query203_00000", + "negative_test_query334_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "apositive_test_query311_00004", + "negative_test_query308_00004", + "negative_test_query58_00015", + "negative_test_query27_00017", + "negative_test_query240_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "negative_test_query116_00000", + "negative_test_query59_00011", + "negative_test_query59_00014", + "negative_test_query59_00015", + "negative_test_query202_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query6", + "ranked_ids": [ + "negative_test_query179_00007", + "negative_test_query6_00012", + "apositive_test_query6_00000", + "negative_test_query252_00004", + "apositive_test_query109_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.125 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00011", + "negative_test_query31_00004", + "apositive_test_query218_00004", + "apositive_test_query60_00006", + "apositive_test_query285_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query61", + "ranked_ids": [ + "apositive_test_query173_00004", + "negative_test_query327_00016", + "apositive_test_query61_00003", + "negative_test_query61_00002", + "negative_test_query129_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00003", + "negative_test_query62_00002", + "negative_test_query62_00000", + "apositive_test_query62_00001", + "negative_test_query219_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00006", + "negative_test_query321_00002", + "negative_test_query297_00006", + "negative_test_query63_00003", + "apositive_test_query63_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query120_00000", + "apositive_test_query160_00003", + "negative_test_query149_00001", + "negative_test_query120_00003", + "negative_test_query64_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query65_00000", + "negative_test_query268_00012", + "apositive_test_query356_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query66", + "ranked_ids": [ + "negative_test_query9_00012", + "apositive_test_query66_00004", + "apositive_test_query20_00005", + "negative_test_query188_00006", + "apositive_test_query66_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query67", + "ranked_ids": [ + "apositive_test_query286_00004", + "apositive_test_query286_00010", + "negative_test_query123_00008", + "negative_test_query67_00007", + "negative_test_query260_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query68", + "ranked_ids": [ + "negative_test_query330_00009", + "apositive_test_query68_00000", + "negative_test_query288_00004", + "negative_test_query330_00011", + "negative_test_query68_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.15789473684210525 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00010", + "apositive_test_query257_00003", + "apositive_test_query69_00005", + "negative_test_query18_00010", + "apositive_test_query69_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "apositive_test_query159_00001", + "negative_test_query264_00013", + "apositive_test_query38_00003", + "negative_test_query159_00010", + "negative_test_query7_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00006", + "apositive_test_query70_00001", + "negative_test_query227_00010", + "apositive_test_query227_00000", + "negative_test_query348_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query71", + "ranked_ids": [ + "negative_test_query71_00002", + "negative_test_query71_00015", + "apositive_test_query30_00005", + "negative_test_query71_00009", + "negative_test_query71_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "negative_test_query17_00009", + "negative_test_query72_00015", + "negative_test_query72_00006", + "negative_test_query72_00003", + "negative_test_query72_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query73", + "ranked_ids": [ + "negative_test_query73_00013", + "apositive_test_query333_00000", + "negative_test_query73_00008", + "negative_test_query233_00015", + "negative_test_query128_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query60_00006", + "negative_test_query74_00001", + "apositive_test_query113_00001", + "apositive_test_query74_00002", + "negative_test_query74_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query75", + "ranked_ids": [ + "negative_test_query99_00006", + "apositive_test_query75_00001", + "negative_test_query209_00004", + "negative_test_query67_00017", + "negative_test_query75_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query76", + "ranked_ids": [ + "negative_test_query76_00009", + "apositive_test_query76_00000", + "negative_test_query147_00007", + "apositive_test_query76_00002", + "negative_test_query181_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query329_00015", + "negative_test_query77_00013", + "negative_test_query342_00007", + "negative_test_query98_00014", + "negative_test_query301_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query78", + "ranked_ids": [ + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query214_00010", + "apositive_test_query38_00003", + "apositive_test_query326_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "negative_test_query38_00007", + "negative_test_query313_00013", + "apositive_test_query79_00004", + "apositive_test_query295_00001", + "negative_test_query157_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "apositive_test_query192_00002", + "negative_test_query329_00005", + "negative_test_query8_00000", + "negative_test_query8_00014", + "negative_test_query96_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query289_00005", + "apositive_test_query179_00003", + "negative_test_query287_00005", + "negative_test_query80_00009", + "negative_test_query80_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query81", + "ranked_ids": [ + "negative_test_query81_00005", + "apositive_test_query81_00003", + "negative_test_query81_00004", + "negative_test_query145_00004", + "apositive_test_query81_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query146_00014", + "apositive_test_query168_00000", + "negative_test_query279_00012", + "negative_test_query168_00007", + "negative_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query84", + "ranked_ids": [ + "negative_test_query84_00013", + "negative_test_query268_00009", + "negative_test_query84_00006", + "negative_test_query84_00005", + "negative_test_query84_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query286_00004", + "apositive_test_query85_00002", + "apositive_test_query81_00008", + "apositive_test_query85_00000", + "negative_test_query85_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query86", + "ranked_ids": [ + "negative_test_query86_00001", + "apositive_test_query86_00000", + "negative_test_query86_00003", + "negative_test_query86_00000", + "negative_test_query86_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query87", + "ranked_ids": [ + "apositive_test_query87_00001", + "negative_test_query87_00002", + "negative_test_query182_00017", + "negative_test_query200_00011", + "negative_test_query315_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "apositive_test_query88_00001", + "negative_test_query88_00002", + "negative_test_query88_00010", + "negative_test_query88_00012", + "negative_test_query105_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query89_00005", + "negative_test_query68_00004", + "negative_test_query233_00005", + "apositive_test_query255_00000", + "negative_test_query89_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00018", + "negative_test_query9_00013", + "negative_test_query9_00005", + "negative_test_query9_00016", + "negative_test_query9_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query227_00006", + "negative_test_query90_00012", + "negative_test_query227_00001", + "negative_test_query127_00001", + "negative_test_query90_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query91", + "ranked_ids": [ + "negative_test_query358_00004", + "apositive_test_query342_00001", + "apositive_test_query11_00006", + "negative_test_query279_00011", + "negative_test_query303_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query92", + "ranked_ids": [ + "apositive_test_query100_00001", + "negative_test_query122_00018", + "apositive_test_query92_00001", + "apositive_test_query99_00003", + "negative_test_query283_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query243_00006", + "apositive_test_query93_00003", + "apositive_test_query335_00003", + "negative_test_query108_00003", + "negative_test_query280_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.375 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00010", + "apositive_test_query94_00002", + "apositive_test_query94_00006", + "negative_test_query94_00002", + "apositive_test_query94_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "negative_test_query244_00010", + "apositive_test_query292_00001", + "negative_test_query95_00008", + "apositive_test_query177_00000", + "negative_test_query95_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004", + "apositive_test_query339_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "negative_test_query113_00009", + "negative_test_query113_00006", + "negative_test_query209_00008", + "negative_test_query16_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00018", + "negative_test_query98_00011", + "negative_test_query98_00005", + "apositive_test_query14_00002", + "negative_test_query23_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query99", + "ranked_ids": [ + "apositive_test_query193_00010", + "apositive_test_query99_00001", + "apositive_test_query333_00003", + "apositive_test_query253_00003", + "apositive_test_query153_00012" + ], + "valid": true + } + ], + "query_id_filter": [], + "schema_version": 1, + "seed": 13, + "set_size": 3, + "stride": 10, + "timeout": null, + "top_k": 5, + "tourrank_rounds": 2, + "tourrank_stage_policy": "paper_top_100_else_single_group_halving", + "window_size": 20 +} diff --git a/benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json b/benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json new file mode 100644 index 0000000..41b2954 --- /dev/null +++ b/benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json @@ -0,0 +1,84577 @@ +{ + "aggregate": [ + { + "algorithm": "original_bm25", + "case_count": 361, + "invalid_rate": 0.0, + "metrics": { + "mrr@5": 0.5062326869806094, + "ndcg@5": 0.3519713563219895, + "recall@5": 0.28624170216624956 + }, + "valid_count": 361 + }, + { + "algorithm": "single_call_listwise@20", + "case_count": 361, + "invalid_rate": 0.00554016620498615, + "metrics": { + "mrr@5": 0.5541089566020314, + "ndcg@5": 0.40824079766980675, + "recall@5": 0.3345189219319816 + }, + "valid_count": 359 + }, + { + "algorithm": "rankgpt_sw_w5", + "case_count": 361, + "invalid_rate": 0.0, + "metrics": { + "mrr@5": 0.5283010156971376, + "ndcg@5": 0.39728693703247925, + "recall@5": 0.3365996924460257 + }, + "valid_count": 361 + }, + { + "algorithm": "acurank_k5_b1", + "case_count": 361, + "invalid_rate": 0.013850415512465374, + "metrics": { + "mrr@5": 0.5491228070175438, + "ndcg@5": 0.40531184715382623, + "recall@5": 0.33771567900685506 + }, + "valid_count": 356 + }, + { + "algorithm": "tourrank_r2", + "case_count": 361, + "invalid_rate": 0.0, + "metrics": { + "mrr@5": 0.5725300092336103, + "ndcg@5": 0.42364249552168015, + "recall@5": 0.3601459765907722 + }, + "valid_count": 361 + }, + { + "algorithm": "prp_sliding_p1", + "case_count": 361, + "invalid_rate": 0.0, + "metrics": { + "mrr@5": 0.5817636195752539, + "ndcg@5": 0.40645532961841807, + "recall@5": 0.3276757309495532 + }, + "valid_count": 361 + }, + { + "algorithm": "setwise_hs_s10", + "case_count": 361, + "invalid_rate": 0.0, + "metrics": { + "mrr@5": 0.5058633425669437, + "ndcg@5": 0.3652885555170704, + "recall@5": 0.30053939029276283 + }, + "valid_count": 361 + }, + { + "algorithm": "answer_confidence", + "case_count": 361, + "invalid_rate": 0.0, + "metrics": { + "mrr@5": 0.28619575253924284, + "ndcg@5": 0.17217008703277661, + "recall@5": 0.1434797614811208 + }, + "valid_count": 361 + } + ], + "algorithm": [ + "original_bm25", + "single_call_listwise@20", + "rankgpt_sw_w5", + "acurank_k5_b1", + "tourrank_r2", + "prp_sliding_p1", + "setwise_hs_s10", + "answer_confidence" + ], + "benchmark_type": "reranking_with_first_stage_candidates", + "cache_dir": ".benchmark-cache/askubuntu-bm25", + "call_estimates": { + "acurank_k5_b1": 722, + "answer_confidence": 7220, + "original_bm25": 0, + "prp_sliding_p1": 13718, + "rankgpt_sw_w5": 3249, + "setwise_hs_s10": 4332, + "single_call_listwise@20": 361, + "tourrank_r2": 2888 + }, + "candidate_count": 20, + "candidate_strategy": "candidate_file", + "candidates": "benchmark-results/pyserini/askubuntu-bm25-top20.trec", + "case_count": 361, + "checkpoint_output": null, + "dataset": "benchmark-cache", + "dataset_name": "askubuntu-bm25", + "fixture": null, + "merged_from": [ + { + "algorithm": [ + "original_bm25", + "single_call_listwise@20", + "rankgpt_sw_w5", + "acurank_k5_b1", + "tourrank_r2", + "prp_sliding_p1", + "setwise_hs_s10" + ], + "divergent_fields": {}, + "file": "askubuntu-bm25-top20-default-live.v3.merged.json" + }, + { + "algorithm": [ + "answer_confidence" + ], + "divergent_fields": { + "cache_dir": "/Users/skiiwoo/Documents/ranksmith/.benchmark-cache/askubuntu-bm25", + "timeout": null + }, + "file": "askubuntu-bm25-top20-answer-confidence.json" + } + ], + "method_settings": { + "acurank_k5_b1": { + "candidate_count": 20, + "max_adaptive_reranker_calls": 1, + "target_rank": 5, + "top_k_early_stop": false, + "window_size": 20 + }, + "answer_confidence": { + "candidate_count": 20, + "stride": 10, + "top_k_early_stop": false, + "window_size": 20 + }, + "original_bm25": { + "candidate_count": 20, + "top_k_early_stop": false + }, + "prp_sliding_p1": { + "candidate_count": 20, + "passes": 1, + "top_k_early_stop": false + }, + "rankgpt_sw_w5": { + "candidate_count": 20, + "stride": 2, + "top_k_early_stop": false, + "window_size": 5 + }, + "setwise_hs_s10": { + "candidate_count": 20, + "set_size": 10, + "top_k": 5, + "top_k_early_stop": true + }, + "single_call_listwise@20": { + "candidate_count": 20, + "top_k_early_stop": false + }, + "tourrank_r2": { + "candidate_count": 20, + "rounds": 2, + "top_k_early_stop": false + } + }, + "passes": 10, + "per_query": [ + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.4166666666666667 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00006", + "apositive_test_query0_00004", + "apositive_test_query0_00003", + "negative_test_query359_00002", + "negative_test_query0_00001", + "negative_test_query9_00017", + "apositive_test_query0_00002", + "negative_test_query221_00006", + "negative_test_query259_00006", + "apositive_test_query107_00004", + "negative_test_query130_00002", + "negative_test_query201_00000", + "negative_test_query357_00000", + "apositive_test_query130_00000", + "apositive_test_query153_00004", + "negative_test_query0_00007", + "negative_test_query0_00000", + "negative_test_query36_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query1", + "ranked_ids": [ + "apositive_test_query1_00000", + "negative_test_query1_00005", + "apositive_test_query348_00003", + "negative_test_query1_00009", + "negative_test_query61_00013", + "negative_test_query1_00001", + "negative_test_query61_00014", + "negative_test_query297_00011", + "negative_test_query184_00015", + "negative_test_query331_00002", + "apositive_test_query188_00002", + "negative_test_query202_00001", + "negative_test_query256_00005", + "negative_test_query61_00002", + "negative_test_query120_00006", + "negative_test_query110_00007", + "negative_test_query27_00017", + "negative_test_query245_00006", + "negative_test_query61_00003", + "negative_test_query157_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query10", + "ranked_ids": [ + "apositive_test_query10_00001", + "apositive_test_query10_00000", + "apositive_test_query10_00002", + "negative_test_query10_00011", + "negative_test_query10_00005", + "negative_test_query10_00000", + "negative_test_query10_00010", + "negative_test_query169_00008", + "negative_test_query135_00014", + "negative_test_query104_00011", + "negative_test_query83_00000", + "negative_test_query266_00009", + "apositive_test_query69_00016", + "apositive_test_query44_00001", + "apositive_test_query69_00000", + "negative_test_query72_00002", + "negative_test_query10_00004", + "negative_test_query10_00007", + "apositive_test_query286_00008", + "negative_test_query10_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query100", + "ranked_ids": [ + "negative_test_query100_00001", + "apositive_test_query100_00000", + "negative_test_query15_00000", + "negative_test_query100_00013", + "negative_test_query100_00009", + "negative_test_query290_00015", + "negative_test_query237_00010", + "negative_test_query100_00010", + "apositive_test_query338_00002", + "negative_test_query258_00017", + "apositive_test_query100_00004", + "apositive_test_query181_00000", + "apositive_test_query338_00005", + "apositive_test_query100_00002", + "apositive_test_query100_00003", + "negative_test_query271_00006", + "negative_test_query100_00014", + "negative_test_query252_00005", + "negative_test_query323_00000", + "negative_test_query147_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query101_00005", + "apositive_test_query101_00002", + "apositive_test_query73_00002", + "negative_test_query173_00013", + "negative_test_query61_00002", + "negative_test_query61_00013", + "negative_test_query241_00005", + "negative_test_query68_00009", + "negative_test_query258_00000", + "negative_test_query53_00000", + "apositive_test_query336_00001", + "apositive_test_query196_00000", + "negative_test_query103_00009", + "negative_test_query61_00014", + "negative_test_query35_00004", + "negative_test_query191_00014", + "negative_test_query134_00008", + "negative_test_query61_00000", + "negative_test_query189_00014", + "apositive_test_query291_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query102_00014", + "apositive_test_query20_00001", + "negative_test_query131_00009", + "negative_test_query313_00016", + "negative_test_query205_00016", + "negative_test_query47_00011", + "apositive_test_query30_00005", + "negative_test_query108_00009", + "negative_test_query183_00012", + "apositive_test_query356_00008", + "apositive_test_query47_00001", + "negative_test_query183_00009", + "negative_test_query203_00007", + "apositive_test_query238_00000", + "negative_test_query209_00006", + "negative_test_query219_00004", + "negative_test_query90_00016", + "negative_test_query47_00010", + "negative_test_query7_00001", + "negative_test_query232_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query103", + "ranked_ids": [ + "apositive_test_query103_00001", + "negative_test_query103_00003", + "negative_test_query103_00011", + "apositive_test_query185_00003", + "negative_test_query185_00005", + "negative_test_query217_00005", + "negative_test_query222_00009", + "negative_test_query103_00005", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query307_00008", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query103_00000", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query48_00014", + "negative_test_query72_00012", + "negative_test_query103_00006", + "apositive_test_query103_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "negative_test_query104_00002", + "negative_test_query104_00018", + "negative_test_query104_00011", + "negative_test_query104_00001", + "negative_test_query104_00013", + "negative_test_query104_00015", + "negative_test_query104_00010", + "negative_test_query104_00003", + "negative_test_query343_00013", + "negative_test_query104_00005", + "negative_test_query104_00012", + "negative_test_query104_00014", + "negative_test_query104_00009", + "negative_test_query104_00016", + "negative_test_query104_00004", + "negative_test_query104_00007", + "negative_test_query48_00010", + "apositive_test_query28_00011", + "apositive_test_query48_00000", + "apositive_test_query104_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query105", + "ranked_ids": [ + "apositive_test_query105_00000", + "negative_test_query79_00002", + "negative_test_query105_00011", + "apositive_test_query83_00004", + "negative_test_query295_00005", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query119_00005", + "negative_test_query105_00005", + "negative_test_query105_00006", + "negative_test_query168_00005", + "negative_test_query142_00010", + "negative_test_query267_00011", + "negative_test_query313_00013", + "apositive_test_query157_00008", + "negative_test_query359_00008", + "negative_test_query97_00007", + "negative_test_query79_00004", + "negative_test_query56_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query106", + "ranked_ids": [ + "apositive_test_query106_00003", + "apositive_test_query106_00005", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query326_00008", + "apositive_test_query106_00000", + "negative_test_query286_00000", + "negative_test_query36_00001", + "negative_test_query114_00003", + "apositive_test_query42_00006", + "negative_test_query106_00012", + "apositive_test_query106_00001", + "negative_test_query212_00002", + "negative_test_query106_00000", + "negative_test_query106_00006", + "apositive_test_query286_00005", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "apositive_test_query357_00003", + "apositive_test_query36_00002", + "negative_test_query146_00000", + "apositive_test_query276_00003", + "negative_test_query46_00012", + "negative_test_query160_00001", + "negative_test_query296_00008", + "negative_test_query333_00005", + "apositive_test_query221_00001", + "negative_test_query105_00015", + "negative_test_query142_00000", + "negative_test_query31_00013", + "negative_test_query305_00015", + "negative_test_query124_00005", + "apositive_test_query142_00002", + "negative_test_query105_00008", + "apositive_test_query211_00001", + "negative_test_query296_00001", + "apositive_test_query142_00004", + "negative_test_query142_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.375 + }, + "query_id": "test_query108", + "ranked_ids": [ + "apositive_test_query108_00000", + "apositive_test_query108_00002", + "apositive_test_query108_00003", + "negative_test_query57_00009", + "negative_test_query108_00011", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query102_00012", + "negative_test_query108_00010", + "negative_test_query311_00006", + "negative_test_query108_00008", + "apositive_test_query108_00005", + "apositive_test_query108_00007", + "negative_test_query168_00001", + "negative_test_query108_00001", + "negative_test_query250_00009", + "negative_test_query250_00011", + "negative_test_query108_00006", + "negative_test_query102_00007", + "negative_test_query108_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query109", + "ranked_ids": [ + "apositive_test_query109_00000", + "negative_test_query109_00003", + "negative_test_query109_00000", + "negative_test_query109_00012", + "negative_test_query268_00010", + "apositive_test_query109_00001", + "negative_test_query109_00013", + "negative_test_query109_00014", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query248_00006", + "negative_test_query109_00009", + "negative_test_query109_00008", + "negative_test_query109_00011", + "negative_test_query109_00004", + "negative_test_query6_00006", + "negative_test_query38_00006", + "negative_test_query190_00003", + "apositive_test_query109_00003", + "negative_test_query51_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.375 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00001", + "apositive_test_query11_00006", + "apositive_test_query11_00007", + "negative_test_query132_00003", + "negative_test_query309_00001", + "negative_test_query226_00010", + "negative_test_query11_00004", + "negative_test_query309_00004", + "negative_test_query309_00002", + "negative_test_query358_00005", + "apositive_test_query11_00004", + "apositive_test_query358_00006", + "apositive_test_query358_00007", + "apositive_test_query11_00000", + "apositive_test_query301_00003", + "negative_test_query123_00007", + "negative_test_query132_00013", + "negative_test_query358_00001", + "apositive_test_query11_00003", + "negative_test_query11_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "negative_test_query89_00006", + "negative_test_query110_00013", + "negative_test_query110_00005", + "negative_test_query110_00002", + "negative_test_query110_00014", + "apositive_test_query344_00002", + "apositive_test_query11_00002", + "negative_test_query110_00006", + "negative_test_query311_00011", + "negative_test_query110_00004", + "apositive_test_query138_00000", + "negative_test_query17_00008", + "negative_test_query110_00007", + "apositive_test_query180_00001", + "negative_test_query343_00001", + "negative_test_query260_00015", + "apositive_test_query242_00006", + "apositive_test_query291_00009", + "apositive_test_query330_00001", + "negative_test_query10_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query137_00010", + "negative_test_query134_00006", + "negative_test_query111_00015", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query213_00011", + "negative_test_query223_00000", + "negative_test_query111_00018", + "apositive_test_query162_00005", + "negative_test_query65_00013", + "negative_test_query245_00001", + "negative_test_query65_00007", + "negative_test_query111_00008", + "apositive_test_query174_00008", + "apositive_test_query308_00003", + "negative_test_query14_00015", + "apositive_test_query306_00002", + "negative_test_query58_00013", + "negative_test_query243_00001", + "negative_test_query292_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "negative_test_query36_00014", + "negative_test_query112_00004", + "negative_test_query112_00010", + "negative_test_query246_00016", + "apositive_test_query36_00003", + "apositive_test_query286_00005", + "apositive_test_query106_00003", + "apositive_test_query286_00006", + "negative_test_query112_00016", + "negative_test_query216_00002", + "apositive_test_query246_00001", + "negative_test_query112_00015", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "negative_test_query67_00003", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query6_00003", + "negative_test_query85_00002", + "negative_test_query36_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query113", + "ranked_ids": [ + "apositive_test_query113_00001", + "apositive_test_query113_00000", + "negative_test_query67_00014", + "negative_test_query7_00016", + "negative_test_query113_00003", + "apositive_test_query113_00002", + "negative_test_query67_00002", + "negative_test_query201_00008", + "negative_test_query105_00007", + "apositive_test_query67_00000", + "negative_test_query188_00005", + "negative_test_query223_00003", + "negative_test_query38_00003", + "apositive_test_query186_00005", + "apositive_test_query79_00001", + "negative_test_query6_00016", + "negative_test_query105_00000", + "negative_test_query47_00005", + "negative_test_query216_00011", + "apositive_test_query188_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query114", + "ranked_ids": [ + "apositive_test_query114_00000", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "apositive_test_query114_00001", + "negative_test_query114_00008", + "negative_test_query114_00002", + "apositive_test_query114_00003", + "apositive_test_query124_00004", + "negative_test_query114_00005", + "negative_test_query114_00004", + "negative_test_query114_00007", + "negative_test_query114_00010", + "negative_test_query114_00001", + "negative_test_query114_00003", + "apositive_test_query114_00006", + "apositive_test_query106_00003", + "negative_test_query106_00009", + "negative_test_query349_00003", + "negative_test_query81_00009", + "negative_test_query262_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.5 + }, + "query_id": "test_query115", + "ranked_ids": [ + "apositive_test_query115_00002", + "apositive_test_query115_00001", + "apositive_test_query115_00004", + "negative_test_query115_00003", + "negative_test_query115_00002", + "negative_test_query115_00006", + "apositive_test_query115_00003", + "apositive_test_query115_00000", + "negative_test_query115_00007", + "apositive_test_query348_00000", + "negative_test_query115_00012", + "apositive_test_query348_00009", + "negative_test_query115_00004", + "negative_test_query276_00006", + "negative_test_query139_00000", + "negative_test_query205_00004", + "apositive_test_query121_00001", + "negative_test_query115_00010", + "apositive_test_query133_00010", + "negative_test_query121_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "apositive_test_query116_00000", + "apositive_test_query49_00000", + "negative_test_query116_00015", + "negative_test_query59_00013", + "negative_test_query116_00013", + "negative_test_query116_00008", + "negative_test_query116_00003", + "negative_test_query59_00007", + "negative_test_query116_00002", + "negative_test_query116_00000", + "negative_test_query116_00004", + "negative_test_query261_00013", + "negative_test_query116_00011", + "negative_test_query98_00005", + "negative_test_query116_00006", + "negative_test_query116_00007", + "negative_test_query116_00009", + "negative_test_query116_00010", + "negative_test_query116_00014", + "negative_test_query116_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00003", + "negative_test_query117_00010", + "negative_test_query117_00009", + "apositive_test_query354_00003", + "negative_test_query194_00002", + "negative_test_query63_00001", + "apositive_test_query354_00000", + "apositive_test_query117_00004", + "apositive_test_query117_00006", + "negative_test_query48_00000", + "negative_test_query117_00004", + "apositive_test_query117_00007", + "negative_test_query117_00007", + "negative_test_query117_00001", + "negative_test_query117_00008", + "apositive_test_query117_00000", + "apositive_test_query134_00004", + "negative_test_query202_00006", + "negative_test_query117_00003", + "negative_test_query175_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "apositive_test_query155_00005", + "negative_test_query340_00007", + "negative_test_query296_00007", + "negative_test_query84_00012", + "negative_test_query23_00005", + "apositive_test_query174_00010", + "apositive_test_query310_00001", + "negative_test_query171_00005", + "negative_test_query171_00016", + "apositive_test_query171_00001", + "apositive_test_query174_00003", + "apositive_test_query174_00004", + "apositive_test_query174_00005", + "apositive_test_query340_00002", + "negative_test_query296_00010", + "negative_test_query88_00007", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "apositive_test_query174_00006", + "apositive_test_query217_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query119", + "ranked_ids": [ + "apositive_test_query221_00004", + "apositive_test_query119_00005", + "negative_test_query119_00005", + "apositive_test_query306_00010", + "negative_test_query56_00002", + "negative_test_query158_00001", + "negative_test_query158_00000", + "negative_test_query79_00002", + "apositive_test_query113_00002", + "apositive_test_query79_00004", + "apositive_test_query119_00001", + "negative_test_query152_00004", + "negative_test_query359_00005", + "negative_test_query157_00001", + "negative_test_query157_00007", + "negative_test_query119_00002", + "negative_test_query223_00011", + "apositive_test_query26_00000", + "apositive_test_query105_00000", + "apositive_test_query119_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query12_00016", + "negative_test_query12_00014", + "negative_test_query12_00003", + "negative_test_query12_00004", + "negative_test_query12_00013", + "negative_test_query12_00017", + "negative_test_query12_00007", + "negative_test_query12_00008", + "apositive_test_query12_00000", + "negative_test_query12_00015", + "negative_test_query12_00011", + "negative_test_query12_00001", + "negative_test_query12_00006", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query231_00013", + "negative_test_query234_00014", + "apositive_test_query185_00000", + "apositive_test_query217_00005", + "negative_test_query175_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00002", + "negative_test_query120_00008", + "negative_test_query1_00009", + "apositive_test_query120_00003", + "negative_test_query352_00013", + "apositive_test_query120_00002", + "negative_test_query120_00001", + "negative_test_query121_00002", + "apositive_test_query120_00000", + "apositive_test_query120_00001", + "apositive_test_query117_00007", + "negative_test_query303_00018", + "negative_test_query120_00012", + "negative_test_query121_00012", + "negative_test_query148_00014", + "negative_test_query120_00006", + "apositive_test_query32_00007", + "negative_test_query120_00010", + "negative_test_query196_00006", + "apositive_test_query352_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00002", + "apositive_test_query121_00001", + "negative_test_query121_00008", + "apositive_test_query121_00004", + "apositive_test_query121_00005", + "apositive_test_query121_00000", + "negative_test_query121_00013", + "negative_test_query121_00000", + "negative_test_query121_00012", + "apositive_test_query121_00003", + "negative_test_query308_00009", + "negative_test_query44_00012", + "negative_test_query121_00003", + "negative_test_query256_00003", + "negative_test_query53_00003", + "negative_test_query121_00007", + "negative_test_query313_00006", + "negative_test_query115_00009", + "negative_test_query133_00001", + "apositive_test_query307_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query61_00015", + "negative_test_query86_00006", + "negative_test_query17_00012", + "apositive_test_query254_00002", + "negative_test_query210_00003", + "apositive_test_query173_00000", + "apositive_test_query122_00000", + "negative_test_query195_00008", + "apositive_test_query134_00000", + "negative_test_query86_00001", + "negative_test_query64_00017", + "negative_test_query173_00003", + "negative_test_query297_00000", + "negative_test_query179_00002", + "negative_test_query5_00001", + "negative_test_query179_00014", + "negative_test_query358_00000", + "apositive_test_query175_00004", + "negative_test_query134_00007", + "apositive_test_query173_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query123", + "ranked_ids": [ + "negative_test_query123_00013", + "apositive_test_query123_00002", + "negative_test_query123_00008", + "negative_test_query155_00001", + "negative_test_query123_00005", + "apositive_test_query123_00004", + "apositive_test_query123_00003", + "negative_test_query123_00001", + "negative_test_query123_00004", + "negative_test_query123_00010", + "negative_test_query161_00009", + "negative_test_query340_00012", + "negative_test_query309_00008", + "negative_test_query123_00007", + "negative_test_query123_00006", + "negative_test_query269_00006", + "negative_test_query123_00000", + "negative_test_query123_00002", + "apositive_test_query123_00000", + "negative_test_query155_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "negative_test_query187_00003", + "apositive_test_query13_00004", + "negative_test_query347_00010", + "negative_test_query124_00003", + "apositive_test_query194_00000", + "apositive_test_query13_00000", + "negative_test_query42_00005", + "negative_test_query123_00009", + "negative_test_query42_00006", + "apositive_test_query162_00002", + "negative_test_query262_00011", + "negative_test_query187_00006", + "negative_test_query309_00012", + "negative_test_query144_00010", + "negative_test_query76_00004", + "apositive_test_query162_00001", + "negative_test_query349_00017", + "negative_test_query171_00002", + "negative_test_query339_00008", + "negative_test_query93_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.3191471544989227, + "recall@5": 0.5 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query125_00014", + "negative_test_query140_00017", + "negative_test_query360_00016", + "apositive_test_query125_00000", + "apositive_test_query125_00003", + "negative_test_query125_00003", + "negative_test_query125_00013", + "negative_test_query125_00004", + "negative_test_query125_00000", + "negative_test_query3_00000", + "negative_test_query125_00009", + "apositive_test_query273_00003", + "negative_test_query301_00012", + "negative_test_query125_00011", + "negative_test_query125_00006", + "negative_test_query12_00009", + "apositive_test_query286_00009", + "negative_test_query6_00003", + "apositive_test_query286_00006", + "negative_test_query260_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query126", + "ranked_ids": [ + "apositive_test_query126_00000", + "negative_test_query27_00013", + "negative_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query131_00009", + "apositive_test_query330_00001", + "negative_test_query151_00006", + "negative_test_query190_00005", + "negative_test_query311_00005", + "negative_test_query220_00000", + "negative_test_query12_00008", + "apositive_test_query330_00002", + "apositive_test_query5_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00004", + "apositive_test_query127_00008", + "apositive_test_query127_00000", + "apositive_test_query127_00003", + "apositive_test_query127_00007", + "negative_test_query94_00009", + "apositive_test_query127_00012", + "apositive_test_query127_00016", + "apositive_test_query94_00005", + "apositive_test_query127_00009", + "negative_test_query267_00009", + "apositive_test_query127_00006", + "negative_test_query94_00003", + "apositive_test_query127_00014", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "apositive_test_query127_00013", + "apositive_test_query163_00000", + "apositive_test_query311_00003", + "negative_test_query198_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query128", + "ranked_ids": [ + "negative_test_query256_00011", + "apositive_test_query128_00001", + "apositive_test_query128_00000", + "negative_test_query21_00017", + "negative_test_query283_00006", + "negative_test_query21_00000", + "apositive_test_query21_00000", + "negative_test_query128_00006", + "negative_test_query8_00017", + "apositive_test_query216_00002", + "apositive_test_query196_00004", + "apositive_test_query32_00001", + "apositive_test_query142_00000", + "apositive_test_query80_00005", + "negative_test_query310_00009", + "apositive_test_query8_00000", + "negative_test_query128_00010", + "negative_test_query128_00011", + "negative_test_query192_00006", + "negative_test_query75_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query129", + "ranked_ids": [ + "apositive_test_query129_00000", + "negative_test_query211_00012", + "negative_test_query156_00017", + "apositive_test_query244_00001", + "negative_test_query63_00002", + "negative_test_query63_00006", + "negative_test_query129_00010", + "negative_test_query165_00012", + "negative_test_query245_00014", + "apositive_test_query356_00006", + "apositive_test_query129_00001", + "negative_test_query117_00006", + "negative_test_query84_00008", + "apositive_test_query150_00000", + "apositive_test_query308_00004", + "negative_test_query16_00007", + "negative_test_query187_00012", + "negative_test_query342_00013", + "apositive_test_query155_00002", + "apositive_test_query249_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query13", + "ranked_ids": [ + "negative_test_query13_00001", + "apositive_test_query13_00006", + "negative_test_query13_00002", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "apositive_test_query13_00002", + "apositive_test_query13_00000", + "negative_test_query13_00003", + "negative_test_query13_00000", + "apositive_test_query13_00007", + "apositive_test_query13_00009", + "apositive_test_query13_00008", + "apositive_test_query13_00010", + "negative_test_query13_00004", + "apositive_test_query13_00004", + "apositive_test_query13_00012", + "negative_test_query13_00006", + "apositive_test_query13_00003", + "negative_test_query13_00005", + "negative_test_query245_00015" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query130", + "ranked_ids": [ + "apositive_test_query130_00005", + "negative_test_query130_00001", + "negative_test_query198_00011", + "negative_test_query130_00008", + "negative_test_query198_00003", + "negative_test_query345_00002", + "negative_test_query90_00014", + "apositive_test_query284_00001", + "negative_test_query130_00006", + "negative_test_query284_00001", + "apositive_test_query284_00003", + "apositive_test_query90_00001", + "apositive_test_query323_00004", + "apositive_test_query130_00002", + "negative_test_query284_00013", + "negative_test_query284_00009", + "apositive_test_query233_00000", + "apositive_test_query275_00000", + "apositive_test_query30_00002", + "negative_test_query289_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query131", + "ranked_ids": [ + "apositive_test_query131_00000", + "negative_test_query131_00001", + "apositive_test_query131_00001", + "negative_test_query186_00012", + "negative_test_query131_00002", + "negative_test_query247_00004", + "negative_test_query313_00008", + "apositive_test_query106_00000", + "apositive_test_query344_00003", + "negative_test_query284_00002", + "negative_test_query357_00001", + "negative_test_query284_00006", + "apositive_test_query50_00008", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query158_00001", + "negative_test_query168_00013", + "negative_test_query184_00012", + "negative_test_query345_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "negative_test_query132_00006", + "negative_test_query132_00000", + "negative_test_query132_00018", + "negative_test_query81_00005", + "negative_test_query123_00012", + "negative_test_query81_00001", + "negative_test_query132_00010", + "negative_test_query304_00000", + "negative_test_query181_00008", + "negative_test_query132_00012", + "negative_test_query132_00004", + "negative_test_query184_00015", + "negative_test_query11_00006", + "negative_test_query240_00011", + "apositive_test_query161_00007", + "negative_test_query148_00011", + "negative_test_query132_00016", + "apositive_test_query187_00000", + "apositive_test_query3_00004", + "negative_test_query246_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.2 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query287_00001", + "apositive_test_query133_00001", + "apositive_test_query133_00000", + "apositive_test_query129_00004", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00014", + "apositive_test_query91_00002", + "apositive_test_query133_00008", + "negative_test_query206_00017", + "negative_test_query310_00002", + "negative_test_query59_00008", + "apositive_test_query133_00002", + "negative_test_query12_00014", + "apositive_test_query129_00006", + "apositive_test_query174_00011", + "apositive_test_query129_00002", + "negative_test_query260_00006", + "apositive_test_query129_00007", + "apositive_test_query129_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query134", + "ranked_ids": [ + "negative_test_query134_00010", + "negative_test_query134_00008", + "negative_test_query134_00006", + "apositive_test_query134_00002", + "apositive_test_query213_00001", + "negative_test_query359_00011", + "negative_test_query108_00003", + "apositive_test_query357_00000", + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query142_00001", + "negative_test_query238_00001", + "negative_test_query46_00011", + "negative_test_query107_00003", + "apositive_test_query259_00011", + "negative_test_query46_00004", + "negative_test_query134_00000", + "negative_test_query218_00009", + "apositive_test_query238_00001", + "apositive_test_query276_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00003", + "negative_test_query135_00017", + "apositive_test_query135_00001", + "negative_test_query135_00011", + "apositive_test_query133_00011", + "negative_test_query135_00001", + "negative_test_query135_00005", + "negative_test_query287_00009", + "negative_test_query135_00014", + "negative_test_query135_00015", + "negative_test_query135_00013", + "negative_test_query135_00004", + "negative_test_query135_00009", + "negative_test_query297_00017", + "negative_test_query135_00007", + "negative_test_query135_00008", + "apositive_test_query135_00000", + "apositive_test_query28_00008", + "negative_test_query325_00012", + "negative_test_query135_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query136", + "ranked_ids": [ + "negative_test_query128_00011", + "apositive_test_query136_00002", + "apositive_test_query136_00000", + "apositive_test_query136_00011", + "apositive_test_query136_00014", + "apositive_test_query136_00001", + "apositive_test_query136_00007", + "apositive_test_query136_00005", + "apositive_test_query136_00010", + "apositive_test_query136_00009", + "apositive_test_query136_00013", + "negative_test_query136_00003", + "negative_test_query301_00001", + "apositive_test_query136_00004", + "negative_test_query129_00001", + "apositive_test_query60_00014", + "negative_test_query60_00002", + "negative_test_query8_00010", + "negative_test_query245_00003", + "negative_test_query149_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query137", + "ranked_ids": [ + "negative_test_query137_00014", + "apositive_test_query137_00000", + "apositive_test_query253_00000", + "negative_test_query95_00004", + "negative_test_query137_00003", + "negative_test_query349_00006", + "negative_test_query207_00006", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query137_00012", + "negative_test_query13_00002", + "negative_test_query253_00007", + "apositive_test_query137_00002", + "negative_test_query137_00000", + "apositive_test_query137_00003", + "negative_test_query74_00001", + "negative_test_query149_00013", + "negative_test_query262_00009", + "negative_test_query74_00003", + "negative_test_query74_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query138", + "ranked_ids": [ + "apositive_test_query138_00000", + "negative_test_query138_00014", + "negative_test_query138_00003", + "negative_test_query138_00013", + "negative_test_query138_00006", + "negative_test_query138_00015", + "negative_test_query138_00008", + "negative_test_query138_00004", + "apositive_test_query138_00001", + "negative_test_query138_00001", + "apositive_test_query138_00002", + "negative_test_query138_00005", + "apositive_test_query244_00005", + "apositive_test_query23_00003", + "apositive_test_query356_00003", + "negative_test_query132_00009", + "negative_test_query213_00006", + "negative_test_query240_00000", + "negative_test_query183_00004", + "negative_test_query184_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query35_00001", + "apositive_test_query139_00000", + "apositive_test_query35_00000", + "apositive_test_query139_00001", + "apositive_test_query146_00007", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query123_00006", + "negative_test_query231_00012", + "negative_test_query269_00006", + "negative_test_query101_00006", + "negative_test_query262_00002", + "negative_test_query81_00002", + "negative_test_query320_00007", + "negative_test_query35_00010", + "apositive_test_query354_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query14", + "ranked_ids": [ + "negative_test_query155_00009", + "negative_test_query14_00007", + "negative_test_query14_00005", + "negative_test_query260_00014", + "apositive_test_query296_00005", + "apositive_test_query14_00003", + "negative_test_query53_00003", + "negative_test_query155_00001", + "negative_test_query14_00006", + "apositive_test_query216_00002", + "apositive_test_query77_00000", + "negative_test_query14_00001", + "negative_test_query179_00001", + "apositive_test_query64_00000", + "negative_test_query14_00004", + "negative_test_query102_00011", + "negative_test_query260_00000", + "negative_test_query199_00009", + "negative_test_query350_00015", + "apositive_test_query26_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00005", + "apositive_test_query281_00013", + "negative_test_query140_00001", + "negative_test_query140_00003", + "negative_test_query140_00016", + "apositive_test_query273_00005", + "negative_test_query140_00010", + "negative_test_query140_00018", + "negative_test_query140_00002", + "negative_test_query140_00014", + "negative_test_query140_00013", + "negative_test_query257_00012", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00009", + "negative_test_query18_00006", + "negative_test_query140_00007", + "apositive_test_query18_00003", + "negative_test_query18_00013", + "apositive_test_query18_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query141", + "ranked_ids": [ + "apositive_test_query288_00003", + "negative_test_query245_00008", + "apositive_test_query141_00006", + "apositive_test_query141_00005", + "negative_test_query245_00016", + "negative_test_query57_00002", + "negative_test_query141_00006", + "apositive_test_query141_00000", + "negative_test_query176_00003", + "negative_test_query25_00006", + "negative_test_query325_00008", + "negative_test_query308_00005", + "negative_test_query220_00011", + "negative_test_query344_00013", + "apositive_test_query288_00005", + "apositive_test_query288_00007", + "negative_test_query141_00010", + "negative_test_query245_00013", + "apositive_test_query246_00001", + "negative_test_query105_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query142", + "ranked_ids": [ + "apositive_test_query142_00001", + "negative_test_query238_00001", + "negative_test_query102_00001", + "negative_test_query142_00003", + "negative_test_query46_00002", + "negative_test_query142_00009", + "negative_test_query46_00001", + "negative_test_query46_00012", + "negative_test_query105_00006", + "apositive_test_query142_00003", + "negative_test_query203_00003", + "negative_test_query223_00005", + "negative_test_query359_00005", + "negative_test_query39_00008", + "negative_test_query142_00007", + "apositive_test_query142_00002", + "negative_test_query105_00008", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query142_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query143", + "ranked_ids": [ + "apositive_test_query143_00000", + "negative_test_query143_00006", + "negative_test_query143_00011", + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query143_00005", + "negative_test_query143_00009", + "apositive_test_query25_00006", + "negative_test_query143_00000", + "negative_test_query143_00012", + "negative_test_query90_00014", + "negative_test_query143_00010", + "negative_test_query143_00013", + "negative_test_query143_00015", + "negative_test_query143_00007", + "negative_test_query143_00014", + "negative_test_query143_00016", + "apositive_test_query25_00005", + "negative_test_query56_00005", + "apositive_test_query308_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query41_00009", + "negative_test_query144_00010", + "negative_test_query144_00012", + "negative_test_query331_00015", + "negative_test_query144_00018", + "negative_test_query41_00001", + "negative_test_query315_00011", + "negative_test_query144_00011", + "negative_test_query144_00016", + "negative_test_query144_00001", + "negative_test_query144_00003", + "negative_test_query144_00006", + "negative_test_query144_00017", + "negative_test_query166_00001", + "apositive_test_query41_00000", + "negative_test_query144_00009", + "negative_test_query226_00011", + "negative_test_query41_00011", + "negative_test_query154_00009", + "negative_test_query198_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query145", + "ranked_ids": [ + "negative_test_query230_00002", + "negative_test_query53_00012", + "apositive_test_query213_00004", + "apositive_test_query160_00010", + "negative_test_query118_00000", + "negative_test_query79_00005", + "apositive_test_query28_00008", + "apositive_test_query225_00002", + "negative_test_query112_00007", + "negative_test_query171_00003", + "apositive_test_query145_00002", + "apositive_test_query107_00001", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "apositive_test_query207_00004", + "apositive_test_query305_00002", + "negative_test_query187_00014", + "negative_test_query190_00012", + "negative_test_query211_00013", + "negative_test_query305_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.0625 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query146_00004", + "apositive_test_query3_00010", + "apositive_test_query153_00008", + "negative_test_query358_00008", + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query146_00010", + "apositive_test_query314_00004", + "apositive_test_query146_00003", + "negative_test_query144_00008", + "apositive_test_query161_00007", + "apositive_test_query314_00001", + "negative_test_query65_00006", + "apositive_test_query314_00003", + "apositive_test_query146_00005", + "apositive_test_query161_00008", + "negative_test_query146_00001", + "negative_test_query99_00001", + "apositive_test_query161_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query147", + "ranked_ids": [ + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query147_00008", + "negative_test_query309_00014", + "apositive_test_query161_00004", + "negative_test_query11_00009", + "negative_test_query147_00013", + "negative_test_query258_00008", + "negative_test_query53_00009", + "negative_test_query200_00011", + "apositive_test_query147_00002", + "apositive_test_query76_00001", + "negative_test_query309_00016", + "negative_test_query354_00007", + "apositive_test_query147_00003", + "negative_test_query147_00005", + "apositive_test_query76_00004", + "negative_test_query181_00008", + "negative_test_query17_00009", + "negative_test_query180_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query148_00005", + "negative_test_query124_00010", + "negative_test_query148_00012", + "negative_test_query148_00018", + "negative_test_query210_00013", + "negative_test_query89_00007", + "negative_test_query148_00017", + "negative_test_query210_00008", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query112_00013", + "negative_test_query148_00008", + "negative_test_query196_00004", + "negative_test_query210_00006", + "negative_test_query315_00002", + "negative_test_query321_00013", + "negative_test_query80_00007", + "apositive_test_query304_00001", + "negative_test_query148_00014", + "negative_test_query210_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "negative_test_query149_00001", + "negative_test_query149_00012", + "negative_test_query149_00010", + "negative_test_query149_00011", + "negative_test_query149_00002", + "negative_test_query149_00008", + "apositive_test_query225_00004", + "negative_test_query332_00008", + "apositive_test_query145_00001", + "negative_test_query149_00017", + "negative_test_query225_00006", + "negative_test_query149_00015", + "negative_test_query63_00009", + "negative_test_query149_00013", + "apositive_test_query113_00003", + "apositive_test_query149_00000", + "apositive_test_query149_00001", + "negative_test_query332_00004", + "negative_test_query74_00007", + "negative_test_query85_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00010", + "negative_test_query15_00012", + "negative_test_query15_00008", + "negative_test_query15_00007", + "negative_test_query15_00013", + "negative_test_query15_00014", + "apositive_test_query325_00000", + "negative_test_query176_00011", + "negative_test_query15_00015", + "apositive_test_query15_00001", + "negative_test_query15_00004", + "negative_test_query15_00011", + "negative_test_query325_00011", + "negative_test_query15_00005", + "negative_test_query15_00017", + "negative_test_query266_00013", + "negative_test_query266_00006", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query266_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query150", + "ranked_ids": [ + "negative_test_query150_00010", + "apositive_test_query150_00003", + "negative_test_query250_00013", + "negative_test_query139_00005", + "negative_test_query306_00004", + "negative_test_query1_00012", + "apositive_test_query20_00010", + "negative_test_query62_00009", + "negative_test_query301_00007", + "negative_test_query213_00007", + "negative_test_query346_00008", + "negative_test_query63_00004", + "negative_test_query267_00006", + "apositive_test_query89_00000", + "negative_test_query254_00003", + "negative_test_query350_00018", + "apositive_test_query158_00001", + "negative_test_query313_00017", + "apositive_test_query112_00000", + "apositive_test_query20_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "negative_test_query190_00003", + "apositive_test_query126_00000", + "apositive_test_query300_00001", + "negative_test_query190_00014", + "negative_test_query151_00006", + "negative_test_query190_00005", + "negative_test_query293_00008", + "negative_test_query151_00003", + "negative_test_query44_00002", + "negative_test_query151_00015", + "negative_test_query190_00015", + "negative_test_query201_00008", + "negative_test_query151_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query237_00011", + "apositive_test_query152_00004", + "apositive_test_query152_00009", + "negative_test_query237_00000", + "negative_test_query312_00014", + "negative_test_query237_00004", + "negative_test_query124_00010", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query152_00005", + "negative_test_query349_00001", + "negative_test_query342_00005", + "negative_test_query95_00017", + "negative_test_query349_00005", + "negative_test_query253_00004", + "negative_test_query182_00004", + "negative_test_query250_00002", + "apositive_test_query194_00008", + "negative_test_query237_00009", + "negative_test_query262_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.1875 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00003", + "apositive_test_query153_00002", + "apositive_test_query153_00004", + "negative_test_query283_00015", + "negative_test_query112_00007", + "negative_test_query278_00006", + "apositive_test_query153_00000", + "apositive_test_query153_00015", + "negative_test_query359_00003", + "negative_test_query278_00007", + "apositive_test_query153_00007", + "apositive_test_query153_00001", + "apositive_test_query20_00006", + "negative_test_query112_00015", + "negative_test_query228_00002", + "negative_test_query2_00001", + "negative_test_query322_00001", + "apositive_test_query213_00004", + "apositive_test_query176_00000", + "negative_test_query250_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query154", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query20_00003", + "apositive_test_query20_00007", + "apositive_test_query23_00004", + "apositive_test_query356_00005", + "apositive_test_query154_00006", + "apositive_test_query33_00000", + "negative_test_query41_00007", + "negative_test_query331_00012", + "apositive_test_query358_00008", + "negative_test_query133_00002", + "apositive_test_query20_00010", + "apositive_test_query154_00005", + "negative_test_query245_00006", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "negative_test_query334_00000", + "negative_test_query311_00005", + "negative_test_query68_00015", + "negative_test_query102_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00006", + "negative_test_query155_00005", + "negative_test_query155_00003", + "negative_test_query155_00009", + "negative_test_query155_00004", + "apositive_test_query155_00005", + "negative_test_query155_00010", + "negative_test_query155_00001", + "negative_test_query155_00012", + "apositive_test_query355_00007", + "negative_test_query119_00001", + "negative_test_query199_00016", + "negative_test_query111_00016", + "apositive_test_query3_00008", + "negative_test_query155_00000", + "apositive_test_query40_00004", + "negative_test_query99_00001", + "apositive_test_query155_00001", + "negative_test_query123_00014", + "negative_test_query155_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "apositive_test_query156_00000", + "negative_test_query92_00010", + "negative_test_query1_00012", + "negative_test_query283_00001", + "negative_test_query92_00003", + "negative_test_query283_00009", + "apositive_test_query330_00002", + "negative_test_query49_00005", + "apositive_test_query39_00002", + "apositive_test_query154_00005", + "negative_test_query92_00005", + "apositive_test_query210_00001", + "negative_test_query330_00006", + "negative_test_query156_00001", + "apositive_test_query25_00009", + "negative_test_query246_00016", + "negative_test_query140_00008", + "negative_test_query15_00010", + "negative_test_query299_00014", + "negative_test_query32_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "negative_test_query237_00004", + "negative_test_query237_00009", + "negative_test_query157_00005", + "apositive_test_query237_00000", + "apositive_test_query152_00001", + "apositive_test_query152_00005", + "negative_test_query157_00010", + "apositive_test_query157_00001", + "negative_test_query152_00003", + "negative_test_query237_00002", + "negative_test_query152_00008", + "apositive_test_query182_00000", + "apositive_test_query157_00002", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query271_00003", + "apositive_test_query237_00002", + "apositive_test_query152_00000", + "apositive_test_query237_00004", + "negative_test_query312_00017" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.1 + }, + "query_id": "test_query158", + "ranked_ids": [ + "negative_test_query102_00015", + "negative_test_query334_00016", + "negative_test_query223_00007", + "negative_test_query26_00004", + "apositive_test_query158_00000", + "negative_test_query223_00002", + "negative_test_query26_00005", + "apositive_test_query158_00003", + "apositive_test_query186_00003", + "negative_test_query105_00013", + "negative_test_query158_00006", + "negative_test_query164_00004", + "negative_test_query247_00010", + "negative_test_query26_00003", + "negative_test_query102_00003", + "negative_test_query213_00002", + "negative_test_query26_00000", + "negative_test_query334_00009", + "negative_test_query26_00010", + "apositive_test_query158_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query159", + "ranked_ids": [ + "apositive_test_query159_00001", + "negative_test_query264_00013", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query159_00016", + "negative_test_query38_00010", + "negative_test_query159_00005", + "apositive_test_query50_00004", + "apositive_test_query212_00001", + "apositive_test_query326_00007", + "apositive_test_query38_00003", + "apositive_test_query212_00002", + "negative_test_query264_00003", + "negative_test_query66_00003", + "negative_test_query7_00005", + "apositive_test_query212_00003", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query122_00002", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query16", + "ranked_ids": [ + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query308_00004", + "apositive_test_query16_00004", + "negative_test_query16_00008", + "negative_test_query16_00001", + "negative_test_query16_00000", + "negative_test_query16_00010", + "negative_test_query16_00009", + "apositive_test_query16_00002", + "negative_test_query16_00012", + "negative_test_query16_00011", + "negative_test_query280_00007", + "negative_test_query16_00002", + "negative_test_query240_00003", + "negative_test_query237_00010", + "negative_test_query16_00003", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query208_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query160", + "ranked_ids": [ + "negative_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query330_00005", + "apositive_test_query160_00009", + "negative_test_query160_00004", + "negative_test_query211_00009", + "negative_test_query64_00009", + "negative_test_query64_00015", + "negative_test_query149_00007", + "apositive_test_query64_00000", + "apositive_test_query355_00007", + "negative_test_query271_00010", + "negative_test_query304_00007", + "apositive_test_query120_00002", + "negative_test_query214_00013", + "negative_test_query215_00010", + "apositive_test_query160_00004", + "apositive_test_query260_00000", + "negative_test_query106_00004", + "negative_test_query176_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query161", + "ranked_ids": [ + "apositive_test_query161_00006", + "negative_test_query155_00012", + "apositive_test_query3_00006", + "apositive_test_query276_00010", + "apositive_test_query305_00002", + "apositive_test_query161_00008", + "apositive_test_query161_00005", + "negative_test_query22_00015", + "apositive_test_query37_00004", + "negative_test_query199_00004", + "negative_test_query71_00006", + "negative_test_query260_00004", + "negative_test_query57_00001", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "apositive_test_query261_00004", + "negative_test_query301_00015", + "apositive_test_query314_00004", + "apositive_test_query301_00000", + "apositive_test_query314_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.25 + }, + "query_id": "test_query162", + "ranked_ids": [ + "negative_test_query177_00002", + "apositive_test_query162_00010", + "apositive_test_query162_00009", + "negative_test_query162_00004", + "apositive_test_query162_00003", + "apositive_test_query162_00000", + "negative_test_query162_00001", + "apositive_test_query162_00006", + "negative_test_query95_00006", + "negative_test_query240_00008", + "apositive_test_query162_00005", + "apositive_test_query194_00006", + "apositive_test_query162_00004", + "negative_test_query95_00013", + "negative_test_query177_00005", + "apositive_test_query194_00009", + "negative_test_query343_00012", + "apositive_test_query114_00001", + "apositive_test_query194_00003", + "negative_test_query230_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query197_00002", + "negative_test_query163_00008", + "negative_test_query90_00011", + "negative_test_query58_00015", + "negative_test_query328_00006", + "negative_test_query47_00001", + "negative_test_query265_00015", + "negative_test_query9_00005", + "negative_test_query202_00002", + "apositive_test_query172_00006", + "apositive_test_query357_00004", + "negative_test_query16_00013", + "negative_test_query163_00011", + "negative_test_query245_00001", + "negative_test_query163_00003", + "negative_test_query163_00013", + "negative_test_query197_00001", + "negative_test_query224_00010", + "negative_test_query350_00017", + "apositive_test_query152_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query164_00000", + "negative_test_query131_00014", + "negative_test_query2_00000", + "apositive_test_query146_00014", + "negative_test_query332_00001", + "apositive_test_query13_00000", + "negative_test_query158_00008", + "negative_test_query164_00015", + "negative_test_query165_00003", + "negative_test_query101_00005", + "negative_test_query208_00014", + "negative_test_query360_00012", + "negative_test_query100_00013", + "negative_test_query1_00003", + "negative_test_query245_00005", + "negative_test_query321_00011", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query309_00012", + "apositive_test_query297_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query165_00013", + "negative_test_query122_00003", + "negative_test_query86_00006", + "negative_test_query86_00010", + "negative_test_query165_00007", + "negative_test_query165_00000", + "negative_test_query285_00007", + "negative_test_query285_00004", + "negative_test_query348_00001", + "negative_test_query264_00016", + "negative_test_query215_00013", + "negative_test_query149_00012", + "negative_test_query299_00011", + "negative_test_query327_00010", + "negative_test_query159_00017", + "negative_test_query264_00017", + "apositive_test_query165_00001", + "negative_test_query327_00016", + "negative_test_query38_00010", + "apositive_test_query219_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query166", + "ranked_ids": [ + "apositive_test_query166_00005", + "negative_test_query166_00000", + "negative_test_query166_00013", + "negative_test_query166_00012", + "negative_test_query166_00006", + "apositive_test_query166_00002", + "apositive_test_query166_00003", + "negative_test_query166_00007", + "negative_test_query166_00005", + "negative_test_query166_00004", + "apositive_test_query166_00001", + "negative_test_query166_00003", + "negative_test_query166_00001", + "negative_test_query249_00000", + "negative_test_query224_00014", + "apositive_test_query224_00000", + "negative_test_query165_00015", + "apositive_test_query166_00000", + "negative_test_query166_00008", + "negative_test_query255_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query167_00002", + "negative_test_query167_00016", + "negative_test_query167_00004", + "negative_test_query167_00011", + "negative_test_query167_00005", + "negative_test_query167_00010", + "negative_test_query167_00012", + "negative_test_query167_00014", + "negative_test_query167_00003", + "negative_test_query167_00001", + "negative_test_query167_00013", + "negative_test_query167_00000", + "negative_test_query167_00008", + "negative_test_query167_00015", + "negative_test_query167_00007", + "negative_test_query167_00009", + "negative_test_query278_00016", + "negative_test_query278_00010", + "apositive_test_query288_00005", + "negative_test_query141_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6096199500078984, + "recall@5": 0.75 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query83_00003", + "apositive_test_query168_00000", + "apositive_test_query168_00001", + "apositive_test_query168_00002", + "apositive_test_query286_00007", + "apositive_test_query190_00001", + "negative_test_query6_00001", + "negative_test_query168_00000", + "negative_test_query6_00000", + "apositive_test_query340_00000", + "negative_test_query168_00014", + "negative_test_query83_00003", + "negative_test_query270_00003", + "negative_test_query6_00005", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query168_00007", + "apositive_test_query83_00004", + "negative_test_query260_00011", + "negative_test_query168_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "negative_test_query49_00017", + "negative_test_query72_00013", + "negative_test_query104_00001", + "apositive_test_query69_00000", + "negative_test_query72_00001", + "negative_test_query72_00016", + "apositive_test_query286_00008", + "apositive_test_query180_00001", + "negative_test_query343_00001", + "negative_test_query89_00006", + "negative_test_query260_00015", + "apositive_test_query242_00006", + "apositive_test_query291_00009", + "apositive_test_query330_00001", + "negative_test_query10_00005", + "negative_test_query180_00004", + "apositive_test_query266_00000", + "apositive_test_query80_00003", + "negative_test_query10_00013", + "negative_test_query169_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00005", + "negative_test_query17_00002", + "negative_test_query17_00014", + "negative_test_query17_00000", + "negative_test_query10_00005", + "negative_test_query34_00003", + "apositive_test_query225_00007", + "negative_test_query17_00015", + "negative_test_query17_00016", + "negative_test_query148_00001", + "negative_test_query17_00003", + "negative_test_query17_00012", + "negative_test_query305_00004", + "negative_test_query312_00004", + "negative_test_query302_00011", + "negative_test_query130_00007", + "negative_test_query164_00006", + "negative_test_query292_00010", + "apositive_test_query153_00013", + "apositive_test_query355_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query170", + "ranked_ids": [ + "apositive_test_query170_00006", + "apositive_test_query170_00000", + "apositive_test_query170_00002", + "apositive_test_query170_00005", + "negative_test_query170_00001", + "negative_test_query170_00005", + "apositive_test_query170_00004", + "apositive_test_query297_00000", + "negative_test_query0_00006", + "negative_test_query170_00007", + "negative_test_query170_00008", + "negative_test_query348_00001", + "negative_test_query297_00005", + "negative_test_query227_00002", + "apositive_test_query170_00001", + "negative_test_query170_00000", + "negative_test_query170_00004", + "apositive_test_query170_00007", + "apositive_test_query70_00003", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query171", + "ranked_ids": [ + "negative_test_query171_00013", + "negative_test_query171_00000", + "negative_test_query171_00005", + "apositive_test_query171_00001", + "negative_test_query171_00004", + "negative_test_query171_00011", + "negative_test_query219_00010", + "negative_test_query171_00015", + "negative_test_query277_00007", + "negative_test_query261_00000", + "apositive_test_query224_00000", + "negative_test_query171_00001", + "negative_test_query126_00013", + "negative_test_query171_00007", + "negative_test_query171_00017", + "negative_test_query191_00006", + "negative_test_query24_00007", + "negative_test_query346_00005", + "negative_test_query171_00003", + "negative_test_query219_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.25 + }, + "query_id": "test_query172", + "ranked_ids": [ + "apositive_test_query172_00003", + "apositive_test_query172_00004", + "negative_test_query350_00009", + "negative_test_query172_00007", + "apositive_test_query254_00003", + "apositive_test_query136_00011", + "negative_test_query208_00015", + "negative_test_query173_00006", + "apositive_test_query136_00001", + "negative_test_query8_00000", + "negative_test_query64_00006", + "apositive_test_query136_00009", + "negative_test_query192_00000", + "negative_test_query208_00007", + "negative_test_query349_00001", + "negative_test_query192_00010", + "apositive_test_query124_00002", + "apositive_test_query172_00006", + "apositive_test_query13_00003", + "apositive_test_query172_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query64_00017", + "negative_test_query173_00006", + "negative_test_query173_00003", + "negative_test_query64_00006", + "apositive_test_query173_00002", + "apositive_test_query173_00003", + "negative_test_query173_00011", + "negative_test_query173_00013", + "apositive_test_query173_00000", + "negative_test_query350_00002", + "negative_test_query173_00000", + "apositive_test_query173_00004", + "negative_test_query339_00006", + "apositive_test_query173_00001", + "apositive_test_query134_00000", + "apositive_test_query101_00004", + "negative_test_query303_00005", + "negative_test_query64_00004", + "negative_test_query64_00013", + "negative_test_query222_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.11764705882352941 + }, + "query_id": "test_query174", + "ranked_ids": [ + "negative_test_query19_00012", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "apositive_test_query4_00012", + "negative_test_query262_00005", + "apositive_test_query174_00012", + "apositive_test_query174_00002", + "apositive_test_query174_00003", + "apositive_test_query174_00004", + "apositive_test_query174_00005", + "apositive_test_query174_00006", + "apositive_test_query174_00008", + "apositive_test_query174_00009", + "apositive_test_query174_00010", + "apositive_test_query174_00007", + "apositive_test_query174_00011", + "apositive_test_query174_00015", + "apositive_test_query174_00014", + "apositive_test_query174_00013", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.5 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00000", + "apositive_test_query175_00003", + "negative_test_query175_00013", + "apositive_test_query175_00005", + "negative_test_query175_00010", + "apositive_test_query50_00015", + "apositive_test_query175_00004", + "negative_test_query38_00006", + "negative_test_query105_00012", + "apositive_test_query353_00005", + "apositive_test_query204_00010", + "negative_test_query204_00002", + "negative_test_query97_00007", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "negative_test_query105_00016", + "negative_test_query204_00000", + "negative_test_query7_00000", + "negative_test_query175_00000", + "negative_test_query204_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query176_00015", + "negative_test_query195_00006", + "apositive_test_query128_00007", + "negative_test_query195_00012", + "negative_test_query224_00003", + "negative_test_query221_00007", + "negative_test_query9_00003", + "negative_test_query191_00008", + "negative_test_query200_00005", + "negative_test_query337_00005", + "apositive_test_query344_00003", + "negative_test_query284_00002", + "negative_test_query284_00006", + "negative_test_query24_00014", + "apositive_test_query92_00002", + "negative_test_query144_00018", + "negative_test_query35_00005", + "negative_test_query93_00000", + "apositive_test_query56_00006", + "negative_test_query9_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query172_00008", + "apositive_test_query177_00004", + "negative_test_query177_00004", + "negative_test_query124_00012", + "negative_test_query177_00008", + "apositive_test_query177_00006", + "negative_test_query177_00006", + "apositive_test_query162_00009", + "negative_test_query147_00002", + "negative_test_query147_00005", + "apositive_test_query162_00010", + "apositive_test_query230_00005", + "negative_test_query152_00005", + "negative_test_query147_00009", + "apositive_test_query229_00008", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query124_00001", + "negative_test_query147_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query178", + "ranked_ids": [ + "negative_test_query178_00007", + "negative_test_query178_00001", + "apositive_test_query178_00005", + "apositive_test_query285_00006", + "negative_test_query165_00001", + "negative_test_query285_00009", + "negative_test_query274_00001", + "apositive_test_query178_00004", + "negative_test_query214_00002", + "negative_test_query43_00001", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query178_00003", + "apositive_test_query352_00001", + "negative_test_query327_00004", + "negative_test_query321_00017", + "negative_test_query178_00000", + "negative_test_query178_00010", + "negative_test_query178_00012", + "negative_test_query285_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "negative_test_query179_00014", + "apositive_test_query173_00000", + "apositive_test_query173_00003", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query303_00005", + "negative_test_query297_00000", + "apositive_test_query316_00001", + "negative_test_query316_00001", + "negative_test_query279_00010", + "negative_test_query206_00009", + "negative_test_query14_00001", + "negative_test_query173_00013", + "negative_test_query179_00001", + "negative_test_query191_00003", + "negative_test_query260_00012", + "negative_test_query64_00003", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query18", + "ranked_ids": [ + "apositive_test_query18_00001", + "apositive_test_query18_00000", + "apositive_test_query18_00003", + "apositive_test_query18_00002", + "apositive_test_query18_00004", + "negative_test_query18_00006", + "negative_test_query18_00011", + "negative_test_query18_00012", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00003", + "negative_test_query18_00013", + "apositive_test_query281_00013", + "negative_test_query88_00002", + "apositive_test_query288_00004", + "negative_test_query156_00018", + "negative_test_query257_00012", + "negative_test_query287_00004", + "negative_test_query88_00012", + "negative_test_query300_00015" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query180", + "ranked_ids": [ + "negative_test_query180_00002", + "apositive_test_query180_00000", + "apositive_test_query180_00001", + "negative_test_query180_00004", + "apositive_test_query262_00000", + "apositive_test_query262_00003", + "negative_test_query312_00018", + "negative_test_query180_00011", + "negative_test_query180_00007", + "negative_test_query180_00016", + "negative_test_query246_00003", + "negative_test_query180_00006", + "negative_test_query112_00014", + "negative_test_query169_00015", + "negative_test_query114_00003", + "negative_test_query207_00007", + "apositive_test_query180_00002", + "negative_test_query95_00014", + "negative_test_query176_00001", + "apositive_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00006", + "negative_test_query181_00017", + "negative_test_query181_00012", + "negative_test_query181_00005", + "negative_test_query181_00002", + "negative_test_query181_00014", + "negative_test_query181_00010", + "negative_test_query181_00004", + "negative_test_query181_00001", + "negative_test_query181_00018", + "negative_test_query181_00007", + "negative_test_query181_00016", + "apositive_test_query256_00003", + "negative_test_query96_00006", + "negative_test_query181_00009", + "negative_test_query348_00007", + "negative_test_query181_00008", + "negative_test_query320_00007", + "apositive_test_query119_00011", + "negative_test_query183_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "apositive_test_query182_00000", + "negative_test_query182_00011", + "negative_test_query182_00017", + "negative_test_query182_00004", + "negative_test_query182_00018", + "negative_test_query240_00001", + "negative_test_query237_00004", + "negative_test_query337_00012", + "negative_test_query230_00008", + "apositive_test_query152_00009", + "negative_test_query182_00002", + "negative_test_query237_00006", + "negative_test_query296_00010", + "negative_test_query237_00001", + "negative_test_query237_00007", + "negative_test_query342_00001", + "negative_test_query342_00003", + "negative_test_query342_00004", + "negative_test_query342_00009", + "negative_test_query77_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00006", + "negative_test_query183_00009", + "negative_test_query183_00014", + "negative_test_query7_00007", + "apositive_test_query7_00001", + "negative_test_query183_00012", + "negative_test_query183_00013", + "negative_test_query238_00007", + "negative_test_query183_00007", + "negative_test_query183_00008", + "negative_test_query7_00009", + "negative_test_query183_00018", + "negative_test_query7_00008", + "negative_test_query7_00014", + "negative_test_query183_00015", + "negative_test_query183_00000", + "apositive_test_query178_00001", + "negative_test_query112_00017", + "negative_test_query183_00010", + "negative_test_query7_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "negative_test_query216_00014", + "negative_test_query216_00007", + "apositive_test_query72_00000", + "negative_test_query158_00008", + "negative_test_query216_00008", + "apositive_test_query216_00004", + "apositive_test_query316_00000", + "apositive_test_query33_00006", + "negative_test_query101_00007", + "negative_test_query156_00001", + "negative_test_query344_00010", + "apositive_test_query33_00004", + "apositive_test_query216_00000", + "apositive_test_query216_00001", + "negative_test_query228_00011", + "negative_test_query322_00000", + "negative_test_query216_00006", + "negative_test_query41_00013", + "apositive_test_query20_00010", + "negative_test_query216_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query185", + "ranked_ids": [ + "apositive_test_query307_00008", + "apositive_test_query185_00003", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query72_00009", + "negative_test_query185_00002", + "negative_test_query350_00007", + "negative_test_query103_00000", + "negative_test_query103_00001", + "negative_test_query185_00000", + "negative_test_query72_00011", + "negative_test_query185_00005", + "negative_test_query72_00017", + "negative_test_query139_00014", + "negative_test_query48_00009", + "apositive_test_query43_00006", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query48_00014", + "negative_test_query72_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "negative_test_query247_00012", + "negative_test_query46_00001", + "apositive_test_query306_00003", + "negative_test_query186_00005", + "negative_test_query247_00007", + "negative_test_query47_00009", + "apositive_test_query79_00005", + "negative_test_query336_00008", + "negative_test_query313_00015", + "negative_test_query142_00009", + "negative_test_query29_00003", + "negative_test_query313_00005", + "negative_test_query105_00001", + "negative_test_query259_00003", + "apositive_test_query50_00000", + "apositive_test_query50_00001", + "negative_test_query57_00007", + "apositive_test_query50_00010", + "negative_test_query313_00014", + "negative_test_query334_00017" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query245_00006", + "negative_test_query176_00008", + "negative_test_query306_00004", + "negative_test_query230_00000", + "apositive_test_query311_00002", + "apositive_test_query176_00000", + "apositive_test_query358_00008", + "negative_test_query62_00009", + "negative_test_query168_00002", + "negative_test_query279_00002", + "apositive_test_query127_00014", + "apositive_test_query154_00005", + "negative_test_query184_00015", + "negative_test_query331_00002", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query33_00008", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "apositive_test_query20_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query188", + "ranked_ids": [ + "negative_test_query188_00008", + "negative_test_query20_00006", + "negative_test_query188_00010", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "apositive_test_query20_00006", + "negative_test_query344_00007", + "apositive_test_query20_00009", + "negative_test_query188_00009", + "negative_test_query112_00017", + "negative_test_query188_00003", + "negative_test_query188_00013", + "negative_test_query220_00011", + "negative_test_query344_00013", + "negative_test_query188_00004", + "negative_test_query188_00011", + "apositive_test_query188_00000", + "negative_test_query188_00014", + "apositive_test_query220_00004", + "apositive_test_query220_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00007", + "negative_test_query189_00004", + "apositive_test_query189_00002", + "negative_test_query189_00001", + "negative_test_query189_00006", + "negative_test_query189_00013", + "negative_test_query189_00000", + "negative_test_query85_00007", + "apositive_test_query85_00002", + "apositive_test_query189_00003", + "negative_test_query140_00016", + "negative_test_query44_00005", + "negative_test_query55_00003", + "apositive_test_query189_00001", + "negative_test_query260_00007", + "negative_test_query189_00002", + "negative_test_query18_00006", + "negative_test_query190_00001", + "negative_test_query287_00000", + "apositive_test_query254_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query19", + "ranked_ids": [ + "negative_test_query19_00003", + "apositive_test_query19_00000", + "negative_test_query15_00000", + "negative_test_query19_00006", + "negative_test_query19_00008", + "negative_test_query266_00010", + "negative_test_query4_00001", + "apositive_test_query174_00014", + "negative_test_query229_00002", + "apositive_test_query4_00000", + "apositive_test_query4_00010", + "negative_test_query169_00014", + "apositive_test_query174_00007", + "negative_test_query106_00004", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query141_00000", + "negative_test_query174_00001", + "negative_test_query141_00001", + "negative_test_query167_00018" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query271_00005", + "apositive_test_query311_00000", + "negative_test_query260_00007", + "negative_test_query85_00006", + "apositive_test_query260_00003", + "negative_test_query109_00006", + "negative_test_query6_00007", + "negative_test_query345_00006", + "apositive_test_query234_00001", + "negative_test_query190_00013", + "negative_test_query181_00003", + "negative_test_query246_00002", + "negative_test_query264_00008", + "negative_test_query233_00008", + "negative_test_query265_00004", + "negative_test_query85_00014", + "negative_test_query205_00016", + "apositive_test_query85_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "apositive_test_query191_00001", + "negative_test_query191_00010", + "negative_test_query96_00006", + "apositive_test_query63_00003", + "negative_test_query60_00000", + "apositive_test_query191_00002", + "negative_test_query332_00012", + "apositive_test_query341_00003", + "negative_test_query191_00012", + "negative_test_query297_00002", + "negative_test_query181_00013", + "negative_test_query256_00009", + "negative_test_query241_00007", + "apositive_test_query341_00005", + "negative_test_query191_00003", + "negative_test_query274_00002", + "negative_test_query191_00002", + "negative_test_query64_00003", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00002", + "negative_test_query120_00004", + "negative_test_query192_00001", + "negative_test_query192_00002", + "negative_test_query192_00013", + "apositive_test_query192_00000", + "negative_test_query172_00005", + "negative_test_query192_00008", + "negative_test_query192_00009", + "negative_test_query192_00010", + "negative_test_query350_00010", + "negative_test_query120_00007", + "negative_test_query350_00013", + "apositive_test_query192_00004", + "apositive_test_query254_00007", + "negative_test_query192_00004", + "negative_test_query192_00012", + "negative_test_query192_00014", + "negative_test_query322_00008", + "negative_test_query192_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query193", + "ranked_ids": [ + "apositive_test_query99_00003", + "apositive_test_query193_00009", + "apositive_test_query193_00001", + "apositive_test_query193_00004", + "apositive_test_query193_00011", + "apositive_test_query193_00005", + "negative_test_query99_00009", + "apositive_test_query153_00012", + "apositive_test_query193_00000", + "apositive_test_query99_00002", + "negative_test_query146_00001", + "apositive_test_query193_00008", + "negative_test_query193_00002", + "apositive_test_query146_00008", + "apositive_test_query276_00006", + "negative_test_query146_00003", + "negative_test_query193_00003", + "apositive_test_query193_00002", + "negative_test_query193_00006", + "negative_test_query193_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query194_00002", + "apositive_test_query194_00007", + "negative_test_query177_00007", + "negative_test_query162_00000", + "negative_test_query95_00016", + "apositive_test_query254_00006", + "negative_test_query148_00009", + "apositive_test_query124_00000", + "negative_test_query62_00014", + "apositive_test_query162_00006", + "apositive_test_query194_00006", + "negative_test_query262_00011", + "apositive_test_query194_00000", + "negative_test_query194_00003", + "negative_test_query89_00007", + "negative_test_query137_00011", + "apositive_test_query194_00001", + "apositive_test_query42_00001", + "negative_test_query262_00007", + "apositive_test_query114_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query195", + "ranked_ids": [ + "negative_test_query222_00003", + "apositive_test_query195_00002", + "negative_test_query195_00002", + "apositive_test_query195_00001", + "negative_test_query195_00013", + "apositive_test_query195_00005", + "negative_test_query109_00008", + "negative_test_query195_00001", + "negative_test_query285_00000", + "negative_test_query321_00005", + "negative_test_query149_00012", + "negative_test_query195_00004", + "negative_test_query321_00008", + "apositive_test_query249_00012", + "negative_test_query205_00006", + "apositive_test_query251_00000", + "negative_test_query195_00005", + "negative_test_query195_00008", + "negative_test_query166_00009", + "negative_test_query199_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query196_00003", + "apositive_test_query32_00011", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "negative_test_query64_00005", + "apositive_test_query32_00006", + "negative_test_query235_00008", + "negative_test_query32_00002", + "apositive_test_query32_00004", + "negative_test_query14_00001", + "negative_test_query179_00001", + "negative_test_query139_00011", + "negative_test_query173_00012", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query341_00008", + "negative_test_query325_00004", + "negative_test_query96_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.38364913903790027, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query197", + "ranked_ids": [ + "negative_test_query163_00001", + "negative_test_query197_00010", + "negative_test_query197_00016", + "apositive_test_query197_00000", + "apositive_test_query197_00002", + "negative_test_query197_00006", + "negative_test_query197_00013", + "apositive_test_query189_00001", + "negative_test_query178_00008", + "apositive_test_query34_00000", + "negative_test_query197_00004", + "negative_test_query321_00006", + "negative_test_query34_00012", + "negative_test_query28_00001", + "apositive_test_query163_00001", + "apositive_test_query197_00001", + "negative_test_query163_00002", + "apositive_test_query163_00000", + "negative_test_query163_00009", + "negative_test_query163_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query198_00000", + "negative_test_query112_00005", + "negative_test_query239_00011", + "negative_test_query239_00013", + "negative_test_query294_00002", + "negative_test_query239_00007", + "negative_test_query198_00010", + "negative_test_query239_00001", + "negative_test_query343_00002", + "negative_test_query62_00016", + "negative_test_query293_00010", + "negative_test_query293_00011", + "negative_test_query198_00002", + "negative_test_query198_00008", + "negative_test_query239_00010", + "negative_test_query352_00002", + "negative_test_query219_00007", + "negative_test_query198_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query199_00015", + "negative_test_query54_00003", + "negative_test_query305_00007", + "negative_test_query199_00014", + "negative_test_query139_00014", + "negative_test_query199_00005", + "negative_test_query199_00013", + "negative_test_query140_00014", + "negative_test_query250_00006", + "apositive_test_query43_00006", + "negative_test_query199_00006", + "negative_test_query185_00002", + "apositive_test_query244_00002", + "negative_test_query350_00007", + "negative_test_query199_00002", + "negative_test_query221_00008", + "negative_test_query213_00011", + "negative_test_query72_00005", + "negative_test_query199_00001", + "negative_test_query134_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query190_00016", + "negative_test_query2_00013", + "negative_test_query2_00012", + "negative_test_query44_00008", + "negative_test_query2_00009", + "negative_test_query2_00002", + "negative_test_query2_00011", + "negative_test_query352_00003", + "negative_test_query2_00010", + "apositive_test_query217_00006", + "negative_test_query197_00004", + "negative_test_query343_00004", + "negative_test_query85_00007", + "negative_test_query2_00016", + "negative_test_query15_00000", + "negative_test_query348_00005", + "negative_test_query125_00012", + "negative_test_query126_00014", + "negative_test_query104_00007", + "apositive_test_query57_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query154_00006", + "apositive_test_query20_00010", + "apositive_test_query20_00007", + "apositive_test_query23_00004", + "apositive_test_query356_00005", + "negative_test_query154_00002", + "apositive_test_query20_00003", + "apositive_test_query154_00005", + "negative_test_query66_00002", + "apositive_test_query20_00009", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "apositive_test_query33_00008", + "negative_test_query188_00000", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "negative_test_query220_00005", + "apositive_test_query33_00000", + "negative_test_query33_00008", + "apositive_test_query154_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query200_00009", + "negative_test_query200_00000", + "negative_test_query200_00015", + "negative_test_query337_00009", + "negative_test_query337_00004", + "negative_test_query264_00006", + "negative_test_query200_00006", + "negative_test_query200_00012", + "negative_test_query200_00004", + "negative_test_query200_00013", + "negative_test_query38_00005", + "negative_test_query200_00005", + "negative_test_query200_00010", + "negative_test_query337_00005", + "negative_test_query8_00008", + "negative_test_query264_00007", + "negative_test_query88_00013", + "negative_test_query200_00007", + "negative_test_query337_00007", + "negative_test_query303_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query201", + "ranked_ids": [ + "negative_test_query259_00002", + "negative_test_query71_00013", + "negative_test_query47_00012", + "apositive_test_query201_00006", + "negative_test_query0_00004", + "apositive_test_query30_00005", + "negative_test_query71_00000", + "negative_test_query71_00002", + "negative_test_query71_00005", + "negative_test_query65_00002", + "apositive_test_query71_00001", + "negative_test_query209_00008", + "negative_test_query71_00009", + "apositive_test_query20_00005", + "apositive_test_query30_00001", + "apositive_test_query0_00011", + "negative_test_query47_00013", + "negative_test_query159_00009", + "apositive_test_query201_00002", + "negative_test_query231_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query235_00001", + "negative_test_query202_00000", + "negative_test_query202_00004", + "negative_test_query202_00001", + "apositive_test_query206_00000", + "negative_test_query202_00008", + "negative_test_query202_00006", + "negative_test_query312_00012", + "apositive_test_query302_00006", + "negative_test_query202_00009", + "negative_test_query202_00016", + "negative_test_query176_00011", + "negative_test_query321_00018", + "negative_test_query202_00010", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query15_00010", + "apositive_test_query309_00000", + "negative_test_query188_00011", + "apositive_test_query61_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.1 + }, + "query_id": "test_query203", + "ranked_ids": [ + "negative_test_query203_00006", + "negative_test_query203_00008", + "negative_test_query203_00005", + "negative_test_query250_00005", + "apositive_test_query203_00004", + "apositive_test_query110_00003", + "apositive_test_query284_00003", + "negative_test_query284_00014", + "apositive_test_query130_00002", + "apositive_test_query203_00009", + "negative_test_query209_00000", + "apositive_test_query203_00003", + "negative_test_query130_00001", + "negative_test_query250_00009", + "negative_test_query250_00011", + "apositive_test_query30_00005", + "negative_test_query35_00004", + "apositive_test_query203_00007", + "negative_test_query250_00016", + "apositive_test_query203_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.058823529411764705 + }, + "query_id": "test_query204", + "ranked_ids": [ + "apositive_test_query353_00005", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "negative_test_query204_00002", + "negative_test_query97_00007", + "apositive_test_query204_00002", + "negative_test_query353_00003", + "negative_test_query204_00001", + "apositive_test_query175_00004", + "apositive_test_query204_00009", + "apositive_test_query353_00000", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "negative_test_query204_00000", + "apositive_test_query353_00011", + "apositive_test_query97_00001", + "apositive_test_query204_00007", + "negative_test_query353_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query96_00011", + "negative_test_query228_00016", + "negative_test_query205_00007", + "negative_test_query256_00015", + "negative_test_query341_00003", + "negative_test_query256_00000", + "negative_test_query205_00015", + "negative_test_query256_00007", + "negative_test_query73_00000", + "negative_test_query73_00011", + "negative_test_query256_00006", + "apositive_test_query205_00000", + "negative_test_query35_00007", + "negative_test_query205_00008", + "negative_test_query31_00015", + "negative_test_query35_00008", + "apositive_test_query256_00002", + "negative_test_query115_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query206_00002", + "negative_test_query206_00007", + "negative_test_query206_00015", + "negative_test_query206_00014", + "negative_test_query303_00015", + "negative_test_query206_00000", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00001", + "negative_test_query206_00005", + "negative_test_query303_00003", + "apositive_test_query348_00001", + "negative_test_query70_00011", + "negative_test_query206_00003", + "negative_test_query299_00001", + "negative_test_query2_00016", + "negative_test_query206_00018", + "negative_test_query299_00002", + "negative_test_query64_00016", + "negative_test_query206_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00000", + "negative_test_query207_00006", + "apositive_test_query207_00004", + "negative_test_query207_00002", + "negative_test_query207_00011", + "negative_test_query207_00009", + "negative_test_query207_00005", + "apositive_test_query207_00005", + "apositive_test_query207_00002", + "negative_test_query207_00001", + "apositive_test_query207_00001", + "negative_test_query207_00008", + "negative_test_query207_00012", + "negative_test_query207_00000", + "negative_test_query207_00003", + "negative_test_query207_00007", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query37_00004", + "negative_test_query207_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "apositive_test_query208_00000", + "negative_test_query128_00003", + "negative_test_query208_00016", + "negative_test_query208_00000", + "negative_test_query208_00010", + "negative_test_query208_00017", + "negative_test_query208_00018", + "apositive_test_query128_00002", + "negative_test_query208_00008", + "negative_test_query73_00008", + "negative_test_query128_00001", + "apositive_test_query128_00007", + "negative_test_query256_00011", + "negative_test_query208_00003", + "negative_test_query208_00012", + "negative_test_query61_00012", + "negative_test_query208_00004", + "negative_test_query208_00006", + "negative_test_query208_00009", + "negative_test_query211_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query209", + "ranked_ids": [ + "apositive_test_query209_00001", + "apositive_test_query37_00001", + "negative_test_query209_00007", + "negative_test_query102_00010", + "negative_test_query231_00006", + "apositive_test_query26_00007", + "negative_test_query347_00004", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query223_00003", + "negative_test_query26_00002", + "apositive_test_query119_00008", + "apositive_test_query158_00004", + "apositive_test_query158_00009", + "negative_test_query102_00016", + "negative_test_query209_00001", + "negative_test_query223_00007", + "negative_test_query26_00004", + "negative_test_query313_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "negative_test_query224_00015", + "negative_test_query10_00008", + "apositive_test_query128_00003", + "apositive_test_query128_00002", + "negative_test_query234_00007", + "negative_test_query21_00010", + "negative_test_query8_00006", + "negative_test_query128_00004", + "negative_test_query208_00000", + "negative_test_query21_00005", + "negative_test_query21_00009", + "negative_test_query330_00000", + "apositive_test_query21_00000", + "negative_test_query128_00006", + "apositive_test_query128_00004", + "negative_test_query208_00017", + "negative_test_query21_00012", + "negative_test_query234_00006", + "negative_test_query21_00001", + "negative_test_query350_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query210_00013", + "negative_test_query210_00015", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query148_00014", + "negative_test_query202_00014", + "negative_test_query148_00008", + "negative_test_query156_00015", + "negative_test_query17_00002", + "negative_test_query300_00000", + "negative_test_query288_00007", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query210_00007", + "negative_test_query210_00008", + "negative_test_query148_00007", + "negative_test_query210_00010", + "negative_test_query199_00015", + "negative_test_query288_00006", + "negative_test_query335_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query211", + "ranked_ids": [ + "negative_test_query159_00006", + "apositive_test_query211_00001", + "negative_test_query239_00014", + "negative_test_query137_00012", + "apositive_test_query176_00000", + "apositive_test_query198_00000", + "negative_test_query189_00008", + "negative_test_query32_00005", + "negative_test_query100_00012", + "negative_test_query195_00007", + "apositive_test_query162_00010", + "negative_test_query167_00015", + "negative_test_query293_00013", + "negative_test_query211_00012", + "negative_test_query202_00006", + "negative_test_query83_00010", + "apositive_test_query101_00000", + "negative_test_query211_00000", + "negative_test_query211_00015", + "apositive_test_query163_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query212", + "ranked_ids": [ + "apositive_test_query212_00004", + "apositive_test_query326_00004", + "apositive_test_query326_00000", + "negative_test_query167_00016", + "apositive_test_query98_00000", + "negative_test_query167_00017", + "negative_test_query183_00009", + "negative_test_query346_00006", + "apositive_test_query244_00007", + "negative_test_query183_00012", + "negative_test_query282_00010", + "negative_test_query344_00008", + "apositive_test_query66_00005", + "negative_test_query282_00009", + "apositive_test_query71_00001", + "negative_test_query126_00017", + "negative_test_query190_00017", + "negative_test_query77_00004", + "apositive_test_query11_00007", + "apositive_test_query89_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00000", + "apositive_test_query213_00002", + "apositive_test_query213_00004", + "negative_test_query213_00001", + "negative_test_query213_00005", + "negative_test_query137_00015", + "negative_test_query213_00011", + "negative_test_query224_00015", + "negative_test_query138_00016", + "negative_test_query213_00004", + "apositive_test_query213_00003", + "apositive_test_query305_00003", + "apositive_test_query36_00002", + "apositive_test_query259_00001", + "apositive_test_query259_00000", + "negative_test_query138_00001", + "negative_test_query112_00003", + "apositive_test_query158_00001", + "negative_test_query313_00017", + "negative_test_query357_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "apositive_test_query214_00000", + "negative_test_query214_00013", + "negative_test_query214_00001", + "negative_test_query214_00003", + "apositive_test_query323_00003", + "negative_test_query214_00004", + "apositive_test_query234_00001", + "negative_test_query214_00015", + "negative_test_query17_00013", + "apositive_test_query89_00000", + "negative_test_query324_00002", + "negative_test_query214_00010", + "apositive_test_query323_00002", + "negative_test_query214_00016", + "negative_test_query9_00008", + "negative_test_query324_00012", + "apositive_test_query160_00009", + "apositive_test_query75_00003", + "apositive_test_query17_00001", + "negative_test_query103_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00012", + "apositive_test_query215_00000", + "negative_test_query215_00008", + "negative_test_query215_00006", + "negative_test_query215_00007", + "negative_test_query215_00018", + "negative_test_query215_00009", + "negative_test_query215_00010", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query215_00015", + "negative_test_query215_00001", + "negative_test_query263_00001", + "negative_test_query215_00011", + "negative_test_query257_00001", + "negative_test_query293_00013", + "negative_test_query295_00004", + "negative_test_query215_00017", + "apositive_test_query80_00005", + "negative_test_query215_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query216_00000", + "negative_test_query216_00007", + "negative_test_query216_00002", + "negative_test_query216_00008", + "negative_test_query246_00016", + "apositive_test_query154_00005", + "negative_test_query216_00014", + "negative_test_query216_00006", + "apositive_test_query216_00004", + "apositive_test_query216_00001", + "apositive_test_query216_00002", + "negative_test_query216_00000", + "apositive_test_query20_00001", + "negative_test_query112_00002", + "negative_test_query44_00002", + "negative_test_query216_00005", + "negative_test_query216_00009", + "apositive_test_query33_00004", + "apositive_test_query182_00000", + "apositive_test_query330_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "negative_test_query217_00004", + "negative_test_query217_00011", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00000", + "apositive_test_query213_00004", + "negative_test_query28_00002", + "apositive_test_query307_00010", + "negative_test_query112_00007", + "negative_test_query230_00002", + "negative_test_query53_00012", + "negative_test_query76_00003", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query72_00005", + "apositive_test_query107_00001", + "apositive_test_query160_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query218", + "ranked_ids": [ + "negative_test_query134_00003", + "negative_test_query218_00009", + "apositive_test_query218_00004", + "negative_test_query218_00000", + "negative_test_query218_00012", + "apositive_test_query218_00003", + "negative_test_query218_00003", + "negative_test_query218_00002", + "negative_test_query274_00008", + "negative_test_query43_00006", + "apositive_test_query218_00005", + "negative_test_query304_00005", + "apositive_test_query314_00015", + "apositive_test_query134_00000", + "negative_test_query58_00010", + "negative_test_query134_00009", + "apositive_test_query60_00012", + "negative_test_query43_00010", + "apositive_test_query218_00000", + "negative_test_query165_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "apositive_test_query219_00000", + "negative_test_query219_00014", + "negative_test_query219_00016", + "apositive_test_query219_00001", + "negative_test_query187_00015", + "negative_test_query308_00006", + "negative_test_query165_00010", + "negative_test_query59_00000", + "negative_test_query219_00008", + "negative_test_query219_00012", + "negative_test_query219_00015", + "apositive_test_query177_00004", + "negative_test_query219_00011", + "negative_test_query227_00007", + "negative_test_query219_00017", + "negative_test_query305_00005", + "negative_test_query183_00016", + "negative_test_query219_00005", + "negative_test_query171_00005", + "negative_test_query24_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "negative_test_query22_00003", + "apositive_test_query22_00000", + "negative_test_query22_00008", + "negative_test_query22_00015", + "negative_test_query22_00000", + "negative_test_query22_00005", + "negative_test_query22_00007", + "negative_test_query22_00009", + "negative_test_query22_00017", + "negative_test_query22_00018", + "negative_test_query22_00004", + "negative_test_query22_00002", + "negative_test_query22_00012", + "negative_test_query22_00013", + "negative_test_query22_00016", + "negative_test_query66_00004", + "negative_test_query22_00006", + "negative_test_query22_00001", + "apositive_test_query162_00007", + "apositive_test_query177_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.5 + }, + "query_id": "test_query220", + "ranked_ids": [ + "negative_test_query344_00007", + "apositive_test_query220_00003", + "apositive_test_query220_00000", + "apositive_test_query220_00001", + "negative_test_query220_00011", + "negative_test_query344_00013", + "apositive_test_query188_00000", + "negative_test_query188_00014", + "apositive_test_query220_00004", + "negative_test_query188_00013", + "negative_test_query188_00009", + "negative_test_query20_00002", + "negative_test_query220_00004", + "negative_test_query188_00004", + "negative_test_query154_00009", + "negative_test_query188_00008", + "negative_test_query188_00007", + "apositive_test_query20_00009", + "apositive_test_query146_00009", + "negative_test_query161_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "negative_test_query65_00007", + "apositive_test_query142_00004", + "negative_test_query277_00010", + "apositive_test_query174_00006", + "apositive_test_query180_00002", + "negative_test_query278_00004", + "apositive_test_query306_00006", + "negative_test_query15_00005", + "negative_test_query184_00001", + "negative_test_query171_00008", + "negative_test_query12_00005", + "apositive_test_query244_00008", + "apositive_test_query162_00005", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "negative_test_query180_00011", + "negative_test_query244_00006", + "negative_test_query65_00013", + "apositive_test_query92_00003", + "negative_test_query39_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8503449055347546, + "recall@5": 1.0 + }, + "query_id": "test_query222", + "ranked_ids": [ + "apositive_test_query222_00001", + "negative_test_query222_00014", + "negative_test_query222_00006", + "negative_test_query222_00011", + "apositive_test_query222_00000", + "negative_test_query222_00017", + "apositive_test_query275_00001", + "apositive_test_query289_00002", + "negative_test_query289_00006", + "negative_test_query290_00004", + "apositive_test_query275_00006", + "negative_test_query271_00015", + "apositive_test_query275_00000", + "negative_test_query271_00004", + "negative_test_query275_00002", + "apositive_test_query275_00005", + "apositive_test_query275_00008", + "apositive_test_query302_00000", + "negative_test_query275_00004", + "apositive_test_query275_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "apositive_test_query259_00011", + "negative_test_query359_00008", + "negative_test_query47_00001", + "apositive_test_query259_00008", + "negative_test_query359_00014", + "negative_test_query259_00006", + "apositive_test_query259_00003", + "negative_test_query323_00005", + "negative_test_query345_00002", + "negative_test_query164_00011", + "apositive_test_query201_00006", + "negative_test_query47_00002", + "negative_test_query51_00003", + "apositive_test_query107_00005", + "negative_test_query107_00004", + "negative_test_query47_00007", + "apositive_test_query201_00004", + "apositive_test_query260_00003", + "negative_test_query109_00006", + "negative_test_query6_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query224", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query224_00012", + "negative_test_query224_00010", + "negative_test_query350_00017", + "apositive_test_query224_00003", + "negative_test_query205_00009", + "apositive_test_query256_00001", + "apositive_test_query179_00002", + "negative_test_query350_00016", + "negative_test_query224_00014", + "apositive_test_query83_00000", + "negative_test_query224_00008", + "negative_test_query350_00015", + "negative_test_query98_00018", + "negative_test_query212_00013", + "negative_test_query149_00008", + "negative_test_query224_00011", + "negative_test_query304_00001", + "apositive_test_query32_00002", + "negative_test_query196_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00000", + "apositive_test_query225_00001", + "negative_test_query145_00000", + "apositive_test_query225_00003", + "apositive_test_query225_00007", + "apositive_test_query225_00009", + "apositive_test_query225_00004", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query183_00015", + "apositive_test_query225_00005", + "negative_test_query343_00009", + "negative_test_query149_00011", + "negative_test_query161_00004", + "apositive_test_query145_00005", + "negative_test_query225_00008", + "apositive_test_query225_00002", + "negative_test_query225_00000", + "negative_test_query54_00002", + "apositive_test_query169_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query226", + "ranked_ids": [ + "apositive_test_query226_00005", + "negative_test_query226_00001", + "negative_test_query226_00007", + "negative_test_query226_00009", + "negative_test_query226_00012", + "negative_test_query226_00003", + "negative_test_query226_00010", + "negative_test_query226_00013", + "negative_test_query226_00004", + "negative_test_query226_00005", + "negative_test_query226_00006", + "negative_test_query226_00011", + "apositive_test_query226_00001", + "negative_test_query226_00000", + "apositive_test_query226_00000", + "apositive_test_query226_00003", + "apositive_test_query226_00004", + "negative_test_query360_00000", + "apositive_test_query226_00002", + "apositive_test_query153_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00003", + "apositive_test_query227_00000", + "negative_test_query227_00006", + "negative_test_query227_00012", + "negative_test_query227_00004", + "apositive_test_query70_00001", + "negative_test_query227_00000", + "negative_test_query227_00010", + "negative_test_query227_00005", + "negative_test_query227_00002", + "negative_test_query227_00011", + "negative_test_query70_00011", + "negative_test_query227_00014", + "negative_test_query321_00004", + "apositive_test_query227_00001", + "negative_test_query235_00006", + "negative_test_query227_00013", + "negative_test_query115_00007", + "apositive_test_query348_00004", + "negative_test_query59_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query228", + "ranked_ids": [ + "negative_test_query228_00012", + "negative_test_query165_00011", + "apositive_test_query228_00000", + "negative_test_query228_00000", + "negative_test_query73_00016", + "negative_test_query228_00004", + "negative_test_query228_00002", + "negative_test_query228_00014", + "negative_test_query2_00001", + "negative_test_query322_00001", + "negative_test_query216_00013", + "negative_test_query228_00011", + "negative_test_query228_00003", + "apositive_test_query254_00000", + "apositive_test_query318_00001", + "apositive_test_query322_00003", + "apositive_test_query322_00007", + "apositive_test_query333_00001", + "apositive_test_query73_00001", + "negative_test_query179_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query229_00003", + "apositive_test_query229_00000", + "apositive_test_query229_00002", + "negative_test_query229_00005", + "negative_test_query229_00006", + "negative_test_query229_00008", + "negative_test_query229_00000", + "negative_test_query229_00009", + "negative_test_query229_00007", + "apositive_test_query229_00007", + "apositive_test_query18_00004", + "apositive_test_query69_00009", + "negative_test_query355_00001", + "negative_test_query44_00005", + "apositive_test_query158_00006", + "negative_test_query0_00001", + "apositive_test_query94_00007", + "negative_test_query94_00010", + "apositive_test_query146_00004", + "apositive_test_query158_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.6 + }, + "query_id": "test_query23", + "ranked_ids": [ + "apositive_test_query23_00000", + "apositive_test_query23_00002", + "apositive_test_query23_00001", + "negative_test_query259_00007", + "negative_test_query356_00008", + "apositive_test_query39_00002", + "negative_test_query239_00012", + "apositive_test_query146_00015", + "negative_test_query23_00014", + "apositive_test_query54_00007", + "apositive_test_query225_00000", + "apositive_test_query54_00004", + "negative_test_query133_00003", + "negative_test_query292_00013", + "negative_test_query307_00002", + "negative_test_query66_00010", + "apositive_test_query52_00000", + "apositive_test_query52_00001", + "apositive_test_query52_00002", + "apositive_test_query52_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query230_00007", + "negative_test_query230_00010", + "negative_test_query230_00000", + "apositive_test_query230_00003", + "apositive_test_query3_00001", + "negative_test_query329_00000", + "negative_test_query269_00010", + "negative_test_query23_00012", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "apositive_test_query3_00006", + "negative_test_query99_00003", + "apositive_test_query3_00009", + "negative_test_query3_00004", + "negative_test_query3_00005", + "negative_test_query280_00006", + "apositive_test_query280_00002", + "negative_test_query280_00004", + "negative_test_query3_00000", + "negative_test_query3_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query231", + "ranked_ids": [ + "apositive_test_query130_00000", + "apositive_test_query0_00001", + "apositive_test_query231_00003", + "negative_test_query345_00011", + "apositive_test_query201_00006", + "apositive_test_query201_00002", + "negative_test_query231_00001", + "negative_test_query47_00006", + "negative_test_query306_00006", + "apositive_test_query130_00004", + "negative_test_query130_00010", + "negative_test_query75_00000", + "negative_test_query213_00004", + "apositive_test_query119_00006", + "negative_test_query130_00000", + "negative_test_query359_00008", + "negative_test_query201_00011", + "negative_test_query47_00005", + "negative_test_query9_00017", + "negative_test_query221_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "apositive_test_query232_00000", + "negative_test_query232_00013", + "negative_test_query232_00014", + "negative_test_query232_00000", + "negative_test_query232_00011", + "negative_test_query232_00003", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query232_00008", + "negative_test_query232_00015", + "negative_test_query232_00018", + "negative_test_query232_00002", + "negative_test_query232_00004", + "negative_test_query232_00006", + "negative_test_query232_00010", + "negative_test_query232_00001", + "negative_test_query232_00012", + "negative_test_query232_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query233", + "ranked_ids": [ + "apositive_test_query233_00000", + "negative_test_query233_00001", + "apositive_test_query233_00002", + "negative_test_query233_00010", + "negative_test_query2_00010", + "negative_test_query233_00005", + "negative_test_query233_00015", + "negative_test_query233_00006", + "negative_test_query190_00007", + "negative_test_query233_00004", + "negative_test_query233_00009", + "negative_test_query246_00001", + "negative_test_query263_00009", + "negative_test_query219_00015", + "apositive_test_query2_00000", + "apositive_test_query275_00003", + "negative_test_query130_00008", + "negative_test_query2_00014", + "negative_test_query130_00001", + "negative_test_query176_00017" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00015", + "negative_test_query234_00005", + "negative_test_query234_00011", + "negative_test_query304_00001", + "negative_test_query304_00005", + "negative_test_query214_00008", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query214_00006", + "negative_test_query234_00010", + "negative_test_query304_00008", + "apositive_test_query43_00005", + "apositive_test_query304_00003", + "negative_test_query304_00012", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query304_00013", + "apositive_test_query218_00002", + "negative_test_query10_00008", + "apositive_test_query304_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00002", + "negative_test_query235_00014", + "apositive_test_query32_00006", + "negative_test_query235_00008", + "negative_test_query235_00004", + "negative_test_query61_00001", + "negative_test_query235_00009", + "apositive_test_query32_00000", + "negative_test_query196_00000", + "negative_test_query235_00000", + "negative_test_query235_00011", + "apositive_test_query32_00004", + "negative_test_query173_00005", + "negative_test_query235_00015", + "apositive_test_query235_00000", + "apositive_test_query235_00001", + "apositive_test_query235_00003", + "negative_test_query235_00013", + "negative_test_query24_00017", + "negative_test_query219_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "negative_test_query236_00000", + "negative_test_query236_00014", + "negative_test_query236_00009", + "negative_test_query236_00012", + "negative_test_query279_00012", + "apositive_test_query279_00002", + "negative_test_query236_00006", + "negative_test_query236_00013", + "negative_test_query236_00004", + "negative_test_query236_00011", + "negative_test_query279_00003", + "negative_test_query279_00005", + "negative_test_query22_00016", + "negative_test_query236_00005", + "negative_test_query236_00007", + "negative_test_query236_00015", + "negative_test_query236_00018", + "negative_test_query222_00000", + "negative_test_query236_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "negative_test_query237_00006", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query237_00002", + "negative_test_query237_00007", + "negative_test_query152_00008", + "negative_test_query237_00001", + "negative_test_query261_00012", + "negative_test_query237_00009", + "apositive_test_query237_00001", + "apositive_test_query237_00000", + "negative_test_query237_00004", + "negative_test_query237_00005", + "negative_test_query290_00010", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query312_00001", + "apositive_test_query157_00001", + "negative_test_query152_00003", + "negative_test_query237_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "negative_test_query123_00001", + "apositive_test_query146_00013", + "apositive_test_query276_00008", + "negative_test_query46_00012", + "apositive_test_query29_00007", + "negative_test_query111_00013", + "negative_test_query56_00000", + "negative_test_query238_00011", + "apositive_test_query29_00001", + "negative_test_query56_00008", + "negative_test_query29_00006", + "apositive_test_query142_00003", + "apositive_test_query306_00003", + "negative_test_query102_00007", + "negative_test_query108_00007", + "negative_test_query111_00008", + "negative_test_query13_00003", + "negative_test_query142_00011", + "negative_test_query203_00003", + "negative_test_query223_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "apositive_test_query239_00000", + "negative_test_query239_00015", + "negative_test_query239_00001", + "negative_test_query239_00003", + "negative_test_query198_00002", + "negative_test_query239_00014", + "negative_test_query239_00000", + "negative_test_query163_00004", + "negative_test_query239_00007", + "negative_test_query268_00005", + "negative_test_query198_00008", + "negative_test_query239_00010", + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query239_00011", + "negative_test_query239_00013", + "negative_test_query352_00002", + "negative_test_query61_00004", + "negative_test_query183_00015", + "negative_test_query167_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "negative_test_query24_00000", + "apositive_test_query226_00004", + "negative_test_query15_00000", + "negative_test_query24_00004", + "apositive_test_query57_00004", + "apositive_test_query24_00000", + "negative_test_query24_00008", + "apositive_test_query106_00002", + "negative_test_query24_00002", + "negative_test_query24_00009", + "negative_test_query250_00008", + "negative_test_query128_00002", + "negative_test_query356_00003", + "negative_test_query323_00000", + "negative_test_query252_00005", + "negative_test_query198_00006", + "negative_test_query24_00012", + "negative_test_query52_00012", + "negative_test_query190_00016", + "negative_test_query24_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "apositive_test_query240_00000", + "negative_test_query240_00012", + "negative_test_query240_00006", + "negative_test_query240_00004", + "negative_test_query240_00002", + "negative_test_query240_00015", + "negative_test_query240_00017", + "negative_test_query240_00007", + "negative_test_query240_00018", + "negative_test_query240_00003", + "negative_test_query240_00011", + "negative_test_query246_00002", + "apositive_test_query311_00000", + "negative_test_query27_00010", + "negative_test_query240_00010", + "negative_test_query282_00013", + "negative_test_query6_00000", + "negative_test_query293_00004", + "negative_test_query240_00016", + "negative_test_query297_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "apositive_test_query241_00000", + "negative_test_query241_00005", + "negative_test_query241_00010", + "negative_test_query241_00016", + "negative_test_query241_00017", + "negative_test_query241_00002", + "negative_test_query241_00008", + "negative_test_query173_00013", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query206_00011", + "negative_test_query256_00000", + "negative_test_query258_00000", + "negative_test_query53_00000", + "negative_test_query206_00009", + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query298_00014", + "negative_test_query191_00014", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query291_00000", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query242_00001", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query242_00004", + "apositive_test_query242_00010", + "negative_test_query291_00004", + "negative_test_query291_00003", + "negative_test_query291_00005", + "apositive_test_query242_00005", + "apositive_test_query291_00008", + "apositive_test_query247_00005", + "apositive_test_query242_00013", + "apositive_test_query291_00005", + "apositive_test_query242_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query243_00000", + "apositive_test_query243_00002", + "negative_test_query274_00014", + "negative_test_query289_00000", + "negative_test_query243_00000", + "apositive_test_query243_00005", + "apositive_test_query243_00012", + "negative_test_query205_00007", + "negative_test_query250_00013", + "apositive_test_query243_00003", + "negative_test_query303_00005", + "negative_test_query243_00002", + "negative_test_query192_00010", + "negative_test_query192_00007", + "apositive_test_query173_00004", + "apositive_test_query160_00003", + "negative_test_query64_00003", + "apositive_test_query243_00011", + "negative_test_query184_00004", + "negative_test_query223_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query244", + "ranked_ids": [ + "negative_test_query244_00001", + "negative_test_query244_00005", + "negative_test_query244_00008", + "apositive_test_query244_00004", + "negative_test_query244_00004", + "negative_test_query244_00009", + "apositive_test_query244_00000", + "apositive_test_query244_00001", + "negative_test_query244_00003", + "negative_test_query244_00000", + "negative_test_query244_00010", + "negative_test_query244_00002", + "apositive_test_query244_00002", + "apositive_test_query244_00005", + "apositive_test_query244_00007", + "apositive_test_query244_00006", + "apositive_test_query42_00002", + "negative_test_query126_00015", + "apositive_test_query23_00003", + "apositive_test_query356_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query245_00003", + "negative_test_query245_00010", + "negative_test_query64_00013", + "negative_test_query135_00013", + "negative_test_query245_00000", + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query245_00000", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query245_00001", + "negative_test_query202_00000", + "negative_test_query121_00002", + "negative_test_query293_00003", + "negative_test_query122_00002", + "negative_test_query159_00003", + "negative_test_query336_00002", + "negative_test_query202_00004", + "apositive_test_query141_00002", + "apositive_test_query141_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query252_00003", + "negative_test_query246_00012", + "negative_test_query246_00001", + "negative_test_query246_00002", + "negative_test_query190_00003", + "negative_test_query135_00017", + "negative_test_query246_00016", + "negative_test_query6_00014", + "negative_test_query44_00002", + "negative_test_query317_00014", + "negative_test_query6_00006", + "apositive_test_query0_00001", + "negative_test_query198_00012", + "negative_test_query293_00000", + "apositive_test_query216_00000", + "negative_test_query311_00005", + "apositive_test_query68_00002", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "negative_test_query291_00005", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "negative_test_query291_00003", + "apositive_test_query242_00001", + "apositive_test_query247_00005", + "negative_test_query242_00000", + "negative_test_query291_00000", + "negative_test_query247_00005", + "negative_test_query186_00010", + "apositive_test_query291_00000", + "negative_test_query105_00012", + "negative_test_query247_00011", + "negative_test_query291_00001", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "apositive_test_query242_00012", + "negative_test_query247_00003", + "apositive_test_query242_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00008", + "apositive_test_query248_00002", + "apositive_test_query248_00005", + "negative_test_query248_00010", + "negative_test_query251_00017", + "apositive_test_query248_00007", + "negative_test_query248_00002", + "negative_test_query248_00011", + "apositive_test_query248_00003", + "negative_test_query248_00006", + "negative_test_query248_00004", + "negative_test_query29_00001", + "apositive_test_query29_00000", + "apositive_test_query29_00004", + "apositive_test_query248_00001", + "apositive_test_query29_00005", + "negative_test_query248_00009", + "negative_test_query205_00017", + "negative_test_query248_00000", + "negative_test_query248_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query249", + "ranked_ids": [ + "negative_test_query156_00002", + "negative_test_query249_00001", + "negative_test_query193_00001", + "negative_test_query71_00006", + "apositive_test_query112_00000", + "negative_test_query325_00015", + "apositive_test_query177_00004", + "apositive_test_query249_00000", + "apositive_test_query281_00001", + "negative_test_query210_00010", + "negative_test_query326_00005", + "negative_test_query302_00007", + "negative_test_query278_00016", + "negative_test_query301_00013", + "apositive_test_query249_00004", + "negative_test_query156_00007", + "negative_test_query72_00017", + "apositive_test_query204_00003", + "apositive_test_query353_00004", + "negative_test_query122_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query25", + "ranked_ids": [ + "negative_test_query25_00007", + "negative_test_query45_00014", + "apositive_test_query25_00000", + "apositive_test_query25_00006", + "apositive_test_query25_00008", + "negative_test_query143_00000", + "negative_test_query25_00002", + "negative_test_query25_00001", + "apositive_test_query25_00005", + "apositive_test_query25_00003", + "negative_test_query25_00005", + "apositive_test_query25_00007", + "apositive_test_query25_00009", + "apositive_test_query25_00001", + "apositive_test_query25_00010", + "apositive_test_query25_00004", + "negative_test_query30_00003", + "negative_test_query346_00006", + "negative_test_query9_00001", + "negative_test_query328_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query250", + "ranked_ids": [ + "negative_test_query186_00007", + "negative_test_query250_00009", + "negative_test_query250_00011", + "negative_test_query250_00013", + "negative_test_query111_00015", + "apositive_test_query203_00000", + "negative_test_query79_00013", + "negative_test_query57_00009", + "negative_test_query102_00010", + "apositive_test_query250_00000", + "negative_test_query250_00001", + "negative_test_query159_00014", + "negative_test_query192_00009", + "negative_test_query101_00008", + "negative_test_query61_00007", + "apositive_test_query203_00006", + "negative_test_query184_00015", + "negative_test_query64_00005", + "negative_test_query250_00003", + "negative_test_query159_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query251", + "ranked_ids": [ + "negative_test_query189_00011", + "apositive_test_query134_00003", + "negative_test_query322_00002", + "negative_test_query56_00005", + "negative_test_query234_00006", + "negative_test_query251_00003", + "negative_test_query251_00004", + "negative_test_query298_00003", + "apositive_test_query29_00002", + "negative_test_query143_00001", + "negative_test_query160_00001", + "negative_test_query8_00009", + "negative_test_query326_00000", + "apositive_test_query251_00001", + "negative_test_query268_00011", + "apositive_test_query347_00000", + "negative_test_query14_00000", + "negative_test_query170_00008", + "negative_test_query269_00007", + "negative_test_query347_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query252", + "ranked_ids": [ + "apositive_test_query252_00000", + "negative_test_query252_00009", + "negative_test_query252_00012", + "negative_test_query126_00001", + "negative_test_query151_00000", + "apositive_test_query252_00003", + "negative_test_query212_00000", + "negative_test_query330_00006", + "apositive_test_query125_00001", + "negative_test_query252_00011", + "negative_test_query318_00006", + "negative_test_query252_00007", + "negative_test_query252_00001", + "negative_test_query321_00004", + "apositive_test_query252_00002", + "apositive_test_query110_00001", + "negative_test_query16_00006", + "negative_test_query252_00010", + "negative_test_query44_00011", + "negative_test_query14_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2 + }, + "query_id": "test_query253", + "ranked_ids": [ + "apositive_test_query253_00003", + "apositive_test_query194_00008", + "apositive_test_query253_00004", + "negative_test_query253_00005", + "apositive_test_query162_00008", + "negative_test_query177_00000", + "apositive_test_query253_00007", + "apositive_test_query186_00004", + "apositive_test_query74_00002", + "apositive_test_query13_00012", + "apositive_test_query74_00003", + "apositive_test_query349_00000", + "negative_test_query124_00001", + "negative_test_query226_00013", + "negative_test_query253_00003", + "negative_test_query95_00017", + "negative_test_query194_00001", + "negative_test_query253_00004", + "negative_test_query124_00002", + "negative_test_query172_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query254", + "ranked_ids": [ + "apositive_test_query254_00004", + "apositive_test_query254_00000", + "negative_test_query350_00000", + "negative_test_query270_00010", + "negative_test_query254_00008", + "apositive_test_query254_00005", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00010", + "apositive_test_query254_00003", + "negative_test_query136_00003", + "apositive_test_query254_00001", + "apositive_test_query254_00002", + "apositive_test_query254_00006", + "negative_test_query228_00016", + "negative_test_query21_00009", + "apositive_test_query333_00004", + "negative_test_query254_00004", + "negative_test_query35_00015", + "negative_test_query316_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "apositive_test_query255_00000", + "negative_test_query255_00017", + "apositive_test_query322_00001", + "negative_test_query255_00010", + "negative_test_query246_00003", + "negative_test_query255_00006", + "negative_test_query255_00000", + "negative_test_query255_00018", + "apositive_test_query25_00004", + "negative_test_query208_00008", + "negative_test_query278_00015", + "apositive_test_query313_00000", + "negative_test_query255_00013", + "negative_test_query326_00006", + "negative_test_query89_00004", + "negative_test_query255_00004", + "negative_test_query255_00005", + "negative_test_query110_00001", + "apositive_test_query278_00000", + "negative_test_query278_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.592512031964586, + "recall@5": 0.75 + }, + "query_id": "test_query256", + "ranked_ids": [ + "negative_test_query256_00003", + "apositive_test_query256_00002", + "apositive_test_query256_00001", + "negative_test_query256_00011", + "apositive_test_query256_00000", + "apositive_test_query119_00000", + "negative_test_query183_00008", + "negative_test_query347_00007", + "negative_test_query7_00009", + "apositive_test_query43_00001", + "apositive_test_query216_00002", + "negative_test_query279_00000", + "apositive_test_query256_00003", + "negative_test_query256_00008", + "negative_test_query319_00001", + "negative_test_query82_00001", + "negative_test_query5_00010", + "negative_test_query135_00016", + "apositive_test_query142_00002", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query257", + "ranked_ids": [ + "negative_test_query257_00005", + "apositive_test_query257_00001", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query52_00005", + "apositive_test_query257_00000", + "apositive_test_query286_00014", + "apositive_test_query257_00003", + "apositive_test_query257_00002", + "apositive_test_query257_00004", + "apositive_test_query257_00005", + "apositive_test_query69_00007", + "negative_test_query257_00007", + "negative_test_query52_00004", + "negative_test_query52_00007", + "apositive_test_query52_00001", + "apositive_test_query54_00003", + "negative_test_query52_00002", + "negative_test_query54_00005", + "apositive_test_query151_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query228_00005", + "negative_test_query206_00016", + "negative_test_query311_00002", + "negative_test_query162_00004", + "negative_test_query167_00007", + "negative_test_query320_00002", + "negative_test_query319_00000", + "negative_test_query82_00000", + "negative_test_query259_00001", + "negative_test_query181_00002", + "negative_test_query38_00002", + "apositive_test_query301_00002", + "negative_test_query144_00010", + "negative_test_query101_00001", + "negative_test_query169_00016", + "negative_test_query171_00002", + "negative_test_query173_00010", + "negative_test_query339_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.25 + }, + "query_id": "test_query259", + "ranked_ids": [ + "negative_test_query259_00002", + "apositive_test_query259_00007", + "apositive_test_query259_00006", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "negative_test_query259_00006", + "apositive_test_query259_00003", + "apositive_test_query231_00001", + "apositive_test_query345_00000", + "apositive_test_query54_00011", + "negative_test_query111_00010", + "negative_test_query47_00004", + "negative_test_query250_00015", + "negative_test_query359_00012", + "apositive_test_query259_00008", + "negative_test_query259_00001", + "negative_test_query30_00009", + "negative_test_query213_00004", + "negative_test_query359_00003", + "apositive_test_query0_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "apositive_test_query26_00005", + "negative_test_query26_00008", + "negative_test_query223_00002", + "negative_test_query26_00005", + "apositive_test_query102_00000", + "apositive_test_query158_00003", + "apositive_test_query186_00003", + "apositive_test_query26_00004", + "negative_test_query105_00013", + "negative_test_query158_00006", + "negative_test_query164_00004", + "negative_test_query247_00010", + "negative_test_query26_00003", + "negative_test_query119_00006", + "negative_test_query158_00009", + "negative_test_query26_00009", + "negative_test_query26_00001", + "negative_test_query102_00015", + "negative_test_query238_00002", + "negative_test_query334_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "negative_test_query260_00011", + "negative_test_query260_00009", + "apositive_test_query6_00002", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query252_00004", + "negative_test_query6_00005", + "negative_test_query6_00009", + "apositive_test_query6_00001", + "negative_test_query6_00004", + "negative_test_query6_00011", + "apositive_test_query6_00000", + "negative_test_query125_00001", + "negative_test_query2_00004", + "negative_test_query6_00002", + "apositive_test_query186_00001", + "negative_test_query260_00002", + "apositive_test_query109_00000", + "apositive_test_query109_00001", + "negative_test_query6_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query261", + "ranked_ids": [ + "negative_test_query261_00006", + "negative_test_query261_00005", + "negative_test_query116_00004", + "negative_test_query261_00010", + "negative_test_query261_00013", + "negative_test_query261_00008", + "negative_test_query261_00007", + "negative_test_query261_00004", + "negative_test_query261_00002", + "negative_test_query261_00003", + "negative_test_query261_00000", + "negative_test_query261_00001", + "negative_test_query261_00011", + "apositive_test_query261_00002", + "negative_test_query118_00006", + "negative_test_query2_00015", + "apositive_test_query261_00000", + "negative_test_query261_00009", + "apositive_test_query261_00003", + "apositive_test_query29_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query262", + "ranked_ids": [ + "negative_test_query266_00003", + "negative_test_query106_00005", + "negative_test_query301_00005", + "negative_test_query262_00000", + "negative_test_query312_00000", + "negative_test_query312_00003", + "apositive_test_query106_00003", + "apositive_test_query262_00000", + "apositive_test_query262_00003", + "negative_test_query106_00010", + "negative_test_query312_00018", + "apositive_test_query177_00008", + "apositive_test_query194_00002", + "apositive_test_query42_00006", + "negative_test_query147_00001", + "negative_test_query152_00006", + "negative_test_query162_00007", + "negative_test_query172_00000", + "negative_test_query326_00008", + "negative_test_query124_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query263", + "ranked_ids": [ + "negative_test_query135_00004", + "negative_test_query169_00014", + "negative_test_query190_00009", + "apositive_test_query155_00003", + "apositive_test_query242_00011", + "apositive_test_query291_00010", + "negative_test_query253_00009", + "negative_test_query125_00012", + "negative_test_query307_00005", + "apositive_test_query37_00002", + "negative_test_query71_00006", + "negative_test_query88_00003", + "apositive_test_query263_00002", + "negative_test_query263_00005", + "apositive_test_query231_00001", + "apositive_test_query292_00002", + "negative_test_query201_00010", + "negative_test_query213_00001", + "negative_test_query213_00005", + "negative_test_query357_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "negative_test_query264_00011", + "negative_test_query264_00004", + "negative_test_query264_00014", + "negative_test_query200_00002", + "negative_test_query200_00017", + "negative_test_query264_00002", + "negative_test_query264_00018", + "negative_test_query159_00013", + "negative_test_query264_00006", + "negative_test_query200_00011", + "apositive_test_query38_00003", + "negative_test_query337_00002", + "apositive_test_query264_00000", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query264_00007", + "negative_test_query337_00012", + "negative_test_query264_00016", + "negative_test_query264_00008", + "negative_test_query159_00017" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query265", + "ranked_ids": [ + "apositive_test_query265_00000", + "negative_test_query265_00002", + "negative_test_query265_00000", + "apositive_test_query265_00001", + "negative_test_query274_00008", + "apositive_test_query156_00000", + "negative_test_query165_00016", + "negative_test_query265_00009", + "negative_test_query86_00008", + "negative_test_query165_00007", + "negative_test_query271_00005", + "negative_test_query92_00012", + "negative_test_query265_00007", + "apositive_test_query234_00001", + "negative_test_query52_00000", + "apositive_test_query78_00002", + "negative_test_query66_00010", + "negative_test_query190_00013", + "negative_test_query139_00000", + "negative_test_query86_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "negative_test_query266_00018", + "negative_test_query266_00008", + "apositive_test_query19_00002", + "negative_test_query141_00000", + "negative_test_query106_00004", + "negative_test_query266_00009", + "negative_test_query266_00001", + "negative_test_query266_00000", + "apositive_test_query54_00009", + "negative_test_query266_00017", + "negative_test_query266_00005", + "negative_test_query266_00002", + "apositive_test_query266_00000", + "negative_test_query176_00018", + "negative_test_query266_00010", + "negative_test_query266_00014", + "negative_test_query169_00014", + "negative_test_query106_00001", + "negative_test_query174_00001", + "negative_test_query266_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "negative_test_query79_00004", + "negative_test_query267_00002", + "apositive_test_query157_00008", + "apositive_test_query295_00001", + "negative_test_query98_00013", + "negative_test_query295_00002", + "negative_test_query345_00013", + "negative_test_query359_00005", + "negative_test_query5_00012", + "negative_test_query79_00002", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query250_00000", + "negative_test_query186_00006", + "negative_test_query313_00016", + "negative_test_query30_00000", + "apositive_test_query50_00004", + "negative_test_query204_00001", + "apositive_test_query113_00002", + "negative_test_query115_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query268", + "ranked_ids": [ + "apositive_test_query268_00004", + "negative_test_query268_00012", + "apositive_test_query268_00000", + "apositive_test_query201_00003", + "negative_test_query268_00013", + "negative_test_query47_00017", + "negative_test_query268_00004", + "negative_test_query268_00003", + "apositive_test_query51_00003", + "apositive_test_query268_00002", + "negative_test_query268_00008", + "negative_test_query323_00002", + "negative_test_query268_00000", + "negative_test_query268_00007", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query268_00005", + "negative_test_query210_00007", + "negative_test_query228_00007", + "negative_test_query335_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query269", + "ranked_ids": [ + "apositive_test_query269_00000", + "negative_test_query329_00014", + "negative_test_query269_00001", + "negative_test_query269_00009", + "negative_test_query269_00000", + "negative_test_query269_00015", + "negative_test_query269_00004", + "negative_test_query269_00012", + "negative_test_query329_00004", + "negative_test_query269_00008", + "negative_test_query269_00002", + "negative_test_query269_00010", + "negative_test_query269_00007", + "apositive_test_query269_00003", + "negative_test_query296_00003", + "apositive_test_query296_00005", + "negative_test_query269_00003", + "negative_test_query269_00005", + "negative_test_query269_00011", + "apositive_test_query296_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "negative_test_query27_00009", + "negative_test_query164_00007", + "apositive_test_query59_00000", + "negative_test_query147_00000", + "negative_test_query27_00004", + "negative_test_query259_00006", + "negative_test_query138_00006", + "negative_test_query358_00003", + "negative_test_query76_00009", + "negative_test_query315_00001", + "negative_test_query130_00006", + "negative_test_query30_00009", + "apositive_test_query29_00004", + "apositive_test_query81_00007", + "negative_test_query27_00014", + "apositive_test_query127_00010", + "apositive_test_query154_00000", + "negative_test_query33_00004", + "negative_test_query11_00008", + "apositive_test_query16_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query270", + "ranked_ids": [ + "negative_test_query270_00012", + "negative_test_query270_00000", + "negative_test_query270_00009", + "negative_test_query270_00004", + "negative_test_query270_00008", + "negative_test_query270_00002", + "negative_test_query270_00014", + "negative_test_query270_00011", + "apositive_test_query270_00001", + "negative_test_query270_00003", + "negative_test_query270_00010", + "negative_test_query270_00015", + "apositive_test_query270_00000", + "apositive_test_query270_00002", + "negative_test_query270_00005", + "negative_test_query270_00006", + "negative_test_query270_00016", + "apositive_test_query81_00000", + "negative_test_query76_00005", + "negative_test_query81_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query271", + "ranked_ids": [ + "apositive_test_query271_00001", + "negative_test_query330_00008", + "apositive_test_query271_00000", + "apositive_test_query10_00002", + "negative_test_query120_00003", + "negative_test_query327_00010", + "negative_test_query78_00010", + "negative_test_query84_00014", + "apositive_test_query314_00007", + "negative_test_query169_00009", + "negative_test_query271_00007", + "negative_test_query198_00015", + "negative_test_query221_00012", + "negative_test_query241_00018", + "apositive_test_query189_00000", + "apositive_test_query109_00001", + "negative_test_query90_00001", + "negative_test_query227_00009", + "apositive_test_query67_00000", + "negative_test_query112_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00003", + "negative_test_query272_00015", + "negative_test_query144_00000", + "negative_test_query272_00001", + "negative_test_query272_00010", + "negative_test_query272_00002", + "apositive_test_query272_00000", + "negative_test_query272_00016", + "negative_test_query328_00000", + "negative_test_query272_00014", + "negative_test_query272_00008", + "negative_test_query272_00017", + "negative_test_query18_00014", + "negative_test_query272_00000", + "negative_test_query272_00009", + "negative_test_query272_00013", + "negative_test_query332_00013", + "negative_test_query272_00012", + "negative_test_query41_00008", + "apositive_test_query251_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00008", + "apositive_test_query273_00005", + "negative_test_query18_00011", + "negative_test_query273_00000", + "apositive_test_query273_00000", + "apositive_test_query281_00008", + "negative_test_query273_00006", + "apositive_test_query18_00003", + "negative_test_query273_00009", + "apositive_test_query273_00004", + "apositive_test_query18_00001", + "apositive_test_query18_00004", + "negative_test_query18_00006", + "negative_test_query18_00012", + "negative_test_query166_00009", + "apositive_test_query18_00000", + "negative_test_query273_00004", + "negative_test_query300_00015", + "negative_test_query140_00013", + "apositive_test_query36_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query274", + "ranked_ids": [ + "apositive_test_query274_00000", + "negative_test_query274_00000", + "negative_test_query274_00014", + "negative_test_query274_00011", + "negative_test_query274_00012", + "negative_test_query34_00014", + "negative_test_query352_00012", + "negative_test_query274_00005", + "apositive_test_query274_00003", + "negative_test_query256_00015", + "apositive_test_query274_00001", + "negative_test_query218_00011", + "negative_test_query274_00007", + "negative_test_query200_00009", + "negative_test_query274_00009", + "apositive_test_query327_00002", + "negative_test_query274_00004", + "negative_test_query78_00007", + "negative_test_query352_00009", + "negative_test_query34_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query275_00000", + "apositive_test_query275_00005", + "apositive_test_query275_00004", + "apositive_test_query275_00007", + "negative_test_query275_00000", + "apositive_test_query275_00006", + "negative_test_query202_00005", + "apositive_test_query275_00008", + "negative_test_query275_00004", + "negative_test_query289_00013", + "negative_test_query263_00009", + "apositive_test_query275_00003", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query275_00007", + "apositive_test_query110_00001", + "negative_test_query149_00016", + "apositive_test_query93_00003", + "negative_test_query16_00001", + "negative_test_query87_00017" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query22_00004", + "apositive_test_query276_00005", + "apositive_test_query276_00004", + "negative_test_query199_00012", + "negative_test_query280_00002", + "negative_test_query153_00001", + "apositive_test_query153_00006", + "negative_test_query276_00001", + "apositive_test_query161_00002", + "negative_test_query146_00001", + "negative_test_query251_00012", + "apositive_test_query314_00007", + "negative_test_query146_00000", + "apositive_test_query314_00005", + "apositive_test_query314_00006", + "apositive_test_query276_00006", + "apositive_test_query3_00010", + "negative_test_query113_00011", + "negative_test_query146_00003", + "apositive_test_query146_00015" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query277", + "ranked_ids": [ + "apositive_test_query277_00001", + "apositive_test_query277_00003", + "apositive_test_query277_00000", + "negative_test_query277_00007", + "negative_test_query277_00004", + "negative_test_query277_00014", + "negative_test_query277_00015", + "negative_test_query313_00009", + "negative_test_query277_00008", + "negative_test_query277_00003", + "apositive_test_query107_00007", + "apositive_test_query277_00002", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "negative_test_query345_00000", + "negative_test_query277_00000", + "negative_test_query277_00006", + "negative_test_query277_00009", + "negative_test_query277_00010", + "negative_test_query277_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query140_00006", + "negative_test_query278_00002", + "apositive_test_query263_00005", + "negative_test_query278_00011", + "apositive_test_query4_00013", + "apositive_test_query36_00001", + "negative_test_query260_00002", + "negative_test_query324_00011", + "negative_test_query158_00003", + "negative_test_query260_00011", + "apositive_test_query186_00001", + "negative_test_query64_00007", + "negative_test_query15_00007", + "negative_test_query177_00001", + "negative_test_query260_00009", + "negative_test_query278_00009", + "negative_test_query255_00018", + "negative_test_query255_00016", + "negative_test_query110_00011", + "negative_test_query27_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query279_00000", + "apositive_test_query279_00001", + "negative_test_query279_00014", + "apositive_test_query279_00003", + "negative_test_query279_00009", + "apositive_test_query348_00007", + "apositive_test_query279_00004", + "apositive_test_query177_00004", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "negative_test_query279_00013", + "negative_test_query181_00008", + "negative_test_query309_00011", + "apositive_test_query3_00000", + "apositive_test_query76_00005", + "negative_test_query279_00007", + "apositive_test_query358_00007", + "apositive_test_query162_00009", + "negative_test_query132_00013", + "apositive_test_query76_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query28", + "ranked_ids": [ + "negative_test_query72_00004", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00001", + "apositive_test_query28_00003", + "apositive_test_query217_00000", + "negative_test_query72_00018", + "apositive_test_query28_00009", + "negative_test_query217_00004", + "negative_test_query307_00003", + "apositive_test_query28_00007", + "apositive_test_query217_00003", + "negative_test_query28_00002", + "negative_test_query48_00009", + "apositive_test_query28_00008", + "negative_test_query185_00001", + "negative_test_query185_00003", + "negative_test_query217_00011", + "negative_test_query307_00004", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "negative_test_query280_00004", + "negative_test_query280_00005", + "apositive_test_query280_00004", + "negative_test_query280_00008", + "negative_test_query106_00006", + "negative_test_query280_00010", + "negative_test_query9_00012", + "apositive_test_query280_00000", + "negative_test_query280_00006", + "apositive_test_query253_00009", + "negative_test_query280_00002", + "negative_test_query280_00013", + "negative_test_query129_00008", + "apositive_test_query74_00001", + "negative_test_query236_00018", + "apositive_test_query280_00003", + "negative_test_query280_00012", + "negative_test_query280_00003", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query281_00004", + "apositive_test_query273_00002", + "apositive_test_query281_00010", + "apositive_test_query281_00007", + "apositive_test_query281_00012", + "apositive_test_query281_00003", + "apositive_test_query281_00001", + "apositive_test_query281_00008", + "apositive_test_query281_00016", + "negative_test_query257_00012", + "apositive_test_query281_00011", + "negative_test_query273_00003", + "apositive_test_query273_00008", + "apositive_test_query281_00014", + "negative_test_query273_00004", + "negative_test_query249_00000", + "apositive_test_query273_00003", + "apositive_test_query273_00000", + "negative_test_query273_00007", + "negative_test_query125_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5706417189553201, + "recall@5": 1.0 + }, + "query_id": "test_query282", + "ranked_ids": [ + "negative_test_query344_00001", + "negative_test_query282_00007", + "apositive_test_query282_00001", + "apositive_test_query282_00000", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query282_00016", + "negative_test_query282_00010", + "negative_test_query344_00008", + "negative_test_query282_00000", + "negative_test_query344_00000", + "negative_test_query282_00012", + "negative_test_query282_00009", + "apositive_test_query20_00004", + "negative_test_query282_00003", + "apositive_test_query220_00002", + "negative_test_query344_00004", + "negative_test_query282_00004", + "negative_test_query282_00001", + "negative_test_query282_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query93_00002", + "negative_test_query283_00004", + "negative_test_query283_00007", + "negative_test_query93_00004", + "negative_test_query283_00003", + "negative_test_query93_00010", + "negative_test_query283_00016", + "negative_test_query31_00010", + "negative_test_query93_00009", + "negative_test_query93_00006", + "negative_test_query265_00013", + "negative_test_query283_00012", + "negative_test_query283_00014", + "negative_test_query138_00009", + "negative_test_query283_00000", + "negative_test_query92_00010", + "negative_test_query138_00005", + "negative_test_query270_00006", + "apositive_test_query93_00002", + "negative_test_query299_00017" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.4 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "negative_test_query284_00009", + "negative_test_query284_00001", + "apositive_test_query284_00003", + "apositive_test_query198_00002", + "negative_test_query14_00014", + "apositive_test_query162_00005", + "apositive_test_query284_00001", + "apositive_test_query163_00003", + "negative_test_query114_00006", + "apositive_test_query141_00002", + "apositive_test_query141_00006", + "apositive_test_query163_00002", + "negative_test_query109_00009", + "negative_test_query284_00012", + "negative_test_query288_00011", + "apositive_test_query79_00004", + "negative_test_query284_00004", + "apositive_test_query259_00009", + "negative_test_query141_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.25 + }, + "query_id": "test_query285", + "ranked_ids": [ + "apositive_test_query285_00003", + "negative_test_query285_00002", + "negative_test_query285_00004", + "apositive_test_query285_00004", + "negative_test_query285_00008", + "apositive_test_query285_00001", + "apositive_test_query285_00007", + "negative_test_query285_00005", + "negative_test_query304_00003", + "apositive_test_query285_00002", + "apositive_test_query285_00006", + "negative_test_query285_00000", + "negative_test_query285_00011", + "negative_test_query304_00009", + "apositive_test_query285_00005", + "negative_test_query285_00010", + "negative_test_query86_00001", + "negative_test_query285_00001", + "negative_test_query285_00007", + "negative_test_query285_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.2 + }, + "query_id": "test_query286", + "ranked_ids": [ + "negative_test_query286_00000", + "negative_test_query36_00001", + "apositive_test_query286_00010", + "apositive_test_query286_00013", + "apositive_test_query286_00012", + "negative_test_query257_00005", + "negative_test_query257_00001", + "negative_test_query52_00001", + "apositive_test_query286_00006", + "apositive_test_query80_00000", + "negative_test_query286_00003", + "negative_test_query54_00004", + "negative_test_query36_00013", + "apositive_test_query286_00005", + "negative_test_query125_00012", + "negative_test_query52_00005", + "apositive_test_query257_00001", + "negative_test_query139_00008", + "apositive_test_query257_00000", + "apositive_test_query286_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.4 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query287_00001", + "apositive_test_query133_00002", + "apositive_test_query133_00001", + "negative_test_query301_00010", + "apositive_test_query133_00014", + "apositive_test_query133_00006", + "apositive_test_query133_00013", + "apositive_test_query287_00004", + "apositive_test_query133_00004", + "negative_test_query153_00003", + "negative_test_query359_00011", + "apositive_test_query133_00012", + "negative_test_query133_00002", + "apositive_test_query153_00003", + "apositive_test_query4_00009", + "negative_test_query134_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query288", + "ranked_ids": [ + "apositive_test_query141_00002", + "negative_test_query288_00006", + "negative_test_query288_00002", + "apositive_test_query210_00002", + "apositive_test_query210_00001", + "apositive_test_query141_00006", + "apositive_test_query288_00002", + "apositive_test_query304_00001", + "negative_test_query148_00014", + "negative_test_query210_00005", + "negative_test_query213_00007", + "negative_test_query288_00010", + "negative_test_query288_00001", + "negative_test_query148_00004", + "negative_test_query17_00002", + "negative_test_query288_00007", + "negative_test_query199_00015", + "negative_test_query210_00013", + "negative_test_query199_00010", + "negative_test_query93_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00001", + "negative_test_query289_00013", + "negative_test_query289_00006", + "negative_test_query334_00014", + "apositive_test_query275_00005", + "apositive_test_query289_00000", + "negative_test_query289_00000", + "negative_test_query305_00011", + "negative_test_query267_00005", + "apositive_test_query157_00008", + "apositive_test_query289_00002", + "negative_test_query47_00009", + "apositive_test_query289_00004", + "negative_test_query113_00009", + "negative_test_query247_00001", + "negative_test_query289_00004", + "negative_test_query7_00006", + "negative_test_query250_00016", + "negative_test_query107_00004", + "negative_test_query289_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.375 + }, + "query_id": "test_query29", + "ranked_ids": [ + "negative_test_query29_00009", + "negative_test_query29_00008", + "apositive_test_query29_00004", + "apositive_test_query29_00000", + "apositive_test_query29_00005", + "negative_test_query248_00009", + "apositive_test_query29_00007", + "apositive_test_query29_00001", + "apositive_test_query29_00003", + "negative_test_query248_00007", + "negative_test_query29_00007", + "negative_test_query39_00011", + "negative_test_query29_00006", + "negative_test_query29_00010", + "negative_test_query213_00012", + "apositive_test_query29_00002", + "negative_test_query292_00015", + "apositive_test_query29_00006", + "negative_test_query29_00000", + "negative_test_query29_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query290", + "ranked_ids": [ + "apositive_test_query290_00001", + "negative_test_query290_00005", + "negative_test_query98_00003", + "apositive_test_query290_00002", + "negative_test_query342_00005", + "negative_test_query290_00009", + "negative_test_query237_00010", + "negative_test_query290_00014", + "apositive_test_query237_00002", + "negative_test_query290_00003", + "apositive_test_query290_00003", + "negative_test_query290_00000", + "negative_test_query290_00006", + "negative_test_query290_00008", + "negative_test_query59_00002", + "negative_test_query116_00002", + "negative_test_query290_00002", + "negative_test_query157_00005", + "negative_test_query237_00013", + "negative_test_query312_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query242_00001", + "apositive_test_query291_00000", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query291_00005", + "apositive_test_query242_00005", + "apositive_test_query291_00008", + "apositive_test_query242_00010", + "negative_test_query291_00004", + "negative_test_query291_00003", + "apositive_test_query242_00013", + "apositive_test_query291_00005", + "apositive_test_query247_00005", + "negative_test_query291_00007", + "negative_test_query186_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query292", + "ranked_ids": [ + "negative_test_query292_00010", + "apositive_test_query292_00000", + "apositive_test_query292_00001", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query292_00000", + "negative_test_query292_00012", + "negative_test_query292_00005", + "negative_test_query292_00007", + "negative_test_query292_00008", + "negative_test_query292_00014", + "negative_test_query292_00015", + "negative_test_query298_00000", + "negative_test_query292_00016", + "negative_test_query139_00001", + "negative_test_query205_00000", + "negative_test_query61_00011", + "negative_test_query292_00006", + "negative_test_query99_00004", + "negative_test_query151_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query293", + "ranked_ids": [ + "apositive_test_query293_00000", + "apositive_test_query293_00001", + "negative_test_query293_00015", + "negative_test_query293_00009", + "negative_test_query27_00008", + "negative_test_query27_00011", + "negative_test_query27_00002", + "negative_test_query293_00003", + "apositive_test_query44_00001", + "apositive_test_query293_00002", + "apositive_test_query178_00003", + "negative_test_query293_00000", + "apositive_test_query52_00005", + "negative_test_query293_00016", + "negative_test_query293_00006", + "negative_test_query89_00002", + "negative_test_query167_00013", + "negative_test_query293_00010", + "negative_test_query293_00011", + "negative_test_query220_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00008", + "apositive_test_query294_00001", + "negative_test_query294_00003", + "negative_test_query294_00011", + "negative_test_query294_00013", + "negative_test_query294_00006", + "negative_test_query294_00002", + "negative_test_query314_00001", + "negative_test_query294_00004", + "negative_test_query294_00012", + "negative_test_query294_00016", + "negative_test_query294_00005", + "negative_test_query294_00007", + "negative_test_query294_00001", + "negative_test_query277_00005", + "negative_test_query58_00003", + "negative_test_query111_00015", + "apositive_test_query170_00005", + "apositive_test_query177_00004", + "negative_test_query156_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query295", + "ranked_ids": [ + "negative_test_query26_00002", + "apositive_test_query295_00000", + "negative_test_query295_00008", + "negative_test_query295_00002", + "apositive_test_query57_00000", + "apositive_test_query295_00002", + "negative_test_query295_00013", + "apositive_test_query223_00001", + "negative_test_query295_00011", + "apositive_test_query295_00001", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query295_00016", + "negative_test_query168_00005", + "negative_test_query115_00005", + "negative_test_query295_00001", + "negative_test_query295_00004", + "negative_test_query295_00012", + "negative_test_query57_00009", + "negative_test_query295_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query296", + "ranked_ids": [ + "apositive_test_query296_00002", + "negative_test_query340_00009", + "negative_test_query296_00005", + "apositive_test_query296_00001", + "negative_test_query296_00001", + "apositive_test_query296_00000", + "apositive_test_query296_00004", + "negative_test_query296_00009", + "apositive_test_query296_00005", + "negative_test_query340_00014", + "apositive_test_query296_00003", + "negative_test_query296_00008", + "negative_test_query296_00003", + "negative_test_query296_00013", + "negative_test_query329_00011", + "negative_test_query296_00000", + "negative_test_query296_00002", + "negative_test_query0_00004", + "negative_test_query296_00012", + "negative_test_query112_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query297_00013", + "negative_test_query297_00004", + "negative_test_query297_00012", + "negative_test_query297_00014", + "apositive_test_query139_00001", + "negative_test_query297_00001", + "negative_test_query297_00003", + "negative_test_query191_00003", + "negative_test_query297_00002", + "negative_test_query64_00003", + "negative_test_query191_00011", + "negative_test_query297_00008", + "negative_test_query64_00005", + "negative_test_query70_00005", + "negative_test_query297_00007", + "negative_test_query191_00010", + "negative_test_query329_00007", + "negative_test_query297_00010", + "apositive_test_query297_00000", + "negative_test_query117_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "apositive_test_query298_00002", + "negative_test_query216_00010", + "apositive_test_query298_00000", + "negative_test_query202_00011", + "negative_test_query298_00006", + "negative_test_query347_00014", + "negative_test_query182_00011", + "apositive_test_query298_00001", + "apositive_test_query347_00001", + "negative_test_query182_00010", + "negative_test_query240_00001", + "negative_test_query298_00001", + "apositive_test_query56_00002", + "negative_test_query175_00009", + "negative_test_query193_00006", + "negative_test_query298_00003", + "negative_test_query298_00004", + "negative_test_query347_00000", + "negative_test_query347_00016", + "apositive_test_query347_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query299_00005", + "apositive_test_query346_00001", + "apositive_test_query265_00002", + "negative_test_query299_00006", + "apositive_test_query325_00002", + "negative_test_query300_00000", + "negative_test_query351_00009", + "negative_test_query125_00005", + "apositive_test_query101_00003", + "negative_test_query134_00000", + "negative_test_query305_00007", + "negative_test_query92_00010", + "apositive_test_query141_00002", + "negative_test_query210_00008", + "negative_test_query288_00000", + "negative_test_query3_00003", + "negative_test_query58_00005", + "negative_test_query89_00008", + "negative_test_query288_00007", + "apositive_test_query227_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query3", + "ranked_ids": [ + "apositive_test_query3_00001", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "negative_test_query3_00000", + "negative_test_query230_00007", + "negative_test_query230_00000", + "negative_test_query329_00000", + "negative_test_query83_00007", + "negative_test_query23_00012", + "negative_test_query230_00010", + "negative_test_query3_00005", + "negative_test_query147_00014", + "negative_test_query279_00000", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "negative_test_query147_00011", + "negative_test_query3_00007", + "apositive_test_query230_00003", + "apositive_test_query3_00007", + "apositive_test_query3_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query30", + "ranked_ids": [ + "apositive_test_query71_00002", + "apositive_test_query30_00001", + "apositive_test_query306_00002", + "apositive_test_query30_00004", + "apositive_test_query71_00000", + "negative_test_query231_00008", + "negative_test_query71_00010", + "negative_test_query71_00005", + "apositive_test_query209_00005", + "negative_test_query259_00002", + "apositive_test_query30_00005", + "apositive_test_query30_00003", + "negative_test_query182_00014", + "negative_test_query201_00005", + "negative_test_query357_00002", + "apositive_test_query30_00000", + "negative_test_query231_00000", + "negative_test_query356_00010", + "negative_test_query231_00002", + "negative_test_query97_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query15_00000", + "negative_test_query300_00009", + "apositive_test_query300_00001", + "negative_test_query266_00010", + "negative_test_query312_00011", + "negative_test_query300_00013", + "negative_test_query190_00004", + "negative_test_query185_00001", + "negative_test_query300_00008", + "negative_test_query252_00005", + "negative_test_query167_00018", + "negative_test_query300_00006", + "negative_test_query300_00011", + "apositive_test_query300_00002", + "negative_test_query141_00000", + "negative_test_query190_00016", + "negative_test_query174_00001", + "negative_test_query141_00001", + "negative_test_query300_00005", + "negative_test_query300_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.592512031964586, + "recall@5": 0.75 + }, + "query_id": "test_query301", + "ranked_ids": [ + "negative_test_query301_00011", + "apositive_test_query301_00002", + "apositive_test_query301_00001", + "negative_test_query301_00006", + "apositive_test_query301_00000", + "negative_test_query2_00008", + "negative_test_query301_00002", + "negative_test_query301_00001", + "negative_test_query301_00000", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query301_00015", + "negative_test_query301_00013", + "negative_test_query301_00007", + "apositive_test_query171_00000", + "apositive_test_query86_00002", + "negative_test_query148_00009", + "apositive_test_query86_00000", + "negative_test_query297_00015", + "negative_test_query21_00018" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query302", + "ranked_ids": [ + "apositive_test_query302_00000", + "apositive_test_query302_00003", + "negative_test_query302_00010", + "apositive_test_query302_00002", + "negative_test_query302_00001", + "negative_test_query302_00012", + "apositive_test_query302_00001", + "apositive_test_query302_00005", + "apositive_test_query302_00004", + "negative_test_query308_00008", + "negative_test_query302_00000", + "apositive_test_query324_00002", + "negative_test_query308_00003", + "apositive_test_query308_00001", + "apositive_test_query308_00008", + "apositive_test_query324_00001", + "negative_test_query308_00002", + "negative_test_query308_00009", + "apositive_test_query308_00009", + "negative_test_query308_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query35_00003", + "negative_test_query303_00017", + "apositive_test_query303_00000", + "negative_test_query173_00001", + "negative_test_query316_00009", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query206_00011", + "negative_test_query303_00015", + "negative_test_query206_00009", + "negative_test_query243_00000", + "negative_test_query307_00006", + "negative_test_query61_00011", + "negative_test_query303_00005", + "negative_test_query274_00014", + "negative_test_query196_00007", + "negative_test_query8_00013", + "apositive_test_query86_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query304", + "ranked_ids": [ + "negative_test_query304_00005", + "negative_test_query214_00002", + "negative_test_query43_00001", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query304_00013", + "apositive_test_query304_00001", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query234_00010", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "apositive_test_query304_00003", + "negative_test_query304_00012", + "negative_test_query304_00008", + "negative_test_query304_00004", + "negative_test_query304_00007", + "negative_test_query304_00010", + "apositive_test_query285_00005", + "negative_test_query43_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query305", + "ranked_ids": [ + "negative_test_query305_00014", + "apositive_test_query357_00000", + "negative_test_query323_00003", + "apositive_test_query305_00001", + "negative_test_query0_00007", + "negative_test_query203_00006", + "negative_test_query0_00004", + "negative_test_query0_00003", + "apositive_test_query259_00011", + "apositive_test_query127_00005", + "negative_test_query310_00000", + "negative_test_query164_00011", + "negative_test_query224_00015", + "negative_test_query134_00006", + "negative_test_query56_00004", + "negative_test_query51_00003", + "apositive_test_query71_00003", + "negative_test_query305_00005", + "negative_test_query47_00001", + "negative_test_query65_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query306", + "ranked_ids": [ + "apositive_test_query71_00002", + "apositive_test_query209_00005", + "negative_test_query305_00014", + "negative_test_query71_00012", + "apositive_test_query71_00001", + "negative_test_query204_00001", + "negative_test_query159_00009", + "negative_test_query259_00002", + "apositive_test_query306_00002", + "apositive_test_query119_00006", + "negative_test_query71_00011", + "apositive_test_query71_00003", + "apositive_test_query289_00003", + "apositive_test_query30_00001", + "negative_test_query182_00014", + "apositive_test_query30_00002", + "apositive_test_query127_00007", + "negative_test_query209_00008", + "negative_test_query142_00002", + "negative_test_query71_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "apositive_test_query279_00002", + "apositive_test_query3_00003", + "negative_test_query251_00001", + "negative_test_query70_00010", + "negative_test_query144_00004", + "negative_test_query187_00011", + "negative_test_query346_00017", + "apositive_test_query13_00005", + "negative_test_query207_00004", + "apositive_test_query279_00003", + "negative_test_query103_00012", + "negative_test_query48_00018", + "apositive_test_query170_00005", + "negative_test_query163_00010", + "negative_test_query18_00005", + "negative_test_query307_00006", + "negative_test_query360_00003", + "negative_test_query132_00008", + "negative_test_query217_00003", + "apositive_test_query177_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query308_00002", + "apositive_test_query308_00000", + "apositive_test_query308_00001", + "apositive_test_query308_00008", + "apositive_test_query324_00000", + "negative_test_query159_00013", + "negative_test_query308_00008", + "apositive_test_query308_00009", + "negative_test_query324_00012", + "negative_test_query324_00014", + "apositive_test_query324_00002", + "negative_test_query308_00003", + "apositive_test_query308_00003", + "apositive_test_query324_00001", + "negative_test_query308_00009", + "negative_test_query308_00007", + "negative_test_query94_00011", + "apositive_test_query310_00008", + "apositive_test_query7_00000", + "apositive_test_query341_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "negative_test_query309_00005", + "negative_test_query354_00002", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "negative_test_query309_00015", + "negative_test_query309_00012", + "apositive_test_query309_00000", + "apositive_test_query358_00007", + "negative_test_query309_00003", + "negative_test_query123_00007", + "negative_test_query354_00008", + "apositive_test_query161_00008", + "apositive_test_query358_00006", + "negative_test_query81_00002", + "negative_test_query113_00013", + "negative_test_query113_00011", + "negative_test_query42_00009", + "apositive_test_query358_00003", + "negative_test_query309_00006", + "apositive_test_query76_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query31_00003", + "negative_test_query313_00007", + "apositive_test_query31_00000", + "negative_test_query31_00011", + "negative_test_query31_00015", + "negative_test_query129_00000", + "negative_test_query31_00001", + "negative_test_query31_00006", + "negative_test_query31_00002", + "negative_test_query31_00007", + "negative_test_query31_00013", + "negative_test_query31_00010", + "negative_test_query31_00008", + "apositive_test_query60_00011", + "negative_test_query31_00000", + "negative_test_query31_00004", + "apositive_test_query213_00005", + "apositive_test_query249_00009", + "negative_test_query326_00005", + "negative_test_query38_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5 + }, + "query_id": "test_query310", + "ranked_ids": [ + "apositive_test_query310_00001", + "apositive_test_query310_00000", + "apositive_test_query310_00009", + "apositive_test_query310_00008", + "apositive_test_query310_00002", + "apositive_test_query218_00005", + "apositive_test_query310_00003", + "negative_test_query329_00003", + "negative_test_query241_00001", + "negative_test_query179_00013", + "negative_test_query74_00002", + "negative_test_query191_00010", + "negative_test_query330_00007", + "apositive_test_query310_00006", + "negative_test_query148_00003", + "negative_test_query148_00010", + "negative_test_query292_00007", + "negative_test_query310_00001", + "negative_test_query36_00015", + "negative_test_query59_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query311_00005", + "negative_test_query311_00008", + "apositive_test_query311_00000", + "negative_test_query311_00012", + "negative_test_query240_00014", + "apositive_test_query330_00001", + "negative_test_query202_00017", + "negative_test_query246_00006", + "negative_test_query240_00008", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query12_00008", + "apositive_test_query5_00002", + "apositive_test_query330_00002", + "negative_test_query288_00004", + "negative_test_query330_00009", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "apositive_test_query212_00004", + "apositive_test_query326_00004", + "negative_test_query312_00006", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query182_00002", + "negative_test_query157_00002", + "apositive_test_query98_00000", + "negative_test_query246_00015", + "apositive_test_query312_00000", + "negative_test_query212_00003", + "negative_test_query191_00000", + "negative_test_query206_00010", + "negative_test_query343_00016", + "negative_test_query242_00001", + "negative_test_query294_00015", + "negative_test_query305_00015", + "apositive_test_query237_00001", + "negative_test_query283_00016", + "apositive_test_query220_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query334_00001", + "negative_test_query46_00005", + "negative_test_query46_00003", + "negative_test_query46_00011", + "negative_test_query313_00010", + "negative_test_query57_00001", + "negative_test_query359_00005", + "apositive_test_query142_00005", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "negative_test_query56_00001", + "negative_test_query336_00001", + "apositive_test_query108_00001", + "negative_test_query313_00002", + "negative_test_query158_00000", + "negative_test_query111_00013", + "apositive_test_query155_00003", + "negative_test_query119_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.11764705882352941 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query3_00009", + "apositive_test_query314_00016", + "apositive_test_query314_00001", + "negative_test_query314_00001", + "apositive_test_query314_00003", + "apositive_test_query314_00007", + "apositive_test_query276_00007", + "apositive_test_query314_00005", + "apositive_test_query314_00006", + "apositive_test_query314_00002", + "apositive_test_query314_00004", + "apositive_test_query314_00011", + "negative_test_query37_00006", + "negative_test_query314_00002", + "apositive_test_query314_00010", + "apositive_test_query314_00012", + "apositive_test_query314_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query315_00000", + "apositive_test_query315_00002", + "negative_test_query333_00007", + "negative_test_query85_00013", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query345_00006", + "apositive_test_query319_00005", + "apositive_test_query82_00005", + "negative_test_query85_00014", + "apositive_test_query85_00003", + "negative_test_query85_00005", + "negative_test_query315_00009", + "apositive_test_query315_00001", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "negative_test_query269_00004", + "negative_test_query319_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query316", + "ranked_ids": [ + "apositive_test_query316_00000", + "negative_test_query316_00011", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query316_00004", + "negative_test_query316_00000", + "negative_test_query316_00008", + "negative_test_query316_00001", + "negative_test_query316_00016", + "negative_test_query316_00005", + "apositive_test_query63_00007", + "negative_test_query301_00005", + "negative_test_query101_00007", + "apositive_test_query316_00001", + "apositive_test_query216_00000", + "apositive_test_query72_00000", + "negative_test_query228_00011", + "negative_test_query322_00000", + "negative_test_query316_00002", + "negative_test_query158_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00004", + "negative_test_query317_00005", + "negative_test_query317_00013", + "negative_test_query317_00008", + "negative_test_query317_00014", + "apositive_test_query317_00000", + "negative_test_query317_00010", + "apositive_test_query317_00001", + "negative_test_query317_00006", + "negative_test_query317_00000", + "negative_test_query317_00012", + "negative_test_query116_00014", + "negative_test_query55_00014", + "negative_test_query317_00015", + "negative_test_query116_00012", + "negative_test_query271_00002", + "negative_test_query283_00013", + "negative_test_query331_00009", + "negative_test_query317_00017", + "negative_test_query317_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00000", + "apositive_test_query318_00002", + "apositive_test_query318_00001", + "negative_test_query318_00012", + "negative_test_query73_00012", + "negative_test_query318_00015", + "apositive_test_query318_00003", + "negative_test_query318_00010", + "negative_test_query318_00003", + "negative_test_query318_00000", + "negative_test_query318_00002", + "negative_test_query318_00004", + "negative_test_query318_00005", + "negative_test_query73_00004", + "negative_test_query73_00005", + "negative_test_query318_00008", + "negative_test_query318_00009", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query73_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00003", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query340_00013", + "negative_test_query319_00009", + "negative_test_query82_00009", + "negative_test_query319_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query196_00000", + "negative_test_query189_00014", + "apositive_test_query136_00003", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query357_00000", + "negative_test_query55_00011", + "apositive_test_query17_00000", + "negative_test_query78_00007", + "negative_test_query55_00014", + "apositive_test_query32_00013", + "negative_test_query321_00001", + "negative_test_query341_00010", + "negative_test_query211_00016", + "apositive_test_query192_00000", + "negative_test_query211_00005", + "negative_test_query68_00009", + "negative_test_query83_00001", + "apositive_test_query336_00001", + "negative_test_query339_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query320", + "ranked_ids": [ + "negative_test_query344_00010", + "negative_test_query318_00011", + "negative_test_query100_00001", + "apositive_test_query238_00000", + "negative_test_query212_00006", + "negative_test_query131_00009", + "negative_test_query91_00007", + "negative_test_query320_00005", + "negative_test_query230_00001", + "negative_test_query193_00006", + "negative_test_query258_00012", + "negative_test_query53_00014", + "negative_test_query43_00000", + "apositive_test_query320_00003", + "negative_test_query216_00014", + "apositive_test_query267_00001", + "negative_test_query320_00004", + "negative_test_query304_00005", + "negative_test_query320_00008", + "negative_test_query290_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "negative_test_query321_00010", + "negative_test_query321_00000", + "apositive_test_query321_00000", + "negative_test_query321_00002", + "apositive_test_query185_00009", + "negative_test_query103_00011", + "negative_test_query34_00012", + "negative_test_query126_00005", + "negative_test_query136_00004", + "negative_test_query141_00009", + "negative_test_query86_00014", + "apositive_test_query217_00006", + "negative_test_query197_00004", + "negative_test_query352_00005", + "negative_test_query15_00013", + "negative_test_query226_00002", + "negative_test_query126_00011", + "negative_test_query322_00002", + "negative_test_query210_00016", + "negative_test_query103_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query322", + "ranked_ids": [ + "apositive_test_query322_00000", + "apositive_test_query322_00005", + "apositive_test_query322_00001", + "negative_test_query255_00010", + "apositive_test_query322_00002", + "negative_test_query322_00006", + "apositive_test_query322_00006", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query322_00002", + "negative_test_query318_00002", + "negative_test_query73_00010", + "negative_test_query322_00005", + "negative_test_query322_00003", + "apositive_test_query228_00001", + "apositive_test_query322_00003", + "apositive_test_query322_00004", + "apositive_test_query254_00006", + "negative_test_query322_00009", + "negative_test_query73_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query323", + "ranked_ids": [ + "negative_test_query323_00003", + "apositive_test_query323_00000", + "negative_test_query323_00012", + "apositive_test_query323_00006", + "apositive_test_query323_00002", + "apositive_test_query323_00001", + "apositive_test_query323_00004", + "negative_test_query323_00007", + "negative_test_query323_00001", + "negative_test_query75_00002", + "negative_test_query209_00004", + "negative_test_query255_00002", + "negative_test_query201_00009", + "negative_test_query75_00007", + "apositive_test_query130_00001", + "negative_test_query323_00000", + "negative_test_query15_00006", + "negative_test_query345_00011", + "negative_test_query75_00004", + "negative_test_query323_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9060254355346823, + "recall@5": 1.0 + }, + "query_id": "test_query324", + "ranked_ids": [ + "apositive_test_query324_00002", + "negative_test_query308_00003", + "apositive_test_query324_00000", + "apositive_test_query324_00001", + "negative_test_query324_00005", + "negative_test_query324_00003", + "negative_test_query324_00004", + "negative_test_query324_00015", + "negative_test_query324_00006", + "negative_test_query324_00016", + "negative_test_query308_00007", + "apositive_test_query308_00003", + "negative_test_query324_00014", + "apositive_test_query308_00001", + "negative_test_query218_00003", + "negative_test_query308_00009", + "apositive_test_query308_00009", + "negative_test_query324_00007", + "apositive_test_query308_00002", + "apositive_test_query308_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query325_00007", + "negative_test_query15_00017", + "negative_test_query15_00015", + "negative_test_query293_00015", + "negative_test_query15_00005", + "negative_test_query15_00016", + "negative_test_query325_00002", + "apositive_test_query19_00002", + "apositive_test_query293_00002", + "negative_test_query325_00013", + "apositive_test_query178_00003", + "apositive_test_query246_00001", + "negative_test_query293_00000", + "negative_test_query266_00014", + "negative_test_query19_00008", + "negative_test_query191_00011", + "negative_test_query297_00008", + "negative_test_query176_00003", + "negative_test_query325_00008", + "negative_test_query169_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query326", + "ranked_ids": [ + "negative_test_query107_00011", + "negative_test_query17_00011", + "negative_test_query156_00013", + "negative_test_query178_00012", + "apositive_test_query326_00003", + "apositive_test_query52_00002", + "apositive_test_query54_00002", + "apositive_test_query326_00001", + "apositive_test_query326_00000", + "negative_test_query265_00007", + "apositive_test_query212_00004", + "apositive_test_query326_00004", + "apositive_test_query355_00002", + "negative_test_query183_00001", + "negative_test_query81_00003", + "apositive_test_query23_00000", + "apositive_test_query294_00001", + "negative_test_query181_00007", + "negative_test_query265_00005", + "negative_test_query45_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query327", + "ranked_ids": [ + "apositive_test_query327_00001", + "negative_test_query327_00014", + "apositive_test_query327_00000", + "negative_test_query321_00014", + "negative_test_query327_00000", + "negative_test_query327_00003", + "negative_test_query195_00007", + "negative_test_query61_00006", + "negative_test_query327_00013", + "negative_test_query303_00010", + "negative_test_query61_00003", + "negative_test_query327_00010", + "apositive_test_query327_00002", + "negative_test_query327_00012", + "negative_test_query227_00009", + "apositive_test_query271_00000", + "negative_test_query299_00010", + "apositive_test_query271_00001", + "negative_test_query271_00007", + "negative_test_query335_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query328", + "ranked_ids": [ + "negative_test_query328_00003", + "negative_test_query328_00000", + "apositive_test_query328_00002", + "negative_test_query24_00006", + "negative_test_query92_00009", + "negative_test_query328_00007", + "negative_test_query328_00006", + "apositive_test_query328_00007", + "apositive_test_query40_00003", + "negative_test_query138_00013", + "negative_test_query144_00005", + "negative_test_query59_00017", + "negative_test_query222_00016", + "negative_test_query261_00009", + "apositive_test_query328_00001", + "negative_test_query129_00002", + "negative_test_query217_00005", + "negative_test_query328_00002", + "apositive_test_query227_00001", + "apositive_test_query328_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query329", + "ranked_ids": [ + "negative_test_query329_00006", + "negative_test_query329_00009", + "apositive_test_query329_00001", + "negative_test_query269_00000", + "negative_test_query329_00004", + "negative_test_query329_00001", + "negative_test_query269_00012", + "negative_test_query329_00010", + "apositive_test_query329_00002", + "negative_test_query62_00017", + "negative_test_query329_00003", + "negative_test_query329_00002", + "negative_test_query329_00011", + "negative_test_query329_00013", + "negative_test_query298_00015", + "apositive_test_query7_00001", + "negative_test_query183_00006", + "negative_test_query35_00000", + "negative_test_query202_00016", + "negative_test_query38_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00004", + "apositive_test_query33_00006", + "apositive_test_query20_00010", + "apositive_test_query154_00007", + "apositive_test_query33_00004", + "negative_test_query33_00006", + "negative_test_query154_00006", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "apositive_test_query33_00009", + "negative_test_query344_00010", + "apositive_test_query284_00004", + "negative_test_query356_00006", + "apositive_test_query220_00003", + "apositive_test_query33_00003", + "negative_test_query216_00004", + "negative_test_query216_00014", + "apositive_test_query220_00000", + "negative_test_query161_00008", + "negative_test_query220_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query330", + "ranked_ids": [ + "negative_test_query330_00010", + "negative_test_query252_00007", + "apositive_test_query330_00002", + "negative_test_query330_00009", + "negative_test_query330_00008", + "negative_test_query27_00013", + "negative_test_query330_00001", + "negative_test_query288_00006", + "apositive_test_query60_00014", + "negative_test_query27_00018", + "negative_test_query330_00000", + "negative_test_query330_00012", + "apositive_test_query281_00012", + "negative_test_query311_00009", + "apositive_test_query284_00003", + "apositive_test_query288_00002", + "negative_test_query129_00003", + "negative_test_query180_00000", + "negative_test_query27_00011", + "negative_test_query325_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query331", + "ranked_ids": [ + "negative_test_query331_00013", + "apositive_test_query331_00000", + "negative_test_query250_00016", + "apositive_test_query331_00002", + "negative_test_query202_00001", + "negative_test_query303_00001", + "apositive_test_query223_00001", + "apositive_test_query331_00001", + "negative_test_query331_00000", + "negative_test_query113_00000", + "negative_test_query312_00012", + "negative_test_query154_00000", + "negative_test_query331_00012", + "negative_test_query110_00007", + "negative_test_query214_00015", + "negative_test_query331_00008", + "negative_test_query251_00009", + "negative_test_query3_00008", + "negative_test_query331_00007", + "negative_test_query334_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00003", + "negative_test_query202_00010", + "apositive_test_query142_00004", + "negative_test_query12_00005", + "negative_test_query171_00008", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "negative_test_query65_00007", + "negative_test_query180_00011", + "negative_test_query245_00001", + "apositive_test_query348_00008", + "negative_test_query115_00008", + "apositive_test_query162_00005", + "negative_test_query332_00009", + "negative_test_query244_00006", + "negative_test_query202_00002", + "apositive_test_query332_00001", + "negative_test_query145_00010", + "apositive_test_query332_00002", + "negative_test_query332_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00002", + "negative_test_query333_00008", + "negative_test_query216_00013", + "apositive_test_query322_00003", + "negative_test_query322_00009", + "negative_test_query73_00008", + "negative_test_query333_00005", + "negative_test_query246_00000", + "negative_test_query318_00005", + "negative_test_query67_00000", + "negative_test_query73_00005", + "apositive_test_query322_00000", + "apositive_test_query333_00000", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00005", + "negative_test_query136_00003", + "negative_test_query73_00014", + "apositive_test_query228_00000", + "negative_test_query318_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query334_00010", + "negative_test_query37_00000", + "apositive_test_query276_00009", + "negative_test_query334_00012", + "negative_test_query155_00010", + "negative_test_query111_00017", + "negative_test_query213_00009", + "negative_test_query107_00001", + "negative_test_query155_00013", + "apositive_test_query57_00006", + "negative_test_query111_00005", + "negative_test_query111_00010", + "negative_test_query170_00004", + "apositive_test_query26_00001", + "negative_test_query209_00003", + "negative_test_query111_00006", + "negative_test_query102_00015", + "negative_test_query164_00005", + "negative_test_query334_00016", + "negative_test_query223_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.4 + }, + "query_id": "test_query335", + "ranked_ids": [ + "apositive_test_query335_00000", + "negative_test_query335_00006", + "negative_test_query335_00013", + "negative_test_query335_00000", + "apositive_test_query335_00002", + "apositive_test_query335_00001", + "negative_test_query335_00010", + "negative_test_query335_00004", + "negative_test_query335_00007", + "apositive_test_query335_00003", + "apositive_test_query335_00004", + "negative_test_query335_00003", + "negative_test_query335_00001", + "negative_test_query335_00009", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query92_00002", + "negative_test_query335_00011", + "negative_test_query45_00010", + "negative_test_query210_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "negative_test_query336_00009", + "apositive_test_query50_00005", + "negative_test_query336_00004", + "apositive_test_query50_00015", + "negative_test_query336_00007", + "apositive_test_query186_00002", + "negative_test_query108_00005", + "negative_test_query247_00006", + "apositive_test_query50_00012", + "apositive_test_query38_00005", + "negative_test_query247_00000", + "negative_test_query247_00008", + "apositive_test_query242_00012", + "negative_test_query247_00003", + "negative_test_query336_00015", + "negative_test_query336_00014", + "negative_test_query105_00011", + "negative_test_query186_00012", + "apositive_test_query113_00005", + "apositive_test_query186_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00010", + "apositive_test_query337_00000", + "negative_test_query200_00001", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query200_00007", + "negative_test_query337_00007", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query200_00000", + "negative_test_query200_00009", + "negative_test_query200_00015", + "apositive_test_query38_00003", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query337_00000", + "negative_test_query159_00013", + "negative_test_query337_00004", + "negative_test_query38_00008", + "negative_test_query337_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query338", + "ranked_ids": [ + "apositive_test_query338_00002", + "apositive_test_query338_00008", + "apositive_test_query338_00005", + "apositive_test_query338_00001", + "negative_test_query114_00004", + "negative_test_query114_00007", + "apositive_test_query338_00003", + "negative_test_query81_00004", + "negative_test_query95_00004", + "apositive_test_query338_00000", + "negative_test_query119_00003", + "apositive_test_query338_00006", + "negative_test_query338_00003", + "apositive_test_query81_00000", + "negative_test_query76_00005", + "negative_test_query81_00006", + "apositive_test_query338_00004", + "negative_test_query53_00008", + "apositive_test_query114_00005", + "apositive_test_query114_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query339_00001", + "negative_test_query339_00008", + "negative_test_query339_00012", + "apositive_test_query339_00000", + "apositive_test_query96_00001", + "apositive_test_query96_00002", + "negative_test_query192_00011", + "negative_test_query339_00001", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query339_00016", + "negative_test_query96_00004", + "negative_test_query96_00016", + "negative_test_query96_00003", + "negative_test_query339_00009", + "negative_test_query339_00017", + "negative_test_query339_00004", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query339_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00000", + "apositive_test_query352_00002", + "negative_test_query34_00002", + "negative_test_query352_00004", + "apositive_test_query34_00004", + "apositive_test_query352_00003", + "negative_test_query34_00008", + "negative_test_query352_00010", + "apositive_test_query352_00001", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "negative_test_query34_00000", + "negative_test_query352_00000", + "apositive_test_query34_00001", + "negative_test_query352_00008", + "negative_test_query352_00013", + "negative_test_query34_00009", + "apositive_test_query34_00000", + "negative_test_query34_00014", + "negative_test_query352_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.4 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00004", + "negative_test_query340_00014", + "negative_test_query340_00008", + "negative_test_query340_00007", + "apositive_test_query340_00000", + "negative_test_query340_00012", + "negative_test_query340_00011", + "negative_test_query88_00013", + "negative_test_query88_00006", + "apositive_test_query340_00002", + "negative_test_query88_00007", + "apositive_test_query340_00001", + "apositive_test_query174_00002", + "negative_test_query88_00000", + "negative_test_query340_00001", + "negative_test_query340_00006", + "apositive_test_query340_00003", + "apositive_test_query174_00012", + "negative_test_query23_00012", + "negative_test_query88_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query96_00005", + "apositive_test_query341_00003", + "negative_test_query214_00005", + "negative_test_query341_00003", + "negative_test_query316_00008", + "negative_test_query196_00009", + "negative_test_query32_00003", + "negative_test_query341_00009", + "apositive_test_query341_00002", + "apositive_test_query341_00005", + "apositive_test_query341_00001", + "negative_test_query341_00000", + "negative_test_query73_00014", + "negative_test_query341_00002", + "negative_test_query96_00013", + "negative_test_query96_00007", + "apositive_test_query32_00007", + "negative_test_query196_00006", + "negative_test_query341_00005", + "negative_test_query101_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query212_00009", + "negative_test_query342_00009", + "negative_test_query342_00015", + "negative_test_query68_00005", + "negative_test_query134_00013", + "negative_test_query200_00009", + "negative_test_query342_00016", + "negative_test_query49_00002", + "apositive_test_query342_00000", + "negative_test_query84_00007", + "negative_test_query136_00003", + "negative_test_query78_00015", + "negative_test_query98_00017", + "negative_test_query245_00001", + "negative_test_query98_00011", + "negative_test_query324_00015", + "negative_test_query325_00015", + "negative_test_query77_00004", + "negative_test_query342_00003", + "negative_test_query77_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query343", + "ranked_ids": [ + "negative_test_query103_00010", + "apositive_test_query279_00003", + "negative_test_query307_00006", + "negative_test_query225_00008", + "apositive_test_query307_00008", + "apositive_test_query136_00004", + "apositive_test_query133_00005", + "apositive_test_query287_00003", + "apositive_test_query343_00001", + "negative_test_query360_00017", + "apositive_test_query185_00001", + "apositive_test_query343_00000", + "negative_test_query103_00008", + "negative_test_query343_00008", + "apositive_test_query279_00002", + "negative_test_query227_00010", + "apositive_test_query185_00008", + "apositive_test_query217_00002", + "apositive_test_query217_00004", + "apositive_test_query28_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query282_00008", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query282_00004", + "negative_test_query282_00001", + "negative_test_query344_00002", + "negative_test_query344_00004", + "apositive_test_query221_00002", + "negative_test_query83_00005", + "negative_test_query344_00012", + "apositive_test_query83_00006", + "negative_test_query223_00016", + "negative_test_query344_00005", + "negative_test_query345_00012", + "negative_test_query83_00008", + "negative_test_query344_00006", + "negative_test_query188_00000", + "negative_test_query231_00015", + "negative_test_query282_00000", + "negative_test_query344_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query345_00003", + "negative_test_query90_00003", + "negative_test_query9_00002", + "apositive_test_query345_00004", + "negative_test_query184_00009", + "negative_test_query47_00008", + "apositive_test_query0_00008", + "negative_test_query345_00001", + "negative_test_query345_00007", + "negative_test_query345_00010", + "apositive_test_query9_00000", + "negative_test_query130_00004", + "negative_test_query221_00002", + "negative_test_query259_00007", + "negative_test_query47_00003", + "negative_test_query345_00011", + "apositive_test_query203_00005", + "apositive_test_query345_00000", + "apositive_test_query345_00001", + "apositive_test_query90_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query346_00009", + "negative_test_query346_00002", + "negative_test_query157_00005", + "negative_test_query346_00001", + "negative_test_query49_00007", + "negative_test_query346_00010", + "negative_test_query346_00012", + "apositive_test_query346_00001", + "negative_test_query346_00014", + "apositive_test_query346_00000", + "negative_test_query346_00003", + "negative_test_query346_00017", + "negative_test_query346_00008", + "negative_test_query346_00006", + "negative_test_query174_00001", + "negative_test_query19_00012", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "negative_test_query15_00000", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query347", + "ranked_ids": [ + "apositive_test_query347_00000", + "negative_test_query347_00000", + "negative_test_query347_00001", + "negative_test_query347_00007", + "negative_test_query31_00003", + "negative_test_query347_00002", + "negative_test_query347_00005", + "negative_test_query347_00008", + "negative_test_query347_00014", + "negative_test_query298_00006", + "negative_test_query347_00006", + "negative_test_query347_00009", + "negative_test_query347_00010", + "negative_test_query347_00012", + "negative_test_query298_00003", + "negative_test_query347_00016", + "apositive_test_query347_00001", + "negative_test_query347_00011", + "negative_test_query251_00007", + "negative_test_query347_00015" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query348_00001", + "apositive_test_query348_00006", + "apositive_test_query348_00005", + "apositive_test_query348_00004", + "apositive_test_query348_00009", + "negative_test_query348_00008", + "negative_test_query348_00006", + "apositive_test_query115_00002", + "apositive_test_query227_00002", + "apositive_test_query115_00004", + "negative_test_query115_00002", + "negative_test_query115_00006", + "negative_test_query359_00014", + "negative_test_query297_00017", + "apositive_test_query115_00001", + "negative_test_query341_00008", + "apositive_test_query348_00000", + "negative_test_query115_00012", + "negative_test_query351_00010", + "negative_test_query31_00017" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "negative_test_query253_00000", + "negative_test_query312_00016", + "negative_test_query349_00015", + "negative_test_query253_00007", + "negative_test_query74_00001", + "negative_test_query349_00003", + "apositive_test_query74_00003", + "apositive_test_query253_00005", + "apositive_test_query253_00009", + "negative_test_query124_00013", + "apositive_test_query355_00007", + "apositive_test_query253_00008", + "negative_test_query253_00009", + "apositive_test_query74_00002", + "negative_test_query157_00009", + "negative_test_query13_00002", + "apositive_test_query137_00002", + "negative_test_query194_00001", + "negative_test_query349_00012", + "apositive_test_query42_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.44149241373678083, + "recall@5": 0.5 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query139_00000", + "apositive_test_query35_00001", + "apositive_test_query35_00000", + "negative_test_query205_00005", + "apositive_test_query146_00007", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "apositive_test_query35_00003", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query139_00016", + "negative_test_query35_00002", + "negative_test_query35_00012", + "negative_test_query123_00006", + "negative_test_query231_00012", + "negative_test_query269_00006", + "negative_test_query101_00006", + "negative_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query350_00015", + "negative_test_query350_00014", + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query350_00010", + "negative_test_query149_00005", + "negative_test_query14_00001", + "negative_test_query179_00001", + "negative_test_query63_00000", + "negative_test_query331_00012", + "negative_test_query192_00004", + "negative_test_query350_00001", + "negative_test_query350_00006", + "negative_test_query192_00005", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query350_00011", + "negative_test_query192_00013", + "negative_test_query316_00006", + "apositive_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query351_00000", + "negative_test_query156_00006", + "apositive_test_query299_00000", + "negative_test_query351_00002", + "negative_test_query92_00005", + "apositive_test_query311_00000", + "negative_test_query351_00006", + "negative_test_query351_00014", + "apositive_test_query113_00001", + "apositive_test_query60_00003", + "apositive_test_query60_00000", + "apositive_test_query60_00010", + "apositive_test_query218_00004", + "apositive_test_query335_00004", + "apositive_test_query60_00012", + "negative_test_query234_00009", + "negative_test_query53_00011", + "negative_test_query8_00003", + "apositive_test_query193_00009", + "apositive_test_query60_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query34_00001", + "apositive_test_query352_00003", + "apositive_test_query352_00001", + "negative_test_query34_00002", + "negative_test_query352_00004", + "apositive_test_query352_00000", + "negative_test_query352_00013", + "negative_test_query34_00008", + "negative_test_query352_00010", + "apositive_test_query34_00004", + "apositive_test_query352_00002", + "negative_test_query34_00000", + "negative_test_query352_00000", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "apositive_test_query34_00002", + "negative_test_query352_00008", + "negative_test_query34_00012", + "apositive_test_query34_00000", + "negative_test_query34_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query353_00011", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00012", + "negative_test_query353_00002", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00009", + "apositive_test_query353_00000", + "apositive_test_query204_00002", + "apositive_test_query353_00005", + "negative_test_query353_00003", + "apositive_test_query353_00009", + "apositive_test_query51_00002", + "apositive_test_query204_00015", + "apositive_test_query353_00007", + "negative_test_query175_00012", + "negative_test_query71_00008", + "negative_test_query108_00002", + "apositive_test_query204_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "apositive_test_query354_00006", + "apositive_test_query354_00004", + "negative_test_query309_00013", + "negative_test_query354_00004", + "negative_test_query194_00002", + "negative_test_query309_00014", + "negative_test_query358_00003", + "apositive_test_query354_00001", + "negative_test_query280_00007", + "negative_test_query331_00014", + "apositive_test_query16_00003", + "apositive_test_query16_00001", + "apositive_test_query320_00006", + "negative_test_query148_00009", + "apositive_test_query354_00005", + "negative_test_query309_00000", + "negative_test_query354_00006", + "apositive_test_query147_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query355_00001", + "negative_test_query355_00009", + "negative_test_query355_00007", + "apositive_test_query355_00002", + "negative_test_query355_00008", + "negative_test_query355_00005", + "negative_test_query159_00017", + "negative_test_query264_00017", + "apositive_test_query355_00005", + "negative_test_query355_00003", + "negative_test_query355_00002", + "negative_test_query87_00004", + "negative_test_query215_00001", + "negative_test_query264_00003", + "negative_test_query68_00000", + "negative_test_query104_00001", + "apositive_test_query355_00003", + "apositive_test_query38_00006", + "apositive_test_query355_00006", + "apositive_test_query100_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query356_00002", + "apositive_test_query356_00004", + "negative_test_query253_00001", + "negative_test_query15_00006", + "negative_test_query102_00014", + "apositive_test_query356_00000", + "apositive_test_query30_00005", + "negative_test_query47_00011", + "negative_test_query102_00010", + "apositive_test_query66_00003", + "negative_test_query280_00004", + "apositive_test_query356_00008", + "apositive_test_query33_00000", + "apositive_test_query20_00001", + "negative_test_query334_00000", + "apositive_test_query127_00007", + "apositive_test_query295_00000", + "apositive_test_query203_00006", + "apositive_test_query88_00000", + "apositive_test_query154_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00002", + "apositive_test_query221_00000", + "apositive_test_query117_00004", + "apositive_test_query221_00002", + "apositive_test_query314_00000", + "negative_test_query300_00007", + "apositive_test_query119_00001", + "negative_test_query58_00009", + "apositive_test_query119_00000", + "apositive_test_query119_00010", + "apositive_test_query221_00004", + "negative_test_query58_00008", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query119_00005", + "apositive_test_query119_00009", + "apositive_test_query357_00004", + "negative_test_query46_00004", + "apositive_test_query119_00012", + "apositive_test_query357_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2 + }, + "query_id": "test_query358", + "ranked_ids": [ + "negative_test_query309_00005", + "apositive_test_query358_00006", + "negative_test_query123_00007", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "apositive_test_query309_00000", + "negative_test_query309_00006", + "apositive_test_query11_00007", + "apositive_test_query161_00000", + "apositive_test_query358_00003", + "apositive_test_query358_00007", + "apositive_test_query358_00000", + "negative_test_query132_00001", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "negative_test_query279_00010", + "negative_test_query354_00008", + "apositive_test_query358_00009", + "negative_test_query309_00015", + "negative_test_query42_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00006", + "apositive_test_query0_00004", + "apositive_test_query0_00003", + "negative_test_query359_00002", + "negative_test_query0_00001", + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "negative_test_query0_00007", + "negative_test_query184_00016", + "negative_test_query268_00014", + "negative_test_query305_00000", + "negative_test_query359_00007", + "apositive_test_query0_00002", + "negative_test_query259_00006", + "apositive_test_query259_00008", + "apositive_test_query0_00007", + "apositive_test_query259_00004", + "apositive_test_query259_00005", + "apositive_test_query359_00002", + "negative_test_query130_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query36", + "ranked_ids": [ + "negative_test_query112_00004", + "apositive_test_query36_00003", + "apositive_test_query286_00006", + "negative_test_query112_00016", + "negative_test_query36_00014", + "negative_test_query112_00010", + "negative_test_query216_00002", + "apositive_test_query286_00005", + "negative_test_query112_00015", + "apositive_test_query106_00003", + "negative_test_query296_00013", + "negative_test_query6_00003", + "negative_test_query36_00010", + "negative_test_query260_00009", + "negative_test_query286_00000", + "negative_test_query36_00001", + "negative_test_query112_00013", + "negative_test_query260_00010", + "negative_test_query80_00007", + "apositive_test_query136_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00001", + "apositive_test_query310_00006", + "apositive_test_query360_00000", + "apositive_test_query152_00002", + "negative_test_query189_00013", + "negative_test_query360_00005", + "apositive_test_query142_00004", + "apositive_test_query153_00011", + "negative_test_query240_00010", + "apositive_test_query69_00014", + "negative_test_query360_00004", + "negative_test_query148_00018", + "negative_test_query175_00009", + "negative_test_query216_00011", + "negative_test_query312_00004", + "negative_test_query189_00014", + "negative_test_query246_00001", + "negative_test_query281_00000", + "negative_test_query360_00009", + "negative_test_query5_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query123_00001", + "apositive_test_query177_00000", + "negative_test_query209_00001", + "apositive_test_query37_00001", + "negative_test_query231_00006", + "negative_test_query155_00001", + "negative_test_query123_00011", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query20_00003", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query209_00002", + "negative_test_query152_00008", + "negative_test_query57_00009", + "apositive_test_query158_00007", + "apositive_test_query213_00006", + "negative_test_query209_00000", + "apositive_test_query108_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query38", + "ranked_ids": [ + "apositive_test_query38_00003", + "negative_test_query159_00010", + "negative_test_query267_00016", + "negative_test_query38_00010", + "negative_test_query267_00015", + "negative_test_query47_00016", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query337_00000", + "apositive_test_query38_00007", + "apositive_test_query337_00000", + "apositive_test_query38_00002", + "apositive_test_query247_00005", + "negative_test_query337_00017", + "negative_test_query38_00006", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query175_00005", + "negative_test_query79_00002", + "negative_test_query94_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query39", + "ranked_ids": [ + "apositive_test_query39_00003", + "negative_test_query18_00004", + "negative_test_query39_00000", + "negative_test_query39_00006", + "negative_test_query39_00012", + "apositive_test_query39_00001", + "apositive_test_query39_00002", + "negative_test_query331_00010", + "apositive_test_query39_00004", + "negative_test_query39_00003", + "negative_test_query39_00009", + "negative_test_query39_00007", + "apositive_test_query39_00000", + "negative_test_query39_00004", + "negative_test_query66_00008", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query29_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.35714285714285715 + }, + "query_id": "test_query4", + "ranked_ids": [ + "apositive_test_query4_00003", + "apositive_test_query4_00000", + "apositive_test_query4_00010", + "apositive_test_query4_00012", + "apositive_test_query4_00001", + "negative_test_query266_00009", + "negative_test_query174_00001", + "negative_test_query4_00002", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "apositive_test_query4_00004", + "apositive_test_query4_00005", + "negative_test_query4_00001", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "negative_test_query149_00015", + "negative_test_query19_00004", + "negative_test_query266_00004", + "negative_test_query266_00012", + "negative_test_query15_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00004", + "negative_test_query40_00006", + "negative_test_query40_00007", + "negative_test_query40_00001", + "negative_test_query40_00010", + "negative_test_query40_00013", + "negative_test_query40_00000", + "negative_test_query40_00003", + "negative_test_query40_00005", + "negative_test_query66_00005", + "negative_test_query214_00015", + "negative_test_query40_00011", + "negative_test_query347_00003", + "negative_test_query40_00012", + "negative_test_query328_00006", + "negative_test_query106_00009", + "apositive_test_query40_00002", + "negative_test_query40_00009", + "apositive_test_query223_00001", + "negative_test_query324_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query41", + "ranked_ids": [ + "apositive_test_query41_00002", + "negative_test_query41_00000", + "negative_test_query41_00004", + "negative_test_query41_00007", + "negative_test_query282_00003", + "apositive_test_query41_00000", + "negative_test_query41_00002", + "negative_test_query41_00012", + "negative_test_query41_00010", + "apositive_test_query41_00001", + "negative_test_query41_00006", + "apositive_test_query41_00003", + "apositive_test_query249_00012", + "negative_test_query144_00016", + "negative_test_query226_00011", + "negative_test_query41_00011", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query98_00000", + "negative_test_query41_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query42_00008", + "apositive_test_query42_00003", + "negative_test_query42_00002", + "negative_test_query42_00003", + "negative_test_query137_00005", + "apositive_test_query42_00001", + "apositive_test_query42_00005", + "negative_test_query42_00001", + "apositive_test_query42_00000", + "negative_test_query42_00000", + "negative_test_query42_00007", + "apositive_test_query42_00002", + "negative_test_query42_00005", + "negative_test_query42_00010", + "negative_test_query42_00006", + "apositive_test_query276_00012", + "apositive_test_query42_00004", + "negative_test_query42_00011", + "negative_test_query250_00010", + "apositive_test_query162_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query185_00002", + "negative_test_query48_00017", + "apositive_test_query43_00000", + "negative_test_query43_00009", + "apositive_test_query146_00004", + "apositive_test_query43_00002", + "apositive_test_query146_00003", + "apositive_test_query218_00002", + "negative_test_query333_00001", + "apositive_test_query161_00002", + "negative_test_query179_00006", + "negative_test_query43_00000", + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query153_00008", + "apositive_test_query153_00014", + "apositive_test_query161_00005", + "negative_test_query319_00002", + "negative_test_query82_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query44", + "ranked_ids": [ + "apositive_test_query44_00001", + "negative_test_query27_00011", + "negative_test_query27_00002", + "negative_test_query293_00015", + "negative_test_query293_00003", + "negative_test_query293_00009", + "negative_test_query330_00009", + "negative_test_query44_00002", + "negative_test_query27_00008", + "negative_test_query293_00000", + "apositive_test_query293_00000", + "apositive_test_query141_00005", + "apositive_test_query245_00000", + "negative_test_query10_00012", + "apositive_test_query20_00003", + "negative_test_query21_00009", + "negative_test_query330_00012", + "negative_test_query190_00011", + "negative_test_query141_00007", + "negative_test_query180_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query45_00013", + "apositive_test_query25_00000", + "negative_test_query45_00004", + "negative_test_query45_00007", + "negative_test_query182_00016", + "negative_test_query45_00001", + "apositive_test_query45_00000", + "negative_test_query328_00005", + "negative_test_query45_00005", + "negative_test_query45_00003", + "negative_test_query143_00009", + "negative_test_query45_00006", + "negative_test_query342_00005", + "negative_test_query143_00014", + "apositive_test_query25_00005", + "negative_test_query143_00011", + "negative_test_query45_00016", + "apositive_test_query165_00000", + "negative_test_query143_00004", + "negative_test_query45_00009" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "negative_test_query39_00011", + "negative_test_query108_00003", + "negative_test_query46_00001", + "negative_test_query56_00001", + "apositive_test_query142_00001", + "negative_test_query105_00006", + "negative_test_query213_00012", + "negative_test_query238_00001", + "negative_test_query313_00010", + "negative_test_query46_00011", + "negative_test_query46_00003", + "apositive_test_query146_00013", + "negative_test_query168_00003", + "negative_test_query334_00004", + "negative_test_query359_00005", + "negative_test_query79_00012", + "apositive_test_query276_00008", + "apositive_test_query186_00001", + "negative_test_query46_00012", + "apositive_test_query79_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query47_00001", + "negative_test_query47_00013", + "negative_test_query47_00006", + "negative_test_query144_00018", + "negative_test_query325_00007", + "apositive_test_query201_00002", + "negative_test_query41_00009", + "negative_test_query47_00012", + "negative_test_query195_00005", + "negative_test_query331_00014", + "negative_test_query331_00005", + "negative_test_query47_00002", + "apositive_test_query331_00002", + "apositive_test_query201_00006", + "apositive_test_query20_00005", + "negative_test_query47_00011", + "apositive_test_query201_00004", + "negative_test_query345_00002", + "negative_test_query328_00007", + "negative_test_query331_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "negative_test_query307_00005", + "apositive_test_query185_00012", + "apositive_test_query28_00013", + "negative_test_query48_00016", + "negative_test_query48_00009", + "apositive_test_query48_00000", + "negative_test_query48_00012", + "negative_test_query48_00011", + "apositive_test_query185_00004", + "negative_test_query103_00012", + "negative_test_query48_00018", + "apositive_test_query28_00010", + "negative_test_query307_00003", + "negative_test_query48_00004", + "negative_test_query48_00007", + "negative_test_query307_00001", + "negative_test_query49_00001", + "negative_test_query48_00000", + "apositive_test_query307_00002", + "apositive_test_query343_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query49", + "ranked_ids": [ + "apositive_test_query49_00001", + "negative_test_query342_00005", + "apositive_test_query29_00006", + "negative_test_query176_00016", + "negative_test_query169_00000", + "negative_test_query49_00001", + "negative_test_query49_00002", + "negative_test_query328_00009", + "apositive_test_query19_00001", + "negative_test_query15_00008", + "negative_test_query49_00007", + "apositive_test_query54_00009", + "negative_test_query52_00011", + "negative_test_query156_00007", + "negative_test_query72_00017", + "negative_test_query156_00010", + "negative_test_query106_00004", + "apositive_test_query204_00003", + "apositive_test_query353_00004", + "negative_test_query122_00016" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query5", + "ranked_ids": [ + "apositive_test_query5_00000", + "negative_test_query5_00005", + "apositive_test_query5_00001", + "negative_test_query246_00013", + "negative_test_query67_00010", + "negative_test_query246_00002", + "negative_test_query5_00006", + "negative_test_query275_00001", + "negative_test_query5_00004", + "negative_test_query5_00003", + "negative_test_query5_00007", + "apositive_test_query5_00002", + "negative_test_query5_00000", + "negative_test_query5_00001", + "negative_test_query5_00013", + "negative_test_query67_00003", + "negative_test_query5_00010", + "negative_test_query5_00014", + "negative_test_query301_00003", + "negative_test_query246_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00003", + "apositive_test_query50_00014", + "apositive_test_query50_00009", + "negative_test_query336_00011", + "apositive_test_query50_00013", + "apositive_test_query50_00008", + "apositive_test_query186_00005", + "apositive_test_query50_00006", + "apositive_test_query50_00015", + "apositive_test_query50_00001", + "apositive_test_query50_00002", + "negative_test_query50_00001", + "apositive_test_query131_00000", + "apositive_test_query50_00005", + "apositive_test_query50_00012", + "negative_test_query336_00004", + "negative_test_query66_00002", + "negative_test_query131_00001", + "negative_test_query119_00002", + "negative_test_query223_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query51", + "ranked_ids": [ + "negative_test_query51_00006", + "apositive_test_query51_00000", + "apositive_test_query51_00006", + "negative_test_query97_00011", + "negative_test_query97_00002", + "negative_test_query51_00000", + "negative_test_query97_00001", + "apositive_test_query51_00001", + "apositive_test_query51_00007", + "apositive_test_query51_00008", + "negative_test_query97_00007", + "negative_test_query97_00006", + "negative_test_query267_00007", + "apositive_test_query51_00009", + "negative_test_query51_00004", + "apositive_test_query51_00005", + "negative_test_query204_00001", + "negative_test_query209_00008", + "negative_test_query231_00012", + "apositive_test_query267_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query257_00005", + "apositive_test_query286_00014", + "negative_test_query52_00007", + "negative_test_query257_00009", + "negative_test_query54_00004", + "apositive_test_query286_00013", + "apositive_test_query286_00006", + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00001", + "apositive_test_query257_00003", + "apositive_test_query257_00000", + "negative_test_query52_00005", + "apositive_test_query257_00002", + "apositive_test_query257_00004", + "apositive_test_query257_00005", + "apositive_test_query69_00007", + "negative_test_query257_00007", + "negative_test_query52_00004", + "apositive_test_query52_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00016", + "negative_test_query354_00001", + "negative_test_query161_00007", + "negative_test_query150_00015", + "negative_test_query354_00011", + "negative_test_query258_00013", + "negative_test_query258_00000", + "negative_test_query3_00008", + "negative_test_query53_00000", + "negative_test_query169_00003", + "negative_test_query258_00018", + "apositive_test_query217_00004", + "apositive_test_query28_00006", + "negative_test_query348_00001", + "negative_test_query150_00014", + "negative_test_query230_00009", + "apositive_test_query248_00005", + "negative_test_query132_00015", + "negative_test_query180_00012", + "negative_test_query258_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query140_00006", + "negative_test_query54_00006", + "negative_test_query44_00014", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query56_00011", + "apositive_test_query86_00000", + "negative_test_query44_00007", + "negative_test_query51_00006", + "negative_test_query200_00006", + "negative_test_query301_00006", + "negative_test_query97_00002", + "apositive_test_query323_00006", + "apositive_test_query51_00007", + "negative_test_query136_00004", + "apositive_test_query156_00000", + "apositive_test_query204_00002", + "apositive_test_query335_00002", + "apositive_test_query51_00006", + "apositive_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query55", + "ranked_ids": [ + "apositive_test_query55_00000", + "negative_test_query55_00002", + "negative_test_query55_00005", + "apositive_test_query55_00001", + "negative_test_query55_00016", + "negative_test_query55_00013", + "negative_test_query55_00007", + "negative_test_query55_00008", + "apositive_test_query55_00002", + "negative_test_query139_00015", + "negative_test_query55_00010", + "negative_test_query55_00012", + "negative_test_query61_00008", + "negative_test_query55_00015", + "negative_test_query55_00011", + "negative_test_query55_00000", + "negative_test_query234_00011", + "negative_test_query55_00006", + "negative_test_query55_00009", + "negative_test_query55_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query56", + "ranked_ids": [ + "negative_test_query105_00006", + "negative_test_query359_00005", + "apositive_test_query56_00005", + "negative_test_query119_00005", + "negative_test_query56_00005", + "negative_test_query111_00006", + "apositive_test_query56_00006", + "negative_test_query46_00011", + "negative_test_query56_00002", + "negative_test_query46_00014", + "apositive_test_query238_00001", + "negative_test_query186_00007", + "negative_test_query56_00000", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query56_00004", + "negative_test_query56_00003", + "negative_test_query56_00006", + "apositive_test_query46_00001", + "negative_test_query102_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query57", + "ranked_ids": [ + "negative_test_query295_00003", + "apositive_test_query57_00005", + "apositive_test_query57_00003", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query46_00006", + "apositive_test_query57_00006", + "apositive_test_query57_00008", + "apositive_test_query57_00000", + "negative_test_query57_00001", + "negative_test_query198_00016", + "negative_test_query65_00010", + "apositive_test_query259_00006", + "apositive_test_query338_00002", + "negative_test_query334_00001", + "negative_test_query57_00010", + "apositive_test_query146_00000", + "apositive_test_query203_00000", + "negative_test_query111_00010", + "apositive_test_query74_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "negative_test_query58_00010", + "negative_test_query58_00015", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query308_00004", + "negative_test_query346_00000", + "negative_test_query240_00002", + "negative_test_query240_00016", + "negative_test_query240_00008", + "negative_test_query27_00017", + "negative_test_query27_00016", + "negative_test_query311_00012", + "negative_test_query11_00004", + "negative_test_query240_00004", + "negative_test_query57_00004", + "apositive_test_query311_00003", + "apositive_test_query311_00004", + "negative_test_query240_00005", + "negative_test_query240_00015" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "apositive_test_query59_00000", + "negative_test_query59_00012", + "negative_test_query59_00010", + "negative_test_query116_00000", + "negative_test_query59_00006", + "negative_test_query59_00011", + "negative_test_query59_00014", + "negative_test_query59_00009", + "negative_test_query227_00008", + "negative_test_query59_00018", + "negative_test_query59_00017", + "apositive_test_query104_00000", + "negative_test_query116_00013", + "negative_test_query59_00015", + "negative_test_query66_00008", + "negative_test_query116_00003", + "negative_test_query81_00003", + "negative_test_query59_00005", + "negative_test_query280_00003", + "negative_test_query202_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9674679834891693, + "recall@5": 1.0 + }, + "query_id": "test_query6", + "ranked_ids": [ + "apositive_test_query6_00000", + "apositive_test_query6_00001", + "negative_test_query6_00002", + "apositive_test_query6_00002", + "negative_test_query6_00012", + "negative_test_query179_00007", + "apositive_test_query109_00001", + "negative_test_query6_00004", + "apositive_test_query242_00004", + "negative_test_query296_00013", + "apositive_test_query189_00000", + "negative_test_query252_00004", + "negative_test_query6_00001", + "apositive_test_query305_00003", + "negative_test_query112_00010", + "negative_test_query58_00007", + "negative_test_query6_00016", + "negative_test_query6_00011", + "negative_test_query112_00004", + "negative_test_query296_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.25 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00007", + "apositive_test_query60_00003", + "negative_test_query243_00004", + "apositive_test_query60_00013", + "apositive_test_query60_00001", + "apositive_test_query60_00000", + "apositive_test_query218_00004", + "apositive_test_query60_00009", + "apositive_test_query285_00001", + "apositive_test_query285_00007", + "negative_test_query304_00002", + "negative_test_query304_00003", + "negative_test_query60_00001", + "negative_test_query60_00003", + "apositive_test_query128_00001", + "apositive_test_query60_00005", + "apositive_test_query60_00006", + "apositive_test_query60_00011", + "negative_test_query218_00007", + "negative_test_query31_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5654495432396527, + "recall@5": 0.75 + }, + "query_id": "test_query61", + "ranked_ids": [ + "negative_test_query61_00014", + "apositive_test_query61_00003", + "negative_test_query173_00003", + "apositive_test_query61_00002", + "apositive_test_query61_00001", + "negative_test_query129_00005", + "negative_test_query61_00006", + "negative_test_query61_00003", + "negative_test_query61_00005", + "negative_test_query201_00004", + "negative_test_query61_00015", + "negative_test_query61_00000", + "negative_test_query211_00005", + "negative_test_query210_00012", + "apositive_test_query101_00002", + "negative_test_query61_00010", + "apositive_test_query173_00004", + "negative_test_query61_00004", + "negative_test_query327_00016", + "negative_test_query61_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00004", + "negative_test_query62_00002", + "negative_test_query62_00010", + "negative_test_query62_00003", + "negative_test_query62_00000", + "negative_test_query62_00001", + "negative_test_query62_00013", + "negative_test_query62_00008", + "negative_test_query62_00011", + "negative_test_query352_00005", + "negative_test_query62_00006", + "negative_test_query16_00001", + "negative_test_query15_00011", + "negative_test_query287_00007", + "apositive_test_query62_00001", + "negative_test_query219_00006", + "negative_test_query88_00015", + "negative_test_query38_00009", + "negative_test_query222_00007", + "negative_test_query250_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00001", + "negative_test_query224_00007", + "negative_test_query63_00003", + "negative_test_query63_00005", + "negative_test_query63_00001", + "negative_test_query297_00006", + "negative_test_query63_00002", + "negative_test_query321_00002", + "apositive_test_query63_00004", + "negative_test_query63_00010", + "negative_test_query78_00006", + "negative_test_query264_00015", + "negative_test_query331_00010", + "negative_test_query63_00004", + "apositive_test_query288_00004", + "negative_test_query241_00011", + "negative_test_query63_00009", + "apositive_test_query63_00008", + "apositive_test_query63_00006", + "apositive_test_query141_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query205_00009", + "negative_test_query64_00015", + "negative_test_query64_00013", + "negative_test_query149_00017", + "negative_test_query120_00003", + "negative_test_query64_00010", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query120_00000", + "negative_test_query63_00009", + "apositive_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query310_00005", + "negative_test_query64_00016", + "negative_test_query149_00001", + "negative_test_query211_00009", + "negative_test_query234_00010", + "negative_test_query64_00009", + "negative_test_query63_00000", + "negative_test_query214_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "negative_test_query65_00004", + "apositive_test_query65_00001", + "apositive_test_query65_00000", + "apositive_test_query134_00001", + "negative_test_query30_00011", + "apositive_test_query260_00000", + "negative_test_query260_00013", + "negative_test_query6_00008", + "apositive_test_query62_00000", + "negative_test_query230_00001", + "apositive_test_query356_00004", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query47_00005", + "negative_test_query23_00006", + "negative_test_query261_00003", + "negative_test_query6_00003", + "negative_test_query268_00012", + "negative_test_query65_00000", + "negative_test_query260_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.5 + }, + "query_id": "test_query66", + "ranked_ids": [ + "negative_test_query66_00011", + "apositive_test_query66_00004", + "apositive_test_query66_00005", + "apositive_test_query20_00005", + "apositive_test_query66_00003", + "apositive_test_query66_00002", + "negative_test_query220_00001", + "apositive_test_query20_00001", + "apositive_test_query220_00003", + "negative_test_query161_00008", + "negative_test_query220_00010", + "negative_test_query154_00009", + "negative_test_query161_00003", + "negative_test_query9_00012", + "apositive_test_query188_00000", + "negative_test_query188_00014", + "negative_test_query344_00007", + "apositive_test_query220_00000", + "apositive_test_query188_00003", + "negative_test_query188_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query67", + "ranked_ids": [ + "negative_test_query67_00005", + "negative_test_query67_00001", + "negative_test_query67_00008", + "negative_test_query67_00010", + "negative_test_query67_00007", + "negative_test_query246_00016", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "apositive_test_query286_00010", + "negative_test_query67_00015", + "negative_test_query246_00009", + "negative_test_query123_00008", + "negative_test_query260_00013", + "negative_test_query6_00008", + "negative_test_query67_00011", + "negative_test_query65_00008", + "apositive_test_query286_00004", + "apositive_test_query286_00007", + "negative_test_query55_00005", + "apositive_test_query67_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query68", + "ranked_ids": [ + "negative_test_query68_00001", + "negative_test_query68_00013", + "negative_test_query126_00008", + "negative_test_query151_00008", + "apositive_test_query68_00002", + "negative_test_query330_00009", + "negative_test_query68_00014", + "apositive_test_query252_00001", + "negative_test_query68_00016", + "negative_test_query68_00007", + "apositive_test_query68_00000", + "apositive_test_query68_00001", + "apositive_test_query330_00001", + "negative_test_query212_00000", + "negative_test_query311_00005", + "apositive_test_query5_00002", + "apositive_test_query326_00005", + "apositive_test_query89_00003", + "negative_test_query288_00004", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00002", + "apositive_test_query69_00011", + "apositive_test_query257_00003", + "apositive_test_query69_00000", + "apositive_test_query69_00013", + "negative_test_query169_00002", + "apositive_test_query355_00001", + "apositive_test_query69_00009", + "apositive_test_query69_00008", + "negative_test_query18_00010", + "apositive_test_query69_00010", + "apositive_test_query69_00005", + "apositive_test_query174_00016", + "apositive_test_query281_00016", + "apositive_test_query69_00012", + "apositive_test_query107_00007", + "negative_test_query277_00001", + "apositive_test_query52_00001", + "apositive_test_query54_00003", + "apositive_test_query225_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "negative_test_query7_00005", + "apositive_test_query212_00002", + "negative_test_query159_00009", + "apositive_test_query119_00006", + "apositive_test_query212_00001", + "apositive_test_query326_00007", + "negative_test_query84_00007", + "negative_test_query7_00001", + "apositive_test_query159_00001", + "negative_test_query264_00013", + "negative_test_query38_00010", + "negative_test_query79_00002", + "apositive_test_query38_00003", + "negative_test_query327_00011", + "negative_test_query184_00006", + "negative_test_query290_00006", + "negative_test_query231_00011", + "negative_test_query159_00004", + "negative_test_query306_00008", + "negative_test_query159_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00004", + "negative_test_query70_00011", + "negative_test_query227_00006", + "negative_test_query70_00006", + "apositive_test_query70_00001", + "negative_test_query70_00003", + "negative_test_query227_00013", + "apositive_test_query70_00002", + "negative_test_query227_00000", + "negative_test_query348_00007", + "negative_test_query227_00003", + "negative_test_query227_00010", + "negative_test_query227_00012", + "negative_test_query227_00014", + "negative_test_query70_00000", + "apositive_test_query170_00001", + "apositive_test_query227_00000", + "negative_test_query283_00014", + "negative_test_query59_00004", + "negative_test_query70_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query71", + "ranked_ids": [ + "negative_test_query71_00001", + "negative_test_query259_00002", + "negative_test_query71_00013", + "negative_test_query65_00002", + "apositive_test_query71_00003", + "negative_test_query71_00009", + "apositive_test_query71_00001", + "negative_test_query71_00000", + "negative_test_query71_00005", + "negative_test_query71_00011", + "apositive_test_query30_00005", + "apositive_test_query71_00002", + "apositive_test_query209_00005", + "negative_test_query71_00002", + "apositive_test_query71_00000", + "negative_test_query231_00014", + "negative_test_query186_00000", + "negative_test_query247_00002", + "negative_test_query313_00001", + "negative_test_query71_00015" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "negative_test_query72_00001", + "negative_test_query72_00016", + "apositive_test_query72_00000", + "negative_test_query72_00007", + "negative_test_query72_00008", + "negative_test_query72_00015", + "negative_test_query72_00003", + "negative_test_query72_00013", + "negative_test_query17_00009", + "negative_test_query55_00005", + "negative_test_query92_00001", + "apositive_test_query175_00001", + "negative_test_query228_00000", + "negative_test_query285_00009", + "negative_test_query265_00003", + "negative_test_query233_00009", + "negative_test_query72_00006", + "negative_test_query185_00000", + "negative_test_query72_00011", + "negative_test_query185_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query73", + "ranked_ids": [ + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00004", + "negative_test_query73_00001", + "negative_test_query73_00006", + "apositive_test_query322_00003", + "negative_test_query322_00009", + "negative_test_query73_00008", + "apositive_test_query333_00000", + "negative_test_query73_00013", + "apositive_test_query322_00008", + "negative_test_query233_00015", + "negative_test_query228_00013", + "negative_test_query216_00013", + "apositive_test_query333_00005", + "apositive_test_query73_00000", + "negative_test_query73_00010", + "apositive_test_query333_00002", + "negative_test_query333_00005", + "negative_test_query246_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.787702056960637, + "recall@5": 0.75 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query74_00001", + "apositive_test_query74_00000", + "negative_test_query74_00014", + "negative_test_query74_00001", + "apositive_test_query74_00003", + "negative_test_query44_00006", + "apositive_test_query74_00002", + "negative_test_query95_00009", + "apositive_test_query114_00001", + "negative_test_query85_00005", + "apositive_test_query113_00001", + "negative_test_query74_00013", + "negative_test_query74_00000", + "negative_test_query74_00015", + "apositive_test_query60_00006", + "negative_test_query129_00008", + "apositive_test_query238_00003", + "negative_test_query238_00013", + "negative_test_query74_00003", + "negative_test_query203_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query75", + "ranked_ids": [ + "apositive_test_query75_00005", + "apositive_test_query75_00001", + "apositive_test_query75_00000", + "negative_test_query130_00012", + "negative_test_query89_00000", + "negative_test_query75_00006", + "apositive_test_query75_00003", + "negative_test_query75_00003", + "negative_test_query75_00004", + "negative_test_query319_00004", + "negative_test_query82_00004", + "negative_test_query75_00012", + "apositive_test_query75_00006", + "negative_test_query112_00018", + "negative_test_query323_00011", + "negative_test_query99_00006", + "negative_test_query67_00017", + "negative_test_query209_00004", + "negative_test_query260_00013", + "negative_test_query6_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query76", + "ranked_ids": [ + "apositive_test_query76_00004", + "negative_test_query76_00007", + "negative_test_query91_00004", + "apositive_test_query76_00002", + "negative_test_query147_00007", + "apositive_test_query76_00001", + "negative_test_query181_00008", + "negative_test_query76_00010", + "apositive_test_query76_00006", + "negative_test_query76_00009", + "negative_test_query76_00002", + "negative_test_query53_00001", + "negative_test_query76_00008", + "negative_test_query279_00009", + "negative_test_query76_00011", + "negative_test_query309_00011", + "negative_test_query258_00008", + "negative_test_query53_00009", + "negative_test_query147_00005", + "apositive_test_query76_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query216_00006", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query148_00013", + "negative_test_query329_00015", + "negative_test_query342_00007", + "negative_test_query77_00013", + "negative_test_query144_00003", + "negative_test_query14_00010", + "negative_test_query150_00011", + "negative_test_query215_00004", + "negative_test_query216_00012", + "negative_test_query263_00002", + "negative_test_query287_00012", + "negative_test_query301_00003", + "negative_test_query315_00003", + "negative_test_query326_00009", + "negative_test_query47_00010", + "negative_test_query87_00016", + "negative_test_query98_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query78", + "ranked_ids": [ + "negative_test_query214_00010", + "negative_test_query290_00006", + "negative_test_query290_00013", + "apositive_test_query38_00003", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query72_00005", + "apositive_test_query78_00001", + "apositive_test_query326_00000", + "apositive_test_query78_00000", + "negative_test_query78_00016", + "negative_test_query200_00009", + "negative_test_query236_00018", + "negative_test_query329_00002", + "negative_test_query144_00000", + "negative_test_query272_00001", + "negative_test_query274_00014", + "negative_test_query192_00009", + "negative_test_query7_00005", + "negative_test_query78_00007" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "negative_test_query79_00002", + "negative_test_query158_00000", + "apositive_test_query79_00004", + "negative_test_query79_00013", + "apositive_test_query221_00004", + "apositive_test_query119_00005", + "apositive_test_query79_00000", + "apositive_test_query306_00010", + "negative_test_query56_00002", + "negative_test_query303_00016", + "apositive_test_query295_00001", + "negative_test_query79_00004", + "negative_test_query38_00007", + "negative_test_query53_00015", + "apositive_test_query113_00002", + "apositive_test_query157_00008", + "negative_test_query157_00000", + "negative_test_query313_00013", + "negative_test_query79_00009", + "negative_test_query105_00011" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query8_00004", + "negative_test_query8_00014", + "negative_test_query8_00001", + "negative_test_query8_00011", + "negative_test_query205_00007", + "apositive_test_query8_00000", + "negative_test_query192_00006", + "negative_test_query8_00015", + "apositive_test_query192_00003", + "negative_test_query192_00003", + "negative_test_query8_00012", + "apositive_test_query243_00005", + "negative_test_query8_00008", + "apositive_test_query192_00002", + "apositive_test_query243_00010", + "negative_test_query329_00005", + "negative_test_query8_00000", + "negative_test_query96_00010" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query179_00005", + "negative_test_query80_00004", + "apositive_test_query80_00002", + "negative_test_query80_00001", + "apositive_test_query179_00001", + "negative_test_query80_00003", + "apositive_test_query189_00004", + "negative_test_query102_00009", + "negative_test_query36_00010", + "negative_test_query80_00012", + "apositive_test_query179_00003", + "negative_test_query80_00009", + "apositive_test_query136_00013", + "negative_test_query273_00005", + "negative_test_query287_00005", + "negative_test_query289_00005", + "negative_test_query85_00007", + "negative_test_query179_00006", + "negative_test_query270_00003", + "apositive_test_query36_00000" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query81", + "ranked_ids": [ + "apositive_test_query81_00000", + "negative_test_query81_00005", + "apositive_test_query81_00001", + "negative_test_query81_00006", + "negative_test_query81_00000", + "apositive_test_query81_00004", + "apositive_test_query81_00005", + "apositive_test_query81_00006", + "apositive_test_query81_00008", + "negative_test_query81_00008", + "negative_test_query81_00004", + "negative_test_query81_00007", + "negative_test_query81_00009", + "negative_test_query81_00001", + "apositive_test_query81_00003", + "negative_test_query81_00010", + "negative_test_query207_00008", + "negative_test_query27_00006", + "negative_test_query246_00017", + "negative_test_query145_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00003", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query340_00013", + "negative_test_query319_00009", + "negative_test_query82_00009", + "negative_test_query319_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query83_00002", + "apositive_test_query229_00001", + "apositive_test_query83_00004", + "apositive_test_query95_00000", + "negative_test_query348_00006", + "negative_test_query168_00007", + "apositive_test_query168_00001", + "apositive_test_query168_00000", + "apositive_test_query146_00014", + "apositive_test_query164_00000", + "negative_test_query28_00005", + "negative_test_query48_00006", + "apositive_test_query83_00000", + "negative_test_query309_00016", + "apositive_test_query83_00003", + "apositive_test_query217_00006", + "negative_test_query279_00012", + "negative_test_query0_00006", + "negative_test_query357_00002", + "negative_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query84", + "ranked_ids": [ + "negative_test_query84_00017", + "apositive_test_query84_00001", + "apositive_test_query84_00000", + "negative_test_query84_00000", + "negative_test_query84_00007", + "negative_test_query84_00006", + "negative_test_query84_00012", + "negative_test_query84_00002", + "negative_test_query84_00005", + "negative_test_query84_00001", + "negative_test_query84_00016", + "negative_test_query84_00009", + "negative_test_query43_00002", + "negative_test_query84_00013", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query84_00008", + "negative_test_query268_00009", + "apositive_test_query249_00013", + "apositive_test_query155_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query85_00000", + "negative_test_query85_00000", + "negative_test_query85_00012", + "negative_test_query85_00003", + "apositive_test_query85_00002", + "apositive_test_query109_00004", + "apositive_test_query85_00001", + "apositive_test_query85_00003", + "negative_test_query85_00009", + "negative_test_query85_00011", + "apositive_test_query284_00004", + "negative_test_query106_00003", + "negative_test_query85_00001", + "negative_test_query112_00008", + "apositive_test_query286_00006", + "apositive_test_query81_00008", + "negative_test_query44_00008", + "apositive_test_query133_00013", + "apositive_test_query286_00004", + "apositive_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.44149241373678083, + "recall@5": 0.5 + }, + "query_id": "test_query86", + "ranked_ids": [ + "negative_test_query310_00007", + "apositive_test_query86_00001", + "apositive_test_query86_00003", + "negative_test_query86_00001", + "negative_test_query86_00013", + "negative_test_query86_00012", + "negative_test_query299_00007", + "negative_test_query65_00013", + "negative_test_query86_00014", + "negative_test_query86_00008", + "apositive_test_query86_00002", + "apositive_test_query86_00000", + "negative_test_query86_00000", + "negative_test_query86_00007", + "negative_test_query86_00002", + "negative_test_query210_00012", + "negative_test_query245_00012", + "negative_test_query86_00011", + "negative_test_query188_00013", + "negative_test_query86_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query87", + "ranked_ids": [ + "negative_test_query87_00013", + "apositive_test_query87_00000", + "apositive_test_query87_00001", + "negative_test_query87_00004", + "negative_test_query87_00000", + "negative_test_query87_00007", + "negative_test_query87_00011", + "negative_test_query87_00008", + "negative_test_query87_00014", + "negative_test_query87_00015", + "negative_test_query87_00010", + "negative_test_query87_00006", + "negative_test_query87_00016", + "negative_test_query275_00006", + "negative_test_query315_00006", + "negative_test_query87_00002", + "negative_test_query200_00011", + "negative_test_query182_00017", + "negative_test_query263_00002", + "apositive_test_query302_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "negative_test_query88_00005", + "apositive_test_query88_00001", + "negative_test_query88_00013", + "negative_test_query325_00006", + "apositive_test_query250_00002", + "apositive_test_query147_00002", + "negative_test_query88_00003", + "negative_test_query88_00016", + "negative_test_query88_00017", + "negative_test_query105_00014", + "negative_test_query340_00012", + "negative_test_query88_00002", + "negative_test_query184_00013", + "negative_test_query88_00008", + "negative_test_query88_00010", + "apositive_test_query223_00000", + "negative_test_query240_00009", + "negative_test_query334_00006", + "negative_test_query203_00000", + "negative_test_query88_00012" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query110_00011", + "negative_test_query326_00006", + "negative_test_query89_00004", + "apositive_test_query89_00004", + "negative_test_query89_00008", + "negative_test_query89_00000", + "negative_test_query68_00004", + "negative_test_query212_00000", + "negative_test_query89_00005", + "apositive_test_query255_00000", + "negative_test_query156_00013", + "negative_test_query130_00003", + "negative_test_query89_00007", + "negative_test_query278_00009", + "negative_test_query278_00000", + "apositive_test_query75_00006", + "negative_test_query325_00012", + "apositive_test_query68_00001", + "apositive_test_query89_00000", + "negative_test_query233_00005" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00008", + "negative_test_query9_00006", + "negative_test_query90_00004", + "negative_test_query9_00018", + "negative_test_query9_00004", + "negative_test_query9_00016", + "negative_test_query90_00009", + "apositive_test_query9_00000", + "negative_test_query221_00002", + "negative_test_query9_00005", + "negative_test_query9_00001", + "negative_test_query9_00011", + "negative_test_query9_00015", + "negative_test_query9_00013", + "apositive_test_query90_00001", + "negative_test_query221_00011", + "negative_test_query90_00007", + "negative_test_query305_00009", + "negative_test_query221_00007", + "negative_test_query9_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query310_00007", + "apositive_test_query90_00000", + "negative_test_query231_00002", + "negative_test_query90_00011", + "negative_test_query90_00010", + "negative_test_query90_00002", + "negative_test_query90_00017", + "negative_test_query227_00012", + "negative_test_query127_00001", + "negative_test_query90_00012", + "negative_test_query227_00001", + "apositive_test_query304_00000", + "negative_test_query90_00015", + "negative_test_query90_00000", + "negative_test_query90_00006", + "negative_test_query231_00014", + "negative_test_query261_00002", + "negative_test_query227_00006", + "negative_test_query9_00008", + "apositive_test_query14_00002" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query91", + "ranked_ids": [ + "apositive_test_query91_00000", + "apositive_test_query91_00002", + "negative_test_query279_00011", + "negative_test_query292_00009", + "negative_test_query358_00004", + "apositive_test_query342_00001", + "negative_test_query315_00013", + "negative_test_query319_00002", + "negative_test_query350_00004", + "negative_test_query82_00002", + "apositive_test_query11_00006", + "negative_test_query130_00010", + "negative_test_query91_00010", + "apositive_test_query157_00001", + "apositive_test_query91_00006", + "negative_test_query102_00002", + "apositive_test_query157_00000", + "negative_test_query118_00015", + "negative_test_query303_00011", + "negative_test_query59_00006" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.5 + }, + "query_id": "test_query92", + "ranked_ids": [ + "negative_test_query92_00003", + "apositive_test_query92_00005", + "apositive_test_query92_00003", + "negative_test_query283_00001", + "apositive_test_query92_00000", + "apositive_test_query92_00001", + "negative_test_query105_00016", + "apositive_test_query92_00002", + "apositive_test_query92_00004", + "apositive_test_query198_00001", + "negative_test_query1_00012", + "negative_test_query283_00009", + "negative_test_query92_00013", + "apositive_test_query100_00001", + "apositive_test_query99_00003", + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query122_00018", + "negative_test_query100_00012", + "negative_test_query100_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query93_00000", + "apositive_test_query93_00004", + "apositive_test_query243_00006", + "negative_test_query219_00000", + "negative_test_query195_00006", + "negative_test_query335_00003", + "negative_test_query280_00012", + "negative_test_query93_00003", + "apositive_test_query93_00003", + "negative_test_query92_00012", + "negative_test_query108_00003", + "negative_test_query195_00001", + "negative_test_query277_00003", + "negative_test_query321_00001", + "negative_test_query335_00011", + "negative_test_query5_00015", + "apositive_test_query323_00003", + "apositive_test_query335_00003", + "apositive_test_query160_00007", + "negative_test_query129_00003" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.375 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00001", + "apositive_test_query94_00001", + "apositive_test_query94_00005", + "apositive_test_query94_00002", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "negative_test_query94_00009", + "negative_test_query94_00002", + "apositive_test_query94_00004", + "apositive_test_query127_00010", + "apositive_test_query127_00006", + "negative_test_query94_00003", + "apositive_test_query94_00006", + "apositive_test_query127_00004", + "apositive_test_query94_00007", + "negative_test_query94_00005", + "apositive_test_query127_00008", + "apositive_test_query94_00000", + "negative_test_query155_00008", + "apositive_test_query127_00015" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "apositive_test_query95_00001", + "apositive_test_query95_00000", + "apositive_test_query253_00007", + "negative_test_query95_00005", + "negative_test_query95_00015", + "negative_test_query95_00012", + "negative_test_query95_00010", + "negative_test_query95_00002", + "negative_test_query95_00008", + "negative_test_query12_00015", + "negative_test_query95_00003", + "negative_test_query95_00014", + "negative_test_query95_00013", + "negative_test_query12_00017", + "negative_test_query244_00010", + "negative_test_query334_00012", + "apositive_test_query177_00000", + "negative_test_query57_00007", + "negative_test_query95_00006", + "apositive_test_query292_00001" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query339_00000", + "apositive_test_query96_00001", + "apositive_test_query96_00002", + "negative_test_query192_00011", + "negative_test_query339_00001", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query339_00016", + "negative_test_query96_00004", + "negative_test_query96_00016", + "negative_test_query339_00008", + "negative_test_query96_00003", + "apositive_test_query339_00001", + "negative_test_query339_00009", + "negative_test_query339_00017", + "negative_test_query339_00004", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query339_00006", + "negative_test_query96_00013" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.5 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "apositive_test_query97_00003", + "apositive_test_query97_00001", + "negative_test_query97_00009", + "apositive_test_query97_00002", + "negative_test_query113_00006", + "negative_test_query97_00006", + "apositive_test_query51_00000", + "negative_test_query84_00008", + "negative_test_query97_00011", + "apositive_test_query97_00005", + "negative_test_query209_00008", + "negative_test_query113_00009", + "negative_test_query16_00014", + "negative_test_query97_00012", + "apositive_test_query51_00002", + "apositive_test_query289_00003", + "apositive_test_query267_00000", + "apositive_test_query201_00006", + "negative_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00007", + "negative_test_query98_00016", + "apositive_test_query342_00000", + "negative_test_query98_00004", + "negative_test_query231_00014", + "negative_test_query98_00018", + "negative_test_query98_00017", + "apositive_test_query14_00003", + "negative_test_query98_00011", + "apositive_test_query14_00002", + "negative_test_query197_00007", + "negative_test_query98_00005", + "negative_test_query129_00003", + "negative_test_query212_00006", + "negative_test_query23_00013", + "apositive_test_query201_00004", + "negative_test_query39_00005", + "negative_test_query345_00002", + "negative_test_query300_00000", + "negative_test_query316_00014" + ], + "valid": true + }, + { + "algorithm": "original_bm25", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query99", + "ranked_ids": [ + "negative_test_query296_00013", + "apositive_test_query193_00008", + "negative_test_query193_00002", + "negative_test_query174_00002", + "negative_test_query193_00001", + "negative_test_query129_00001", + "apositive_test_query193_00012", + "negative_test_query193_00005", + "negative_test_query27_00018", + "negative_test_query330_00008", + "apositive_test_query253_00003", + "apositive_test_query193_00010", + "apositive_test_query333_00003", + "apositive_test_query99_00001", + "apositive_test_query153_00012", + "apositive_test_query347_00002", + "apositive_test_query99_00007", + "negative_test_query256_00002", + "negative_test_query129_00003", + "negative_test_query27_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.4166666666666667 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00006", + "apositive_test_query0_00004", + "apositive_test_query0_00003", + "negative_test_query359_00002", + "apositive_test_query0_00002", + "apositive_test_query107_00004", + "negative_test_query0_00000", + "negative_test_query36_00011", + "negative_test_query9_00017", + "negative_test_query221_00006", + "apositive_test_query130_00000", + "apositive_test_query153_00004", + "negative_test_query0_00007", + "negative_test_query259_00006", + "negative_test_query130_00002", + "negative_test_query201_00000", + "negative_test_query357_00000", + "negative_test_query0_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query1", + "ranked_ids": [ + "negative_test_query61_00013", + "apositive_test_query1_00000", + "negative_test_query1_00001", + "negative_test_query61_00002", + "negative_test_query157_00007", + "negative_test_query1_00009", + "apositive_test_query348_00003", + "negative_test_query1_00005", + "negative_test_query120_00006", + "negative_test_query61_00003", + "negative_test_query297_00011", + "negative_test_query256_00005", + "negative_test_query27_00017", + "negative_test_query245_00006", + "negative_test_query110_00007", + "negative_test_query61_00014", + "negative_test_query184_00015", + "negative_test_query331_00002", + "apositive_test_query188_00002", + "negative_test_query202_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query10", + "ranked_ids": [ + "apositive_test_query10_00001", + "apositive_test_query10_00000", + "apositive_test_query10_00002", + "negative_test_query10_00011", + "negative_test_query10_00005", + "negative_test_query10_00000", + "negative_test_query10_00010", + "negative_test_query169_00008", + "negative_test_query10_00004", + "negative_test_query10_00007", + "apositive_test_query286_00008", + "apositive_test_query69_00000", + "negative_test_query72_00002", + "negative_test_query266_00009", + "apositive_test_query69_00016", + "apositive_test_query44_00001", + "negative_test_query135_00014", + "negative_test_query104_00011", + "negative_test_query83_00000", + "negative_test_query10_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query100", + "ranked_ids": [ + "negative_test_query100_00001", + "apositive_test_query100_00000", + "apositive_test_query100_00004", + "negative_test_query100_00013", + "negative_test_query100_00014", + "apositive_test_query100_00003", + "apositive_test_query100_00002", + "negative_test_query271_00006", + "negative_test_query290_00015", + "negative_test_query100_00010", + "negative_test_query100_00009", + "negative_test_query252_00005", + "negative_test_query15_00000", + "negative_test_query323_00000", + "apositive_test_query338_00002", + "apositive_test_query338_00005", + "apositive_test_query181_00000", + "negative_test_query258_00017", + "negative_test_query147_00000", + "negative_test_query237_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query101_00002", + "apositive_test_query73_00002", + "apositive_test_query101_00005", + "negative_test_query61_00013", + "negative_test_query241_00005", + "negative_test_query35_00004", + "negative_test_query189_00014", + "negative_test_query61_00014", + "negative_test_query191_00014", + "negative_test_query61_00002", + "negative_test_query173_00013", + "apositive_test_query196_00000", + "negative_test_query103_00009", + "negative_test_query134_00008", + "apositive_test_query291_00000", + "negative_test_query61_00000", + "negative_test_query68_00009", + "negative_test_query258_00000", + "negative_test_query53_00000", + "apositive_test_query336_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query108_00009", + "apositive_test_query238_00000", + "negative_test_query102_00014", + "negative_test_query203_00007", + "negative_test_query183_00012", + "negative_test_query183_00009", + "negative_test_query131_00009", + "negative_test_query313_00016", + "apositive_test_query30_00005", + "negative_test_query209_00006", + "negative_test_query7_00001", + "apositive_test_query20_00001", + "apositive_test_query356_00008", + "apositive_test_query47_00001", + "negative_test_query47_00011", + "negative_test_query219_00004", + "negative_test_query205_00016", + "negative_test_query90_00016", + "negative_test_query47_00010", + "negative_test_query232_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "negative_test_query103_00003", + "negative_test_query103_00011", + "apositive_test_query185_00003", + "negative_test_query185_00005", + "apositive_test_query103_00001", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query307_00008", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query217_00005", + "negative_test_query222_00009", + "apositive_test_query103_00005", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query103_00000", + "negative_test_query48_00014", + "negative_test_query72_00012", + "negative_test_query103_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "apositive_test_query104_00000", + "negative_test_query104_00002", + "negative_test_query104_00018", + "negative_test_query104_00003", + "negative_test_query104_00015", + "negative_test_query104_00013", + "negative_test_query104_00010", + "negative_test_query104_00011", + "negative_test_query104_00001", + "negative_test_query104_00005", + "negative_test_query104_00012", + "negative_test_query104_00014", + "negative_test_query104_00009", + "negative_test_query104_00007", + "negative_test_query48_00010", + "apositive_test_query28_00011", + "apositive_test_query48_00000", + "negative_test_query343_00013", + "negative_test_query104_00016", + "negative_test_query104_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query105", + "ranked_ids": [ + "apositive_test_query105_00000", + "negative_test_query105_00011", + "negative_test_query97_00007", + "negative_test_query79_00004", + "apositive_test_query157_00008", + "negative_test_query105_00006", + "negative_test_query119_00005", + "negative_test_query79_00002", + "negative_test_query313_00013", + "negative_test_query295_00005", + "negative_test_query359_00008", + "negative_test_query105_00005", + "negative_test_query142_00010", + "negative_test_query267_00011", + "apositive_test_query83_00004", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query168_00005", + "negative_test_query56_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query106", + "ranked_ids": [ + "apositive_test_query106_00003", + "negative_test_query36_00001", + "negative_test_query286_00000", + "apositive_test_query286_00005", + "apositive_test_query106_00005", + "apositive_test_query106_00000", + "apositive_test_query177_00008", + "negative_test_query326_00008", + "negative_test_query106_00012", + "negative_test_query106_00000", + "negative_test_query162_00007", + "negative_test_query147_00001", + "negative_test_query212_00002", + "apositive_test_query106_00001", + "negative_test_query106_00006", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query106_00005", + "negative_test_query114_00003", + "apositive_test_query42_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "apositive_test_query357_00003", + "negative_test_query142_00004", + "apositive_test_query221_00001", + "negative_test_query31_00013", + "negative_test_query105_00015", + "negative_test_query142_00000", + "negative_test_query105_00008", + "apositive_test_query211_00001", + "apositive_test_query142_00002", + "negative_test_query124_00005", + "negative_test_query305_00015", + "negative_test_query160_00001", + "negative_test_query46_00012", + "apositive_test_query276_00003", + "negative_test_query146_00000", + "apositive_test_query36_00002", + "negative_test_query296_00001", + "negative_test_query296_00008", + "negative_test_query333_00005", + "apositive_test_query142_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.5 + }, + "query_id": "test_query108", + "ranked_ids": [ + "apositive_test_query108_00007", + "apositive_test_query108_00000", + "negative_test_query311_00006", + "apositive_test_query108_00002", + "apositive_test_query108_00003", + "negative_test_query108_00008", + "apositive_test_query108_00005", + "negative_test_query108_00011", + "negative_test_query168_00001", + "negative_test_query57_00009", + "negative_test_query108_00001", + "negative_test_query102_00012", + "negative_test_query108_00010", + "negative_test_query108_00006", + "negative_test_query102_00007", + "negative_test_query108_00007", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query250_00009", + "negative_test_query250_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query109", + "ranked_ids": [ + "negative_test_query190_00003", + "apositive_test_query109_00003", + "negative_test_query51_00004", + "apositive_test_query109_00001", + "negative_test_query109_00003", + "apositive_test_query109_00000", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query6_00006", + "negative_test_query38_00006", + "negative_test_query109_00011", + "negative_test_query109_00014", + "negative_test_query109_00013", + "negative_test_query109_00000", + "negative_test_query109_00012", + "negative_test_query268_00010", + "negative_test_query109_00004", + "negative_test_query109_00009", + "negative_test_query109_00008", + "apositive_test_query248_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.25 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00000", + "apositive_test_query11_00001", + "negative_test_query11_00002", + "negative_test_query132_00003", + "negative_test_query226_00010", + "negative_test_query309_00002", + "negative_test_query358_00005", + "apositive_test_query11_00004", + "apositive_test_query358_00006", + "negative_test_query11_00004", + "apositive_test_query11_00006", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "negative_test_query309_00001", + "apositive_test_query11_00007", + "negative_test_query309_00004", + "negative_test_query123_00007", + "negative_test_query132_00013", + "apositive_test_query358_00007", + "apositive_test_query11_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "negative_test_query89_00006", + "negative_test_query110_00013", + "negative_test_query343_00001", + "negative_test_query10_00005", + "apositive_test_query180_00001", + "apositive_test_query330_00001", + "negative_test_query110_00002", + "negative_test_query110_00014", + "negative_test_query110_00005", + "negative_test_query17_00008", + "negative_test_query110_00007", + "negative_test_query110_00006", + "apositive_test_query138_00000", + "apositive_test_query344_00002", + "negative_test_query311_00011", + "negative_test_query110_00004", + "apositive_test_query11_00002", + "negative_test_query260_00015", + "apositive_test_query242_00006", + "apositive_test_query291_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query111_00015", + "negative_test_query223_00000", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query134_00006", + "negative_test_query213_00011", + "negative_test_query111_00018", + "negative_test_query65_00013", + "negative_test_query137_00010", + "apositive_test_query162_00005", + "apositive_test_query306_00002", + "negative_test_query58_00013", + "apositive_test_query174_00008", + "negative_test_query111_00008", + "negative_test_query292_00010", + "negative_test_query14_00015", + "apositive_test_query308_00003", + "negative_test_query245_00001", + "negative_test_query65_00007", + "negative_test_query243_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "negative_test_query36_00014", + "negative_test_query112_00010", + "apositive_test_query286_00005", + "apositive_test_query106_00003", + "negative_test_query112_00016", + "negative_test_query216_00002", + "negative_test_query246_00016", + "apositive_test_query36_00003", + "apositive_test_query286_00006", + "apositive_test_query246_00000", + "negative_test_query112_00015", + "negative_test_query112_00004", + "apositive_test_query246_00001", + "negative_test_query36_00010", + "apositive_test_query67_00001", + "negative_test_query67_00003", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query6_00003", + "negative_test_query85_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query113", + "ranked_ids": [ + "negative_test_query216_00011", + "negative_test_query67_00014", + "apositive_test_query113_00000", + "negative_test_query7_00016", + "negative_test_query38_00003", + "apositive_test_query186_00005", + "negative_test_query113_00003", + "negative_test_query201_00008", + "apositive_test_query113_00001", + "apositive_test_query113_00002", + "negative_test_query6_00016", + "negative_test_query105_00000", + "negative_test_query47_00005", + "negative_test_query67_00002", + "apositive_test_query188_00001", + "negative_test_query188_00005", + "negative_test_query223_00003", + "negative_test_query105_00007", + "apositive_test_query67_00000", + "apositive_test_query79_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query114", + "ranked_ids": [ + "apositive_test_query114_00003", + "apositive_test_query114_00000", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "apositive_test_query114_00001", + "negative_test_query106_00009", + "negative_test_query262_00007", + "apositive_test_query106_00003", + "negative_test_query349_00003", + "apositive_test_query124_00004", + "apositive_test_query114_00006", + "negative_test_query114_00003", + "negative_test_query114_00002", + "negative_test_query114_00008", + "negative_test_query81_00009", + "negative_test_query114_00005", + "negative_test_query114_00007", + "negative_test_query114_00004", + "negative_test_query114_00010", + "negative_test_query114_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query115", + "ranked_ids": [ + "apositive_test_query115_00002", + "apositive_test_query115_00001", + "apositive_test_query115_00004", + "negative_test_query115_00003", + "apositive_test_query115_00000", + "negative_test_query115_00007", + "apositive_test_query115_00003", + "negative_test_query115_00006", + "negative_test_query115_00002", + "apositive_test_query348_00000", + "negative_test_query115_00012", + "apositive_test_query348_00009", + "negative_test_query276_00006", + "negative_test_query139_00000", + "negative_test_query205_00004", + "negative_test_query115_00004", + "negative_test_query115_00010", + "apositive_test_query133_00010", + "negative_test_query121_00008", + "apositive_test_query121_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "apositive_test_query116_00000", + "apositive_test_query49_00000", + "negative_test_query116_00015", + "negative_test_query59_00013", + "negative_test_query116_00013", + "negative_test_query116_00008", + "negative_test_query116_00003", + "negative_test_query59_00007", + "negative_test_query116_00002", + "negative_test_query116_00000", + "negative_test_query116_00004", + "negative_test_query261_00013", + "negative_test_query116_00011", + "negative_test_query98_00005", + "negative_test_query116_00006", + "negative_test_query116_00007", + "negative_test_query116_00009", + "negative_test_query116_00010", + "negative_test_query116_00014", + "negative_test_query116_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00004", + "negative_test_query117_00010", + "apositive_test_query117_00003", + "negative_test_query117_00001", + "negative_test_query117_00009", + "apositive_test_query354_00003", + "negative_test_query117_00007", + "apositive_test_query117_00006", + "negative_test_query48_00000", + "negative_test_query117_00004", + "apositive_test_query117_00007", + "apositive_test_query117_00000", + "apositive_test_query134_00004", + "negative_test_query63_00001", + "apositive_test_query354_00000", + "negative_test_query194_00002", + "negative_test_query117_00008", + "negative_test_query175_00013", + "negative_test_query202_00006", + "negative_test_query117_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "negative_test_query23_00005", + "apositive_test_query155_00005", + "negative_test_query340_00007", + "negative_test_query296_00007", + "apositive_test_query340_00002", + "negative_test_query88_00007", + "apositive_test_query171_00001", + "apositive_test_query174_00003", + "apositive_test_query174_00004", + "apositive_test_query174_00005", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "apositive_test_query174_00006", + "apositive_test_query174_00010", + "negative_test_query171_00016", + "negative_test_query171_00005", + "apositive_test_query310_00001", + "negative_test_query296_00010", + "negative_test_query84_00012", + "apositive_test_query217_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query119", + "ranked_ids": [ + "apositive_test_query119_00005", + "negative_test_query119_00005", + "apositive_test_query221_00004", + "negative_test_query158_00001", + "negative_test_query158_00000", + "negative_test_query152_00004", + "apositive_test_query119_00001", + "apositive_test_query119_00012", + "apositive_test_query306_00010", + "negative_test_query157_00001", + "negative_test_query119_00002", + "negative_test_query223_00011", + "apositive_test_query26_00000", + "negative_test_query56_00002", + "apositive_test_query105_00000", + "apositive_test_query113_00002", + "apositive_test_query79_00004", + "negative_test_query359_00005", + "negative_test_query79_00002", + "negative_test_query157_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query12_00006", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query12_00016", + "negative_test_query12_00014", + "negative_test_query12_00003", + "negative_test_query12_00004", + "negative_test_query12_00013", + "negative_test_query12_00017", + "negative_test_query12_00007", + "negative_test_query12_00008", + "apositive_test_query12_00000", + "negative_test_query12_00015", + "negative_test_query12_00011", + "negative_test_query12_00001", + "negative_test_query231_00013", + "negative_test_query234_00014", + "apositive_test_query185_00000", + "apositive_test_query217_00005", + "negative_test_query175_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00008", + "negative_test_query1_00009", + "negative_test_query120_00002", + "negative_test_query121_00002", + "negative_test_query120_00012", + "negative_test_query120_00010", + "apositive_test_query32_00007", + "negative_test_query196_00006", + "apositive_test_query352_00000", + "negative_test_query148_00014", + "negative_test_query120_00006", + "apositive_test_query120_00002", + "negative_test_query352_00013", + "apositive_test_query120_00003", + "negative_test_query120_00001", + "apositive_test_query120_00000", + "apositive_test_query120_00001", + "apositive_test_query117_00007", + "negative_test_query303_00018", + "negative_test_query121_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.8333333333333334 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00002", + "apositive_test_query121_00000", + "apositive_test_query121_00005", + "apositive_test_query121_00003", + "apositive_test_query121_00001", + "apositive_test_query121_00004", + "negative_test_query121_00008", + "negative_test_query308_00009", + "negative_test_query121_00003", + "negative_test_query121_00013", + "apositive_test_query307_00010", + "negative_test_query121_00000", + "negative_test_query121_00012", + "negative_test_query121_00007", + "negative_test_query313_00006", + "negative_test_query44_00012", + "negative_test_query53_00003", + "negative_test_query256_00003", + "negative_test_query115_00009", + "negative_test_query133_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query61_00015", + "negative_test_query17_00012", + "apositive_test_query254_00002", + "negative_test_query195_00008", + "apositive_test_query134_00000", + "negative_test_query86_00001", + "negative_test_query64_00017", + "negative_test_query86_00006", + "negative_test_query173_00003", + "negative_test_query297_00000", + "negative_test_query210_00003", + "apositive_test_query173_00000", + "apositive_test_query122_00000", + "negative_test_query179_00002", + "negative_test_query5_00001", + "negative_test_query179_00014", + "negative_test_query358_00000", + "apositive_test_query175_00004", + "negative_test_query134_00007", + "apositive_test_query173_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query123", + "ranked_ids": [ + "negative_test_query123_00013", + "negative_test_query123_00005", + "apositive_test_query123_00002", + "negative_test_query123_00010", + "negative_test_query161_00009", + "negative_test_query123_00008", + "apositive_test_query123_00003", + "apositive_test_query123_00000", + "negative_test_query123_00004", + "negative_test_query123_00001", + "apositive_test_query123_00004", + "negative_test_query123_00000", + "negative_test_query123_00002", + "negative_test_query155_00001", + "negative_test_query123_00006", + "negative_test_query269_00006", + "negative_test_query155_00009", + "negative_test_query340_00012", + "negative_test_query309_00008", + "negative_test_query123_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "apositive_test_query194_00000", + "negative_test_query347_00010", + "apositive_test_query13_00004", + "apositive_test_query13_00000", + "negative_test_query124_00003", + "apositive_test_query162_00002", + "negative_test_query349_00017", + "negative_test_query42_00006", + "negative_test_query42_00005", + "apositive_test_query162_00001", + "negative_test_query262_00011", + "negative_test_query187_00003", + "negative_test_query187_00006", + "negative_test_query123_00009", + "negative_test_query171_00002", + "negative_test_query309_00012", + "negative_test_query76_00004", + "negative_test_query339_00008", + "negative_test_query93_00010", + "negative_test_query144_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query125_00014", + "negative_test_query140_00017", + "negative_test_query360_00016", + "apositive_test_query286_00009", + "apositive_test_query125_00000", + "apositive_test_query125_00003", + "negative_test_query125_00011", + "apositive_test_query273_00003", + "negative_test_query125_00013", + "negative_test_query125_00003", + "negative_test_query125_00006", + "negative_test_query12_00009", + "negative_test_query6_00003", + "apositive_test_query286_00006", + "negative_test_query260_00012", + "negative_test_query125_00004", + "negative_test_query125_00000", + "negative_test_query3_00000", + "negative_test_query125_00009", + "negative_test_query301_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query126", + "ranked_ids": [ + "negative_test_query126_00000", + "apositive_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00017", + "negative_test_query190_00017", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query151_00006", + "negative_test_query190_00005", + "negative_test_query27_00013", + "negative_test_query311_00005", + "apositive_test_query330_00001", + "negative_test_query131_00009", + "negative_test_query12_00008", + "apositive_test_query330_00002", + "apositive_test_query5_00002", + "negative_test_query220_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00004", + "apositive_test_query127_00007", + "apositive_test_query127_00012", + "apositive_test_query127_00016", + "apositive_test_query127_00014", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "apositive_test_query127_00009", + "negative_test_query267_00009", + "apositive_test_query127_00008", + "apositive_test_query127_00000", + "apositive_test_query127_00003", + "apositive_test_query127_00006", + "negative_test_query94_00003", + "negative_test_query94_00009", + "apositive_test_query94_00005", + "apositive_test_query127_00013", + "apositive_test_query163_00000", + "apositive_test_query311_00003", + "negative_test_query198_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query128", + "ranked_ids": [ + "apositive_test_query128_00000", + "negative_test_query21_00017", + "apositive_test_query128_00001", + "negative_test_query256_00011", + "negative_test_query283_00006", + "negative_test_query310_00009", + "apositive_test_query216_00002", + "apositive_test_query80_00005", + "apositive_test_query142_00000", + "negative_test_query21_00000", + "negative_test_query128_00011", + "negative_test_query8_00017", + "negative_test_query128_00010", + "apositive_test_query21_00000", + "negative_test_query128_00006", + "apositive_test_query8_00000", + "negative_test_query192_00006", + "apositive_test_query196_00004", + "apositive_test_query32_00001", + "negative_test_query75_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.25 + }, + "query_id": "test_query129", + "ranked_ids": [ + "apositive_test_query129_00001", + "apositive_test_query129_00000", + "negative_test_query211_00012", + "negative_test_query117_00006", + "negative_test_query156_00017", + "negative_test_query63_00002", + "negative_test_query63_00006", + "negative_test_query129_00010", + "negative_test_query165_00012", + "negative_test_query245_00014", + "apositive_test_query244_00001", + "apositive_test_query356_00006", + "negative_test_query84_00008", + "apositive_test_query308_00004", + "apositive_test_query249_00003", + "apositive_test_query150_00000", + "negative_test_query16_00007", + "negative_test_query342_00013", + "apositive_test_query155_00002", + "negative_test_query187_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query13", + "ranked_ids": [ + "apositive_test_query13_00002", + "negative_test_query13_00001", + "apositive_test_query13_00000", + "apositive_test_query13_00004", + "apositive_test_query13_00003", + "apositive_test_query13_00008", + "apositive_test_query13_00010", + "apositive_test_query13_00006", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query13_00004", + "apositive_test_query13_00012", + "negative_test_query13_00006", + "negative_test_query13_00002", + "apositive_test_query13_00007", + "apositive_test_query13_00009", + "negative_test_query13_00000", + "negative_test_query13_00003", + "negative_test_query13_00005", + "negative_test_query245_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query130", + "ranked_ids": [ + "negative_test_query130_00001", + "apositive_test_query130_00005", + "negative_test_query198_00003", + "negative_test_query198_00011", + "negative_test_query345_00002", + "negative_test_query130_00008", + "negative_test_query284_00001", + "negative_test_query130_00006", + "apositive_test_query284_00003", + "apositive_test_query90_00001", + "apositive_test_query130_00002", + "apositive_test_query323_00004", + "negative_test_query284_00013", + "negative_test_query284_00009", + "apositive_test_query284_00001", + "negative_test_query90_00014", + "apositive_test_query233_00000", + "apositive_test_query275_00000", + "apositive_test_query30_00002", + "negative_test_query289_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query131", + "ranked_ids": [ + "apositive_test_query131_00000", + "negative_test_query186_00012", + "apositive_test_query131_00001", + "negative_test_query131_00001", + "negative_test_query313_00008", + "apositive_test_query106_00000", + "apositive_test_query158_00002", + "negative_test_query247_00004", + "negative_test_query131_00002", + "apositive_test_query50_00008", + "negative_test_query284_00006", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query158_00001", + "negative_test_query168_00013", + "negative_test_query184_00012", + "negative_test_query345_00009", + "negative_test_query284_00002", + "negative_test_query357_00001", + "apositive_test_query344_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "negative_test_query132_00006", + "negative_test_query81_00005", + "negative_test_query123_00012", + "negative_test_query81_00001", + "negative_test_query132_00010", + "negative_test_query132_00012", + "negative_test_query132_00016", + "negative_test_query132_00018", + "negative_test_query132_00000", + "negative_test_query132_00004", + "negative_test_query181_00008", + "negative_test_query11_00006", + "negative_test_query246_00016", + "negative_test_query240_00011", + "apositive_test_query161_00007", + "apositive_test_query3_00004", + "negative_test_query184_00015", + "negative_test_query304_00000", + "negative_test_query148_00011", + "apositive_test_query187_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query287_00001", + "apositive_test_query133_00000", + "apositive_test_query133_00008", + "apositive_test_query133_00002", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00001", + "apositive_test_query133_00014", + "apositive_test_query129_00002", + "apositive_test_query129_00004", + "negative_test_query12_00014", + "negative_test_query206_00017", + "apositive_test_query174_00011", + "apositive_test_query129_00006", + "apositive_test_query129_00007", + "apositive_test_query129_00005", + "negative_test_query59_00008", + "negative_test_query260_00006", + "apositive_test_query91_00002", + "negative_test_query310_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query134", + "ranked_ids": [ + "apositive_test_query213_00001", + "apositive_test_query276_00004", + "negative_test_query134_00006", + "negative_test_query134_00010", + "negative_test_query359_00011", + "apositive_test_query134_00002", + "negative_test_query107_00003", + "negative_test_query46_00004", + "negative_test_query218_00009", + "negative_test_query134_00000", + "apositive_test_query259_00011", + "negative_test_query134_00008", + "negative_test_query108_00003", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query46_00011", + "apositive_test_query142_00001", + "negative_test_query238_00001", + "apositive_test_query357_00000", + "apositive_test_query238_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00003", + "negative_test_query135_00007", + "apositive_test_query135_00000", + "negative_test_query135_00017", + "negative_test_query135_00011", + "apositive_test_query135_00001", + "negative_test_query325_00012", + "negative_test_query135_00008", + "negative_test_query135_00010", + "negative_test_query135_00004", + "negative_test_query135_00001", + "apositive_test_query133_00011", + "negative_test_query135_00005", + "negative_test_query287_00009", + "negative_test_query135_00014", + "negative_test_query135_00015", + "negative_test_query135_00013", + "negative_test_query135_00009", + "negative_test_query297_00017", + "apositive_test_query28_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query136", + "ranked_ids": [ + "negative_test_query128_00011", + "apositive_test_query136_00002", + "apositive_test_query136_00000", + "apositive_test_query136_00011", + "apositive_test_query136_00014", + "apositive_test_query136_00001", + "apositive_test_query136_00007", + "apositive_test_query136_00005", + "apositive_test_query136_00010", + "apositive_test_query136_00009", + "apositive_test_query136_00013", + "apositive_test_query136_00004", + "negative_test_query301_00001", + "negative_test_query136_00003", + "negative_test_query129_00001", + "apositive_test_query60_00014", + "negative_test_query60_00002", + "negative_test_query8_00010", + "negative_test_query245_00003", + "negative_test_query149_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query137", + "ranked_ids": [ + "negative_test_query137_00003", + "apositive_test_query137_00000", + "apositive_test_query253_00000", + "negative_test_query74_00001", + "negative_test_query253_00007", + "apositive_test_query137_00002", + "negative_test_query349_00006", + "apositive_test_query137_00003", + "negative_test_query95_00004", + "negative_test_query137_00000", + "negative_test_query137_00014", + "negative_test_query74_00012", + "negative_test_query13_00002", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query262_00009", + "negative_test_query137_00012", + "negative_test_query149_00013", + "negative_test_query207_00006", + "negative_test_query74_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query138", + "ranked_ids": [ + "apositive_test_query138_00000", + "apositive_test_query138_00001", + "negative_test_query183_00004", + "apositive_test_query356_00003", + "negative_test_query138_00008", + "negative_test_query138_00006", + "negative_test_query240_00000", + "negative_test_query184_00005", + "negative_test_query213_00006", + "apositive_test_query23_00003", + "negative_test_query138_00003", + "negative_test_query138_00001", + "negative_test_query138_00013", + "negative_test_query132_00009", + "negative_test_query138_00015", + "negative_test_query138_00014", + "negative_test_query138_00004", + "negative_test_query138_00005", + "apositive_test_query138_00002", + "apositive_test_query244_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query35_00001", + "apositive_test_query139_00000", + "apositive_test_query35_00000", + "apositive_test_query139_00001", + "apositive_test_query146_00007", + "negative_test_query101_00006", + "negative_test_query35_00010", + "apositive_test_query354_00001", + "negative_test_query320_00007", + "negative_test_query262_00002", + "negative_test_query81_00002", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query123_00006", + "negative_test_query231_00012", + "negative_test_query269_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query14", + "ranked_ids": [ + "apositive_test_query77_00000", + "apositive_test_query14_00003", + "apositive_test_query216_00002", + "negative_test_query155_00009", + "negative_test_query14_00001", + "negative_test_query179_00001", + "apositive_test_query64_00000", + "negative_test_query53_00003", + "negative_test_query155_00001", + "negative_test_query260_00014", + "apositive_test_query296_00005", + "negative_test_query199_00009", + "negative_test_query14_00006", + "negative_test_query14_00007", + "negative_test_query14_00005", + "negative_test_query350_00015", + "negative_test_query14_00004", + "negative_test_query102_00011", + "negative_test_query260_00000", + "apositive_test_query26_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00005", + "negative_test_query140_00003", + "apositive_test_query281_00013", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00001", + "negative_test_query140_00016", + "negative_test_query140_00014", + "negative_test_query140_00013", + "negative_test_query140_00010", + "negative_test_query140_00018", + "apositive_test_query273_00005", + "negative_test_query257_00012", + "negative_test_query18_00006", + "negative_test_query140_00009", + "apositive_test_query18_00003", + "negative_test_query18_00013", + "apositive_test_query18_00001", + "negative_test_query140_00002", + "negative_test_query140_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query141", + "ranked_ids": [ + "apositive_test_query288_00003", + "negative_test_query245_00008", + "apositive_test_query141_00005", + "apositive_test_query141_00000", + "negative_test_query245_00016", + "negative_test_query57_00002", + "apositive_test_query288_00005", + "negative_test_query344_00013", + "negative_test_query220_00011", + "apositive_test_query141_00006", + "negative_test_query176_00003", + "negative_test_query25_00006", + "negative_test_query325_00008", + "negative_test_query308_00005", + "negative_test_query141_00006", + "apositive_test_query288_00007", + "negative_test_query245_00013", + "negative_test_query141_00010", + "apositive_test_query246_00001", + "negative_test_query105_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query142", + "ranked_ids": [ + "apositive_test_query142_00002", + "negative_test_query105_00008", + "negative_test_query142_00007", + "apositive_test_query142_00001", + "negative_test_query238_00001", + "negative_test_query102_00001", + "negative_test_query203_00003", + "negative_test_query223_00005", + "negative_test_query46_00012", + "negative_test_query46_00001", + "negative_test_query46_00002", + "negative_test_query142_00003", + "negative_test_query142_00009", + "apositive_test_query142_00003", + "negative_test_query359_00005", + "negative_test_query39_00008", + "negative_test_query105_00006", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query142_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query143", + "ranked_ids": [ + "apositive_test_query143_00000", + "negative_test_query143_00006", + "negative_test_query143_00011", + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query143_00005", + "negative_test_query143_00009", + "apositive_test_query25_00006", + "negative_test_query143_00000", + "negative_test_query143_00012", + "negative_test_query90_00014", + "negative_test_query143_00010", + "negative_test_query143_00013", + "negative_test_query143_00015", + "negative_test_query143_00007", + "negative_test_query143_00014", + "negative_test_query143_00016", + "apositive_test_query25_00005", + "negative_test_query56_00005", + "apositive_test_query308_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query144_00010", + "negative_test_query144_00012", + "negative_test_query41_00001", + "negative_test_query41_00009", + "negative_test_query144_00018", + "negative_test_query144_00003", + "negative_test_query144_00006", + "negative_test_query144_00001", + "negative_test_query144_00016", + "apositive_test_query41_00000", + "negative_test_query144_00009", + "negative_test_query154_00009", + "negative_test_query198_00016", + "negative_test_query166_00001", + "negative_test_query315_00011", + "negative_test_query144_00011", + "negative_test_query41_00011", + "negative_test_query226_00011", + "negative_test_query331_00015", + "negative_test_query144_00017" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query145", + "ranked_ids": [ + "apositive_test_query225_00002", + "apositive_test_query145_00002", + "negative_test_query230_00002", + "negative_test_query53_00012", + "apositive_test_query305_00002", + "negative_test_query305_00002", + "apositive_test_query160_00010", + "negative_test_query112_00007", + "apositive_test_query213_00004", + "apositive_test_query107_00001", + "negative_test_query79_00005", + "negative_test_query118_00000", + "apositive_test_query207_00004", + "negative_test_query171_00003", + "negative_test_query187_00014", + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "negative_test_query190_00012", + "negative_test_query211_00013" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.125 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query146_00010", + "apositive_test_query153_00008", + "apositive_test_query3_00010", + "negative_test_query358_00008", + "apositive_test_query146_00004", + "apositive_test_query314_00004", + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query146_00003", + "negative_test_query65_00006", + "apositive_test_query146_00005", + "negative_test_query146_00001", + "apositive_test_query314_00003", + "negative_test_query144_00008", + "apositive_test_query161_00007", + "apositive_test_query161_00008", + "negative_test_query99_00001", + "apositive_test_query161_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query147", + "ranked_ids": [ + "negative_test_query147_00013", + "apositive_test_query147_00002", + "apositive_test_query147_00004", + "negative_test_query76_00000", + "apositive_test_query76_00001", + "negative_test_query258_00008", + "negative_test_query53_00009", + "negative_test_query354_00007", + "negative_test_query309_00014", + "negative_test_query309_00016", + "apositive_test_query161_00004", + "apositive_test_query147_00003", + "negative_test_query17_00009", + "negative_test_query181_00008", + "negative_test_query180_00012", + "apositive_test_query76_00004", + "negative_test_query147_00005", + "negative_test_query11_00009", + "negative_test_query147_00008", + "negative_test_query200_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query124_00010", + "negative_test_query148_00018", + "negative_test_query148_00005", + "negative_test_query148_00017", + "negative_test_query210_00013", + "negative_test_query148_00012", + "negative_test_query210_00005", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query315_00002", + "negative_test_query80_00007", + "negative_test_query112_00013", + "negative_test_query148_00008", + "apositive_test_query304_00001", + "apositive_test_query304_00002", + "apositive_test_query234_00003", + "negative_test_query148_00014", + "negative_test_query196_00004", + "negative_test_query89_00007", + "negative_test_query210_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "apositive_test_query149_00001", + "negative_test_query149_00001", + "apositive_test_query149_00000", + "negative_test_query149_00012", + "negative_test_query149_00015", + "negative_test_query63_00009", + "negative_test_query332_00004", + "negative_test_query149_00002", + "negative_test_query149_00008", + "negative_test_query149_00010", + "negative_test_query149_00017", + "negative_test_query85_00009", + "negative_test_query149_00013", + "negative_test_query149_00011", + "apositive_test_query225_00004", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query332_00008", + "negative_test_query74_00007", + "apositive_test_query113_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00010", + "negative_test_query266_00008", + "apositive_test_query325_00000", + "negative_test_query15_00012", + "apositive_test_query15_00001", + "negative_test_query15_00004", + "negative_test_query15_00011", + "negative_test_query325_00011", + "negative_test_query15_00005", + "negative_test_query15_00007", + "negative_test_query15_00013", + "negative_test_query15_00017", + "negative_test_query266_00013", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query15_00014", + "negative_test_query15_00015", + "negative_test_query266_00006", + "negative_test_query15_00008", + "negative_test_query176_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query150", + "ranked_ids": [ + "apositive_test_query20_00007", + "negative_test_query250_00013", + "apositive_test_query150_00003", + "negative_test_query306_00004", + "apositive_test_query20_00010", + "negative_test_query1_00012", + "apositive_test_query89_00000", + "negative_test_query150_00010", + "negative_test_query267_00006", + "negative_test_query62_00009", + "negative_test_query213_00007", + "negative_test_query301_00007", + "negative_test_query139_00005", + "apositive_test_query158_00001", + "negative_test_query313_00017", + "apositive_test_query112_00000", + "negative_test_query346_00008", + "negative_test_query63_00004", + "negative_test_query254_00003", + "negative_test_query350_00018" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "negative_test_query190_00003", + "apositive_test_query126_00000", + "negative_test_query151_00006", + "negative_test_query190_00005", + "apositive_test_query300_00001", + "negative_test_query293_00008", + "negative_test_query151_00003", + "negative_test_query44_00002", + "negative_test_query151_00015", + "negative_test_query190_00015", + "negative_test_query201_00008", + "negative_test_query151_00004", + "negative_test_query190_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query237_00011", + "apositive_test_query152_00004", + "apositive_test_query152_00009", + "negative_test_query237_00000", + "negative_test_query312_00014", + "negative_test_query237_00004", + "negative_test_query124_00010", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query152_00005", + "negative_test_query349_00001", + "negative_test_query342_00005", + "negative_test_query95_00017", + "negative_test_query349_00005", + "negative_test_query253_00004", + "negative_test_query182_00004", + "negative_test_query250_00002", + "apositive_test_query194_00008", + "negative_test_query237_00009", + "negative_test_query262_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.25 + }, + "query_id": "test_query153", + "ranked_ids": [ + "negative_test_query250_00006", + "apositive_test_query153_00003", + "apositive_test_query153_00001", + "apositive_test_query153_00015", + "apositive_test_query153_00002", + "negative_test_query112_00007", + "apositive_test_query153_00007", + "apositive_test_query153_00004", + "negative_test_query359_00003", + "apositive_test_query213_00004", + "apositive_test_query153_00000", + "negative_test_query283_00015", + "apositive_test_query176_00000", + "negative_test_query278_00006", + "negative_test_query278_00007", + "apositive_test_query20_00006", + "negative_test_query112_00015", + "negative_test_query228_00002", + "negative_test_query2_00001", + "negative_test_query322_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query154", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query20_00003", + "apositive_test_query20_00007", + "apositive_test_query23_00004", + "apositive_test_query356_00005", + "apositive_test_query154_00006", + "negative_test_query331_00012", + "apositive_test_query33_00000", + "apositive_test_query358_00008", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "apositive_test_query20_00010", + "apositive_test_query154_00005", + "negative_test_query245_00006", + "negative_test_query334_00000", + "negative_test_query311_00005", + "negative_test_query68_00015", + "negative_test_query102_00010", + "negative_test_query41_00007", + "negative_test_query133_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00009", + "negative_test_query155_00003", + "negative_test_query155_00006", + "negative_test_query155_00005", + "negative_test_query123_00014", + "apositive_test_query155_00001", + "negative_test_query155_00002", + "apositive_test_query3_00008", + "negative_test_query99_00001", + "apositive_test_query155_00005", + "negative_test_query155_00004", + "negative_test_query155_00012", + "apositive_test_query355_00007", + "negative_test_query155_00001", + "apositive_test_query40_00004", + "negative_test_query199_00016", + "negative_test_query119_00001", + "negative_test_query111_00016", + "negative_test_query155_00010", + "negative_test_query155_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "negative_test_query330_00006", + "apositive_test_query210_00001", + "negative_test_query156_00001", + "negative_test_query140_00008", + "apositive_test_query25_00009", + "negative_test_query246_00016", + "apositive_test_query154_00005", + "apositive_test_query330_00002", + "apositive_test_query156_00000", + "negative_test_query92_00010", + "negative_test_query92_00005", + "negative_test_query15_00010", + "negative_test_query299_00014", + "negative_test_query1_00012", + "negative_test_query32_00004", + "negative_test_query283_00001", + "negative_test_query92_00003", + "negative_test_query283_00009", + "negative_test_query49_00005", + "apositive_test_query39_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query152_00005", + "negative_test_query237_00004", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "apositive_test_query152_00000", + "negative_test_query152_00003", + "negative_test_query237_00002", + "apositive_test_query237_00000", + "negative_test_query152_00008", + "negative_test_query271_00003", + "apositive_test_query237_00002", + "apositive_test_query237_00004", + "negative_test_query312_00017", + "negative_test_query237_00009", + "negative_test_query157_00005", + "apositive_test_query157_00001", + "apositive_test_query157_00002", + "apositive_test_query182_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query158", + "ranked_ids": [ + "negative_test_query334_00016", + "negative_test_query223_00007", + "negative_test_query102_00015", + "negative_test_query105_00013", + "negative_test_query158_00006", + "negative_test_query164_00004", + "negative_test_query26_00003", + "negative_test_query247_00010", + "apositive_test_query186_00003", + "negative_test_query213_00002", + "negative_test_query26_00000", + "negative_test_query334_00009", + "negative_test_query26_00010", + "negative_test_query102_00003", + "apositive_test_query158_00003", + "negative_test_query26_00005", + "negative_test_query26_00004", + "apositive_test_query158_00000", + "negative_test_query223_00002", + "apositive_test_query158_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query159", + "ranked_ids": [ + "negative_test_query264_00013", + "negative_test_query159_00011", + "apositive_test_query159_00001", + "negative_test_query159_00005", + "apositive_test_query50_00004", + "apositive_test_query212_00002", + "apositive_test_query326_00007", + "apositive_test_query212_00001", + "apositive_test_query38_00003", + "negative_test_query264_00003", + "negative_test_query38_00010", + "negative_test_query159_00016", + "negative_test_query264_00012", + "negative_test_query66_00003", + "apositive_test_query212_00003", + "negative_test_query7_00005", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query122_00002", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query16", + "ranked_ids": [ + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query308_00004", + "apositive_test_query16_00004", + "negative_test_query16_00008", + "negative_test_query16_00000", + "negative_test_query16_00010", + "negative_test_query16_00009", + "apositive_test_query16_00002", + "negative_test_query16_00001", + "negative_test_query16_00002", + "negative_test_query240_00003", + "negative_test_query16_00003", + "negative_test_query280_00007", + "negative_test_query16_00011", + "negative_test_query16_00012", + "negative_test_query237_00010", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query208_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query160", + "ranked_ids": [ + "negative_test_query160_00003", + "apositive_test_query160_00000", + "apositive_test_query160_00009", + "negative_test_query330_00005", + "negative_test_query160_00004", + "negative_test_query214_00013", + "negative_test_query149_00007", + "apositive_test_query160_00004", + "negative_test_query215_00010", + "apositive_test_query355_00007", + "negative_test_query106_00004", + "negative_test_query176_00016", + "apositive_test_query260_00000", + "apositive_test_query64_00000", + "apositive_test_query120_00002", + "negative_test_query211_00009", + "negative_test_query64_00009", + "negative_test_query64_00015", + "negative_test_query271_00010", + "negative_test_query304_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query161", + "ranked_ids": [ + "apositive_test_query161_00006", + "negative_test_query155_00012", + "apositive_test_query3_00006", + "apositive_test_query276_00010", + "apositive_test_query161_00005", + "negative_test_query199_00004", + "apositive_test_query305_00002", + "apositive_test_query161_00008", + "negative_test_query22_00015", + "apositive_test_query37_00004", + "negative_test_query71_00006", + "negative_test_query260_00004", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "apositive_test_query261_00004", + "negative_test_query301_00015", + "apositive_test_query314_00004", + "apositive_test_query301_00000", + "apositive_test_query314_00006", + "negative_test_query57_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.25 + }, + "query_id": "test_query162", + "ranked_ids": [ + "negative_test_query177_00002", + "apositive_test_query162_00010", + "apositive_test_query162_00009", + "negative_test_query162_00004", + "apositive_test_query162_00003", + "apositive_test_query162_00000", + "negative_test_query162_00001", + "apositive_test_query162_00006", + "negative_test_query95_00006", + "negative_test_query240_00008", + "apositive_test_query162_00005", + "apositive_test_query194_00006", + "apositive_test_query162_00004", + "negative_test_query95_00013", + "negative_test_query177_00005", + "apositive_test_query194_00009", + "negative_test_query343_00012", + "apositive_test_query114_00001", + "apositive_test_query194_00003", + "negative_test_query230_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query197_00002", + "negative_test_query163_00008", + "negative_test_query163_00011", + "negative_test_query163_00003", + "negative_test_query197_00001", + "negative_test_query90_00011", + "negative_test_query163_00013", + "negative_test_query224_00010", + "negative_test_query350_00017", + "apositive_test_query357_00004", + "negative_test_query47_00001", + "negative_test_query16_00013", + "negative_test_query202_00002", + "apositive_test_query172_00006", + "negative_test_query328_00006", + "negative_test_query58_00015", + "negative_test_query245_00001", + "negative_test_query265_00015", + "negative_test_query9_00005", + "apositive_test_query152_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query164_00000", + "negative_test_query131_00014", + "negative_test_query158_00008", + "apositive_test_query146_00014", + "negative_test_query85_00010", + "negative_test_query344_00009", + "negative_test_query2_00000", + "negative_test_query164_00015", + "negative_test_query332_00001", + "apositive_test_query13_00000", + "apositive_test_query297_00000", + "negative_test_query101_00005", + "negative_test_query165_00003", + "negative_test_query245_00005", + "negative_test_query321_00011", + "negative_test_query360_00012", + "negative_test_query1_00003", + "negative_test_query100_00013", + "negative_test_query208_00014", + "negative_test_query309_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query165_00013", + "negative_test_query122_00003", + "negative_test_query149_00012", + "apositive_test_query165_00001", + "negative_test_query165_00000", + "negative_test_query327_00016", + "negative_test_query86_00010", + "negative_test_query86_00006", + "negative_test_query165_00007", + "negative_test_query285_00007", + "negative_test_query285_00004", + "negative_test_query348_00001", + "negative_test_query264_00016", + "negative_test_query327_00010", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query38_00010", + "negative_test_query299_00011", + "negative_test_query215_00013", + "apositive_test_query219_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query166", + "ranked_ids": [ + "apositive_test_query166_00005", + "negative_test_query166_00012", + "negative_test_query166_00000", + "negative_test_query166_00006", + "apositive_test_query166_00001", + "negative_test_query166_00005", + "negative_test_query166_00004", + "apositive_test_query166_00003", + "negative_test_query166_00013", + "negative_test_query165_00015", + "apositive_test_query166_00000", + "apositive_test_query166_00002", + "negative_test_query166_00008", + "negative_test_query255_00003", + "negative_test_query166_00007", + "negative_test_query249_00000", + "negative_test_query166_00003", + "negative_test_query166_00001", + "apositive_test_query224_00000", + "negative_test_query224_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query167_00012", + "negative_test_query278_00016", + "negative_test_query278_00010", + "apositive_test_query288_00005", + "negative_test_query141_00010", + "negative_test_query167_00014", + "negative_test_query167_00002", + "negative_test_query167_00016", + "negative_test_query167_00004", + "negative_test_query167_00011", + "negative_test_query167_00015", + "negative_test_query167_00009", + "negative_test_query167_00010", + "negative_test_query167_00003", + "negative_test_query167_00005", + "negative_test_query167_00001", + "negative_test_query167_00007", + "negative_test_query167_00013", + "negative_test_query167_00000", + "negative_test_query167_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6096199500078984, + "recall@5": 0.75 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query83_00003", + "apositive_test_query168_00000", + "apositive_test_query168_00001", + "apositive_test_query168_00002", + "apositive_test_query190_00001", + "negative_test_query168_00014", + "negative_test_query6_00001", + "negative_test_query168_00000", + "apositive_test_query340_00000", + "negative_test_query270_00003", + "negative_test_query6_00005", + "negative_test_query6_00000", + "apositive_test_query286_00007", + "negative_test_query168_00007", + "apositive_test_query83_00004", + "negative_test_query168_00009", + "negative_test_query83_00003", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "apositive_test_query80_00003", + "apositive_test_query69_00000", + "apositive_test_query286_00008", + "negative_test_query104_00001", + "negative_test_query72_00013", + "negative_test_query72_00001", + "negative_test_query72_00016", + "negative_test_query49_00017", + "apositive_test_query180_00001", + "negative_test_query180_00004", + "negative_test_query10_00005", + "negative_test_query10_00013", + "negative_test_query169_00001", + "negative_test_query89_00006", + "negative_test_query260_00015", + "apositive_test_query242_00006", + "apositive_test_query291_00009", + "apositive_test_query330_00001", + "negative_test_query343_00001", + "apositive_test_query266_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00000", + "negative_test_query130_00007", + "negative_test_query292_00010", + "apositive_test_query153_00013", + "negative_test_query164_00006", + "apositive_test_query355_00007", + "negative_test_query302_00011", + "negative_test_query312_00004", + "negative_test_query17_00005", + "negative_test_query17_00002", + "negative_test_query17_00012", + "negative_test_query17_00016", + "negative_test_query17_00014", + "negative_test_query305_00004", + "negative_test_query10_00005", + "negative_test_query34_00003", + "apositive_test_query225_00007", + "negative_test_query17_00003", + "negative_test_query148_00001", + "negative_test_query17_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5555555555555556 + }, + "query_id": "test_query170", + "ranked_ids": [ + "apositive_test_query170_00006", + "apositive_test_query170_00000", + "apositive_test_query170_00005", + "apositive_test_query170_00004", + "apositive_test_query170_00002", + "apositive_test_query170_00007", + "negative_test_query170_00001", + "negative_test_query0_00006", + "negative_test_query170_00000", + "negative_test_query170_00007", + "negative_test_query170_00008", + "negative_test_query170_00004", + "negative_test_query170_00005", + "apositive_test_query297_00000", + "negative_test_query348_00001", + "apositive_test_query170_00001", + "negative_test_query227_00002", + "negative_test_query297_00005", + "apositive_test_query70_00003", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query171", + "ranked_ids": [ + "negative_test_query171_00005", + "apositive_test_query171_00001", + "negative_test_query171_00004", + "negative_test_query171_00013", + "negative_test_query171_00011", + "negative_test_query219_00010", + "negative_test_query24_00007", + "negative_test_query171_00003", + "negative_test_query219_00002", + "negative_test_query171_00000", + "negative_test_query346_00005", + "negative_test_query277_00007", + "negative_test_query261_00000", + "negative_test_query171_00007", + "negative_test_query171_00017", + "negative_test_query191_00006", + "apositive_test_query224_00000", + "negative_test_query171_00001", + "negative_test_query171_00015", + "negative_test_query126_00013" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query172", + "ranked_ids": [ + "negative_test_query172_00007", + "apositive_test_query172_00003", + "apositive_test_query254_00003", + "negative_test_query192_00010", + "apositive_test_query124_00002", + "apositive_test_query172_00004", + "negative_test_query350_00009", + "apositive_test_query136_00011", + "apositive_test_query136_00001", + "apositive_test_query136_00009", + "negative_test_query8_00000", + "negative_test_query192_00000", + "apositive_test_query172_00006", + "apositive_test_query13_00003", + "negative_test_query349_00001", + "apositive_test_query172_00002", + "negative_test_query208_00007", + "negative_test_query208_00015", + "negative_test_query173_00006", + "negative_test_query64_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query173_00011", + "negative_test_query64_00017", + "apositive_test_query173_00004", + "negative_test_query173_00006", + "apositive_test_query173_00002", + "apositive_test_query173_00001", + "apositive_test_query173_00003", + "negative_test_query64_00006", + "negative_test_query173_00003", + "negative_test_query64_00004", + "negative_test_query64_00013", + "negative_test_query222_00010", + "apositive_test_query173_00000", + "negative_test_query173_00013", + "negative_test_query350_00002", + "apositive_test_query101_00004", + "negative_test_query303_00005", + "apositive_test_query134_00000", + "negative_test_query339_00006", + "negative_test_query173_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00000", + "apositive_test_query174_00005", + "apositive_test_query174_00004", + "apositive_test_query174_00001", + "apositive_test_query174_00012", + "negative_test_query19_00012", + "apositive_test_query174_00006", + "apositive_test_query174_00003", + "apositive_test_query174_00002", + "apositive_test_query174_00008", + "apositive_test_query174_00009", + "apositive_test_query174_00011", + "apositive_test_query174_00010", + "apositive_test_query174_00007", + "apositive_test_query174_00014", + "negative_test_query262_00005", + "apositive_test_query4_00012", + "apositive_test_query174_00015", + "apositive_test_query174_00013", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00004", + "apositive_test_query175_00003", + "negative_test_query175_00010", + "negative_test_query97_00007", + "negative_test_query204_00002", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "apositive_test_query175_00000", + "apositive_test_query204_00010", + "negative_test_query175_00013", + "apositive_test_query175_00005", + "negative_test_query38_00006", + "apositive_test_query353_00005", + "negative_test_query204_00001", + "apositive_test_query50_00015", + "negative_test_query175_00000", + "negative_test_query105_00012", + "negative_test_query7_00000", + "negative_test_query204_00000", + "negative_test_query105_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query195_00006", + "negative_test_query176_00015", + "negative_test_query195_00012", + "negative_test_query224_00003", + "apositive_test_query128_00007", + "negative_test_query24_00014", + "negative_test_query284_00006", + "negative_test_query284_00002", + "apositive_test_query344_00003", + "negative_test_query337_00005", + "negative_test_query200_00005", + "negative_test_query221_00007", + "negative_test_query9_00003", + "negative_test_query191_00008", + "negative_test_query9_00007", + "negative_test_query35_00005", + "negative_test_query144_00018", + "apositive_test_query92_00002", + "negative_test_query93_00000", + "apositive_test_query56_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query172_00008", + "negative_test_query177_00004", + "negative_test_query124_00012", + "negative_test_query177_00008", + "negative_test_query152_00005", + "negative_test_query124_00001", + "apositive_test_query177_00004", + "negative_test_query177_00006", + "negative_test_query147_00011", + "apositive_test_query177_00006", + "apositive_test_query230_00005", + "negative_test_query147_00005", + "negative_test_query147_00009", + "negative_test_query147_00002", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "apositive_test_query229_00008", + "apositive_test_query162_00009", + "apositive_test_query162_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query178", + "ranked_ids": [ + "apositive_test_query178_00004", + "negative_test_query178_00000", + "negative_test_query178_00007", + "negative_test_query178_00010", + "negative_test_query178_00012", + "apositive_test_query178_00005", + "negative_test_query178_00003", + "negative_test_query285_00008", + "negative_test_query178_00001", + "apositive_test_query285_00006", + "negative_test_query285_00009", + "negative_test_query43_00001", + "negative_test_query214_00002", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query321_00017", + "negative_test_query327_00004", + "negative_test_query165_00001", + "apositive_test_query352_00001", + "negative_test_query274_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "negative_test_query179_00014", + "apositive_test_query173_00000", + "apositive_test_query173_00003", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query303_00005", + "negative_test_query297_00000", + "apositive_test_query316_00001", + "negative_test_query316_00001", + "negative_test_query279_00010", + "negative_test_query206_00009", + "negative_test_query14_00001", + "negative_test_query173_00013", + "negative_test_query179_00001", + "negative_test_query191_00003", + "negative_test_query260_00012", + "negative_test_query64_00003", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.8 + }, + "query_id": "test_query18", + "ranked_ids": [ + "apositive_test_query18_00001", + "apositive_test_query18_00000", + "apositive_test_query18_00003", + "apositive_test_query18_00002", + "negative_test_query18_00006", + "negative_test_query18_00011", + "negative_test_query140_00003", + "apositive_test_query281_00013", + "negative_test_query18_00012", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query18_00013", + "negative_test_query257_00012", + "negative_test_query287_00004", + "apositive_test_query18_00004", + "negative_test_query88_00002", + "negative_test_query88_00012", + "negative_test_query300_00015", + "negative_test_query156_00018", + "apositive_test_query288_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query180", + "ranked_ids": [ + "apositive_test_query180_00000", + "apositive_test_query180_00001", + "negative_test_query180_00007", + "negative_test_query180_00016", + "negative_test_query180_00004", + "apositive_test_query180_00002", + "negative_test_query180_00002", + "apositive_test_query262_00003", + "negative_test_query312_00018", + "negative_test_query180_00011", + "negative_test_query114_00003", + "negative_test_query207_00007", + "negative_test_query95_00014", + "apositive_test_query262_00002", + "apositive_test_query262_00000", + "negative_test_query169_00015", + "negative_test_query112_00014", + "negative_test_query180_00006", + "negative_test_query246_00003", + "negative_test_query176_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00006", + "negative_test_query181_00017", + "negative_test_query181_00005", + "negative_test_query181_00012", + "negative_test_query181_00002", + "negative_test_query181_00010", + "negative_test_query181_00004", + "negative_test_query181_00014", + "negative_test_query181_00007", + "negative_test_query181_00016", + "negative_test_query348_00007", + "negative_test_query320_00007", + "negative_test_query181_00008", + "negative_test_query181_00009", + "apositive_test_query119_00011", + "negative_test_query181_00018", + "negative_test_query96_00006", + "apositive_test_query256_00003", + "negative_test_query181_00001", + "negative_test_query183_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "negative_test_query182_00017", + "negative_test_query182_00011", + "apositive_test_query182_00000", + "negative_test_query182_00004", + "negative_test_query182_00002", + "apositive_test_query152_00009", + "negative_test_query237_00004", + "negative_test_query237_00006", + "negative_test_query237_00001", + "negative_test_query237_00007", + "negative_test_query182_00018", + "negative_test_query240_00001", + "negative_test_query230_00008", + "negative_test_query337_00012", + "negative_test_query342_00003", + "negative_test_query77_00002", + "negative_test_query342_00004", + "negative_test_query342_00001", + "negative_test_query296_00010", + "negative_test_query342_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00015", + "negative_test_query183_00009", + "negative_test_query183_00014", + "negative_test_query7_00007", + "negative_test_query183_00012", + "negative_test_query183_00006", + "apositive_test_query7_00001", + "negative_test_query183_00007", + "negative_test_query7_00008", + "negative_test_query7_00014", + "negative_test_query183_00018", + "negative_test_query183_00008", + "negative_test_query7_00009", + "negative_test_query112_00017", + "negative_test_query183_00000", + "negative_test_query183_00010", + "negative_test_query7_00010", + "negative_test_query183_00013", + "negative_test_query238_00007", + "apositive_test_query178_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "negative_test_query216_00014", + "negative_test_query216_00002", + "apositive_test_query33_00004", + "apositive_test_query216_00001", + "negative_test_query216_00008", + "apositive_test_query216_00004", + "apositive_test_query216_00000", + "negative_test_query228_00011", + "negative_test_query216_00007", + "apositive_test_query72_00000", + "negative_test_query158_00008", + "negative_test_query344_00010", + "negative_test_query101_00007", + "negative_test_query156_00001", + "negative_test_query41_00013", + "apositive_test_query20_00010", + "negative_test_query322_00000", + "negative_test_query216_00006", + "apositive_test_query316_00000", + "apositive_test_query33_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query185", + "ranked_ids": [ + "negative_test_query185_00002", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "apositive_test_query185_00003", + "negative_test_query72_00009", + "negative_test_query48_00009", + "negative_test_query103_00001", + "negative_test_query185_00000", + "negative_test_query72_00011", + "negative_test_query185_00005", + "negative_test_query72_00017", + "negative_test_query139_00014", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "apositive_test_query307_00008", + "negative_test_query48_00014", + "negative_test_query72_00012", + "apositive_test_query43_00006", + "negative_test_query103_00000", + "negative_test_query350_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "negative_test_query247_00012", + "negative_test_query186_00005", + "negative_test_query247_00007", + "negative_test_query313_00015", + "apositive_test_query50_00010", + "apositive_test_query50_00000", + "apositive_test_query50_00001", + "apositive_test_query306_00003", + "negative_test_query46_00001", + "negative_test_query313_00014", + "negative_test_query47_00009", + "negative_test_query57_00007", + "negative_test_query142_00009", + "negative_test_query259_00003", + "negative_test_query334_00017", + "apositive_test_query79_00005", + "negative_test_query105_00001", + "negative_test_query336_00008", + "negative_test_query29_00003", + "negative_test_query313_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query306_00004", + "negative_test_query245_00006", + "negative_test_query279_00002", + "negative_test_query176_00008", + "negative_test_query62_00009", + "apositive_test_query127_00014", + "negative_test_query230_00000", + "apositive_test_query311_00002", + "apositive_test_query358_00008", + "apositive_test_query176_00000", + "negative_test_query168_00002", + "apositive_test_query154_00005", + "negative_test_query245_00015", + "negative_test_query13_00005", + "apositive_test_query33_00008", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "apositive_test_query20_00010", + "negative_test_query184_00015", + "negative_test_query331_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "negative_test_query188_00014", + "negative_test_query188_00008", + "apositive_test_query188_00000", + "negative_test_query161_00010", + "apositive_test_query20_00006", + "negative_test_query344_00007", + "apositive_test_query20_00009", + "negative_test_query188_00009", + "apositive_test_query146_00009", + "negative_test_query188_00010", + "negative_test_query112_00017", + "negative_test_query188_00003", + "negative_test_query188_00013", + "negative_test_query188_00011", + "negative_test_query220_00011", + "negative_test_query344_00013", + "negative_test_query188_00004", + "apositive_test_query220_00003", + "apositive_test_query220_00004", + "negative_test_query20_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00001", + "apositive_test_query189_00003", + "apositive_test_query189_00001", + "negative_test_query189_00007", + "negative_test_query189_00004", + "negative_test_query140_00016", + "negative_test_query44_00005", + "negative_test_query287_00000", + "apositive_test_query85_00002", + "apositive_test_query189_00002", + "negative_test_query260_00007", + "negative_test_query189_00013", + "negative_test_query189_00006", + "negative_test_query189_00002", + "negative_test_query18_00006", + "negative_test_query190_00001", + "negative_test_query189_00000", + "negative_test_query85_00007", + "negative_test_query55_00003", + "apositive_test_query254_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query19", + "ranked_ids": [ + "negative_test_query4_00001", + "negative_test_query229_00002", + "apositive_test_query4_00000", + "apositive_test_query4_00010", + "apositive_test_query174_00014", + "apositive_test_query174_00007", + "negative_test_query106_00004", + "apositive_test_query19_00000", + "negative_test_query15_00000", + "negative_test_query169_00014", + "negative_test_query19_00006", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query266_00010", + "negative_test_query19_00003", + "negative_test_query19_00008", + "negative_test_query141_00000", + "negative_test_query174_00001", + "negative_test_query141_00001", + "negative_test_query167_00018" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query271_00005", + "negative_test_query85_00006", + "apositive_test_query260_00003", + "negative_test_query109_00006", + "negative_test_query6_00007", + "negative_test_query260_00007", + "apositive_test_query311_00000", + "negative_test_query345_00006", + "negative_test_query85_00014", + "negative_test_query205_00016", + "apositive_test_query85_00000", + "negative_test_query190_00013", + "negative_test_query181_00003", + "negative_test_query233_00008", + "negative_test_query265_00004", + "apositive_test_query234_00001", + "negative_test_query246_00002", + "negative_test_query264_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.75369761125927, + "recall@5": 0.75 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "negative_test_query191_00010", + "apositive_test_query191_00001", + "apositive_test_query191_00002", + "negative_test_query181_00013", + "negative_test_query256_00009", + "negative_test_query241_00007", + "apositive_test_query341_00005", + "apositive_test_query341_00003", + "apositive_test_query63_00003", + "negative_test_query297_00002", + "negative_test_query191_00011", + "negative_test_query191_00012", + "negative_test_query96_00006", + "negative_test_query60_00000", + "negative_test_query191_00003", + "negative_test_query332_00012", + "negative_test_query274_00002", + "negative_test_query191_00002", + "negative_test_query64_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00000", + "apositive_test_query192_00002", + "negative_test_query120_00004", + "negative_test_query192_00001", + "negative_test_query192_00002", + "negative_test_query192_00013", + "negative_test_query172_00005", + "negative_test_query192_00009", + "negative_test_query192_00010", + "apositive_test_query192_00004", + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query192_00008", + "negative_test_query350_00010", + "apositive_test_query254_00007", + "negative_test_query192_00014", + "negative_test_query192_00004", + "negative_test_query192_00012", + "negative_test_query322_00008", + "negative_test_query192_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query193", + "ranked_ids": [ + "apositive_test_query99_00003", + "apositive_test_query193_00009", + "apositive_test_query193_00004", + "apositive_test_query193_00011", + "apositive_test_query193_00008", + "negative_test_query193_00002", + "apositive_test_query193_00001", + "apositive_test_query193_00005", + "negative_test_query99_00009", + "apositive_test_query153_00012", + "apositive_test_query193_00000", + "apositive_test_query99_00002", + "negative_test_query146_00001", + "apositive_test_query146_00008", + "apositive_test_query276_00006", + "negative_test_query146_00003", + "negative_test_query193_00003", + "apositive_test_query193_00002", + "negative_test_query193_00006", + "negative_test_query193_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query194_00002", + "apositive_test_query194_00007", + "apositive_test_query124_00000", + "apositive_test_query162_00006", + "apositive_test_query194_00000", + "apositive_test_query42_00001", + "apositive_test_query194_00001", + "apositive_test_query114_00001", + "negative_test_query262_00007", + "apositive_test_query194_00006", + "negative_test_query262_00011", + "negative_test_query194_00003", + "negative_test_query162_00000", + "negative_test_query137_00011", + "negative_test_query62_00014", + "negative_test_query177_00007", + "negative_test_query95_00016", + "apositive_test_query254_00006", + "negative_test_query148_00009", + "negative_test_query89_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query195", + "ranked_ids": [ + "apositive_test_query195_00002", + "negative_test_query195_00002", + "negative_test_query195_00008", + "apositive_test_query195_00001", + "negative_test_query195_00013", + "apositive_test_query195_00005", + "negative_test_query195_00001", + "negative_test_query285_00000", + "negative_test_query222_00003", + "negative_test_query195_00005", + "negative_test_query109_00008", + "apositive_test_query251_00000", + "negative_test_query149_00012", + "negative_test_query321_00005", + "negative_test_query195_00004", + "apositive_test_query249_00012", + "negative_test_query321_00008", + "negative_test_query205_00006", + "negative_test_query166_00009", + "negative_test_query199_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query32_00011", + "apositive_test_query196_00003", + "apositive_test_query32_00012", + "apositive_test_query32_00004", + "negative_test_query14_00001", + "apositive_test_query32_00006", + "negative_test_query235_00008", + "negative_test_query196_00011", + "negative_test_query64_00005", + "negative_test_query32_00002", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query139_00011", + "negative_test_query173_00012", + "negative_test_query341_00008", + "negative_test_query325_00004", + "negative_test_query96_00013", + "negative_test_query179_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query197", + "ranked_ids": [ + "negative_test_query197_00010", + "negative_test_query163_00001", + "negative_test_query163_00002", + "apositive_test_query163_00000", + "apositive_test_query197_00000", + "apositive_test_query197_00002", + "negative_test_query197_00006", + "negative_test_query197_00016", + "negative_test_query197_00013", + "negative_test_query197_00004", + "negative_test_query321_00006", + "apositive_test_query189_00001", + "apositive_test_query163_00001", + "apositive_test_query197_00001", + "apositive_test_query34_00000", + "negative_test_query34_00012", + "negative_test_query28_00001", + "negative_test_query178_00008", + "negative_test_query163_00009", + "negative_test_query163_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query239_00011", + "negative_test_query239_00013", + "negative_test_query239_00001", + "negative_test_query239_00010", + "negative_test_query198_00002", + "negative_test_query198_00008", + "negative_test_query352_00002", + "negative_test_query239_00007", + "negative_test_query198_00010", + "negative_test_query198_00004", + "negative_test_query198_00000", + "negative_test_query112_00005", + "negative_test_query294_00002", + "negative_test_query343_00002", + "negative_test_query62_00016", + "negative_test_query293_00010", + "negative_test_query293_00011", + "negative_test_query219_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query199_00014", + "negative_test_query140_00014", + "negative_test_query54_00003", + "negative_test_query199_00001", + "negative_test_query199_00006", + "negative_test_query134_00006", + "negative_test_query199_00002", + "negative_test_query199_00015", + "negative_test_query199_00013", + "negative_test_query199_00005", + "negative_test_query221_00008", + "negative_test_query305_00007", + "negative_test_query139_00014", + "negative_test_query185_00002", + "negative_test_query350_00007", + "apositive_test_query43_00006", + "negative_test_query250_00006", + "negative_test_query213_00011", + "negative_test_query72_00005", + "apositive_test_query244_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query44_00008", + "negative_test_query2_00002", + "negative_test_query2_00016", + "negative_test_query190_00016", + "negative_test_query2_00009", + "negative_test_query15_00000", + "apositive_test_query57_00004", + "negative_test_query2_00011", + "negative_test_query352_00003", + "negative_test_query2_00012", + "negative_test_query2_00013", + "negative_test_query126_00014", + "negative_test_query125_00012", + "negative_test_query104_00007", + "apositive_test_query217_00006", + "negative_test_query197_00004", + "negative_test_query348_00005", + "negative_test_query343_00004", + "negative_test_query85_00007", + "negative_test_query2_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query154_00006", + "apositive_test_query20_00010", + "apositive_test_query20_00007", + "apositive_test_query20_00003", + "apositive_test_query154_00005", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "negative_test_query66_00002", + "apositive_test_query20_00009", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "negative_test_query154_00002", + "apositive_test_query356_00005", + "apositive_test_query23_00004", + "apositive_test_query33_00008", + "negative_test_query188_00000", + "negative_test_query220_00005", + "apositive_test_query33_00000", + "negative_test_query33_00008", + "apositive_test_query154_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query200_00004", + "negative_test_query337_00009", + "negative_test_query200_00000", + "negative_test_query200_00013", + "negative_test_query38_00005", + "negative_test_query200_00009", + "negative_test_query200_00015", + "negative_test_query264_00006", + "negative_test_query200_00012", + "negative_test_query200_00007", + "negative_test_query337_00007", + "negative_test_query200_00010", + "negative_test_query337_00004", + "negative_test_query337_00005", + "negative_test_query200_00005", + "negative_test_query200_00006", + "negative_test_query264_00007", + "negative_test_query88_00013", + "negative_test_query8_00008", + "negative_test_query303_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query201", + "ranked_ids": [ + "apositive_test_query201_00002", + "negative_test_query47_00012", + "negative_test_query47_00013", + "negative_test_query231_00012", + "apositive_test_query30_00005", + "negative_test_query71_00009", + "apositive_test_query0_00011", + "negative_test_query259_00002", + "negative_test_query0_00004", + "negative_test_query71_00013", + "negative_test_query209_00008", + "negative_test_query71_00005", + "negative_test_query71_00002", + "apositive_test_query201_00006", + "negative_test_query159_00009", + "negative_test_query65_00002", + "apositive_test_query30_00001", + "apositive_test_query20_00005", + "negative_test_query71_00000", + "apositive_test_query71_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query202_00004", + "negative_test_query202_00000", + "negative_test_query202_00001", + "negative_test_query202_00010", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query235_00001", + "negative_test_query202_00008", + "negative_test_query202_00006", + "negative_test_query202_00016", + "negative_test_query176_00011", + "negative_test_query202_00009", + "apositive_test_query206_00000", + "negative_test_query15_00010", + "apositive_test_query309_00000", + "negative_test_query312_00012", + "apositive_test_query302_00006", + "negative_test_query321_00018", + "apositive_test_query61_00003", + "negative_test_query188_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query203", + "ranked_ids": [ + "negative_test_query203_00006", + "apositive_test_query110_00003", + "negative_test_query203_00008", + "negative_test_query130_00001", + "negative_test_query203_00005", + "apositive_test_query130_00002", + "apositive_test_query284_00003", + "apositive_test_query30_00005", + "negative_test_query284_00014", + "apositive_test_query203_00009", + "apositive_test_query203_00007", + "negative_test_query209_00000", + "negative_test_query250_00009", + "apositive_test_query203_00003", + "apositive_test_query203_00000", + "negative_test_query250_00016", + "negative_test_query250_00011", + "apositive_test_query203_00004", + "negative_test_query250_00005", + "negative_test_query35_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.058823529411764705 + }, + "query_id": "test_query204", + "ranked_ids": [ + "apositive_test_query204_00000", + "apositive_test_query353_00005", + "negative_test_query204_00002", + "apositive_test_query353_00002", + "apositive_test_query353_00000", + "apositive_test_query204_00009", + "apositive_test_query97_00001", + "apositive_test_query204_00007", + "negative_test_query353_00003", + "negative_test_query204_00001", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "negative_test_query97_00007", + "apositive_test_query204_00002", + "apositive_test_query175_00004", + "apositive_test_query353_00011", + "negative_test_query204_00000", + "negative_test_query353_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query256_00000", + "negative_test_query205_00015", + "apositive_test_query205_00000", + "negative_test_query35_00007", + "negative_test_query31_00015", + "negative_test_query35_00008", + "negative_test_query228_00016", + "negative_test_query205_00007", + "negative_test_query256_00015", + "negative_test_query256_00007", + "apositive_test_query256_00002", + "negative_test_query115_00000", + "negative_test_query96_00011", + "negative_test_query341_00003", + "negative_test_query73_00000", + "negative_test_query73_00011", + "negative_test_query256_00006", + "negative_test_query205_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query206_00002", + "negative_test_query206_00007", + "negative_test_query206_00013", + "negative_test_query303_00015", + "negative_test_query206_00014", + "negative_test_query206_00000", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00001", + "negative_test_query206_00005", + "negative_test_query303_00003", + "apositive_test_query348_00001", + "negative_test_query299_00001", + "negative_test_query70_00011", + "negative_test_query206_00003", + "negative_test_query2_00016", + "negative_test_query206_00018", + "negative_test_query299_00002", + "negative_test_query64_00016", + "negative_test_query206_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00004", + "apositive_test_query207_00005", + "apositive_test_query207_00002", + "apositive_test_query207_00000", + "negative_test_query207_00011", + "negative_test_query207_00002", + "negative_test_query207_00012", + "apositive_test_query207_00001", + "negative_test_query207_00005", + "negative_test_query207_00008", + "negative_test_query207_00006", + "negative_test_query207_00009", + "negative_test_query207_00001", + "negative_test_query207_00000", + "negative_test_query207_00007", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query207_00003", + "negative_test_query207_00004", + "negative_test_query37_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "apositive_test_query208_00000", + "negative_test_query128_00003", + "negative_test_query208_00010", + "negative_test_query208_00017", + "negative_test_query211_00014", + "negative_test_query208_00000", + "negative_test_query208_00016", + "negative_test_query208_00018", + "apositive_test_query128_00002", + "negative_test_query208_00008", + "negative_test_query208_00006", + "negative_test_query208_00004", + "negative_test_query61_00012", + "negative_test_query208_00012", + "negative_test_query208_00009", + "negative_test_query208_00003", + "negative_test_query73_00008", + "negative_test_query128_00001", + "apositive_test_query128_00007", + "negative_test_query256_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query209", + "ranked_ids": [ + "apositive_test_query209_00001", + "apositive_test_query37_00001", + "negative_test_query209_00007", + "negative_test_query102_00010", + "negative_test_query231_00006", + "apositive_test_query26_00007", + "negative_test_query223_00003", + "negative_test_query26_00002", + "apositive_test_query158_00004", + "apositive_test_query158_00009", + "negative_test_query102_00016", + "negative_test_query209_00001", + "negative_test_query223_00007", + "negative_test_query26_00004", + "negative_test_query313_00012", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query347_00004", + "apositive_test_query119_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "apositive_test_query21_00000", + "negative_test_query128_00006", + "apositive_test_query128_00003", + "negative_test_query21_00010", + "negative_test_query234_00007", + "apositive_test_query128_00002", + "negative_test_query128_00004", + "negative_test_query21_00005", + "negative_test_query21_00009", + "negative_test_query330_00000", + "negative_test_query224_00015", + "negative_test_query10_00008", + "negative_test_query8_00006", + "negative_test_query208_00000", + "negative_test_query208_00017", + "apositive_test_query128_00004", + "negative_test_query21_00012", + "negative_test_query234_00006", + "negative_test_query21_00001", + "negative_test_query350_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query335_00004", + "negative_test_query199_00015", + "negative_test_query210_00013", + "negative_test_query156_00015", + "negative_test_query210_00015", + "negative_test_query300_00000", + "negative_test_query321_00013", + "negative_test_query148_00014", + "negative_test_query148_00008", + "negative_test_query202_00014", + "negative_test_query17_00002", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query210_00007", + "negative_test_query288_00006", + "negative_test_query210_00006", + "negative_test_query288_00007", + "negative_test_query210_00010", + "negative_test_query148_00007", + "negative_test_query210_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query211", + "ranked_ids": [ + "negative_test_query211_00015", + "apositive_test_query211_00001", + "apositive_test_query101_00000", + "negative_test_query211_00000", + "negative_test_query211_00012", + "negative_test_query137_00012", + "negative_test_query159_00006", + "negative_test_query32_00005", + "apositive_test_query163_00004", + "apositive_test_query198_00000", + "negative_test_query239_00014", + "negative_test_query100_00012", + "negative_test_query195_00007", + "negative_test_query293_00013", + "apositive_test_query176_00000", + "negative_test_query83_00010", + "negative_test_query189_00008", + "apositive_test_query162_00010", + "negative_test_query167_00015", + "negative_test_query202_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query212", + "ranked_ids": [ + "negative_test_query167_00016", + "apositive_test_query326_00000", + "apositive_test_query326_00004", + "apositive_test_query212_00004", + "apositive_test_query89_00004", + "negative_test_query77_00004", + "negative_test_query183_00012", + "negative_test_query282_00010", + "apositive_test_query11_00007", + "apositive_test_query71_00001", + "negative_test_query126_00017", + "negative_test_query190_00017", + "negative_test_query167_00017", + "negative_test_query183_00009", + "apositive_test_query98_00000", + "apositive_test_query244_00007", + "negative_test_query282_00009", + "negative_test_query346_00006", + "negative_test_query344_00008", + "apositive_test_query66_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00004", + "apositive_test_query213_00000", + "apositive_test_query213_00003", + "apositive_test_query213_00002", + "apositive_test_query259_00001", + "negative_test_query213_00001", + "negative_test_query213_00011", + "apositive_test_query158_00001", + "negative_test_query313_00017", + "apositive_test_query305_00003", + "apositive_test_query36_00002", + "negative_test_query138_00016", + "negative_test_query213_00005", + "negative_test_query112_00003", + "negative_test_query137_00015", + "negative_test_query224_00015", + "negative_test_query213_00004", + "apositive_test_query259_00000", + "negative_test_query138_00001", + "negative_test_query357_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "apositive_test_query214_00000", + "negative_test_query214_00013", + "apositive_test_query234_00001", + "negative_test_query214_00010", + "negative_test_query214_00001", + "negative_test_query214_00003", + "negative_test_query214_00004", + "apositive_test_query160_00009", + "apositive_test_query75_00003", + "negative_test_query214_00015", + "negative_test_query214_00016", + "apositive_test_query323_00003", + "apositive_test_query323_00002", + "negative_test_query9_00008", + "negative_test_query324_00012", + "negative_test_query17_00013", + "apositive_test_query89_00000", + "negative_test_query324_00002", + "apositive_test_query17_00001", + "negative_test_query103_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00010", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query215_00011", + "apositive_test_query80_00005", + "negative_test_query215_00004", + "apositive_test_query215_00000", + "negative_test_query257_00001", + "negative_test_query215_00008", + "negative_test_query215_00012", + "negative_test_query215_00006", + "negative_test_query215_00018", + "negative_test_query215_00007", + "negative_test_query215_00015", + "negative_test_query215_00009", + "negative_test_query215_00001", + "negative_test_query295_00004", + "negative_test_query215_00017", + "negative_test_query293_00013", + "negative_test_query263_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.8 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query216_00000", + "negative_test_query216_00000", + "apositive_test_query216_00001", + "apositive_test_query216_00002", + "apositive_test_query216_00004", + "negative_test_query216_00007", + "negative_test_query216_00014", + "negative_test_query216_00006", + "negative_test_query216_00002", + "negative_test_query216_00008", + "negative_test_query246_00016", + "apositive_test_query154_00005", + "negative_test_query112_00002", + "negative_test_query44_00002", + "negative_test_query216_00005", + "negative_test_query216_00009", + "apositive_test_query20_00001", + "apositive_test_query33_00004", + "apositive_test_query330_00002", + "apositive_test_query182_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "negative_test_query217_00004", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00000", + "negative_test_query217_00011", + "apositive_test_query307_00010", + "negative_test_query28_00002", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query72_00005", + "apositive_test_query213_00004", + "negative_test_query112_00007", + "negative_test_query230_00002", + "negative_test_query53_00012", + "negative_test_query76_00003", + "apositive_test_query107_00001", + "apositive_test_query160_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query218", + "ranked_ids": [ + "negative_test_query134_00003", + "apositive_test_query218_00005", + "apositive_test_query218_00004", + "negative_test_query218_00000", + "apositive_test_query218_00003", + "negative_test_query218_00012", + "negative_test_query218_00009", + "apositive_test_query134_00000", + "negative_test_query58_00010", + "apositive_test_query314_00015", + "negative_test_query134_00009", + "negative_test_query218_00003", + "negative_test_query274_00008", + "negative_test_query43_00006", + "negative_test_query43_00010", + "apositive_test_query218_00000", + "apositive_test_query60_00012", + "negative_test_query304_00005", + "negative_test_query218_00002", + "negative_test_query165_00013" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "apositive_test_query219_00000", + "negative_test_query219_00016", + "apositive_test_query219_00001", + "negative_test_query219_00014", + "negative_test_query187_00015", + "negative_test_query59_00000", + "negative_test_query219_00008", + "negative_test_query219_00012", + "negative_test_query219_00015", + "apositive_test_query177_00004", + "negative_test_query219_00017", + "negative_test_query305_00005", + "negative_test_query219_00011", + "negative_test_query227_00007", + "negative_test_query171_00005", + "negative_test_query24_00007", + "negative_test_query308_00006", + "negative_test_query183_00016", + "negative_test_query219_00005", + "negative_test_query165_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "apositive_test_query22_00000", + "negative_test_query22_00002", + "negative_test_query22_00008", + "negative_test_query22_00015", + "negative_test_query22_00004", + "negative_test_query22_00003", + "negative_test_query22_00018", + "negative_test_query22_00001", + "negative_test_query22_00012", + "negative_test_query22_00005", + "negative_test_query22_00009", + "negative_test_query22_00013", + "negative_test_query22_00000", + "negative_test_query22_00017", + "negative_test_query22_00007", + "negative_test_query22_00006", + "negative_test_query66_00004", + "negative_test_query22_00016", + "apositive_test_query162_00007", + "apositive_test_query177_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.5 + }, + "query_id": "test_query220", + "ranked_ids": [ + "apositive_test_query220_00003", + "negative_test_query344_00007", + "negative_test_query188_00013", + "apositive_test_query220_00001", + "apositive_test_query220_00004", + "apositive_test_query220_00000", + "negative_test_query220_00011", + "negative_test_query188_00004", + "negative_test_query154_00009", + "negative_test_query188_00009", + "negative_test_query188_00008", + "negative_test_query344_00013", + "apositive_test_query188_00000", + "negative_test_query20_00002", + "negative_test_query220_00004", + "negative_test_query188_00014", + "negative_test_query188_00007", + "apositive_test_query20_00009", + "apositive_test_query146_00009", + "negative_test_query161_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query142_00004", + "negative_test_query277_00010", + "negative_test_query65_00013", + "apositive_test_query306_00006", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "negative_test_query65_00007", + "negative_test_query184_00001", + "negative_test_query12_00005", + "apositive_test_query180_00002", + "negative_test_query15_00005", + "apositive_test_query174_00006", + "negative_test_query278_00004", + "apositive_test_query244_00008", + "negative_test_query180_00011", + "apositive_test_query162_00005", + "negative_test_query244_00006", + "negative_test_query171_00008", + "apositive_test_query92_00003", + "negative_test_query39_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query222", + "ranked_ids": [ + "apositive_test_query275_00001", + "apositive_test_query289_00002", + "negative_test_query289_00006", + "negative_test_query290_00004", + "apositive_test_query275_00006", + "negative_test_query271_00015", + "apositive_test_query275_00000", + "negative_test_query271_00004", + "negative_test_query275_00002", + "apositive_test_query275_00005", + "apositive_test_query275_00008", + "apositive_test_query302_00000", + "negative_test_query275_00004", + "apositive_test_query275_00004", + "apositive_test_query222_00000", + "negative_test_query222_00017", + "apositive_test_query222_00001", + "negative_test_query222_00014", + "negative_test_query222_00006", + "negative_test_query222_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query47_00001", + "negative_test_query359_00008", + "negative_test_query323_00005", + "apositive_test_query260_00003", + "negative_test_query109_00006", + "negative_test_query6_00007", + "negative_test_query259_00006", + "apositive_test_query259_00003", + "negative_test_query359_00014", + "apositive_test_query201_00004", + "negative_test_query47_00002", + "negative_test_query51_00003", + "negative_test_query345_00002", + "negative_test_query164_00011", + "apositive_test_query259_00008", + "apositive_test_query259_00011", + "negative_test_query47_00007", + "negative_test_query107_00004", + "apositive_test_query107_00005", + "apositive_test_query201_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query224", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query224_00012", + "negative_test_query350_00016", + "negative_test_query224_00014", + "negative_test_query98_00018", + "negative_test_query149_00008", + "negative_test_query224_00011", + "negative_test_query224_00010", + "apositive_test_query256_00001", + "apositive_test_query179_00002", + "negative_test_query350_00017", + "apositive_test_query224_00003", + "negative_test_query205_00009", + "apositive_test_query83_00000", + "negative_test_query212_00013", + "negative_test_query304_00001", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "negative_test_query350_00015", + "negative_test_query224_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00001", + "negative_test_query145_00000", + "apositive_test_query225_00003", + "apositive_test_query225_00007", + "apositive_test_query225_00009", + "apositive_test_query225_00000", + "apositive_test_query225_00005", + "negative_test_query343_00009", + "negative_test_query149_00011", + "apositive_test_query225_00004", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query183_00015", + "apositive_test_query145_00005", + "negative_test_query225_00008", + "apositive_test_query225_00002", + "negative_test_query161_00004", + "negative_test_query225_00000", + "negative_test_query54_00002", + "apositive_test_query169_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "error": "ranking must be a permutation of 1..20.", + "error_type": "RerankParseError", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query226", + "ranked_ids": [], + "valid": false + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00003", + "negative_test_query227_00004", + "negative_test_query70_00011", + "negative_test_query227_00013", + "negative_test_query227_00000", + "negative_test_query227_00006", + "negative_test_query227_00012", + "apositive_test_query70_00001", + "apositive_test_query227_00000", + "negative_test_query227_00014", + "negative_test_query227_00002", + "apositive_test_query227_00001", + "negative_test_query235_00006", + "negative_test_query59_00004", + "negative_test_query227_00005", + "negative_test_query321_00004", + "negative_test_query227_00011", + "negative_test_query227_00010", + "negative_test_query115_00007", + "apositive_test_query348_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query228", + "ranked_ids": [ + "negative_test_query228_00012", + "negative_test_query165_00011", + "apositive_test_query228_00000", + "negative_test_query228_00003", + "negative_test_query228_00011", + "negative_test_query228_00014", + "negative_test_query228_00004", + "negative_test_query228_00002", + "negative_test_query322_00001", + "negative_test_query216_00013", + "negative_test_query228_00000", + "negative_test_query73_00016", + "negative_test_query2_00001", + "apositive_test_query254_00000", + "apositive_test_query318_00001", + "apositive_test_query322_00003", + "apositive_test_query322_00007", + "apositive_test_query333_00001", + "apositive_test_query73_00001", + "negative_test_query179_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query229_00002", + "apositive_test_query229_00003", + "apositive_test_query229_00000", + "apositive_test_query229_00007", + "negative_test_query229_00008", + "apositive_test_query158_00006", + "apositive_test_query146_00004", + "negative_test_query229_00005", + "negative_test_query229_00007", + "apositive_test_query158_00002", + "negative_test_query229_00006", + "negative_test_query229_00000", + "negative_test_query229_00009", + "negative_test_query0_00001", + "apositive_test_query94_00007", + "negative_test_query94_00010", + "negative_test_query44_00005", + "negative_test_query355_00001", + "apositive_test_query69_00009", + "apositive_test_query18_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.6 + }, + "query_id": "test_query23", + "ranked_ids": [ + "negative_test_query66_00010", + "apositive_test_query23_00002", + "apositive_test_query23_00000", + "apositive_test_query23_00001", + "negative_test_query23_00014", + "negative_test_query356_00008", + "negative_test_query259_00007", + "apositive_test_query52_00001", + "apositive_test_query52_00000", + "apositive_test_query52_00002", + "apositive_test_query52_00004", + "apositive_test_query54_00004", + "apositive_test_query225_00000", + "apositive_test_query54_00007", + "negative_test_query292_00013", + "apositive_test_query146_00015", + "negative_test_query239_00012", + "apositive_test_query39_00002", + "negative_test_query307_00002", + "negative_test_query133_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query230_00007", + "negative_test_query230_00010", + "negative_test_query230_00000", + "apositive_test_query230_00003", + "apositive_test_query3_00001", + "negative_test_query329_00000", + "negative_test_query269_00010", + "negative_test_query23_00012", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "apositive_test_query3_00006", + "negative_test_query99_00003", + "apositive_test_query3_00009", + "negative_test_query3_00004", + "negative_test_query3_00005", + "negative_test_query280_00006", + "apositive_test_query280_00002", + "negative_test_query280_00004", + "negative_test_query3_00000", + "negative_test_query3_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query231", + "ranked_ids": [ + "apositive_test_query130_00000", + "negative_test_query9_00017", + "negative_test_query221_00006", + "negative_test_query345_00011", + "negative_test_query75_00000", + "apositive_test_query0_00001", + "apositive_test_query231_00003", + "negative_test_query213_00004", + "negative_test_query130_00000", + "negative_test_query201_00011", + "negative_test_query359_00008", + "negative_test_query47_00005", + "apositive_test_query201_00002", + "negative_test_query231_00001", + "apositive_test_query130_00004", + "negative_test_query130_00010", + "negative_test_query47_00006", + "negative_test_query306_00006", + "apositive_test_query201_00006", + "apositive_test_query119_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "apositive_test_query232_00000", + "negative_test_query232_00013", + "negative_test_query232_00014", + "negative_test_query232_00000", + "negative_test_query232_00011", + "negative_test_query232_00003", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query232_00008", + "negative_test_query232_00015", + "negative_test_query232_00018", + "negative_test_query232_00002", + "negative_test_query232_00004", + "negative_test_query232_00006", + "negative_test_query232_00010", + "negative_test_query232_00001", + "negative_test_query232_00012", + "negative_test_query232_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query233", + "ranked_ids": [ + "apositive_test_query233_00002", + "apositive_test_query233_00000", + "negative_test_query2_00010", + "negative_test_query2_00014", + "negative_test_query233_00010", + "negative_test_query233_00001", + "negative_test_query233_00004", + "negative_test_query233_00006", + "negative_test_query233_00015", + "apositive_test_query2_00000", + "negative_test_query190_00007", + "negative_test_query246_00001", + "negative_test_query176_00017", + "negative_test_query130_00001", + "apositive_test_query275_00003", + "negative_test_query263_00009", + "negative_test_query130_00008", + "negative_test_query219_00015", + "negative_test_query233_00005", + "negative_test_query233_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00015", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "apositive_test_query304_00001", + "apositive_test_query304_00003", + "negative_test_query234_00005", + "negative_test_query304_00012", + "negative_test_query304_00001", + "negative_test_query214_00008", + "negative_test_query304_00005", + "apositive_test_query43_00005", + "negative_test_query304_00013", + "negative_test_query304_00008", + "apositive_test_query218_00002", + "negative_test_query234_00011", + "negative_test_query214_00006", + "negative_test_query234_00010", + "negative_test_query304_00009", + "negative_test_query285_00011", + "negative_test_query10_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00002", + "apositive_test_query32_00006", + "negative_test_query235_00008", + "apositive_test_query32_00000", + "negative_test_query196_00000", + "negative_test_query235_00000", + "apositive_test_query32_00004", + "apositive_test_query235_00003", + "negative_test_query235_00014", + "apositive_test_query235_00000", + "apositive_test_query235_00001", + "negative_test_query24_00017", + "negative_test_query219_00013", + "negative_test_query235_00009", + "negative_test_query235_00011", + "negative_test_query235_00013", + "negative_test_query235_00004", + "negative_test_query235_00015", + "negative_test_query61_00001", + "negative_test_query173_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "negative_test_query236_00000", + "negative_test_query236_00014", + "negative_test_query236_00009", + "negative_test_query236_00012", + "negative_test_query279_00012", + "apositive_test_query279_00002", + "negative_test_query236_00006", + "negative_test_query236_00013", + "negative_test_query236_00004", + "negative_test_query236_00011", + "negative_test_query279_00003", + "negative_test_query279_00005", + "negative_test_query22_00016", + "negative_test_query236_00005", + "negative_test_query236_00007", + "negative_test_query236_00015", + "negative_test_query236_00018", + "negative_test_query222_00000", + "negative_test_query236_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "apositive_test_query237_00001", + "negative_test_query237_00006", + "negative_test_query237_00004", + "negative_test_query152_00008", + "apositive_test_query152_00001", + "apositive_test_query237_00002", + "negative_test_query152_00003", + "negative_test_query237_00002", + "apositive_test_query237_00000", + "negative_test_query312_00001", + "negative_test_query237_00000", + "apositive_test_query152_00004", + "negative_test_query237_00005", + "negative_test_query237_00009", + "negative_test_query157_00010", + "negative_test_query237_00007", + "negative_test_query237_00001", + "negative_test_query290_00010", + "apositive_test_query157_00001", + "negative_test_query261_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "negative_test_query102_00007", + "negative_test_query108_00007", + "negative_test_query13_00003", + "negative_test_query142_00011", + "apositive_test_query146_00013", + "negative_test_query56_00008", + "negative_test_query56_00000", + "apositive_test_query142_00003", + "negative_test_query223_00005", + "negative_test_query46_00012", + "negative_test_query203_00003", + "apositive_test_query306_00003", + "apositive_test_query276_00008", + "negative_test_query111_00013", + "negative_test_query123_00001", + "negative_test_query111_00008", + "negative_test_query238_00011", + "apositive_test_query29_00001", + "apositive_test_query29_00007", + "negative_test_query29_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "negative_test_query239_00015", + "negative_test_query239_00003", + "apositive_test_query239_00000", + "negative_test_query239_00000", + "negative_test_query239_00007", + "negative_test_query239_00001", + "negative_test_query239_00010", + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query239_00011", + "negative_test_query239_00013", + "negative_test_query198_00002", + "negative_test_query239_00014", + "negative_test_query198_00008", + "negative_test_query352_00002", + "negative_test_query61_00004", + "negative_test_query163_00004", + "negative_test_query183_00015", + "negative_test_query268_00005", + "negative_test_query167_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "negative_test_query24_00000", + "negative_test_query24_00008", + "negative_test_query24_00009", + "negative_test_query250_00008", + "negative_test_query52_00012", + "negative_test_query24_00002", + "negative_test_query24_00012", + "apositive_test_query106_00002", + "apositive_test_query24_00000", + "negative_test_query128_00002", + "apositive_test_query226_00004", + "negative_test_query15_00000", + "negative_test_query24_00004", + "negative_test_query252_00005", + "negative_test_query198_00006", + "negative_test_query190_00016", + "negative_test_query24_00001", + "apositive_test_query57_00004", + "negative_test_query323_00000", + "negative_test_query356_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "negative_test_query240_00012", + "negative_test_query240_00006", + "apositive_test_query240_00000", + "negative_test_query240_00016", + "negative_test_query240_00018", + "negative_test_query240_00004", + "negative_test_query240_00010", + "negative_test_query240_00002", + "negative_test_query240_00015", + "negative_test_query240_00011", + "negative_test_query246_00002", + "apositive_test_query311_00000", + "negative_test_query27_00010", + "negative_test_query240_00007", + "negative_test_query293_00004", + "negative_test_query6_00000", + "negative_test_query282_00013", + "negative_test_query240_00003", + "negative_test_query240_00017", + "negative_test_query297_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "apositive_test_query241_00000", + "negative_test_query241_00005", + "negative_test_query241_00010", + "negative_test_query241_00016", + "negative_test_query241_00017", + "negative_test_query241_00002", + "negative_test_query241_00008", + "negative_test_query173_00013", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query206_00011", + "negative_test_query256_00000", + "negative_test_query258_00000", + "negative_test_query53_00000", + "negative_test_query206_00009", + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query298_00014", + "negative_test_query191_00014", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query291_00000", + "negative_test_query242_00000", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "apositive_test_query242_00000", + "apositive_test_query291_00005", + "apositive_test_query242_00008", + "apositive_test_query242_00013", + "apositive_test_query291_00008", + "apositive_test_query247_00005", + "apositive_test_query242_00005", + "apositive_test_query291_00002", + "negative_test_query242_00004", + "negative_test_query291_00000", + "apositive_test_query242_00001", + "negative_test_query291_00004", + "apositive_test_query242_00010", + "negative_test_query291_00003", + "negative_test_query291_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query243_00000", + "apositive_test_query243_00002", + "negative_test_query274_00014", + "apositive_test_query243_00005", + "apositive_test_query243_00003", + "apositive_test_query243_00012", + "apositive_test_query243_00011", + "negative_test_query243_00002", + "apositive_test_query160_00003", + "negative_test_query64_00003", + "negative_test_query192_00010", + "negative_test_query192_00007", + "negative_test_query243_00000", + "apositive_test_query173_00004", + "negative_test_query205_00007", + "negative_test_query303_00005", + "negative_test_query289_00000", + "negative_test_query250_00013", + "negative_test_query184_00004", + "negative_test_query223_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query244", + "ranked_ids": [ + "apositive_test_query244_00000", + "apositive_test_query244_00001", + "negative_test_query244_00001", + "negative_test_query244_00005", + "apositive_test_query244_00002", + "negative_test_query244_00008", + "negative_test_query244_00009", + "apositive_test_query244_00004", + "negative_test_query244_00002", + "negative_test_query244_00003", + "negative_test_query244_00004", + "negative_test_query244_00000", + "negative_test_query244_00010", + "apositive_test_query244_00006", + "apositive_test_query244_00007", + "apositive_test_query244_00005", + "apositive_test_query42_00002", + "negative_test_query126_00015", + "apositive_test_query23_00003", + "apositive_test_query356_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query245_00003", + "negative_test_query245_00010", + "negative_test_query64_00013", + "negative_test_query135_00013", + "negative_test_query245_00000", + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query245_00000", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query245_00001", + "negative_test_query202_00000", + "negative_test_query121_00002", + "negative_test_query293_00003", + "negative_test_query122_00002", + "negative_test_query159_00003", + "negative_test_query336_00002", + "negative_test_query202_00004", + "apositive_test_query141_00002", + "apositive_test_query141_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query246_00002", + "negative_test_query246_00016", + "negative_test_query6_00014", + "negative_test_query190_00003", + "negative_test_query246_00001", + "negative_test_query246_00012", + "negative_test_query252_00003", + "negative_test_query311_00005", + "apositive_test_query68_00002", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query135_00017", + "negative_test_query6_00006", + "apositive_test_query0_00001", + "negative_test_query198_00012", + "negative_test_query44_00002", + "apositive_test_query216_00000", + "negative_test_query317_00014", + "negative_test_query293_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "negative_test_query291_00003", + "apositive_test_query242_00001", + "apositive_test_query291_00000", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "apositive_test_query242_00002", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query242_00000", + "negative_test_query291_00000", + "negative_test_query247_00005", + "apositive_test_query247_00005", + "negative_test_query291_00005", + "apositive_test_query242_00012", + "negative_test_query247_00003", + "apositive_test_query242_00010", + "negative_test_query186_00010", + "negative_test_query291_00001", + "negative_test_query247_00011", + "negative_test_query105_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00008", + "apositive_test_query248_00002", + "apositive_test_query248_00007", + "negative_test_query248_00002", + "negative_test_query248_00011", + "apositive_test_query248_00003", + "negative_test_query248_00010", + "apositive_test_query248_00005", + "negative_test_query251_00017", + "negative_test_query248_00006", + "negative_test_query248_00004", + "apositive_test_query29_00004", + "apositive_test_query29_00000", + "negative_test_query29_00001", + "negative_test_query205_00017", + "negative_test_query248_00000", + "apositive_test_query248_00001", + "apositive_test_query29_00005", + "negative_test_query248_00009", + "negative_test_query248_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query249", + "ranked_ids": [ + "apositive_test_query249_00004", + "apositive_test_query249_00000", + "negative_test_query249_00001", + "negative_test_query156_00002", + "apositive_test_query281_00001", + "negative_test_query193_00001", + "apositive_test_query204_00003", + "apositive_test_query353_00004", + "negative_test_query122_00016", + "negative_test_query156_00007", + "negative_test_query72_00017", + "negative_test_query326_00005", + "negative_test_query325_00015", + "apositive_test_query177_00004", + "negative_test_query210_00010", + "negative_test_query302_00007", + "negative_test_query301_00013", + "negative_test_query71_00006", + "apositive_test_query112_00000", + "negative_test_query278_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query25", + "ranked_ids": [ + "apositive_test_query25_00000", + "apositive_test_query25_00006", + "apositive_test_query25_00008", + "negative_test_query25_00007", + "negative_test_query143_00000", + "negative_test_query45_00014", + "apositive_test_query25_00003", + "apositive_test_query25_00005", + "negative_test_query25_00005", + "apositive_test_query25_00007", + "apositive_test_query25_00009", + "apositive_test_query25_00001", + "apositive_test_query25_00010", + "apositive_test_query25_00004", + "negative_test_query25_00001", + "negative_test_query30_00003", + "negative_test_query25_00002", + "negative_test_query346_00006", + "negative_test_query9_00001", + "negative_test_query328_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query250", + "ranked_ids": [ + "apositive_test_query250_00000", + "apositive_test_query203_00000", + "negative_test_query250_00009", + "negative_test_query250_00011", + "negative_test_query79_00013", + "apositive_test_query203_00006", + "negative_test_query57_00009", + "negative_test_query111_00015", + "negative_test_query250_00013", + "negative_test_query186_00007", + "negative_test_query184_00015", + "negative_test_query159_00004", + "negative_test_query250_00001", + "negative_test_query102_00010", + "negative_test_query159_00014", + "negative_test_query64_00005", + "negative_test_query250_00003", + "negative_test_query192_00009", + "negative_test_query101_00008", + "negative_test_query61_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query251", + "ranked_ids": [ + "apositive_test_query251_00001", + "negative_test_query189_00011", + "negative_test_query234_00006", + "negative_test_query251_00003", + "negative_test_query251_00004", + "negative_test_query298_00003", + "negative_test_query14_00000", + "negative_test_query347_00014", + "apositive_test_query29_00002", + "negative_test_query56_00005", + "negative_test_query322_00002", + "negative_test_query326_00000", + "apositive_test_query134_00003", + "negative_test_query143_00001", + "negative_test_query160_00001", + "negative_test_query8_00009", + "negative_test_query268_00011", + "apositive_test_query347_00000", + "negative_test_query170_00008", + "negative_test_query269_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query252", + "ranked_ids": [ + "negative_test_query252_00009", + "apositive_test_query252_00000", + "negative_test_query252_00012", + "negative_test_query212_00000", + "apositive_test_query110_00001", + "apositive_test_query252_00002", + "negative_test_query252_00001", + "negative_test_query252_00007", + "negative_test_query14_00006", + "negative_test_query321_00004", + "negative_test_query318_00006", + "negative_test_query252_00011", + "negative_test_query44_00011", + "apositive_test_query125_00001", + "negative_test_query252_00010", + "negative_test_query151_00000", + "negative_test_query126_00001", + "apositive_test_query252_00003", + "negative_test_query16_00006", + "negative_test_query330_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3 + }, + "query_id": "test_query253", + "ranked_ids": [ + "apositive_test_query253_00003", + "apositive_test_query253_00004", + "apositive_test_query253_00007", + "apositive_test_query13_00012", + "negative_test_query194_00001", + "apositive_test_query194_00008", + "apositive_test_query74_00002", + "apositive_test_query162_00008", + "negative_test_query177_00000", + "negative_test_query253_00005", + "negative_test_query226_00013", + "negative_test_query253_00004", + "apositive_test_query186_00004", + "apositive_test_query349_00000", + "apositive_test_query74_00003", + "negative_test_query124_00002", + "negative_test_query172_00008", + "negative_test_query124_00001", + "negative_test_query253_00003", + "negative_test_query95_00017" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query254", + "ranked_ids": [ + "apositive_test_query254_00004", + "apositive_test_query254_00000", + "negative_test_query350_00000", + "negative_test_query270_00010", + "apositive_test_query254_00005", + "apositive_test_query254_00003", + "apositive_test_query254_00001", + "apositive_test_query254_00002", + "apositive_test_query254_00006", + "negative_test_query254_00004", + "negative_test_query35_00015", + "apositive_test_query254_00010", + "negative_test_query136_00003", + "negative_test_query254_00008", + "negative_test_query228_00016", + "negative_test_query21_00009", + "apositive_test_query333_00004", + "negative_test_query21_00005", + "negative_test_query128_00004", + "negative_test_query316_00013" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "negative_test_query278_00015", + "apositive_test_query255_00000", + "negative_test_query255_00006", + "negative_test_query278_00000", + "negative_test_query208_00008", + "apositive_test_query278_00000", + "apositive_test_query25_00004", + "negative_test_query326_00006", + "negative_test_query89_00004", + "negative_test_query246_00003", + "negative_test_query255_00017", + "apositive_test_query322_00001", + "negative_test_query255_00010", + "negative_test_query255_00018", + "negative_test_query110_00001", + "negative_test_query255_00005", + "negative_test_query255_00004", + "negative_test_query255_00013", + "negative_test_query255_00000", + "apositive_test_query313_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query256", + "ranked_ids": [ + "apositive_test_query256_00001", + "negative_test_query256_00011", + "apositive_test_query256_00000", + "negative_test_query256_00008", + "negative_test_query256_00003", + "apositive_test_query256_00003", + "apositive_test_query119_00000", + "negative_test_query279_00000", + "apositive_test_query256_00002", + "negative_test_query319_00001", + "negative_test_query82_00001", + "negative_test_query5_00010", + "negative_test_query135_00016", + "apositive_test_query142_00002", + "negative_test_query105_00008", + "negative_test_query183_00008", + "negative_test_query7_00009", + "negative_test_query347_00007", + "apositive_test_query43_00001", + "apositive_test_query216_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.5 + }, + "query_id": "test_query257", + "ranked_ids": [ + "apositive_test_query257_00001", + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00000", + "apositive_test_query257_00002", + "apositive_test_query257_00004", + "apositive_test_query257_00005", + "apositive_test_query69_00007", + "apositive_test_query286_00014", + "apositive_test_query257_00003", + "negative_test_query52_00005", + "negative_test_query257_00005", + "negative_test_query257_00007", + "negative_test_query52_00007", + "negative_test_query52_00004", + "negative_test_query54_00005", + "apositive_test_query151_00000", + "apositive_test_query52_00001", + "apositive_test_query54_00003", + "negative_test_query52_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query319_00000", + "negative_test_query82_00000", + "negative_test_query320_00002", + "negative_test_query181_00002", + "apositive_test_query301_00002", + "negative_test_query259_00001", + "negative_test_query228_00005", + "negative_test_query206_00016", + "negative_test_query311_00002", + "negative_test_query162_00004", + "negative_test_query167_00007", + "negative_test_query38_00002", + "negative_test_query144_00010", + "negative_test_query101_00001", + "negative_test_query169_00016", + "negative_test_query171_00002", + "negative_test_query173_00010", + "negative_test_query339_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query259", + "ranked_ids": [ + "negative_test_query259_00002", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "apositive_test_query259_00008", + "apositive_test_query231_00001", + "apositive_test_query345_00000", + "apositive_test_query259_00003", + "apositive_test_query259_00006", + "apositive_test_query54_00011", + "negative_test_query259_00001", + "apositive_test_query259_00007", + "negative_test_query259_00006", + "negative_test_query213_00004", + "negative_test_query359_00003", + "apositive_test_query0_00007", + "negative_test_query250_00015", + "negative_test_query359_00012", + "negative_test_query111_00010", + "negative_test_query30_00009", + "negative_test_query47_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "apositive_test_query26_00005", + "negative_test_query26_00008", + "negative_test_query223_00002", + "apositive_test_query158_00003", + "apositive_test_query186_00003", + "apositive_test_query26_00004", + "negative_test_query105_00013", + "negative_test_query158_00006", + "negative_test_query164_00004", + "negative_test_query247_00010", + "negative_test_query26_00003", + "negative_test_query119_00006", + "negative_test_query158_00009", + "negative_test_query26_00009", + "negative_test_query26_00001", + "negative_test_query26_00005", + "apositive_test_query102_00000", + "negative_test_query102_00015", + "negative_test_query334_00016", + "negative_test_query238_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query252_00004", + "apositive_test_query6_00002", + "negative_test_query6_00004", + "negative_test_query6_00011", + "apositive_test_query6_00000", + "negative_test_query125_00001", + "apositive_test_query6_00001", + "negative_test_query6_00002", + "apositive_test_query186_00001", + "negative_test_query6_00009", + "negative_test_query2_00004", + "negative_test_query6_00005", + "negative_test_query260_00009", + "negative_test_query260_00011", + "negative_test_query260_00002", + "apositive_test_query109_00000", + "apositive_test_query109_00001", + "negative_test_query6_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query261", + "ranked_ids": [ + "negative_test_query261_00006", + "negative_test_query116_00004", + "negative_test_query261_00013", + "negative_test_query261_00005", + "negative_test_query261_00010", + "negative_test_query261_00008", + "negative_test_query261_00002", + "negative_test_query261_00003", + "negative_test_query261_00000", + "negative_test_query261_00007", + "negative_test_query261_00004", + "negative_test_query261_00001", + "negative_test_query261_00011", + "apositive_test_query261_00002", + "apositive_test_query261_00003", + "apositive_test_query29_00001", + "negative_test_query261_00009", + "apositive_test_query261_00000", + "negative_test_query2_00015", + "negative_test_query118_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query262", + "ranked_ids": [ + "apositive_test_query262_00003", + "negative_test_query312_00018", + "apositive_test_query262_00000", + "negative_test_query106_00010", + "negative_test_query266_00003", + "negative_test_query262_00000", + "negative_test_query301_00005", + "negative_test_query106_00005", + "apositive_test_query177_00008", + "apositive_test_query194_00002", + "apositive_test_query42_00006", + "negative_test_query147_00001", + "negative_test_query172_00000", + "negative_test_query326_00008", + "negative_test_query124_00012", + "apositive_test_query106_00003", + "negative_test_query162_00007", + "negative_test_query312_00003", + "negative_test_query312_00000", + "negative_test_query152_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query263", + "ranked_ids": [ + "apositive_test_query263_00002", + "negative_test_query263_00005", + "negative_test_query135_00004", + "negative_test_query169_00014", + "negative_test_query190_00009", + "apositive_test_query231_00001", + "negative_test_query201_00010", + "negative_test_query213_00001", + "negative_test_query213_00005", + "negative_test_query357_00006", + "apositive_test_query292_00002", + "negative_test_query71_00006", + "negative_test_query88_00003", + "apositive_test_query242_00011", + "apositive_test_query291_00010", + "negative_test_query125_00012", + "apositive_test_query37_00002", + "apositive_test_query155_00003", + "negative_test_query307_00005", + "negative_test_query253_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "apositive_test_query264_00000", + "negative_test_query264_00011", + "negative_test_query264_00004", + "negative_test_query200_00011", + "negative_test_query264_00006", + "negative_test_query264_00014", + "negative_test_query200_00017", + "negative_test_query264_00018", + "negative_test_query200_00002", + "negative_test_query264_00002", + "negative_test_query159_00011", + "negative_test_query264_00012", + "apositive_test_query38_00003", + "negative_test_query337_00012", + "negative_test_query264_00007", + "negative_test_query159_00013", + "negative_test_query337_00002", + "negative_test_query264_00008", + "negative_test_query264_00016", + "negative_test_query159_00017" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query265", + "ranked_ids": [ + "apositive_test_query265_00000", + "negative_test_query265_00002", + "apositive_test_query265_00001", + "negative_test_query274_00008", + "negative_test_query265_00007", + "negative_test_query265_00000", + "apositive_test_query234_00001", + "apositive_test_query156_00000", + "negative_test_query265_00009", + "negative_test_query86_00008", + "negative_test_query165_00007", + "negative_test_query165_00016", + "negative_test_query271_00005", + "negative_test_query92_00012", + "negative_test_query52_00000", + "apositive_test_query78_00002", + "negative_test_query190_00013", + "negative_test_query139_00000", + "negative_test_query86_00007", + "negative_test_query66_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "apositive_test_query266_00000", + "negative_test_query176_00018", + "negative_test_query266_00008", + "negative_test_query266_00018", + "negative_test_query106_00001", + "negative_test_query174_00001", + "negative_test_query266_00014", + "negative_test_query169_00014", + "negative_test_query106_00004", + "apositive_test_query54_00009", + "negative_test_query266_00009", + "apositive_test_query19_00002", + "negative_test_query141_00000", + "negative_test_query266_00001", + "negative_test_query266_00000", + "negative_test_query266_00010", + "negative_test_query266_00017", + "negative_test_query266_00005", + "negative_test_query266_00007", + "negative_test_query266_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "negative_test_query267_00002", + "negative_test_query79_00004", + "negative_test_query98_00013", + "negative_test_query359_00005", + "negative_test_query79_00002", + "apositive_test_query113_00002", + "negative_test_query115_00007", + "apositive_test_query50_00004", + "apositive_test_query157_00008", + "negative_test_query204_00001", + "apositive_test_query295_00001", + "negative_test_query186_00006", + "negative_test_query295_00002", + "negative_test_query345_00013", + "negative_test_query313_00016", + "negative_test_query30_00000", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "negative_test_query5_00012", + "apositive_test_query250_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query268", + "ranked_ids": [ + "apositive_test_query268_00000", + "negative_test_query268_00007", + "negative_test_query268_00000", + "apositive_test_query201_00003", + "negative_test_query47_00017", + "apositive_test_query268_00004", + "negative_test_query268_00013", + "apositive_test_query51_00003", + "apositive_test_query268_00002", + "negative_test_query268_00012", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query268_00005", + "negative_test_query268_00003", + "negative_test_query268_00004", + "negative_test_query268_00008", + "negative_test_query323_00002", + "negative_test_query210_00007", + "negative_test_query335_00011", + "negative_test_query228_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query269", + "ranked_ids": [ + "negative_test_query269_00001", + "negative_test_query329_00014", + "apositive_test_query269_00000", + "negative_test_query269_00009", + "apositive_test_query296_00005", + "negative_test_query269_00003", + "negative_test_query269_00005", + "negative_test_query269_00000", + "negative_test_query269_00015", + "negative_test_query269_00004", + "negative_test_query269_00007", + "negative_test_query296_00003", + "negative_test_query269_00002", + "apositive_test_query269_00003", + "negative_test_query269_00010", + "negative_test_query269_00012", + "negative_test_query329_00004", + "negative_test_query269_00008", + "negative_test_query269_00011", + "apositive_test_query296_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "negative_test_query27_00004", + "negative_test_query27_00009", + "apositive_test_query154_00000", + "negative_test_query33_00004", + "negative_test_query76_00009", + "negative_test_query147_00000", + "negative_test_query358_00003", + "negative_test_query11_00008", + "apositive_test_query16_00002", + "negative_test_query138_00006", + "apositive_test_query59_00000", + "negative_test_query164_00007", + "negative_test_query259_00006", + "negative_test_query30_00009", + "apositive_test_query127_00010", + "negative_test_query130_00006", + "apositive_test_query29_00004", + "apositive_test_query81_00007", + "negative_test_query315_00001", + "negative_test_query27_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query270", + "ranked_ids": [ + "negative_test_query270_00012", + "negative_test_query270_00000", + "negative_test_query270_00004", + "apositive_test_query270_00001", + "negative_test_query270_00006", + "negative_test_query270_00011", + "apositive_test_query270_00000", + "negative_test_query270_00005", + "negative_test_query270_00016", + "negative_test_query270_00009", + "negative_test_query270_00002", + "apositive_test_query270_00002", + "negative_test_query270_00015", + "negative_test_query270_00008", + "negative_test_query270_00010", + "negative_test_query270_00014", + "negative_test_query270_00003", + "apositive_test_query81_00000", + "negative_test_query76_00005", + "negative_test_query81_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query271", + "ranked_ids": [ + "apositive_test_query271_00001", + "apositive_test_query271_00000", + "negative_test_query271_00007", + "negative_test_query330_00008", + "negative_test_query78_00010", + "negative_test_query120_00003", + "apositive_test_query10_00002", + "negative_test_query227_00009", + "negative_test_query327_00010", + "negative_test_query241_00018", + "apositive_test_query189_00000", + "apositive_test_query109_00001", + "negative_test_query90_00001", + "negative_test_query112_00014", + "negative_test_query221_00012", + "negative_test_query198_00015", + "negative_test_query169_00009", + "apositive_test_query314_00007", + "negative_test_query84_00014", + "apositive_test_query67_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00002", + "negative_test_query272_00015", + "negative_test_query272_00010", + "negative_test_query272_00016", + "negative_test_query144_00000", + "negative_test_query272_00001", + "negative_test_query272_00003", + "negative_test_query272_00008", + "negative_test_query41_00008", + "negative_test_query332_00013", + "negative_test_query272_00012", + "negative_test_query272_00000", + "negative_test_query272_00017", + "negative_test_query272_00013", + "negative_test_query272_00009", + "negative_test_query272_00014", + "apositive_test_query272_00000", + "negative_test_query18_00014", + "negative_test_query328_00000", + "apositive_test_query251_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00008", + "apositive_test_query281_00008", + "negative_test_query273_00004", + "apositive_test_query273_00005", + "apositive_test_query18_00000", + "apositive_test_query18_00004", + "negative_test_query18_00006", + "apositive_test_query18_00001", + "apositive_test_query18_00003", + "negative_test_query18_00011", + "negative_test_query273_00000", + "negative_test_query18_00012", + "apositive_test_query273_00000", + "negative_test_query273_00006", + "negative_test_query273_00009", + "apositive_test_query273_00004", + "negative_test_query300_00015", + "negative_test_query140_00013", + "negative_test_query166_00009", + "apositive_test_query36_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query274", + "ranked_ids": [ + "negative_test_query274_00000", + "negative_test_query274_00014", + "apositive_test_query274_00003", + "negative_test_query274_00005", + "negative_test_query274_00011", + "apositive_test_query274_00000", + "negative_test_query218_00011", + "negative_test_query274_00012", + "negative_test_query274_00009", + "negative_test_query274_00007", + "negative_test_query200_00009", + "negative_test_query78_00007", + "negative_test_query34_00014", + "negative_test_query352_00012", + "negative_test_query256_00015", + "apositive_test_query274_00001", + "apositive_test_query327_00002", + "negative_test_query274_00004", + "negative_test_query352_00009", + "negative_test_query34_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query275_00000", + "apositive_test_query275_00006", + "apositive_test_query275_00005", + "apositive_test_query275_00003", + "negative_test_query263_00009", + "apositive_test_query275_00008", + "apositive_test_query275_00007", + "negative_test_query275_00004", + "negative_test_query289_00013", + "apositive_test_query275_00004", + "negative_test_query275_00000", + "negative_test_query275_00007", + "negative_test_query202_00005", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query149_00016", + "apositive_test_query110_00001", + "apositive_test_query93_00003", + "negative_test_query16_00001", + "negative_test_query87_00017" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query22_00004", + "negative_test_query199_00012", + "apositive_test_query276_00005", + "negative_test_query276_00001", + "negative_test_query153_00001", + "apositive_test_query153_00006", + "apositive_test_query161_00002", + "negative_test_query146_00000", + "apositive_test_query314_00005", + "negative_test_query113_00011", + "apositive_test_query314_00006", + "negative_test_query146_00001", + "apositive_test_query276_00006", + "negative_test_query146_00003", + "apositive_test_query146_00015", + "apositive_test_query3_00010", + "negative_test_query280_00002", + "apositive_test_query276_00004", + "apositive_test_query314_00007", + "negative_test_query251_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9828920819566879, + "recall@5": 1.0 + }, + "query_id": "test_query277", + "ranked_ids": [ + "apositive_test_query277_00001", + "apositive_test_query277_00003", + "apositive_test_query277_00000", + "negative_test_query277_00007", + "apositive_test_query277_00002", + "apositive_test_query107_00007", + "negative_test_query277_00004", + "negative_test_query277_00003", + "negative_test_query277_00015", + "negative_test_query277_00008", + "negative_test_query277_00014", + "negative_test_query277_00000", + "negative_test_query313_00009", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "negative_test_query345_00000", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query277_00009", + "negative_test_query277_00013" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query140_00006", + "negative_test_query278_00002", + "apositive_test_query263_00005", + "negative_test_query278_00011", + "apositive_test_query4_00013", + "negative_test_query260_00009", + "apositive_test_query36_00001", + "negative_test_query260_00002", + "negative_test_query177_00001", + "negative_test_query260_00011", + "negative_test_query15_00007", + "negative_test_query255_00018", + "negative_test_query255_00016", + "negative_test_query278_00009", + "negative_test_query110_00011", + "negative_test_query27_00005", + "negative_test_query324_00011", + "negative_test_query158_00003", + "apositive_test_query186_00001", + "negative_test_query64_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.6 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query177_00004", + "apositive_test_query279_00000", + "apositive_test_query279_00004", + "apositive_test_query279_00003", + "negative_test_query279_00009", + "negative_test_query279_00014", + "apositive_test_query279_00001", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "negative_test_query279_00013", + "negative_test_query181_00008", + "negative_test_query309_00011", + "apositive_test_query3_00000", + "apositive_test_query76_00005", + "negative_test_query279_00007", + "apositive_test_query358_00007", + "apositive_test_query162_00009", + "negative_test_query132_00013", + "apositive_test_query76_00002", + "apositive_test_query348_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query28", + "ranked_ids": [ + "apositive_test_query217_00003", + "negative_test_query48_00009", + "apositive_test_query28_00008", + "apositive_test_query217_00000", + "apositive_test_query28_00009", + "negative_test_query217_00011", + "negative_test_query307_00004", + "negative_test_query185_00003", + "apositive_test_query185_00006", + "negative_test_query72_00004", + "apositive_test_query28_00007", + "apositive_test_query28_00003", + "apositive_test_query217_00001", + "negative_test_query222_00005", + "negative_test_query72_00018", + "negative_test_query28_00002", + "negative_test_query217_00004", + "negative_test_query307_00003", + "negative_test_query185_00001", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "apositive_test_query280_00000", + "negative_test_query280_00012", + "negative_test_query9_00012", + "negative_test_query106_00006", + "negative_test_query280_00010", + "apositive_test_query280_00004", + "negative_test_query280_00003", + "negative_test_query280_00004", + "negative_test_query280_00008", + "negative_test_query280_00006", + "negative_test_query280_00002", + "negative_test_query280_00013", + "negative_test_query280_00005", + "apositive_test_query253_00009", + "apositive_test_query74_00001", + "negative_test_query236_00018", + "apositive_test_query280_00003", + "negative_test_query106_00005", + "negative_test_query129_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query281_00004", + "apositive_test_query273_00002", + "apositive_test_query281_00010", + "apositive_test_query281_00012", + "apositive_test_query281_00003", + "apositive_test_query281_00007", + "apositive_test_query281_00016", + "apositive_test_query273_00000", + "apositive_test_query281_00008", + "negative_test_query273_00007", + "negative_test_query273_00004", + "negative_test_query273_00003", + "apositive_test_query273_00008", + "apositive_test_query281_00014", + "apositive_test_query281_00001", + "apositive_test_query273_00003", + "negative_test_query257_00012", + "negative_test_query249_00000", + "apositive_test_query281_00011", + "negative_test_query125_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6509209298071326, + "recall@5": 1.0 + }, + "query_id": "test_query282", + "ranked_ids": [ + "negative_test_query344_00001", + "apositive_test_query282_00000", + "negative_test_query282_00007", + "apositive_test_query282_00001", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query282_00000", + "negative_test_query344_00000", + "negative_test_query282_00004", + "negative_test_query282_00001", + "negative_test_query282_00006", + "apositive_test_query20_00004", + "negative_test_query282_00010", + "negative_test_query344_00008", + "negative_test_query344_00004", + "negative_test_query282_00016", + "negative_test_query282_00009", + "negative_test_query282_00012", + "negative_test_query282_00003", + "apositive_test_query220_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query283_00012", + "negative_test_query138_00005", + "negative_test_query283_00007", + "negative_test_query138_00009", + "negative_test_query270_00006", + "negative_test_query283_00000", + "negative_test_query93_00004", + "negative_test_query283_00014", + "apositive_test_query93_00002", + "negative_test_query299_00017", + "negative_test_query93_00002", + "negative_test_query283_00004", + "negative_test_query283_00003", + "negative_test_query93_00010", + "negative_test_query93_00009", + "negative_test_query93_00006", + "negative_test_query265_00013", + "negative_test_query283_00016", + "negative_test_query31_00010", + "negative_test_query92_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "apositive_test_query284_00003", + "negative_test_query284_00009", + "negative_test_query284_00001", + "negative_test_query284_00012", + "apositive_test_query284_00001", + "apositive_test_query163_00003", + "negative_test_query114_00006", + "apositive_test_query198_00002", + "negative_test_query288_00011", + "negative_test_query284_00004", + "apositive_test_query79_00004", + "negative_test_query109_00009", + "negative_test_query141_00002", + "negative_test_query14_00014", + "apositive_test_query162_00005", + "apositive_test_query141_00002", + "apositive_test_query141_00006", + "apositive_test_query163_00002", + "apositive_test_query259_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.5 + }, + "query_id": "test_query285", + "ranked_ids": [ + "apositive_test_query285_00004", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "apositive_test_query285_00002", + "apositive_test_query285_00007", + "apositive_test_query285_00003", + "negative_test_query285_00004", + "negative_test_query285_00002", + "negative_test_query285_00008", + "apositive_test_query285_00005", + "apositive_test_query285_00006", + "negative_test_query285_00009", + "negative_test_query285_00005", + "negative_test_query285_00000", + "negative_test_query285_00010", + "negative_test_query86_00001", + "negative_test_query285_00001", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query285_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.2 + }, + "query_id": "test_query286", + "ranked_ids": [ + "apositive_test_query286_00013", + "apositive_test_query286_00012", + "negative_test_query257_00005", + "negative_test_query257_00001", + "apositive_test_query286_00010", + "negative_test_query286_00000", + "negative_test_query36_00001", + "apositive_test_query286_00006", + "apositive_test_query80_00000", + "negative_test_query286_00003", + "negative_test_query54_00004", + "apositive_test_query257_00000", + "apositive_test_query257_00001", + "apositive_test_query286_00014", + "negative_test_query139_00008", + "negative_test_query52_00001", + "apositive_test_query286_00005", + "negative_test_query125_00012", + "negative_test_query52_00005", + "negative_test_query36_00013" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query287_00001", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00002", + "apositive_test_query133_00008", + "apositive_test_query133_00001", + "apositive_test_query133_00014", + "negative_test_query301_00010", + "apositive_test_query133_00006", + "apositive_test_query133_00004", + "negative_test_query133_00002", + "apositive_test_query133_00013", + "apositive_test_query287_00004", + "apositive_test_query133_00012", + "negative_test_query153_00003", + "negative_test_query359_00011", + "apositive_test_query153_00003", + "apositive_test_query4_00009", + "negative_test_query134_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query288", + "ranked_ids": [ + "apositive_test_query141_00002", + "apositive_test_query288_00002", + "negative_test_query199_00015", + "apositive_test_query141_00006", + "negative_test_query288_00006", + "negative_test_query288_00002", + "apositive_test_query210_00001", + "negative_test_query288_00007", + "negative_test_query199_00010", + "negative_test_query93_00000", + "negative_test_query210_00013", + "negative_test_query288_00010", + "negative_test_query288_00001", + "negative_test_query213_00007", + "negative_test_query210_00005", + "negative_test_query148_00014", + "negative_test_query17_00002", + "apositive_test_query304_00001", + "negative_test_query148_00004", + "apositive_test_query210_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query289", + "ranked_ids": [ + "apositive_test_query289_00002", + "negative_test_query289_00006", + "negative_test_query334_00014", + "negative_test_query7_00006", + "negative_test_query289_00013", + "negative_test_query47_00009", + "negative_test_query250_00016", + "negative_test_query107_00004", + "apositive_test_query275_00005", + "negative_test_query289_00004", + "negative_test_query113_00009", + "negative_test_query247_00001", + "negative_test_query289_00001", + "apositive_test_query289_00000", + "apositive_test_query289_00004", + "negative_test_query289_00000", + "apositive_test_query157_00008", + "negative_test_query267_00005", + "negative_test_query289_00003", + "negative_test_query305_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.375 + }, + "query_id": "test_query29", + "ranked_ids": [ + "apositive_test_query29_00004", + "apositive_test_query29_00007", + "negative_test_query39_00011", + "apositive_test_query29_00005", + "negative_test_query248_00009", + "negative_test_query292_00015", + "apositive_test_query29_00000", + "negative_test_query29_00009", + "negative_test_query29_00008", + "apositive_test_query29_00001", + "apositive_test_query29_00003", + "negative_test_query248_00007", + "negative_test_query29_00007", + "negative_test_query29_00006", + "negative_test_query29_00010", + "apositive_test_query29_00006", + "apositive_test_query29_00002", + "negative_test_query213_00012", + "negative_test_query29_00000", + "negative_test_query29_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query290", + "ranked_ids": [ + "apositive_test_query290_00003", + "apositive_test_query290_00001", + "negative_test_query290_00000", + "negative_test_query290_00005", + "negative_test_query98_00003", + "apositive_test_query290_00002", + "negative_test_query290_00009", + "negative_test_query290_00003", + "negative_test_query290_00006", + "negative_test_query290_00008", + "negative_test_query342_00005", + "negative_test_query237_00010", + "negative_test_query290_00014", + "apositive_test_query237_00002", + "negative_test_query157_00005", + "negative_test_query237_00013", + "negative_test_query312_00001", + "negative_test_query290_00002", + "negative_test_query116_00002", + "negative_test_query59_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query242_00001", + "apositive_test_query291_00000", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "negative_test_query242_00000", + "apositive_test_query291_00008", + "apositive_test_query242_00005", + "apositive_test_query242_00010", + "apositive_test_query242_00013", + "apositive_test_query291_00005", + "apositive_test_query247_00005", + "negative_test_query291_00005", + "apositive_test_query291_00002", + "negative_test_query291_00004", + "negative_test_query291_00003", + "negative_test_query291_00000", + "apositive_test_query242_00000", + "negative_test_query291_00007", + "negative_test_query186_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query292", + "ranked_ids": [ + "negative_test_query292_00010", + "apositive_test_query292_00001", + "apositive_test_query292_00000", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query292_00014", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query99_00004", + "negative_test_query298_00000", + "negative_test_query292_00000", + "negative_test_query292_00012", + "negative_test_query292_00005", + "negative_test_query292_00007", + "negative_test_query292_00008", + "negative_test_query61_00011", + "negative_test_query292_00006", + "negative_test_query151_00006", + "negative_test_query139_00001", + "negative_test_query205_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query293", + "ranked_ids": [ + "apositive_test_query293_00002", + "apositive_test_query293_00001", + "apositive_test_query178_00003", + "negative_test_query293_00000", + "negative_test_query293_00011", + "negative_test_query293_00015", + "negative_test_query293_00006", + "negative_test_query89_00002", + "apositive_test_query293_00000", + "negative_test_query293_00009", + "negative_test_query293_00010", + "negative_test_query293_00003", + "apositive_test_query44_00001", + "negative_test_query27_00008", + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query52_00005", + "negative_test_query293_00016", + "negative_test_query167_00013", + "negative_test_query220_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00003", + "negative_test_query294_00013", + "apositive_test_query294_00001", + "negative_test_query294_00008", + "negative_test_query294_00002", + "negative_test_query314_00001", + "negative_test_query111_00015", + "apositive_test_query170_00005", + "negative_test_query58_00003", + "negative_test_query277_00005", + "negative_test_query294_00005", + "negative_test_query294_00007", + "negative_test_query294_00016", + "negative_test_query294_00012", + "negative_test_query294_00006", + "negative_test_query294_00011", + "negative_test_query294_00001", + "negative_test_query294_00004", + "apositive_test_query177_00004", + "negative_test_query156_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query295", + "ranked_ids": [ + "negative_test_query57_00009", + "negative_test_query26_00002", + "apositive_test_query57_00000", + "negative_test_query295_00002", + "apositive_test_query295_00002", + "negative_test_query295_00008", + "apositive_test_query295_00001", + "apositive_test_query295_00000", + "negative_test_query295_00013", + "apositive_test_query223_00001", + "negative_test_query295_00011", + "negative_test_query168_00005", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query295_00016", + "negative_test_query115_00005", + "negative_test_query295_00001", + "negative_test_query295_00004", + "negative_test_query295_00012", + "negative_test_query295_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query296", + "ranked_ids": [ + "negative_test_query340_00009", + "apositive_test_query296_00002", + "apositive_test_query296_00001", + "apositive_test_query296_00004", + "apositive_test_query296_00005", + "apositive_test_query296_00003", + "negative_test_query112_00012", + "negative_test_query340_00014", + "negative_test_query296_00009", + "negative_test_query329_00011", + "negative_test_query296_00001", + "apositive_test_query296_00000", + "negative_test_query296_00005", + "negative_test_query296_00013", + "negative_test_query296_00000", + "negative_test_query296_00008", + "negative_test_query296_00003", + "negative_test_query0_00004", + "negative_test_query296_00012", + "negative_test_query296_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query297_00014", + "negative_test_query117_00006", + "negative_test_query297_00013", + "apositive_test_query139_00001", + "negative_test_query297_00012", + "negative_test_query297_00004", + "negative_test_query297_00007", + "negative_test_query191_00010", + "negative_test_query329_00007", + "negative_test_query70_00005", + "apositive_test_query297_00000", + "negative_test_query297_00010", + "negative_test_query297_00002", + "negative_test_query297_00008", + "negative_test_query64_00005", + "negative_test_query191_00011", + "negative_test_query64_00003", + "negative_test_query297_00001", + "negative_test_query297_00003", + "negative_test_query191_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "apositive_test_query298_00000", + "apositive_test_query298_00002", + "negative_test_query216_00010", + "negative_test_query298_00006", + "negative_test_query347_00014", + "negative_test_query347_00016", + "apositive_test_query347_00001", + "apositive_test_query347_00002", + "negative_test_query298_00004", + "negative_test_query298_00003", + "negative_test_query347_00000", + "negative_test_query240_00001", + "negative_test_query298_00001", + "apositive_test_query298_00001", + "apositive_test_query56_00002", + "negative_test_query182_00010", + "negative_test_query175_00009", + "negative_test_query182_00011", + "negative_test_query193_00006", + "negative_test_query202_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query299_00005", + "negative_test_query299_00006", + "apositive_test_query346_00001", + "apositive_test_query325_00002", + "apositive_test_query265_00002", + "negative_test_query300_00000", + "negative_test_query351_00009", + "apositive_test_query141_00002", + "negative_test_query288_00007", + "negative_test_query288_00000", + "negative_test_query210_00008", + "negative_test_query3_00003", + "negative_test_query89_00008", + "apositive_test_query227_00003", + "negative_test_query58_00005", + "negative_test_query125_00005", + "negative_test_query305_00007", + "negative_test_query134_00000", + "apositive_test_query101_00003", + "negative_test_query92_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query3", + "ranked_ids": [ + "apositive_test_query3_00001", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "negative_test_query3_00000", + "negative_test_query83_00007", + "negative_test_query147_00014", + "negative_test_query230_00000", + "negative_test_query279_00000", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "negative_test_query147_00011", + "negative_test_query3_00007", + "apositive_test_query230_00003", + "apositive_test_query3_00007", + "apositive_test_query3_00008", + "negative_test_query230_00007", + "negative_test_query23_00012", + "negative_test_query230_00010", + "negative_test_query3_00005", + "negative_test_query329_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query30", + "ranked_ids": [ + "apositive_test_query30_00004", + "apositive_test_query71_00000", + "apositive_test_query71_00002", + "apositive_test_query306_00002", + "apositive_test_query30_00001", + "negative_test_query231_00002", + "negative_test_query97_00011", + "apositive_test_query30_00003", + "negative_test_query231_00008", + "negative_test_query71_00010", + "negative_test_query71_00005", + "apositive_test_query209_00005", + "apositive_test_query30_00005", + "negative_test_query259_00002", + "negative_test_query182_00014", + "negative_test_query357_00002", + "negative_test_query201_00005", + "negative_test_query356_00010", + "apositive_test_query30_00000", + "negative_test_query231_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query300_00009", + "negative_test_query15_00000", + "negative_test_query252_00005", + "negative_test_query300_00005", + "apositive_test_query300_00001", + "negative_test_query300_00006", + "negative_test_query300_00010", + "negative_test_query266_00010", + "negative_test_query300_00013", + "negative_test_query190_00004", + "negative_test_query141_00001", + "apositive_test_query300_00002", + "negative_test_query300_00008", + "negative_test_query174_00001", + "negative_test_query312_00011", + "negative_test_query141_00000", + "negative_test_query190_00016", + "negative_test_query185_00001", + "negative_test_query167_00018", + "negative_test_query300_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8048099750039491, + "recall@5": 0.75 + }, + "query_id": "test_query301", + "ranked_ids": [ + "apositive_test_query301_00000", + "apositive_test_query301_00001", + "negative_test_query301_00006", + "apositive_test_query301_00002", + "negative_test_query301_00002", + "negative_test_query301_00011", + "negative_test_query2_00008", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query301_00013", + "negative_test_query301_00015", + "negative_test_query301_00007", + "negative_test_query148_00009", + "apositive_test_query86_00000", + "apositive_test_query86_00002", + "negative_test_query297_00015", + "negative_test_query301_00001", + "negative_test_query301_00000", + "apositive_test_query171_00000", + "negative_test_query21_00018" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query302", + "ranked_ids": [ + "apositive_test_query302_00000", + "apositive_test_query302_00003", + "negative_test_query302_00010", + "apositive_test_query302_00002", + "apositive_test_query302_00001", + "apositive_test_query302_00005", + "apositive_test_query302_00004", + "negative_test_query308_00008", + "apositive_test_query324_00002", + "negative_test_query308_00003", + "apositive_test_query308_00001", + "apositive_test_query308_00008", + "apositive_test_query324_00001", + "negative_test_query308_00002", + "negative_test_query308_00009", + "apositive_test_query308_00009", + "negative_test_query308_00005", + "negative_test_query302_00001", + "negative_test_query302_00012", + "negative_test_query302_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query35_00003", + "negative_test_query303_00017", + "apositive_test_query303_00000", + "negative_test_query173_00001", + "negative_test_query316_00009", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query206_00011", + "negative_test_query303_00015", + "negative_test_query206_00009", + "negative_test_query243_00000", + "negative_test_query307_00006", + "negative_test_query61_00011", + "negative_test_query303_00005", + "negative_test_query274_00014", + "negative_test_query196_00007", + "negative_test_query8_00013", + "apositive_test_query86_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query304_00001", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query214_00002", + "negative_test_query43_00001", + "negative_test_query304_00005", + "negative_test_query43_00006", + "apositive_test_query304_00003", + "negative_test_query304_00012", + "negative_test_query304_00004", + "negative_test_query304_00010", + "negative_test_query304_00013", + "negative_test_query304_00008", + "negative_test_query304_00007", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query234_00010", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "apositive_test_query285_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query305", + "ranked_ids": [ + "apositive_test_query305_00001", + "negative_test_query305_00014", + "negative_test_query305_00005", + "apositive_test_query259_00011", + "negative_test_query323_00003", + "apositive_test_query71_00003", + "negative_test_query0_00007", + "negative_test_query134_00006", + "negative_test_query65_00013", + "apositive_test_query357_00000", + "apositive_test_query127_00005", + "negative_test_query56_00004", + "negative_test_query51_00003", + "negative_test_query203_00006", + "negative_test_query0_00004", + "negative_test_query0_00003", + "negative_test_query164_00011", + "negative_test_query47_00001", + "negative_test_query310_00000", + "negative_test_query224_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query306", + "ranked_ids": [ + "negative_test_query71_00013", + "apositive_test_query71_00002", + "apositive_test_query306_00002", + "negative_test_query209_00008", + "negative_test_query259_00002", + "apositive_test_query209_00005", + "apositive_test_query119_00006", + "negative_test_query71_00012", + "apositive_test_query71_00003", + "negative_test_query305_00014", + "apositive_test_query30_00002", + "apositive_test_query30_00001", + "negative_test_query204_00001", + "apositive_test_query289_00003", + "negative_test_query182_00014", + "negative_test_query71_00011", + "negative_test_query159_00009", + "apositive_test_query71_00001", + "negative_test_query142_00002", + "apositive_test_query127_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "negative_test_query48_00018", + "apositive_test_query170_00005", + "apositive_test_query177_00004", + "apositive_test_query279_00002", + "apositive_test_query3_00003", + "negative_test_query251_00001", + "negative_test_query70_00010", + "negative_test_query217_00003", + "negative_test_query103_00012", + "apositive_test_query279_00003", + "negative_test_query207_00004", + "negative_test_query360_00003", + "negative_test_query18_00005", + "negative_test_query307_00006", + "negative_test_query132_00008", + "negative_test_query163_00010", + "negative_test_query144_00004", + "negative_test_query187_00011", + "negative_test_query346_00017", + "apositive_test_query13_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query308_00000", + "apositive_test_query308_00008", + "apositive_test_query308_00001", + "apositive_test_query308_00002", + "negative_test_query308_00003", + "apositive_test_query324_00001", + "negative_test_query308_00009", + "apositive_test_query324_00002", + "negative_test_query308_00007", + "apositive_test_query308_00009", + "negative_test_query324_00012", + "apositive_test_query324_00000", + "negative_test_query324_00014", + "negative_test_query94_00011", + "negative_test_query159_00013", + "apositive_test_query308_00003", + "negative_test_query308_00008", + "apositive_test_query310_00008", + "apositive_test_query341_00005", + "apositive_test_query7_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "apositive_test_query309_00000", + "negative_test_query123_00007", + "negative_test_query309_00003", + "apositive_test_query358_00007", + "negative_test_query354_00002", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "negative_test_query309_00005", + "negative_test_query354_00008", + "apositive_test_query161_00008", + "apositive_test_query358_00006", + "apositive_test_query358_00003", + "negative_test_query113_00013", + "negative_test_query81_00002", + "negative_test_query113_00011", + "negative_test_query309_00015", + "negative_test_query309_00006", + "negative_test_query309_00012", + "negative_test_query42_00009", + "apositive_test_query76_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query31", + "ranked_ids": [ + "apositive_test_query213_00005", + "negative_test_query31_00003", + "apositive_test_query249_00009", + "negative_test_query313_00007", + "negative_test_query38_00003", + "apositive_test_query31_00000", + "negative_test_query31_00011", + "negative_test_query129_00000", + "negative_test_query31_00015", + "negative_test_query31_00006", + "negative_test_query31_00000", + "negative_test_query31_00013", + "negative_test_query31_00010", + "negative_test_query31_00004", + "negative_test_query31_00002", + "negative_test_query31_00001", + "negative_test_query31_00007", + "negative_test_query31_00008", + "apositive_test_query60_00011", + "negative_test_query326_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.3 + }, + "query_id": "test_query310", + "ranked_ids": [ + "negative_test_query59_00005", + "apositive_test_query310_00001", + "negative_test_query36_00015", + "apositive_test_query310_00008", + "apositive_test_query310_00006", + "apositive_test_query310_00009", + "apositive_test_query310_00002", + "apositive_test_query218_00005", + "negative_test_query148_00003", + "apositive_test_query310_00000", + "apositive_test_query310_00003", + "negative_test_query329_00003", + "negative_test_query241_00001", + "negative_test_query179_00013", + "negative_test_query191_00010", + "negative_test_query330_00007", + "negative_test_query74_00002", + "negative_test_query148_00010", + "negative_test_query292_00007", + "negative_test_query310_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query311_00008", + "negative_test_query240_00014", + "negative_test_query311_00005", + "negative_test_query240_00008", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "apositive_test_query311_00000", + "negative_test_query311_00012", + "apositive_test_query330_00001", + "apositive_test_query330_00002", + "negative_test_query330_00009", + "negative_test_query202_00017", + "negative_test_query246_00006", + "negative_test_query126_00008", + "negative_test_query151_00008", + "apositive_test_query5_00002", + "negative_test_query12_00008", + "negative_test_query288_00004", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "apositive_test_query312_00000", + "negative_test_query212_00003", + "negative_test_query246_00015", + "negative_test_query343_00016", + "negative_test_query206_00010", + "apositive_test_query212_00004", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query182_00002", + "negative_test_query157_00002", + "negative_test_query283_00016", + "apositive_test_query98_00000", + "apositive_test_query220_00005", + "negative_test_query242_00001", + "apositive_test_query326_00004", + "negative_test_query312_00006", + "negative_test_query305_00015", + "apositive_test_query237_00001", + "negative_test_query294_00015", + "negative_test_query191_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query158_00000", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query46_00005", + "negative_test_query46_00003", + "negative_test_query46_00011", + "negative_test_query313_00010", + "negative_test_query57_00001", + "negative_test_query359_00005", + "apositive_test_query142_00005", + "negative_test_query56_00001", + "negative_test_query336_00001", + "negative_test_query334_00001", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "apositive_test_query108_00001", + "negative_test_query313_00002", + "negative_test_query111_00013", + "apositive_test_query155_00003", + "negative_test_query119_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query314_00016", + "apositive_test_query314_00007", + "apositive_test_query276_00007", + "apositive_test_query314_00005", + "apositive_test_query314_00010", + "apositive_test_query314_00012", + "apositive_test_query314_00013", + "apositive_test_query314_00003", + "apositive_test_query314_00006", + "apositive_test_query314_00002", + "apositive_test_query314_00011", + "apositive_test_query314_00004", + "negative_test_query37_00006", + "apositive_test_query3_00009", + "negative_test_query314_00002", + "negative_test_query314_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query315_00002", + "apositive_test_query315_00000", + "apositive_test_query315_00001", + "negative_test_query269_00004", + "negative_test_query333_00007", + "negative_test_query315_00009", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "negative_test_query85_00013", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00005", + "apositive_test_query82_00005", + "negative_test_query319_00001", + "negative_test_query85_00014", + "apositive_test_query85_00003", + "negative_test_query85_00005", + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query345_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query316", + "ranked_ids": [ + "apositive_test_query316_00000", + "negative_test_query316_00011", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query316_00004", + "negative_test_query316_00000", + "negative_test_query158_00008", + "negative_test_query316_00001", + "negative_test_query316_00016", + "negative_test_query316_00005", + "negative_test_query316_00002", + "negative_test_query301_00005", + "apositive_test_query316_00001", + "apositive_test_query63_00007", + "negative_test_query316_00008", + "negative_test_query101_00007", + "apositive_test_query216_00000", + "apositive_test_query72_00000", + "negative_test_query228_00011", + "negative_test_query322_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00004", + "negative_test_query317_00005", + "negative_test_query317_00013", + "negative_test_query317_00014", + "negative_test_query317_00008", + "apositive_test_query317_00000", + "negative_test_query317_00010", + "apositive_test_query317_00001", + "negative_test_query317_00000", + "negative_test_query317_00015", + "negative_test_query317_00012", + "negative_test_query317_00006", + "negative_test_query55_00014", + "negative_test_query116_00014", + "negative_test_query317_00017", + "negative_test_query317_00016", + "negative_test_query331_00009", + "negative_test_query283_00013", + "negative_test_query271_00002", + "negative_test_query116_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8048099750039491, + "recall@5": 0.75 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00002", + "apositive_test_query318_00000", + "negative_test_query318_00015", + "apositive_test_query318_00001", + "negative_test_query318_00009", + "negative_test_query318_00012", + "negative_test_query73_00012", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query73_00000", + "negative_test_query318_00008", + "negative_test_query318_00003", + "negative_test_query318_00000", + "negative_test_query318_00002", + "negative_test_query318_00010", + "negative_test_query318_00005", + "negative_test_query73_00005", + "negative_test_query73_00004", + "negative_test_query318_00004", + "apositive_test_query318_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00003", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query340_00013", + "negative_test_query319_00009", + "negative_test_query82_00009", + "negative_test_query319_00008", + "negative_test_query82_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query196_00000", + "negative_test_query189_00014", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query357_00000", + "negative_test_query211_00016", + "apositive_test_query32_00013", + "negative_test_query55_00011", + "apositive_test_query192_00000", + "apositive_test_query136_00003", + "negative_test_query211_00005", + "negative_test_query341_00010", + "negative_test_query321_00001", + "negative_test_query68_00009", + "apositive_test_query17_00000", + "negative_test_query78_00007", + "negative_test_query55_00014", + "negative_test_query83_00001", + "apositive_test_query336_00001", + "negative_test_query339_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query320", + "ranked_ids": [ + "apositive_test_query320_00003", + "negative_test_query320_00005", + "negative_test_query91_00007", + "negative_test_query320_00004", + "negative_test_query320_00008", + "negative_test_query230_00001", + "negative_test_query318_00011", + "negative_test_query258_00012", + "negative_test_query53_00014", + "negative_test_query304_00005", + "negative_test_query43_00000", + "negative_test_query100_00001", + "negative_test_query193_00006", + "negative_test_query344_00010", + "negative_test_query212_00006", + "negative_test_query131_00009", + "apositive_test_query238_00000", + "apositive_test_query267_00001", + "negative_test_query216_00014", + "negative_test_query290_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "apositive_test_query321_00000", + "negative_test_query321_00002", + "negative_test_query321_00000", + "negative_test_query321_00010", + "negative_test_query34_00012", + "negative_test_query352_00005", + "negative_test_query141_00009", + "apositive_test_query185_00009", + "negative_test_query103_00006", + "negative_test_query103_00011", + "negative_test_query322_00002", + "negative_test_query126_00011", + "negative_test_query126_00005", + "negative_test_query136_00004", + "negative_test_query86_00014", + "negative_test_query226_00002", + "apositive_test_query217_00006", + "negative_test_query15_00013", + "negative_test_query197_00004", + "negative_test_query210_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query322", + "ranked_ids": [ + "apositive_test_query322_00005", + "apositive_test_query322_00001", + "apositive_test_query322_00000", + "apositive_test_query322_00006", + "negative_test_query322_00006", + "negative_test_query73_00010", + "apositive_test_query228_00001", + "negative_test_query322_00003", + "negative_test_query255_00010", + "apositive_test_query322_00002", + "apositive_test_query254_00006", + "negative_test_query322_00009", + "apositive_test_query322_00003", + "apositive_test_query322_00004", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query322_00002", + "negative_test_query318_00002", + "negative_test_query322_00005", + "negative_test_query73_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query323", + "ranked_ids": [ + "apositive_test_query323_00002", + "apositive_test_query323_00001", + "apositive_test_query323_00000", + "negative_test_query323_00012", + "apositive_test_query323_00006", + "negative_test_query323_00000", + "negative_test_query323_00007", + "negative_test_query323_00006", + "negative_test_query323_00001", + "negative_test_query75_00002", + "negative_test_query323_00003", + "apositive_test_query323_00004", + "apositive_test_query130_00001", + "negative_test_query345_00011", + "negative_test_query75_00004", + "negative_test_query209_00004", + "negative_test_query255_00002", + "negative_test_query75_00007", + "negative_test_query15_00006", + "negative_test_query201_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query324", + "ranked_ids": [ + "apositive_test_query324_00001", + "apositive_test_query324_00000", + "negative_test_query308_00003", + "negative_test_query324_00005", + "apositive_test_query308_00009", + "apositive_test_query308_00000", + "apositive_test_query308_00001", + "negative_test_query324_00014", + "negative_test_query218_00003", + "negative_test_query324_00007", + "negative_test_query324_00003", + "negative_test_query324_00004", + "negative_test_query324_00006", + "apositive_test_query308_00003", + "negative_test_query324_00016", + "apositive_test_query324_00002", + "negative_test_query324_00015", + "negative_test_query308_00007", + "negative_test_query308_00009", + "apositive_test_query308_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query176_00003", + "negative_test_query325_00007", + "negative_test_query325_00008", + "negative_test_query169_00014", + "negative_test_query15_00015", + "negative_test_query266_00014", + "apositive_test_query293_00002", + "apositive_test_query178_00003", + "negative_test_query325_00013", + "negative_test_query293_00015", + "negative_test_query293_00000", + "negative_test_query325_00002", + "negative_test_query15_00017", + "negative_test_query15_00005", + "negative_test_query15_00016", + "apositive_test_query19_00002", + "apositive_test_query246_00001", + "negative_test_query19_00008", + "negative_test_query191_00011", + "negative_test_query297_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query326", + "ranked_ids": [ + "apositive_test_query326_00000", + "apositive_test_query212_00004", + "apositive_test_query326_00004", + "apositive_test_query326_00001", + "apositive_test_query326_00003", + "negative_test_query178_00012", + "negative_test_query17_00011", + "negative_test_query156_00013", + "negative_test_query107_00011", + "apositive_test_query52_00002", + "apositive_test_query54_00002", + "negative_test_query265_00007", + "apositive_test_query355_00002", + "apositive_test_query23_00000", + "apositive_test_query294_00001", + "negative_test_query183_00001", + "negative_test_query181_00007", + "negative_test_query81_00003", + "negative_test_query265_00005", + "negative_test_query45_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query327", + "ranked_ids": [ + "apositive_test_query327_00000", + "negative_test_query321_00014", + "apositive_test_query327_00001", + "negative_test_query327_00014", + "negative_test_query327_00000", + "negative_test_query303_00010", + "negative_test_query327_00003", + "negative_test_query61_00003", + "negative_test_query61_00006", + "negative_test_query327_00013", + "negative_test_query327_00012", + "apositive_test_query327_00002", + "negative_test_query327_00010", + "negative_test_query195_00007", + "negative_test_query227_00009", + "apositive_test_query271_00000", + "apositive_test_query271_00001", + "negative_test_query271_00007", + "negative_test_query299_00010", + "negative_test_query335_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query328", + "ranked_ids": [ + "negative_test_query328_00002", + "negative_test_query328_00003", + "apositive_test_query328_00004", + "apositive_test_query328_00001", + "negative_test_query328_00000", + "negative_test_query59_00017", + "apositive_test_query227_00001", + "apositive_test_query328_00002", + "negative_test_query24_00006", + "negative_test_query138_00013", + "apositive_test_query40_00003", + "negative_test_query217_00005", + "apositive_test_query328_00007", + "negative_test_query144_00005", + "negative_test_query261_00009", + "negative_test_query328_00006", + "negative_test_query129_00002", + "negative_test_query222_00016", + "negative_test_query328_00007", + "negative_test_query92_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query329", + "ranked_ids": [ + "negative_test_query269_00000", + "negative_test_query329_00001", + "negative_test_query269_00012", + "negative_test_query329_00010", + "apositive_test_query329_00002", + "negative_test_query329_00004", + "negative_test_query329_00006", + "negative_test_query62_00017", + "negative_test_query329_00011", + "negative_test_query298_00015", + "negative_test_query329_00003", + "negative_test_query329_00002", + "apositive_test_query329_00001", + "negative_test_query329_00009", + "negative_test_query329_00013", + "apositive_test_query7_00001", + "negative_test_query183_00006", + "negative_test_query35_00000", + "negative_test_query38_00009", + "negative_test_query202_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00004", + "apositive_test_query33_00006", + "apositive_test_query33_00004", + "negative_test_query33_00006", + "apositive_test_query154_00007", + "apositive_test_query33_00009", + "negative_test_query220_00010", + "apositive_test_query20_00010", + "negative_test_query154_00006", + "apositive_test_query284_00004", + "negative_test_query161_00008", + "negative_test_query33_00007", + "apositive_test_query154_00002", + "negative_test_query344_00010", + "apositive_test_query33_00003", + "negative_test_query216_00004", + "apositive_test_query220_00003", + "apositive_test_query220_00000", + "negative_test_query356_00006", + "negative_test_query216_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query330", + "ranked_ids": [ + "negative_test_query330_00010", + "apositive_test_query330_00002", + "negative_test_query330_00009", + "negative_test_query330_00008", + "negative_test_query27_00013", + "negative_test_query252_00007", + "negative_test_query330_00001", + "apositive_test_query284_00003", + "negative_test_query325_00016", + "negative_test_query311_00009", + "negative_test_query330_00000", + "negative_test_query27_00018", + "negative_test_query27_00011", + "apositive_test_query60_00014", + "apositive_test_query288_00002", + "negative_test_query288_00006", + "negative_test_query129_00003", + "negative_test_query330_00012", + "apositive_test_query281_00012", + "negative_test_query180_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query331", + "ranked_ids": [ + "apositive_test_query331_00000", + "negative_test_query250_00016", + "negative_test_query331_00013", + "negative_test_query331_00000", + "negative_test_query113_00000", + "apositive_test_query331_00002", + "negative_test_query334_00014", + "negative_test_query110_00007", + "negative_test_query312_00012", + "apositive_test_query331_00001", + "negative_test_query331_00012", + "negative_test_query214_00015", + "negative_test_query331_00008", + "negative_test_query202_00001", + "negative_test_query303_00001", + "apositive_test_query223_00001", + "negative_test_query251_00009", + "negative_test_query3_00008", + "negative_test_query154_00000", + "negative_test_query331_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00001", + "apositive_test_query332_00002", + "negative_test_query332_00002", + "negative_test_query145_00010", + "negative_test_query202_00010", + "negative_test_query202_00002", + "apositive_test_query348_00008", + "negative_test_query115_00008", + "apositive_test_query332_00003", + "negative_test_query245_00001", + "negative_test_query244_00006", + "negative_test_query171_00008", + "negative_test_query180_00011", + "apositive_test_query142_00004", + "negative_test_query332_00009", + "negative_test_query12_00005", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "apositive_test_query162_00005", + "negative_test_query65_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00002", + "negative_test_query318_00010", + "negative_test_query246_00000", + "negative_test_query67_00000", + "negative_test_query73_00005", + "negative_test_query318_00005", + "apositive_test_query322_00000", + "negative_test_query73_00014", + "negative_test_query333_00005", + "negative_test_query73_00008", + "negative_test_query333_00008", + "negative_test_query128_00004", + "negative_test_query21_00005", + "negative_test_query216_00013", + "apositive_test_query254_00005", + "negative_test_query136_00003", + "apositive_test_query322_00003", + "negative_test_query322_00009", + "apositive_test_query333_00000", + "apositive_test_query228_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query155_00010", + "negative_test_query111_00005", + "negative_test_query209_00003", + "negative_test_query111_00006", + "negative_test_query107_00001", + "negative_test_query155_00013", + "apositive_test_query57_00006", + "negative_test_query334_00012", + "negative_test_query37_00000", + "negative_test_query111_00017", + "negative_test_query213_00009", + "negative_test_query111_00010", + "negative_test_query334_00010", + "apositive_test_query276_00009", + "apositive_test_query26_00001", + "negative_test_query102_00015", + "negative_test_query334_00016", + "negative_test_query164_00005", + "negative_test_query170_00004", + "negative_test_query223_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.8 + }, + "query_id": "test_query335", + "ranked_ids": [ + "apositive_test_query335_00000", + "negative_test_query335_00006", + "apositive_test_query335_00003", + "apositive_test_query335_00001", + "apositive_test_query335_00002", + "apositive_test_query335_00004", + "negative_test_query335_00003", + "negative_test_query335_00004", + "negative_test_query335_00013", + "negative_test_query335_00010", + "negative_test_query335_00007", + "negative_test_query335_00000", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query92_00002", + "negative_test_query335_00009", + "negative_test_query335_00011", + "negative_test_query45_00010", + "negative_test_query210_00005", + "negative_test_query335_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "apositive_test_query50_00005", + "negative_test_query336_00004", + "apositive_test_query50_00015", + "negative_test_query336_00009", + "negative_test_query336_00007", + "apositive_test_query38_00005", + "negative_test_query336_00014", + "apositive_test_query50_00012", + "apositive_test_query186_00002", + "negative_test_query247_00006", + "negative_test_query108_00005", + "negative_test_query247_00000", + "negative_test_query247_00008", + "negative_test_query105_00011", + "negative_test_query186_00012", + "apositive_test_query113_00005", + "apositive_test_query186_00000", + "negative_test_query336_00015", + "negative_test_query247_00003", + "apositive_test_query242_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "apositive_test_query337_00000", + "negative_test_query200_00001", + "negative_test_query200_00010", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query337_00000", + "apositive_test_query38_00003", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query200_00007", + "negative_test_query337_00007", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query200_00000", + "negative_test_query337_00004", + "negative_test_query200_00009", + "negative_test_query200_00015", + "negative_test_query159_00013", + "negative_test_query38_00008", + "negative_test_query337_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query338", + "ranked_ids": [ + "apositive_test_query338_00008", + "apositive_test_query338_00002", + "apositive_test_query338_00000", + "apositive_test_query338_00005", + "apositive_test_query338_00001", + "apositive_test_query338_00006", + "negative_test_query338_00003", + "apositive_test_query338_00004", + "negative_test_query53_00008", + "negative_test_query81_00006", + "negative_test_query114_00004", + "apositive_test_query338_00003", + "negative_test_query81_00004", + "negative_test_query114_00007", + "negative_test_query95_00004", + "apositive_test_query114_00006", + "apositive_test_query114_00005", + "apositive_test_query81_00000", + "negative_test_query119_00003", + "negative_test_query76_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8503449055347546, + "recall@5": 1.0 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query339_00001", + "negative_test_query339_00004", + "negative_test_query339_00008", + "negative_test_query339_00009", + "apositive_test_query339_00000", + "negative_test_query339_00012", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query192_00011", + "negative_test_query339_00016", + "negative_test_query96_00016", + "negative_test_query339_00017", + "negative_test_query339_00006", + "apositive_test_query96_00001", + "apositive_test_query96_00002", + "negative_test_query339_00001", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004", + "negative_test_query96_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00002", + "apositive_test_query352_00000", + "negative_test_query34_00002", + "apositive_test_query352_00003", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "apositive_test_query34_00004", + "negative_test_query352_00004", + "negative_test_query34_00008", + "negative_test_query352_00010", + "negative_test_query34_00000", + "negative_test_query352_00000", + "apositive_test_query34_00001", + "apositive_test_query34_00000", + "negative_test_query34_00014", + "negative_test_query352_00012", + "apositive_test_query352_00001", + "negative_test_query34_00009", + "negative_test_query352_00013", + "negative_test_query352_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00000", + "apositive_test_query340_00004", + "negative_test_query88_00006", + "negative_test_query340_00008", + "negative_test_query340_00014", + "apositive_test_query340_00001", + "apositive_test_query340_00002", + "negative_test_query88_00007", + "negative_test_query340_00007", + "negative_test_query340_00012", + "negative_test_query88_00000", + "negative_test_query340_00001", + "negative_test_query340_00006", + "apositive_test_query340_00003", + "apositive_test_query174_00012", + "negative_test_query88_00013", + "negative_test_query340_00011", + "apositive_test_query174_00002", + "negative_test_query23_00012", + "negative_test_query88_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.375 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query96_00005", + "negative_test_query214_00005", + "apositive_test_query341_00005", + "apositive_test_query341_00003", + "apositive_test_query341_00001", + "negative_test_query341_00000", + "apositive_test_query32_00007", + "negative_test_query196_00006", + "apositive_test_query341_00002", + "negative_test_query341_00003", + "negative_test_query341_00009", + "negative_test_query196_00009", + "negative_test_query32_00003", + "negative_test_query73_00014", + "negative_test_query341_00002", + "negative_test_query96_00013", + "negative_test_query96_00007", + "negative_test_query316_00008", + "negative_test_query341_00005", + "negative_test_query101_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query68_00005", + "negative_test_query342_00015", + "negative_test_query84_00007", + "apositive_test_query342_00000", + "negative_test_query212_00009", + "negative_test_query200_00009", + "negative_test_query342_00016", + "negative_test_query49_00002", + "negative_test_query136_00003", + "negative_test_query245_00001", + "negative_test_query134_00013", + "negative_test_query98_00017", + "negative_test_query324_00015", + "negative_test_query342_00009", + "negative_test_query77_00004", + "negative_test_query342_00003", + "negative_test_query77_00002", + "negative_test_query78_00015", + "negative_test_query98_00011", + "negative_test_query325_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query343", + "ranked_ids": [ + "apositive_test_query343_00001", + "apositive_test_query185_00001", + "negative_test_query103_00010", + "negative_test_query343_00008", + "apositive_test_query217_00004", + "apositive_test_query28_00006", + "negative_test_query225_00008", + "apositive_test_query343_00000", + "negative_test_query103_00008", + "apositive_test_query185_00008", + "apositive_test_query217_00002", + "apositive_test_query307_00008", + "negative_test_query360_00017", + "apositive_test_query279_00003", + "apositive_test_query279_00002", + "negative_test_query307_00006", + "apositive_test_query136_00004", + "apositive_test_query133_00005", + "apositive_test_query287_00003", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query344_00009", + "negative_test_query282_00008", + "negative_test_query344_00005", + "negative_test_query83_00008", + "apositive_test_query83_00006", + "negative_test_query345_00012", + "negative_test_query223_00016", + "negative_test_query344_00006", + "negative_test_query83_00005", + "negative_test_query344_00012", + "apositive_test_query221_00002", + "negative_test_query85_00010", + "negative_test_query282_00004", + "negative_test_query344_00004", + "negative_test_query231_00015", + "negative_test_query188_00000", + "negative_test_query344_00002", + "negative_test_query282_00001", + "negative_test_query344_00000", + "negative_test_query282_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query345_00007", + "negative_test_query345_00011", + "negative_test_query345_00003", + "negative_test_query130_00004", + "negative_test_query90_00003", + "negative_test_query9_00002", + "apositive_test_query345_00004", + "negative_test_query184_00009", + "negative_test_query47_00008", + "negative_test_query47_00003", + "apositive_test_query345_00001", + "negative_test_query259_00007", + "apositive_test_query345_00000", + "apositive_test_query203_00005", + "negative_test_query345_00010", + "apositive_test_query90_00001", + "apositive_test_query9_00000", + "negative_test_query221_00002", + "apositive_test_query0_00008", + "negative_test_query345_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query346_00001", + "negative_test_query157_00005", + "negative_test_query346_00008", + "negative_test_query346_00009", + "negative_test_query49_00007", + "negative_test_query346_00010", + "negative_test_query346_00002", + "apositive_test_query346_00001", + "negative_test_query346_00012", + "negative_test_query346_00014", + "apositive_test_query346_00000", + "negative_test_query346_00003", + "negative_test_query346_00017", + "negative_test_query346_00006", + "negative_test_query174_00001", + "negative_test_query19_00012", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "negative_test_query15_00000", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query347", + "ranked_ids": [ + "apositive_test_query347_00000", + "apositive_test_query347_00001", + "negative_test_query298_00003", + "negative_test_query347_00015", + "negative_test_query251_00007", + "negative_test_query347_00000", + "negative_test_query347_00016", + "negative_test_query347_00014", + "negative_test_query298_00006", + "negative_test_query347_00006", + "negative_test_query31_00003", + "negative_test_query347_00002", + "negative_test_query347_00001", + "negative_test_query347_00007", + "negative_test_query347_00011", + "negative_test_query347_00012", + "negative_test_query347_00010", + "negative_test_query347_00009", + "negative_test_query347_00005", + "negative_test_query347_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query348_00004", + "negative_test_query348_00008", + "negative_test_query348_00006", + "apositive_test_query115_00002", + "apositive_test_query115_00004", + "negative_test_query115_00002", + "negative_test_query115_00006", + "negative_test_query359_00014", + "negative_test_query341_00008", + "apositive_test_query348_00001", + "apositive_test_query348_00009", + "apositive_test_query348_00000", + "negative_test_query115_00012", + "negative_test_query351_00010", + "apositive_test_query348_00006", + "apositive_test_query348_00005", + "apositive_test_query227_00002", + "negative_test_query297_00017", + "apositive_test_query115_00001", + "negative_test_query31_00017" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "negative_test_query74_00001", + "negative_test_query253_00007", + "apositive_test_query137_00002", + "negative_test_query194_00001", + "negative_test_query253_00000", + "apositive_test_query253_00008", + "negative_test_query312_00016", + "negative_test_query349_00015", + "apositive_test_query253_00005", + "apositive_test_query74_00002", + "negative_test_query349_00003", + "apositive_test_query42_00004", + "apositive_test_query74_00003", + "negative_test_query124_00013", + "negative_test_query13_00002", + "negative_test_query253_00009", + "apositive_test_query253_00009", + "apositive_test_query355_00007", + "negative_test_query157_00009", + "negative_test_query349_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.592512031964586, + "recall@5": 0.75 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query139_00000", + "apositive_test_query35_00001", + "apositive_test_query35_00000", + "negative_test_query205_00005", + "apositive_test_query35_00003", + "negative_test_query35_00002", + "negative_test_query35_00012", + "negative_test_query139_00016", + "negative_test_query123_00006", + "apositive_test_query146_00007", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query231_00012", + "negative_test_query269_00006", + "negative_test_query101_00006", + "negative_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query350_00015", + "negative_test_query350_00014", + "negative_test_query350_00010", + "negative_test_query192_00004", + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query192_00013", + "negative_test_query192_00005", + "negative_test_query316_00006", + "negative_test_query350_00011", + "negative_test_query149_00005", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query14_00001", + "negative_test_query179_00001", + "negative_test_query63_00000", + "negative_test_query331_00012", + "apositive_test_query33_00008", + "negative_test_query350_00001", + "negative_test_query350_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query351_00000", + "apositive_test_query299_00000", + "negative_test_query156_00006", + "negative_test_query351_00002", + "negative_test_query92_00005", + "negative_test_query351_00014", + "negative_test_query351_00006", + "apositive_test_query311_00000", + "apositive_test_query113_00001", + "apositive_test_query60_00003", + "apositive_test_query60_00000", + "apositive_test_query60_00010", + "apositive_test_query218_00004", + "apositive_test_query335_00004", + "apositive_test_query60_00012", + "negative_test_query234_00009", + "negative_test_query53_00011", + "negative_test_query8_00003", + "apositive_test_query193_00009", + "apositive_test_query60_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query34_00001", + "apositive_test_query352_00003", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "negative_test_query34_00002", + "negative_test_query352_00004", + "apositive_test_query352_00000", + "apositive_test_query34_00004", + "apositive_test_query352_00002", + "negative_test_query34_00000", + "negative_test_query352_00000", + "apositive_test_query34_00000", + "negative_test_query352_00010", + "negative_test_query34_00008", + "apositive_test_query352_00001", + "negative_test_query352_00013", + "apositive_test_query34_00002", + "negative_test_query34_00012", + "negative_test_query352_00008", + "negative_test_query34_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00009", + "apositive_test_query353_00000", + "apositive_test_query204_00002", + "apositive_test_query353_00005", + "negative_test_query353_00003", + "negative_test_query108_00002", + "apositive_test_query353_00011", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00012", + "negative_test_query353_00002", + "apositive_test_query204_00007", + "apositive_test_query353_00009", + "apositive_test_query51_00002", + "apositive_test_query204_00015", + "apositive_test_query353_00007", + "negative_test_query175_00012", + "negative_test_query71_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.625 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00001", + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "apositive_test_query354_00004", + "apositive_test_query354_00006", + "negative_test_query309_00014", + "apositive_test_query354_00005", + "negative_test_query309_00000", + "negative_test_query354_00004", + "negative_test_query358_00003", + "negative_test_query309_00013", + "negative_test_query194_00002", + "negative_test_query354_00006", + "apositive_test_query147_00002", + "negative_test_query148_00009", + "apositive_test_query320_00006", + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query331_00014", + "negative_test_query280_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query355_00001", + "negative_test_query355_00009", + "negative_test_query355_00007", + "apositive_test_query355_00002", + "negative_test_query355_00008", + "negative_test_query355_00005", + "negative_test_query355_00003", + "negative_test_query355_00002", + "apositive_test_query355_00005", + "apositive_test_query355_00003", + "apositive_test_query355_00006", + "apositive_test_query38_00006", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query87_00004", + "negative_test_query215_00001", + "negative_test_query264_00003", + "negative_test_query68_00000", + "negative_test_query104_00001", + "apositive_test_query100_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query356_00000", + "apositive_test_query356_00004", + "apositive_test_query66_00003", + "negative_test_query253_00001", + "apositive_test_query356_00002", + "negative_test_query47_00011", + "apositive_test_query30_00005", + "negative_test_query15_00006", + "negative_test_query102_00014", + "negative_test_query102_00010", + "negative_test_query280_00004", + "apositive_test_query356_00008", + "apositive_test_query20_00001", + "negative_test_query334_00000", + "apositive_test_query33_00000", + "apositive_test_query127_00007", + "apositive_test_query295_00000", + "apositive_test_query203_00006", + "apositive_test_query154_00009", + "apositive_test_query88_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00002", + "apositive_test_query221_00000", + "apositive_test_query117_00004", + "apositive_test_query357_00003", + "apositive_test_query119_00012", + "apositive_test_query119_00010", + "apositive_test_query221_00004", + "negative_test_query58_00008", + "apositive_test_query119_00009", + "apositive_test_query357_00004", + "negative_test_query58_00009", + "apositive_test_query119_00000", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query119_00005", + "apositive_test_query314_00000", + "negative_test_query46_00004", + "negative_test_query300_00007", + "apositive_test_query119_00001", + "apositive_test_query221_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2 + }, + "query_id": "test_query358", + "ranked_ids": [ + "negative_test_query309_00005", + "apositive_test_query358_00006", + "negative_test_query123_00007", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "apositive_test_query309_00000", + "negative_test_query309_00006", + "apositive_test_query11_00007", + "apositive_test_query161_00000", + "apositive_test_query358_00003", + "apositive_test_query358_00007", + "apositive_test_query358_00000", + "negative_test_query132_00001", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "negative_test_query279_00010", + "negative_test_query354_00008", + "apositive_test_query358_00009", + "negative_test_query309_00015", + "negative_test_query42_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00004", + "apositive_test_query0_00003", + "negative_test_query359_00002", + "negative_test_query305_00000", + "negative_test_query359_00007", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "apositive_test_query0_00006", + "negative_test_query268_00014", + "negative_test_query0_00001", + "negative_test_query184_00016", + "apositive_test_query0_00002", + "apositive_test_query259_00008", + "apositive_test_query0_00007", + "apositive_test_query259_00005", + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "negative_test_query0_00007", + "negative_test_query259_00006", + "negative_test_query130_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query36", + "ranked_ids": [ + "negative_test_query112_00004", + "apositive_test_query36_00003", + "apositive_test_query286_00006", + "negative_test_query112_00016", + "negative_test_query36_00014", + "negative_test_query112_00015", + "apositive_test_query106_00003", + "negative_test_query112_00010", + "negative_test_query216_00002", + "apositive_test_query286_00005", + "negative_test_query286_00000", + "negative_test_query36_00001", + "negative_test_query112_00013", + "negative_test_query260_00010", + "negative_test_query80_00007", + "apositive_test_query136_00013", + "negative_test_query6_00003", + "negative_test_query36_00010", + "negative_test_query260_00009", + "negative_test_query296_00013" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00001", + "negative_test_query189_00014", + "apositive_test_query360_00000", + "negative_test_query189_00013", + "apositive_test_query153_00011", + "negative_test_query216_00011", + "apositive_test_query142_00004", + "negative_test_query360_00004", + "apositive_test_query310_00006", + "negative_test_query148_00018", + "negative_test_query175_00009", + "negative_test_query312_00004", + "negative_test_query240_00010", + "negative_test_query281_00000", + "negative_test_query246_00001", + "negative_test_query360_00005", + "apositive_test_query69_00014", + "apositive_test_query152_00002", + "negative_test_query360_00009", + "negative_test_query5_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query231_00006", + "apositive_test_query177_00000", + "negative_test_query209_00001", + "apositive_test_query158_00007", + "apositive_test_query213_00006", + "negative_test_query209_00000", + "negative_test_query57_00009", + "apositive_test_query37_00001", + "negative_test_query123_00001", + "negative_test_query155_00001", + "apositive_test_query209_00002", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query20_00003", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "negative_test_query152_00008", + "negative_test_query123_00011", + "apositive_test_query108_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query38", + "ranked_ids": [ + "apositive_test_query38_00003", + "negative_test_query38_00010", + "negative_test_query267_00015", + "negative_test_query47_00016", + "negative_test_query267_00016", + "negative_test_query159_00010", + "negative_test_query267_00001", + "negative_test_query337_00000", + "apositive_test_query38_00007", + "apositive_test_query38_00000", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query175_00005", + "apositive_test_query337_00000", + "apositive_test_query38_00002", + "negative_test_query337_00017", + "negative_test_query38_00006", + "apositive_test_query247_00005", + "negative_test_query79_00002", + "negative_test_query94_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query39", + "ranked_ids": [ + "apositive_test_query39_00001", + "apositive_test_query39_00004", + "negative_test_query18_00004", + "negative_test_query39_00004", + "negative_test_query29_00004", + "negative_test_query39_00000", + "negative_test_query39_00007", + "apositive_test_query39_00000", + "apositive_test_query39_00002", + "negative_test_query39_00009", + "apositive_test_query39_00003", + "negative_test_query39_00006", + "negative_test_query331_00010", + "negative_test_query39_00003", + "negative_test_query39_00012", + "negative_test_query66_00008", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.35714285714285715 + }, + "query_id": "test_query4", + "ranked_ids": [ + "apositive_test_query4_00003", + "apositive_test_query4_00000", + "apositive_test_query4_00010", + "apositive_test_query4_00001", + "apositive_test_query4_00004", + "apositive_test_query4_00012", + "negative_test_query266_00004", + "negative_test_query4_00001", + "apositive_test_query4_00005", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "negative_test_query149_00015", + "negative_test_query266_00009", + "negative_test_query312_00011", + "apositive_test_query19_00000", + "negative_test_query4_00002", + "negative_test_query174_00001", + "negative_test_query19_00004", + "negative_test_query266_00012", + "negative_test_query15_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00004", + "negative_test_query40_00010", + "negative_test_query40_00011", + "negative_test_query40_00005", + "negative_test_query40_00006", + "negative_test_query66_00005", + "negative_test_query40_00007", + "negative_test_query40_00001", + "negative_test_query40_00013", + "negative_test_query40_00000", + "negative_test_query40_00003", + "apositive_test_query40_00002", + "negative_test_query40_00009", + "negative_test_query40_00012", + "negative_test_query347_00003", + "negative_test_query328_00006", + "negative_test_query106_00009", + "negative_test_query214_00015", + "apositive_test_query223_00001", + "negative_test_query324_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8048099750039491, + "recall@5": 0.75 + }, + "query_id": "test_query41", + "ranked_ids": [ + "apositive_test_query41_00002", + "apositive_test_query41_00000", + "negative_test_query41_00000", + "apositive_test_query41_00001", + "negative_test_query41_00010", + "negative_test_query41_00004", + "negative_test_query41_00007", + "negative_test_query144_00016", + "negative_test_query41_00014", + "apositive_test_query249_00012", + "negative_test_query41_00006", + "negative_test_query41_00012", + "negative_test_query41_00011", + "apositive_test_query41_00003", + "negative_test_query282_00003", + "negative_test_query41_00002", + "negative_test_query226_00011", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query98_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query42_00008", + "negative_test_query42_00002", + "negative_test_query42_00000", + "apositive_test_query42_00003", + "negative_test_query42_00003", + "apositive_test_query42_00001", + "apositive_test_query42_00000", + "apositive_test_query42_00004", + "negative_test_query42_00001", + "negative_test_query42_00005", + "negative_test_query137_00005", + "apositive_test_query42_00005", + "apositive_test_query42_00002", + "negative_test_query42_00010", + "negative_test_query42_00006", + "negative_test_query42_00007", + "negative_test_query42_00011", + "apositive_test_query276_00012", + "apositive_test_query162_00008", + "negative_test_query250_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query43", + "ranked_ids": [ + "negative_test_query43_00009", + "apositive_test_query43_00000", + "apositive_test_query43_00002", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query333_00001", + "negative_test_query43_00000", + "negative_test_query179_00006", + "apositive_test_query146_00004", + "apositive_test_query146_00003", + "apositive_test_query161_00002", + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query153_00008", + "apositive_test_query153_00014", + "apositive_test_query161_00005", + "apositive_test_query218_00002", + "negative_test_query319_00002", + "negative_test_query82_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query44", + "ranked_ids": [ + "negative_test_query27_00008", + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query44_00001", + "negative_test_query141_00007", + "negative_test_query330_00009", + "negative_test_query10_00012", + "negative_test_query293_00015", + "negative_test_query293_00003", + "apositive_test_query20_00003", + "negative_test_query44_00002", + "negative_test_query190_00011", + "negative_test_query293_00000", + "apositive_test_query141_00005", + "negative_test_query180_00000", + "negative_test_query330_00012", + "negative_test_query21_00009", + "apositive_test_query293_00000", + "negative_test_query293_00009", + "apositive_test_query245_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query45_00013", + "apositive_test_query25_00000", + "apositive_test_query45_00000", + "negative_test_query45_00001", + "negative_test_query328_00005", + "negative_test_query45_00007", + "negative_test_query182_00016", + "negative_test_query45_00006", + "negative_test_query45_00004", + "negative_test_query143_00011", + "apositive_test_query25_00005", + "negative_test_query45_00016", + "negative_test_query45_00009", + "negative_test_query143_00004", + "negative_test_query143_00014", + "negative_test_query143_00009", + "negative_test_query45_00003", + "negative_test_query45_00005", + "apositive_test_query165_00000", + "negative_test_query342_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "apositive_test_query79_00005", + "apositive_test_query142_00001", + "negative_test_query56_00001", + "negative_test_query39_00011", + "negative_test_query46_00003", + "negative_test_query46_00011", + "negative_test_query79_00012", + "negative_test_query359_00005", + "negative_test_query334_00004", + "apositive_test_query146_00013", + "negative_test_query313_00010", + "negative_test_query46_00001", + "negative_test_query105_00006", + "negative_test_query168_00003", + "apositive_test_query186_00001", + "negative_test_query46_00012", + "negative_test_query213_00012", + "negative_test_query238_00001", + "negative_test_query108_00003", + "apositive_test_query276_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query47_00001", + "negative_test_query331_00013", + "apositive_test_query201_00002", + "apositive_test_query201_00004", + "negative_test_query345_00002", + "negative_test_query47_00002", + "negative_test_query47_00013", + "negative_test_query47_00012", + "negative_test_query47_00006", + "negative_test_query195_00005", + "apositive_test_query331_00002", + "apositive_test_query201_00006", + "negative_test_query144_00018", + "negative_test_query41_00009", + "negative_test_query331_00014", + "negative_test_query331_00005", + "apositive_test_query20_00005", + "negative_test_query47_00011", + "negative_test_query325_00007", + "negative_test_query328_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "negative_test_query307_00005", + "apositive_test_query185_00012", + "apositive_test_query28_00013", + "negative_test_query48_00016", + "apositive_test_query48_00000", + "negative_test_query48_00007", + "negative_test_query48_00009", + "apositive_test_query343_00001", + "apositive_test_query28_00010", + "apositive_test_query185_00004", + "negative_test_query103_00012", + "negative_test_query48_00018", + "negative_test_query48_00004", + "negative_test_query307_00001", + "negative_test_query48_00011", + "negative_test_query48_00012", + "negative_test_query307_00003", + "negative_test_query49_00001", + "negative_test_query48_00000", + "apositive_test_query307_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query49", + "ranked_ids": [ + "apositive_test_query49_00001", + "negative_test_query49_00002", + "negative_test_query342_00005", + "apositive_test_query29_00006", + "negative_test_query169_00000", + "negative_test_query49_00001", + "apositive_test_query54_00009", + "negative_test_query106_00004", + "apositive_test_query204_00003", + "apositive_test_query353_00004", + "negative_test_query176_00016", + "negative_test_query15_00008", + "negative_test_query49_00007", + "negative_test_query52_00011", + "negative_test_query156_00007", + "negative_test_query72_00017", + "negative_test_query156_00010", + "apositive_test_query19_00001", + "negative_test_query328_00009", + "negative_test_query122_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query5", + "ranked_ids": [ + "apositive_test_query5_00000", + "negative_test_query5_00005", + "negative_test_query67_00010", + "negative_test_query246_00002", + "negative_test_query246_00013", + "apositive_test_query5_00001", + "apositive_test_query5_00002", + "negative_test_query5_00000", + "negative_test_query5_00001", + "negative_test_query5_00014", + "negative_test_query5_00010", + "negative_test_query301_00003", + "negative_test_query5_00007", + "negative_test_query5_00003", + "negative_test_query5_00013", + "negative_test_query5_00004", + "negative_test_query275_00001", + "negative_test_query5_00006", + "negative_test_query67_00003", + "negative_test_query246_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00014", + "apositive_test_query186_00005", + "apositive_test_query50_00003", + "apositive_test_query50_00002", + "apositive_test_query50_00006", + "apositive_test_query50_00005", + "negative_test_query336_00004", + "apositive_test_query50_00008", + "apositive_test_query50_00013", + "apositive_test_query50_00015", + "apositive_test_query50_00012", + "apositive_test_query50_00009", + "negative_test_query336_00011", + "apositive_test_query50_00001", + "negative_test_query50_00001", + "negative_test_query131_00001", + "apositive_test_query131_00000", + "negative_test_query66_00002", + "negative_test_query119_00002", + "negative_test_query223_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.4 + }, + "query_id": "test_query51", + "ranked_ids": [ + "apositive_test_query51_00000", + "negative_test_query51_00006", + "apositive_test_query51_00001", + "apositive_test_query51_00005", + "apositive_test_query51_00008", + "negative_test_query51_00000", + "negative_test_query97_00001", + "apositive_test_query51_00007", + "apositive_test_query51_00006", + "negative_test_query97_00002", + "negative_test_query51_00004", + "negative_test_query97_00006", + "negative_test_query97_00011", + "negative_test_query97_00007", + "negative_test_query204_00001", + "apositive_test_query51_00009", + "negative_test_query267_00007", + "negative_test_query209_00008", + "negative_test_query231_00012", + "apositive_test_query267_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query257_00005", + "apositive_test_query286_00014", + "apositive_test_query286_00013", + "negative_test_query52_00007", + "negative_test_query54_00004", + "apositive_test_query286_00006", + "negative_test_query257_00009", + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00001", + "apositive_test_query257_00002", + "apositive_test_query257_00005", + "apositive_test_query257_00004", + "apositive_test_query69_00007", + "apositive_test_query257_00000", + "apositive_test_query257_00003", + "negative_test_query52_00005", + "negative_test_query257_00007", + "apositive_test_query52_00001", + "negative_test_query52_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00016", + "negative_test_query258_00008", + "negative_test_query161_00007", + "negative_test_query180_00012", + "negative_test_query258_00000", + "negative_test_query53_00000", + "negative_test_query150_00014", + "negative_test_query169_00003", + "apositive_test_query217_00004", + "apositive_test_query28_00006", + "negative_test_query230_00009", + "negative_test_query258_00018", + "negative_test_query354_00001", + "negative_test_query3_00008", + "negative_test_query150_00015", + "negative_test_query354_00011", + "negative_test_query258_00013", + "negative_test_query348_00001", + "negative_test_query132_00015", + "apositive_test_query248_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query54_00006", + "negative_test_query44_00014", + "negative_test_query140_00006", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query56_00011", + "apositive_test_query51_00008", + "negative_test_query51_00006", + "negative_test_query97_00002", + "apositive_test_query51_00007", + "apositive_test_query51_00006", + "apositive_test_query204_00002", + "apositive_test_query323_00006", + "apositive_test_query335_00002", + "negative_test_query200_00006", + "negative_test_query301_00006", + "apositive_test_query86_00000", + "negative_test_query44_00007", + "apositive_test_query156_00000", + "negative_test_query136_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9060254355346823, + "recall@5": 1.0 + }, + "query_id": "test_query55", + "ranked_ids": [ + "apositive_test_query55_00000", + "negative_test_query55_00002", + "apositive_test_query55_00002", + "apositive_test_query55_00001", + "negative_test_query55_00003", + "negative_test_query55_00016", + "negative_test_query55_00009", + "negative_test_query55_00012", + "negative_test_query55_00011", + "negative_test_query61_00008", + "negative_test_query139_00015", + "negative_test_query234_00011", + "negative_test_query55_00006", + "negative_test_query55_00010", + "negative_test_query55_00005", + "negative_test_query55_00015", + "negative_test_query55_00000", + "negative_test_query55_00008", + "negative_test_query55_00013", + "negative_test_query55_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query56", + "ranked_ids": [ + "negative_test_query56_00000", + "negative_test_query56_00006", + "negative_test_query200_00014", + "apositive_test_query46_00001", + "negative_test_query56_00003", + "negative_test_query111_00006", + "negative_test_query56_00005", + "negative_test_query119_00005", + "apositive_test_query56_00005", + "negative_test_query46_00011", + "negative_test_query56_00002", + "negative_test_query46_00014", + "apositive_test_query238_00001", + "negative_test_query186_00007", + "negative_test_query359_00005", + "negative_test_query105_00006", + "apositive_test_query56_00006", + "negative_test_query56_00004", + "negative_test_query102_00001", + "negative_test_query337_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query57", + "ranked_ids": [ + "negative_test_query295_00003", + "apositive_test_query57_00005", + "apositive_test_query57_00003", + "negative_test_query313_00012", + "apositive_test_query57_00008", + "apositive_test_query57_00000", + "apositive_test_query57_00006", + "negative_test_query46_00006", + "negative_test_query57_00001", + "negative_test_query111_00010", + "negative_test_query334_00001", + "negative_test_query334_00013", + "negative_test_query57_00010", + "apositive_test_query203_00000", + "apositive_test_query74_00000", + "apositive_test_query259_00006", + "negative_test_query65_00010", + "apositive_test_query146_00000", + "apositive_test_query338_00002", + "negative_test_query198_00016" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "negative_test_query58_00010", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query240_00016", + "negative_test_query58_00015", + "negative_test_query240_00005", + "apositive_test_query311_00003", + "apositive_test_query311_00004", + "negative_test_query308_00004", + "negative_test_query240_00008", + "negative_test_query27_00017", + "negative_test_query11_00004", + "negative_test_query240_00002", + "negative_test_query27_00016", + "negative_test_query311_00012", + "negative_test_query240_00004", + "negative_test_query57_00004", + "negative_test_query240_00015", + "negative_test_query346_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "apositive_test_query59_00000", + "negative_test_query59_00012", + "negative_test_query59_00010", + "negative_test_query116_00000", + "negative_test_query59_00006", + "negative_test_query59_00011", + "negative_test_query59_00014", + "negative_test_query59_00009", + "negative_test_query227_00008", + "negative_test_query59_00018", + "negative_test_query116_00003", + "negative_test_query116_00013", + "negative_test_query59_00017", + "apositive_test_query104_00000", + "negative_test_query59_00015", + "negative_test_query59_00005", + "negative_test_query81_00003", + "negative_test_query66_00008", + "negative_test_query280_00003", + "negative_test_query202_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query6", + "ranked_ids": [ + "apositive_test_query6_00002", + "apositive_test_query6_00000", + "apositive_test_query6_00001", + "negative_test_query6_00002", + "apositive_test_query109_00001", + "negative_test_query6_00004", + "negative_test_query6_00011", + "apositive_test_query242_00004", + "apositive_test_query305_00003", + "negative_test_query6_00001", + "negative_test_query252_00004", + "negative_test_query179_00007", + "negative_test_query6_00012", + "negative_test_query6_00016", + "negative_test_query296_00013", + "negative_test_query58_00007", + "apositive_test_query189_00000", + "negative_test_query112_00010", + "negative_test_query112_00004", + "negative_test_query296_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.25 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00007", + "apositive_test_query60_00003", + "negative_test_query243_00004", + "apositive_test_query60_00001", + "apositive_test_query60_00013", + "apositive_test_query60_00000", + "apositive_test_query60_00006", + "apositive_test_query60_00011", + "negative_test_query218_00007", + "negative_test_query31_00004", + "apositive_test_query60_00005", + "apositive_test_query128_00001", + "negative_test_query60_00003", + "negative_test_query304_00003", + "apositive_test_query285_00001", + "apositive_test_query285_00007", + "negative_test_query304_00002", + "negative_test_query60_00001", + "apositive_test_query60_00009", + "apositive_test_query218_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5654495432396527, + "recall@5": 0.75 + }, + "query_id": "test_query61", + "ranked_ids": [ + "negative_test_query61_00014", + "apositive_test_query61_00003", + "negative_test_query173_00003", + "apositive_test_query61_00002", + "apositive_test_query61_00001", + "negative_test_query129_00005", + "negative_test_query61_00006", + "negative_test_query61_00003", + "negative_test_query61_00005", + "negative_test_query201_00004", + "negative_test_query61_00015", + "negative_test_query61_00000", + "negative_test_query211_00005", + "negative_test_query210_00012", + "apositive_test_query101_00002", + "negative_test_query61_00010", + "apositive_test_query173_00004", + "negative_test_query61_00004", + "negative_test_query327_00016", + "negative_test_query61_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00004", + "negative_test_query62_00002", + "negative_test_query62_00000", + "apositive_test_query62_00001", + "negative_test_query250_00003", + "negative_test_query62_00010", + "negative_test_query62_00003", + "negative_test_query62_00001", + "negative_test_query62_00013", + "negative_test_query62_00008", + "negative_test_query15_00011", + "negative_test_query287_00007", + "negative_test_query62_00011", + "negative_test_query352_00005", + "negative_test_query62_00006", + "negative_test_query16_00001", + "negative_test_query219_00006", + "negative_test_query88_00015", + "negative_test_query38_00009", + "negative_test_query222_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00001", + "apositive_test_query63_00006", + "apositive_test_query63_00008", + "apositive_test_query141_00001", + "negative_test_query297_00006", + "negative_test_query63_00009", + "negative_test_query321_00002", + "apositive_test_query63_00004", + "negative_test_query224_00007", + "negative_test_query63_00003", + "negative_test_query63_00001", + "negative_test_query63_00005", + "negative_test_query63_00002", + "negative_test_query63_00010", + "negative_test_query78_00006", + "negative_test_query264_00015", + "negative_test_query331_00010", + "negative_test_query63_00004", + "apositive_test_query288_00004", + "negative_test_query241_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query211_00009", + "negative_test_query64_00009", + "negative_test_query64_00015", + "negative_test_query64_00013", + "negative_test_query63_00009", + "negative_test_query205_00009", + "negative_test_query64_00016", + "negative_test_query64_00010", + "negative_test_query149_00017", + "negative_test_query63_00000", + "negative_test_query214_00004", + "apositive_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query120_00003", + "negative_test_query120_00000", + "negative_test_query310_00005", + "negative_test_query234_00010", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query149_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "apositive_test_query65_00001", + "negative_test_query65_00004", + "apositive_test_query134_00001", + "apositive_test_query65_00000", + "negative_test_query23_00006", + "apositive_test_query356_00004", + "negative_test_query47_00005", + "negative_test_query30_00011", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query268_00012", + "negative_test_query6_00003", + "negative_test_query260_00013", + "negative_test_query6_00008", + "apositive_test_query260_00000", + "negative_test_query260_00012", + "negative_test_query261_00003", + "apositive_test_query62_00000", + "negative_test_query230_00001", + "negative_test_query65_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.5 + }, + "query_id": "test_query66", + "ranked_ids": [ + "apositive_test_query66_00004", + "negative_test_query66_00011", + "apositive_test_query66_00003", + "apositive_test_query66_00005", + "negative_test_query188_00014", + "apositive_test_query66_00002", + "negative_test_query220_00001", + "apositive_test_query20_00005", + "apositive_test_query20_00001", + "negative_test_query220_00010", + "negative_test_query161_00008", + "negative_test_query161_00003", + "apositive_test_query220_00003", + "negative_test_query344_00007", + "apositive_test_query220_00000", + "negative_test_query154_00009", + "apositive_test_query188_00000", + "apositive_test_query188_00003", + "negative_test_query188_00006", + "negative_test_query9_00012" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query67", + "ranked_ids": [ + "negative_test_query67_00005", + "negative_test_query67_00011", + "negative_test_query246_00009", + "apositive_test_query67_00000", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "apositive_test_query286_00004", + "apositive_test_query286_00007", + "negative_test_query55_00005", + "apositive_test_query286_00010", + "negative_test_query65_00008", + "negative_test_query67_00001", + "negative_test_query67_00008", + "negative_test_query246_00016", + "negative_test_query67_00007", + "negative_test_query67_00010", + "negative_test_query6_00008", + "negative_test_query260_00013", + "negative_test_query123_00008", + "negative_test_query67_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query68", + "ranked_ids": [ + "negative_test_query330_00009", + "apositive_test_query68_00000", + "apositive_test_query68_00002", + "negative_test_query68_00001", + "apositive_test_query252_00001", + "negative_test_query126_00008", + "negative_test_query68_00014", + "apositive_test_query68_00001", + "apositive_test_query330_00001", + "negative_test_query212_00000", + "negative_test_query68_00013", + "negative_test_query311_00005", + "apositive_test_query5_00002", + "negative_test_query151_00008", + "negative_test_query68_00016", + "negative_test_query68_00007", + "apositive_test_query89_00003", + "apositive_test_query326_00005", + "negative_test_query330_00011", + "negative_test_query288_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00011", + "apositive_test_query69_00002", + "apositive_test_query69_00000", + "apositive_test_query69_00013", + "apositive_test_query257_00003", + "negative_test_query169_00002", + "apositive_test_query355_00001", + "apositive_test_query69_00005", + "apositive_test_query69_00010", + "negative_test_query18_00010", + "apositive_test_query69_00008", + "apositive_test_query69_00009", + "apositive_test_query107_00007", + "negative_test_query277_00001", + "apositive_test_query174_00016", + "apositive_test_query69_00012", + "apositive_test_query281_00016", + "apositive_test_query52_00001", + "apositive_test_query54_00003", + "apositive_test_query225_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "negative_test_query7_00005", + "apositive_test_query212_00001", + "apositive_test_query326_00007", + "apositive_test_query212_00002", + "negative_test_query184_00006", + "negative_test_query306_00008", + "apositive_test_query119_00006", + "negative_test_query159_00010", + "negative_test_query159_00009", + "negative_test_query7_00001", + "negative_test_query231_00011", + "negative_test_query38_00010", + "apositive_test_query159_00001", + "negative_test_query264_00013", + "negative_test_query290_00006", + "negative_test_query159_00004", + "apositive_test_query38_00003", + "negative_test_query327_00011", + "negative_test_query79_00002", + "negative_test_query84_00007" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00004", + "negative_test_query70_00011", + "apositive_test_query70_00002", + "negative_test_query70_00000", + "negative_test_query70_00012", + "negative_test_query227_00013", + "apositive_test_query70_00001", + "negative_test_query227_00006", + "negative_test_query227_00012", + "negative_test_query227_00014", + "negative_test_query70_00006", + "apositive_test_query227_00000", + "negative_test_query227_00000", + "negative_test_query227_00003", + "negative_test_query283_00014", + "negative_test_query70_00003", + "negative_test_query348_00007", + "apositive_test_query170_00001", + "negative_test_query59_00004", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query71", + "ranked_ids": [ + "apositive_test_query71_00003", + "negative_test_query71_00009", + "negative_test_query71_00013", + "negative_test_query259_00002", + "negative_test_query71_00001", + "apositive_test_query71_00001", + "negative_test_query65_00002", + "negative_test_query186_00000", + "negative_test_query247_00002", + "negative_test_query313_00001", + "negative_test_query71_00015", + "negative_test_query231_00014", + "apositive_test_query71_00000", + "apositive_test_query71_00002", + "negative_test_query71_00011", + "apositive_test_query30_00005", + "negative_test_query71_00005", + "apositive_test_query209_00005", + "negative_test_query71_00002", + "negative_test_query71_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "apositive_test_query72_00000", + "negative_test_query72_00006", + "negative_test_query72_00007", + "negative_test_query72_00008", + "negative_test_query185_00005", + "negative_test_query185_00000", + "negative_test_query72_00001", + "negative_test_query72_00016", + "negative_test_query72_00015", + "negative_test_query72_00003", + "negative_test_query72_00013", + "negative_test_query17_00009", + "negative_test_query55_00005", + "negative_test_query92_00001", + "apositive_test_query175_00001", + "negative_test_query228_00000", + "negative_test_query285_00009", + "negative_test_query265_00003", + "negative_test_query233_00009", + "negative_test_query72_00011" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query73", + "ranked_ids": [ + "negative_test_query322_00009", + "apositive_test_query322_00003", + "negative_test_query128_00004", + "negative_test_query73_00008", + "apositive_test_query73_00000", + "negative_test_query73_00010", + "apositive_test_query333_00005", + "apositive_test_query333_00002", + "negative_test_query333_00005", + "negative_test_query246_00000", + "negative_test_query73_00013", + "apositive_test_query322_00008", + "negative_test_query216_00013", + "negative_test_query228_00013", + "negative_test_query233_00015", + "negative_test_query21_00005", + "apositive_test_query254_00004", + "apositive_test_query333_00000", + "negative_test_query73_00001", + "negative_test_query73_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query74_00000", + "apositive_test_query74_00001", + "apositive_test_query74_00002", + "negative_test_query74_00001", + "negative_test_query74_00000", + "negative_test_query74_00013", + "negative_test_query74_00015", + "negative_test_query95_00009", + "apositive_test_query74_00003", + "apositive_test_query114_00001", + "negative_test_query74_00014", + "negative_test_query44_00006", + "negative_test_query85_00005", + "apositive_test_query113_00001", + "apositive_test_query238_00003", + "negative_test_query238_00013", + "negative_test_query203_00006", + "negative_test_query129_00008", + "negative_test_query74_00003", + "apositive_test_query60_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query75", + "ranked_ids": [ + "negative_test_query130_00012", + "apositive_test_query75_00000", + "apositive_test_query75_00001", + "apositive_test_query75_00005", + "negative_test_query89_00000", + "negative_test_query75_00006", + "apositive_test_query75_00006", + "negative_test_query112_00018", + "negative_test_query323_00011", + "negative_test_query99_00006", + "negative_test_query67_00017", + "negative_test_query75_00012", + "negative_test_query82_00004", + "negative_test_query319_00004", + "negative_test_query6_00008", + "negative_test_query260_00013", + "negative_test_query209_00004", + "negative_test_query75_00004", + "negative_test_query75_00003", + "apositive_test_query75_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query76", + "ranked_ids": [ + "apositive_test_query76_00006", + "negative_test_query76_00010", + "apositive_test_query76_00004", + "apositive_test_query76_00001", + "negative_test_query258_00008", + "negative_test_query53_00009", + "apositive_test_query76_00000", + "negative_test_query76_00011", + "negative_test_query76_00009", + "negative_test_query76_00007", + "negative_test_query91_00004", + "negative_test_query147_00007", + "negative_test_query181_00008", + "apositive_test_query76_00002", + "negative_test_query53_00001", + "negative_test_query279_00009", + "negative_test_query309_00011", + "negative_test_query76_00002", + "negative_test_query76_00008", + "negative_test_query147_00005" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query98_00014", + "negative_test_query216_00006", + "negative_test_query263_00002", + "negative_test_query77_00013", + "negative_test_query315_00003", + "negative_test_query87_00016", + "negative_test_query215_00004", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query148_00013", + "negative_test_query287_00012", + "negative_test_query216_00012", + "negative_test_query47_00010", + "negative_test_query329_00015", + "negative_test_query342_00007", + "negative_test_query150_00011", + "negative_test_query144_00003", + "negative_test_query14_00010", + "negative_test_query301_00003", + "negative_test_query326_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query78", + "ranked_ids": [ + "apositive_test_query78_00000", + "negative_test_query290_00013", + "apositive_test_query38_00003", + "negative_test_query78_00007", + "negative_test_query290_00006", + "negative_test_query78_00016", + "apositive_test_query78_00001", + "negative_test_query214_00010", + "negative_test_query200_00009", + "negative_test_query192_00009", + "negative_test_query7_00005", + "apositive_test_query326_00000", + "negative_test_query274_00014", + "negative_test_query272_00001", + "negative_test_query144_00000", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query72_00005", + "negative_test_query236_00018", + "negative_test_query329_00002" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "negative_test_query79_00013", + "negative_test_query79_00002", + "apositive_test_query79_00004", + "negative_test_query56_00002", + "negative_test_query303_00016", + "apositive_test_query221_00004", + "apositive_test_query119_00005", + "apositive_test_query79_00000", + "apositive_test_query306_00010", + "negative_test_query158_00000", + "negative_test_query313_00013", + "negative_test_query79_00009", + "negative_test_query105_00011", + "negative_test_query79_00004", + "negative_test_query38_00007", + "negative_test_query157_00000", + "apositive_test_query157_00008", + "apositive_test_query113_00002", + "apositive_test_query295_00001", + "negative_test_query53_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query8_00001", + "negative_test_query8_00011", + "negative_test_query329_00005", + "negative_test_query8_00004", + "negative_test_query8_00014", + "apositive_test_query192_00002", + "negative_test_query8_00015", + "apositive_test_query192_00003", + "negative_test_query8_00008", + "negative_test_query192_00006", + "apositive_test_query8_00000", + "apositive_test_query243_00005", + "apositive_test_query243_00010", + "negative_test_query205_00007", + "negative_test_query192_00003", + "negative_test_query8_00012", + "negative_test_query8_00000", + "negative_test_query96_00010" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query80_00004", + "negative_test_query179_00005", + "apositive_test_query80_00002", + "apositive_test_query179_00001", + "negative_test_query80_00003", + "apositive_test_query189_00004", + "negative_test_query289_00005", + "negative_test_query85_00007", + "negative_test_query80_00012", + "apositive_test_query36_00000", + "negative_test_query80_00001", + "negative_test_query273_00005", + "negative_test_query36_00010", + "apositive_test_query136_00013", + "negative_test_query179_00006", + "apositive_test_query179_00003", + "negative_test_query80_00009", + "negative_test_query287_00005", + "negative_test_query102_00009", + "negative_test_query270_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query81", + "ranked_ids": [ + "apositive_test_query81_00000", + "apositive_test_query81_00004", + "apositive_test_query81_00005", + "apositive_test_query81_00008", + "negative_test_query81_00006", + "negative_test_query81_00001", + "apositive_test_query81_00006", + "negative_test_query81_00007", + "negative_test_query81_00008", + "negative_test_query81_00009", + "negative_test_query81_00005", + "negative_test_query81_00000", + "apositive_test_query81_00001", + "negative_test_query81_00004", + "negative_test_query81_00010", + "apositive_test_query81_00003", + "negative_test_query207_00008", + "negative_test_query27_00006", + "negative_test_query246_00017", + "negative_test_query145_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00003", + "negative_test_query319_00009", + "negative_test_query82_00009", + "negative_test_query319_00008", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query340_00013", + "negative_test_query82_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query83_00002", + "apositive_test_query229_00001", + "apositive_test_query83_00004", + "apositive_test_query95_00000", + "negative_test_query168_00007", + "apositive_test_query164_00000", + "apositive_test_query146_00014", + "apositive_test_query168_00000", + "apositive_test_query83_00003", + "apositive_test_query168_00001", + "apositive_test_query217_00006", + "negative_test_query348_00006", + "negative_test_query0_00006", + "negative_test_query357_00002", + "negative_test_query279_00012", + "negative_test_query287_00004", + "negative_test_query309_00016", + "apositive_test_query83_00000", + "negative_test_query28_00005", + "negative_test_query48_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "error": "ranking must be a permutation of 1..20.", + "error_type": "RerankParseError", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query84", + "ranked_ids": [], + "valid": false + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query85", + "ranked_ids": [ + "negative_test_query85_00012", + "apositive_test_query85_00000", + "negative_test_query112_00008", + "negative_test_query106_00003", + "negative_test_query85_00000", + "negative_test_query85_00003", + "apositive_test_query85_00002", + "apositive_test_query286_00004", + "negative_test_query85_00009", + "negative_test_query85_00011", + "apositive_test_query286_00006", + "apositive_test_query85_00003", + "apositive_test_query109_00004", + "apositive_test_query284_00004", + "apositive_test_query85_00001", + "negative_test_query85_00001", + "apositive_test_query81_00008", + "apositive_test_query133_00013", + "apositive_test_query287_00004", + "negative_test_query44_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query86", + "ranked_ids": [ + "apositive_test_query86_00003", + "negative_test_query86_00001", + "negative_test_query86_00013", + "negative_test_query86_00012", + "apositive_test_query86_00001", + "negative_test_query299_00007", + "negative_test_query65_00013", + "negative_test_query86_00008", + "apositive_test_query86_00002", + "apositive_test_query86_00000", + "negative_test_query86_00000", + "negative_test_query86_00007", + "negative_test_query86_00002", + "negative_test_query210_00012", + "negative_test_query86_00011", + "negative_test_query310_00007", + "negative_test_query86_00014", + "negative_test_query245_00012", + "negative_test_query188_00013", + "negative_test_query86_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query87", + "ranked_ids": [ + "apositive_test_query87_00001", + "apositive_test_query87_00000", + "negative_test_query87_00004", + "negative_test_query87_00013", + "negative_test_query87_00000", + "negative_test_query87_00007", + "negative_test_query87_00011", + "negative_test_query87_00008", + "negative_test_query87_00014", + "negative_test_query87_00015", + "negative_test_query87_00010", + "negative_test_query87_00006", + "negative_test_query87_00016", + "negative_test_query275_00006", + "negative_test_query315_00006", + "negative_test_query87_00002", + "negative_test_query200_00011", + "negative_test_query182_00017", + "negative_test_query263_00002", + "apositive_test_query302_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "negative_test_query88_00012", + "apositive_test_query88_00001", + "negative_test_query88_00005", + "negative_test_query88_00013", + "negative_test_query325_00006", + "negative_test_query88_00008", + "negative_test_query88_00010", + "negative_test_query88_00003", + "negative_test_query88_00016", + "negative_test_query340_00012", + "negative_test_query88_00002", + "negative_test_query184_00013", + "negative_test_query88_00017", + "negative_test_query105_00014", + "apositive_test_query250_00002", + "apositive_test_query147_00002", + "apositive_test_query223_00000", + "negative_test_query240_00009", + "negative_test_query334_00006", + "negative_test_query203_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query326_00006", + "negative_test_query110_00011", + "apositive_test_query89_00004", + "negative_test_query89_00004", + "negative_test_query89_00008", + "apositive_test_query75_00006", + "negative_test_query89_00000", + "apositive_test_query89_00000", + "negative_test_query89_00005", + "negative_test_query233_00005", + "negative_test_query156_00013", + "negative_test_query130_00003", + "negative_test_query278_00009", + "negative_test_query278_00000", + "apositive_test_query255_00000", + "negative_test_query89_00007", + "negative_test_query212_00000", + "apositive_test_query68_00001", + "negative_test_query325_00012", + "negative_test_query68_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00006", + "negative_test_query90_00004", + "negative_test_query9_00008", + "negative_test_query9_00018", + "apositive_test_query9_00000", + "negative_test_query221_00002", + "negative_test_query9_00001", + "negative_test_query9_00011", + "negative_test_query9_00005", + "negative_test_query9_00004", + "negative_test_query90_00009", + "negative_test_query221_00011", + "negative_test_query305_00009", + "negative_test_query9_00016", + "negative_test_query9_00015", + "negative_test_query9_00013", + "apositive_test_query90_00001", + "negative_test_query90_00007", + "negative_test_query221_00007", + "negative_test_query9_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query231_00002", + "negative_test_query90_00010", + "apositive_test_query90_00000", + "negative_test_query90_00017", + "negative_test_query90_00012", + "negative_test_query90_00011", + "negative_test_query310_00007", + "negative_test_query227_00012", + "negative_test_query127_00001", + "negative_test_query90_00002", + "negative_test_query231_00014", + "negative_test_query9_00008", + "negative_test_query90_00000", + "negative_test_query90_00006", + "negative_test_query90_00015", + "apositive_test_query14_00002", + "negative_test_query261_00002", + "negative_test_query227_00001", + "negative_test_query227_00006", + "apositive_test_query304_00000" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query91", + "ranked_ids": [ + "apositive_test_query91_00006", + "apositive_test_query91_00002", + "negative_test_query91_00010", + "negative_test_query279_00011", + "negative_test_query292_00009", + "negative_test_query319_00002", + "negative_test_query82_00002", + "negative_test_query350_00004", + "apositive_test_query11_00006", + "negative_test_query358_00004", + "apositive_test_query91_00000", + "apositive_test_query157_00001", + "negative_test_query315_00013", + "apositive_test_query342_00001", + "negative_test_query130_00010", + "negative_test_query102_00002", + "apositive_test_query157_00000", + "negative_test_query118_00015", + "negative_test_query303_00011", + "negative_test_query59_00006" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.5 + }, + "query_id": "test_query92", + "ranked_ids": [ + "apositive_test_query92_00003", + "apositive_test_query92_00001", + "negative_test_query283_00001", + "apositive_test_query92_00004", + "negative_test_query92_00003", + "apositive_test_query92_00002", + "apositive_test_query198_00001", + "negative_test_query105_00016", + "apositive_test_query92_00005", + "apositive_test_query92_00000", + "apositive_test_query100_00001", + "negative_test_query100_00014", + "negative_test_query92_00013", + "apositive_test_query99_00003", + "negative_test_query100_00012", + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query122_00018", + "negative_test_query1_00012", + "negative_test_query283_00009" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query93_00004", + "apositive_test_query93_00000", + "apositive_test_query93_00003", + "negative_test_query93_00003", + "apositive_test_query243_00006", + "negative_test_query195_00001", + "negative_test_query277_00003", + "negative_test_query219_00000", + "apositive_test_query160_00007", + "negative_test_query129_00003", + "negative_test_query280_00012", + "apositive_test_query323_00003", + "apositive_test_query335_00003", + "negative_test_query335_00011", + "negative_test_query5_00015", + "negative_test_query335_00003", + "negative_test_query195_00006", + "negative_test_query92_00012", + "negative_test_query321_00001", + "negative_test_query108_00003" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00006", + "negative_test_query94_00003", + "apositive_test_query94_00002", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "apositive_test_query127_00001", + "apositive_test_query94_00001", + "apositive_test_query94_00005", + "apositive_test_query94_00004", + "apositive_test_query127_00010", + "apositive_test_query94_00000", + "apositive_test_query127_00004", + "apositive_test_query94_00007", + "apositive_test_query94_00006", + "negative_test_query94_00005", + "apositive_test_query127_00008", + "negative_test_query94_00009", + "negative_test_query94_00002", + "negative_test_query155_00008", + "apositive_test_query127_00015" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "apositive_test_query95_00001", + "apositive_test_query95_00000", + "negative_test_query95_00012", + "negative_test_query95_00005", + "negative_test_query95_00015", + "negative_test_query95_00003", + "negative_test_query12_00015", + "negative_test_query12_00017", + "apositive_test_query253_00007", + "negative_test_query95_00013", + "negative_test_query95_00002", + "negative_test_query95_00010", + "negative_test_query244_00010", + "apositive_test_query177_00000", + "negative_test_query95_00008", + "negative_test_query95_00006", + "negative_test_query57_00007", + "negative_test_query95_00014", + "negative_test_query334_00012", + "apositive_test_query292_00001" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query339_00000", + "negative_test_query192_00011", + "negative_test_query339_00016", + "negative_test_query96_00016", + "negative_test_query339_00017", + "negative_test_query339_00006", + "negative_test_query339_00008", + "negative_test_query339_00004", + "apositive_test_query339_00001", + "negative_test_query339_00009", + "negative_test_query96_00003", + "negative_test_query96_00013", + "negative_test_query339_00014", + "negative_test_query96_00015", + "apositive_test_query96_00001", + "negative_test_query339_00001", + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "apositive_test_query97_00003", + "apositive_test_query97_00002", + "negative_test_query97_00012", + "apositive_test_query289_00003", + "apositive_test_query97_00001", + "negative_test_query97_00009", + "apositive_test_query97_00005", + "negative_test_query16_00014", + "negative_test_query113_00006", + "negative_test_query97_00006", + "apositive_test_query51_00002", + "apositive_test_query51_00000", + "apositive_test_query267_00000", + "negative_test_query113_00009", + "negative_test_query209_00008", + "negative_test_query97_00011", + "negative_test_query84_00008", + "apositive_test_query201_00006", + "negative_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00017", + "negative_test_query98_00016", + "negative_test_query98_00007", + "negative_test_query98_00004", + "negative_test_query98_00011", + "apositive_test_query342_00000", + "apositive_test_query14_00003", + "negative_test_query231_00014", + "negative_test_query98_00005", + "negative_test_query129_00003", + "negative_test_query212_00006", + "negative_test_query98_00018", + "apositive_test_query201_00004", + "negative_test_query345_00002", + "negative_test_query300_00000", + "negative_test_query197_00007", + "negative_test_query23_00013", + "apositive_test_query14_00002", + "negative_test_query39_00005", + "negative_test_query316_00014" + ], + "valid": true + }, + { + "algorithm": "single_call_listwise@20", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query99", + "ranked_ids": [ + "negative_test_query330_00008", + "apositive_test_query193_00012", + "apositive_test_query193_00008", + "apositive_test_query193_00010", + "apositive_test_query99_00001", + "negative_test_query174_00002", + "negative_test_query193_00002", + "negative_test_query193_00001", + "negative_test_query193_00005", + "negative_test_query296_00013", + "apositive_test_query153_00012", + "apositive_test_query347_00002", + "negative_test_query256_00002", + "apositive_test_query99_00007", + "negative_test_query129_00001", + "apositive_test_query253_00003", + "apositive_test_query333_00003", + "negative_test_query27_00018", + "negative_test_query27_00011", + "negative_test_query129_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.4166666666666667 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00006", + "apositive_test_query0_00004", + "apositive_test_query0_00003", + "apositive_test_query0_00000", + "apositive_test_query107_00004", + "negative_test_query359_00002", + "negative_test_query9_00017", + "apositive_test_query0_00002", + "negative_test_query0_00001", + "negative_test_query0_00000", + "negative_test_query221_00006", + "negative_test_query0_00007", + "negative_test_query259_00006", + "negative_test_query130_00002", + "negative_test_query201_00000", + "negative_test_query357_00000", + "apositive_test_query130_00000", + "apositive_test_query153_00004", + "negative_test_query36_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query1", + "ranked_ids": [ + "apositive_test_query1_00000", + "negative_test_query61_00013", + "negative_test_query1_00009", + "negative_test_query1_00001", + "negative_test_query1_00005", + "apositive_test_query348_00003", + "negative_test_query61_00014", + "negative_test_query297_00011", + "negative_test_query184_00015", + "negative_test_query331_00002", + "negative_test_query61_00002", + "negative_test_query256_00005", + "negative_test_query202_00001", + "apositive_test_query188_00002", + "negative_test_query120_00006", + "negative_test_query27_00017", + "negative_test_query61_00003", + "negative_test_query157_00007", + "negative_test_query245_00006", + "negative_test_query110_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.7328286204777911, + "recall@5": 1.0 + }, + "query_id": "test_query10", + "ranked_ids": [ + "negative_test_query10_00010", + "apositive_test_query10_00001", + "apositive_test_query10_00002", + "apositive_test_query10_00000", + "negative_test_query10_00011", + "negative_test_query10_00007", + "negative_test_query10_00004", + "negative_test_query10_00005", + "negative_test_query169_00008", + "negative_test_query10_00000", + "negative_test_query135_00014", + "negative_test_query104_00011", + "apositive_test_query69_00000", + "negative_test_query83_00000", + "negative_test_query266_00009", + "apositive_test_query69_00016", + "negative_test_query72_00002", + "apositive_test_query44_00001", + "apositive_test_query286_00008", + "negative_test_query10_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query100", + "ranked_ids": [ + "negative_test_query100_00001", + "apositive_test_query100_00003", + "apositive_test_query100_00000", + "negative_test_query100_00013", + "negative_test_query100_00009", + "negative_test_query15_00000", + "negative_test_query290_00015", + "apositive_test_query100_00004", + "negative_test_query100_00010", + "negative_test_query237_00010", + "apositive_test_query338_00002", + "negative_test_query258_00017", + "apositive_test_query181_00000", + "apositive_test_query338_00005", + "negative_test_query100_00014", + "apositive_test_query100_00002", + "negative_test_query271_00006", + "negative_test_query252_00005", + "negative_test_query323_00000", + "negative_test_query147_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query101_00005", + "apositive_test_query101_00002", + "apositive_test_query73_00002", + "negative_test_query61_00013", + "negative_test_query241_00005", + "negative_test_query173_00013", + "negative_test_query191_00014", + "negative_test_query61_00002", + "negative_test_query68_00009", + "negative_test_query258_00000", + "negative_test_query53_00000", + "negative_test_query35_00004", + "apositive_test_query336_00001", + "apositive_test_query196_00000", + "negative_test_query103_00009", + "negative_test_query61_00014", + "negative_test_query189_00014", + "negative_test_query134_00008", + "apositive_test_query291_00000", + "negative_test_query61_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query102_00014", + "apositive_test_query20_00001", + "apositive_test_query30_00005", + "negative_test_query131_00009", + "negative_test_query313_00016", + "negative_test_query108_00009", + "negative_test_query47_00011", + "negative_test_query205_00016", + "apositive_test_query238_00000", + "negative_test_query203_00007", + "negative_test_query183_00012", + "apositive_test_query356_00008", + "negative_test_query183_00009", + "apositive_test_query47_00001", + "negative_test_query209_00006", + "negative_test_query232_00010", + "negative_test_query7_00001", + "negative_test_query47_00010", + "negative_test_query90_00016", + "negative_test_query219_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "apositive_test_query185_00003", + "negative_test_query103_00003", + "apositive_test_query103_00001", + "negative_test_query103_00011", + "negative_test_query185_00005", + "negative_test_query307_00008", + "negative_test_query217_00005", + "negative_test_query222_00009", + "negative_test_query48_00017", + "apositive_test_query185_00002", + "negative_test_query185_00003", + "negative_test_query307_00004", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query103_00006", + "negative_test_query103_00000", + "negative_test_query72_00012", + "negative_test_query48_00014", + "apositive_test_query103_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "apositive_test_query104_00000", + "negative_test_query104_00002", + "negative_test_query104_00003", + "negative_test_query104_00018", + "negative_test_query104_00013", + "negative_test_query104_00011", + "negative_test_query104_00001", + "negative_test_query104_00010", + "negative_test_query104_00015", + "negative_test_query343_00013", + "negative_test_query104_00005", + "negative_test_query104_00012", + "negative_test_query104_00014", + "negative_test_query104_00004", + "negative_test_query104_00009", + "negative_test_query104_00007", + "negative_test_query104_00016", + "negative_test_query48_00010", + "apositive_test_query28_00011", + "apositive_test_query48_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query105", + "ranked_ids": [ + "apositive_test_query105_00000", + "negative_test_query105_00006", + "negative_test_query79_00002", + "negative_test_query105_00011", + "negative_test_query105_00005", + "apositive_test_query83_00004", + "apositive_test_query158_00002", + "negative_test_query295_00005", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query119_00005", + "negative_test_query79_00004", + "negative_test_query313_00013", + "negative_test_query168_00005", + "negative_test_query142_00010", + "negative_test_query267_00011", + "apositive_test_query157_00008", + "negative_test_query359_00008", + "negative_test_query56_00002", + "negative_test_query97_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query106", + "ranked_ids": [ + "apositive_test_query106_00003", + "apositive_test_query106_00005", + "apositive_test_query106_00000", + "negative_test_query326_00008", + "apositive_test_query177_00008", + "apositive_test_query106_00001", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query286_00000", + "negative_test_query36_00001", + "negative_test_query106_00012", + "negative_test_query106_00000", + "apositive_test_query42_00006", + "negative_test_query114_00003", + "negative_test_query212_00002", + "apositive_test_query286_00005", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query106_00006", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "negative_test_query105_00015", + "apositive_test_query36_00002", + "apositive_test_query357_00003", + "negative_test_query146_00000", + "negative_test_query305_00015", + "apositive_test_query276_00003", + "negative_test_query46_00012", + "negative_test_query160_00001", + "negative_test_query142_00000", + "negative_test_query296_00008", + "apositive_test_query221_00001", + "negative_test_query333_00005", + "apositive_test_query142_00002", + "negative_test_query105_00008", + "negative_test_query142_00004", + "negative_test_query31_00013", + "negative_test_query124_00005", + "apositive_test_query211_00001", + "apositive_test_query142_00004", + "negative_test_query296_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.5 + }, + "query_id": "test_query108", + "ranked_ids": [ + "apositive_test_query108_00000", + "negative_test_query311_00006", + "apositive_test_query108_00007", + "apositive_test_query108_00002", + "apositive_test_query108_00003", + "negative_test_query108_00011", + "negative_test_query57_00009", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query102_00012", + "negative_test_query108_00010", + "negative_test_query108_00008", + "negative_test_query108_00006", + "negative_test_query102_00007", + "apositive_test_query108_00005", + "negative_test_query108_00001", + "negative_test_query168_00001", + "negative_test_query250_00009", + "negative_test_query108_00007", + "negative_test_query250_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query109", + "ranked_ids": [ + "apositive_test_query109_00000", + "apositive_test_query109_00003", + "negative_test_query190_00003", + "negative_test_query109_00003", + "negative_test_query109_00013", + "negative_test_query109_00000", + "negative_test_query109_00012", + "negative_test_query268_00010", + "apositive_test_query109_00001", + "apositive_test_query319_00008", + "negative_test_query109_00011", + "negative_test_query109_00014", + "apositive_test_query82_00008", + "apositive_test_query248_00006", + "negative_test_query109_00008", + "negative_test_query109_00009", + "negative_test_query109_00004", + "negative_test_query38_00006", + "negative_test_query6_00006", + "negative_test_query51_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00001", + "apositive_test_query11_00000", + "apositive_test_query11_00004", + "negative_test_query226_00010", + "apositive_test_query11_00006", + "apositive_test_query11_00007", + "negative_test_query132_00003", + "negative_test_query309_00001", + "negative_test_query309_00002", + "negative_test_query11_00004", + "negative_test_query358_00005", + "negative_test_query309_00004", + "apositive_test_query358_00006", + "negative_test_query123_00007", + "apositive_test_query358_00007", + "negative_test_query132_00013", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "apositive_test_query11_00003", + "negative_test_query11_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "negative_test_query89_00006", + "negative_test_query10_00005", + "negative_test_query110_00002", + "negative_test_query110_00005", + "negative_test_query110_00014", + "negative_test_query110_00013", + "apositive_test_query344_00002", + "apositive_test_query180_00001", + "negative_test_query110_00006", + "apositive_test_query11_00002", + "negative_test_query311_00011", + "negative_test_query260_00015", + "negative_test_query110_00004", + "apositive_test_query138_00000", + "negative_test_query110_00007", + "negative_test_query17_00008", + "apositive_test_query330_00001", + "negative_test_query343_00001", + "apositive_test_query291_00009", + "apositive_test_query242_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query111_00015", + "negative_test_query137_00010", + "negative_test_query134_00006", + "negative_test_query65_00013", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query111_00018", + "negative_test_query213_00011", + "negative_test_query223_00000", + "negative_test_query111_00008", + "apositive_test_query306_00002", + "apositive_test_query162_00005", + "negative_test_query65_00007", + "negative_test_query245_00001", + "negative_test_query58_00013", + "apositive_test_query174_00008", + "negative_test_query14_00015", + "apositive_test_query308_00003", + "negative_test_query243_00001", + "negative_test_query292_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "negative_test_query112_00010", + "negative_test_query112_00004", + "apositive_test_query36_00003", + "negative_test_query112_00015", + "negative_test_query36_00014", + "negative_test_query246_00016", + "apositive_test_query286_00005", + "apositive_test_query106_00003", + "negative_test_query112_00016", + "apositive_test_query286_00006", + "apositive_test_query246_00001", + "negative_test_query216_00002", + "negative_test_query112_00013", + "negative_test_query67_00003", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "negative_test_query80_00007", + "negative_test_query6_00003", + "negative_test_query36_00010", + "negative_test_query85_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query113", + "ranked_ids": [ + "apositive_test_query113_00001", + "apositive_test_query113_00000", + "negative_test_query67_00014", + "negative_test_query7_00016", + "negative_test_query38_00003", + "negative_test_query6_00016", + "negative_test_query201_00008", + "negative_test_query113_00003", + "apositive_test_query113_00002", + "negative_test_query67_00002", + "apositive_test_query67_00000", + "negative_test_query105_00007", + "negative_test_query188_00005", + "negative_test_query223_00003", + "negative_test_query216_00011", + "apositive_test_query186_00005", + "apositive_test_query188_00001", + "apositive_test_query79_00001", + "negative_test_query47_00005", + "negative_test_query105_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query114", + "ranked_ids": [ + "apositive_test_query114_00001", + "apositive_test_query114_00000", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "negative_test_query262_00007", + "apositive_test_query114_00003", + "apositive_test_query124_00004", + "negative_test_query114_00008", + "negative_test_query114_00002", + "negative_test_query349_00003", + "apositive_test_query114_00006", + "negative_test_query114_00005", + "negative_test_query114_00004", + "negative_test_query114_00007", + "negative_test_query114_00001", + "negative_test_query114_00010", + "negative_test_query114_00003", + "apositive_test_query106_00003", + "negative_test_query106_00009", + "negative_test_query81_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.5 + }, + "query_id": "test_query115", + "ranked_ids": [ + "negative_test_query115_00002", + "apositive_test_query115_00002", + "apositive_test_query115_00004", + "negative_test_query115_00006", + "apositive_test_query115_00001", + "negative_test_query115_00003", + "apositive_test_query348_00000", + "negative_test_query115_00007", + "apositive_test_query115_00003", + "apositive_test_query115_00000", + "negative_test_query115_00012", + "apositive_test_query348_00009", + "negative_test_query115_00004", + "negative_test_query276_00006", + "negative_test_query139_00000", + "negative_test_query205_00004", + "apositive_test_query121_00001", + "negative_test_query115_00010", + "apositive_test_query133_00010", + "negative_test_query121_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "apositive_test_query116_00000", + "apositive_test_query49_00000", + "negative_test_query116_00015", + "negative_test_query59_00013", + "negative_test_query116_00003", + "negative_test_query116_00013", + "negative_test_query116_00008", + "negative_test_query59_00007", + "negative_test_query116_00002", + "negative_test_query116_00000", + "negative_test_query116_00004", + "negative_test_query261_00013", + "negative_test_query116_00011", + "negative_test_query98_00005", + "negative_test_query116_00006", + "negative_test_query116_00007", + "negative_test_query116_00009", + "negative_test_query116_00010", + "negative_test_query116_00014", + "negative_test_query116_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00003", + "negative_test_query117_00010", + "negative_test_query63_00001", + "negative_test_query117_00009", + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "negative_test_query194_00002", + "apositive_test_query117_00004", + "apositive_test_query117_00006", + "negative_test_query48_00000", + "apositive_test_query117_00000", + "negative_test_query117_00001", + "apositive_test_query117_00007", + "negative_test_query117_00004", + "negative_test_query117_00008", + "negative_test_query117_00007", + "negative_test_query175_00013", + "apositive_test_query134_00004", + "negative_test_query117_00003", + "negative_test_query202_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "apositive_test_query155_00005", + "negative_test_query23_00005", + "negative_test_query296_00007", + "apositive_test_query174_00010", + "apositive_test_query174_00003", + "negative_test_query340_00007", + "apositive_test_query174_00004", + "negative_test_query84_00012", + "apositive_test_query310_00001", + "negative_test_query171_00005", + "negative_test_query171_00016", + "apositive_test_query171_00001", + "apositive_test_query174_00005", + "negative_test_query296_00010", + "apositive_test_query340_00002", + "negative_test_query88_00007", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "apositive_test_query174_00006", + "apositive_test_query217_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query119", + "ranked_ids": [ + "negative_test_query158_00000", + "apositive_test_query221_00004", + "apositive_test_query119_00005", + "negative_test_query119_00005", + "negative_test_query56_00002", + "apositive_test_query119_00001", + "negative_test_query158_00001", + "apositive_test_query306_00010", + "apositive_test_query79_00004", + "negative_test_query79_00002", + "negative_test_query119_00002", + "apositive_test_query113_00002", + "negative_test_query223_00011", + "negative_test_query152_00004", + "negative_test_query157_00001", + "negative_test_query359_00005", + "apositive_test_query119_00012", + "negative_test_query157_00007", + "apositive_test_query26_00000", + "apositive_test_query105_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query12_00016", + "negative_test_query12_00014", + "negative_test_query12_00003", + "negative_test_query12_00004", + "negative_test_query12_00013", + "apositive_test_query12_00000", + "negative_test_query12_00017", + "negative_test_query12_00007", + "negative_test_query12_00008", + "negative_test_query12_00006", + "negative_test_query12_00015", + "negative_test_query12_00011", + "negative_test_query12_00001", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query231_00013", + "negative_test_query234_00014", + "apositive_test_query185_00000", + "apositive_test_query217_00005", + "negative_test_query175_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00002", + "negative_test_query120_00008", + "apositive_test_query120_00003", + "negative_test_query1_00009", + "negative_test_query352_00013", + "apositive_test_query120_00002", + "negative_test_query120_00001", + "negative_test_query121_00002", + "apositive_test_query120_00000", + "negative_test_query120_00010", + "apositive_test_query117_00007", + "negative_test_query120_00012", + "apositive_test_query120_00001", + "negative_test_query303_00018", + "apositive_test_query352_00000", + "negative_test_query121_00012", + "negative_test_query120_00006", + "negative_test_query148_00014", + "apositive_test_query32_00007", + "negative_test_query196_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00002", + "apositive_test_query121_00000", + "apositive_test_query121_00001", + "apositive_test_query121_00003", + "negative_test_query121_00008", + "apositive_test_query121_00004", + "apositive_test_query121_00005", + "negative_test_query121_00003", + "negative_test_query121_00013", + "negative_test_query121_00000", + "negative_test_query121_00012", + "apositive_test_query307_00010", + "negative_test_query44_00012", + "negative_test_query308_00009", + "negative_test_query256_00003", + "negative_test_query313_00006", + "negative_test_query53_00003", + "negative_test_query121_00007", + "negative_test_query133_00001", + "negative_test_query115_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query61_00015", + "negative_test_query195_00008", + "negative_test_query17_00012", + "negative_test_query210_00003", + "negative_test_query86_00006", + "apositive_test_query134_00000", + "apositive_test_query254_00002", + "apositive_test_query173_00000", + "negative_test_query86_00001", + "apositive_test_query122_00000", + "negative_test_query64_00017", + "negative_test_query173_00003", + "negative_test_query297_00000", + "negative_test_query5_00001", + "negative_test_query179_00002", + "apositive_test_query173_00001", + "negative_test_query134_00007", + "negative_test_query179_00014", + "apositive_test_query175_00004", + "negative_test_query358_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.4 + }, + "query_id": "test_query123", + "ranked_ids": [ + "negative_test_query123_00013", + "apositive_test_query123_00000", + "negative_test_query123_00005", + "negative_test_query123_00008", + "apositive_test_query123_00002", + "negative_test_query155_00001", + "negative_test_query123_00004", + "apositive_test_query123_00003", + "negative_test_query123_00001", + "apositive_test_query123_00004", + "negative_test_query123_00010", + "negative_test_query161_00009", + "negative_test_query123_00006", + "negative_test_query340_00012", + "negative_test_query309_00008", + "negative_test_query123_00007", + "negative_test_query269_00006", + "negative_test_query123_00002", + "negative_test_query123_00000", + "negative_test_query155_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "negative_test_query187_00003", + "apositive_test_query194_00000", + "apositive_test_query13_00004", + "apositive_test_query13_00000", + "negative_test_query347_00010", + "negative_test_query124_00003", + "apositive_test_query162_00002", + "negative_test_query42_00006", + "negative_test_query349_00017", + "negative_test_query42_00005", + "negative_test_query123_00009", + "negative_test_query187_00006", + "negative_test_query262_00011", + "apositive_test_query162_00001", + "negative_test_query309_00012", + "negative_test_query76_00004", + "negative_test_query144_00010", + "negative_test_query171_00002", + "negative_test_query339_00008", + "negative_test_query93_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.3191471544989227, + "recall@5": 0.5 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query125_00014", + "negative_test_query140_00017", + "negative_test_query360_00016", + "apositive_test_query125_00003", + "apositive_test_query125_00000", + "negative_test_query125_00003", + "negative_test_query125_00000", + "negative_test_query3_00000", + "negative_test_query125_00004", + "negative_test_query125_00013", + "negative_test_query125_00009", + "negative_test_query6_00003", + "negative_test_query125_00006", + "negative_test_query12_00009", + "negative_test_query301_00012", + "apositive_test_query273_00003", + "apositive_test_query286_00009", + "negative_test_query125_00011", + "apositive_test_query286_00006", + "negative_test_query260_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query126", + "ranked_ids": [ + "negative_test_query126_00017", + "negative_test_query126_00000", + "apositive_test_query126_00000", + "apositive_test_query151_00002", + "negative_test_query27_00013", + "negative_test_query190_00017", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "negative_test_query126_00007", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query190_00005", + "negative_test_query311_00005", + "negative_test_query131_00009", + "negative_test_query151_00006", + "apositive_test_query330_00001", + "apositive_test_query330_00002", + "negative_test_query220_00000", + "apositive_test_query5_00002", + "negative_test_query12_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00004", + "apositive_test_query127_00016", + "apositive_test_query127_00003", + "apositive_test_query127_00000", + "apositive_test_query127_00008", + "apositive_test_query127_00012", + "apositive_test_query127_00007", + "negative_test_query94_00009", + "apositive_test_query127_00009", + "negative_test_query267_00009", + "apositive_test_query94_00005", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "apositive_test_query127_00013", + "apositive_test_query127_00006", + "negative_test_query94_00003", + "apositive_test_query127_00014", + "apositive_test_query311_00003", + "apositive_test_query163_00000", + "negative_test_query198_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.25 + }, + "query_id": "test_query128", + "ranked_ids": [ + "negative_test_query256_00011", + "apositive_test_query128_00000", + "negative_test_query21_00017", + "apositive_test_query128_00001", + "negative_test_query283_00006", + "negative_test_query21_00000", + "apositive_test_query21_00000", + "negative_test_query128_00006", + "negative_test_query310_00009", + "apositive_test_query216_00002", + "negative_test_query8_00017", + "apositive_test_query196_00004", + "apositive_test_query32_00001", + "apositive_test_query8_00000", + "apositive_test_query142_00000", + "negative_test_query128_00010", + "apositive_test_query80_00005", + "negative_test_query128_00011", + "negative_test_query192_00006", + "negative_test_query75_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query129", + "ranked_ids": [ + "apositive_test_query129_00000", + "negative_test_query211_00012", + "negative_test_query156_00017", + "apositive_test_query244_00001", + "negative_test_query245_00014", + "negative_test_query63_00002", + "negative_test_query63_00006", + "negative_test_query165_00012", + "negative_test_query129_00010", + "apositive_test_query129_00001", + "negative_test_query117_00006", + "apositive_test_query356_00006", + "apositive_test_query150_00000", + "negative_test_query84_00008", + "apositive_test_query308_00004", + "negative_test_query16_00007", + "negative_test_query187_00012", + "negative_test_query342_00013", + "apositive_test_query155_00002", + "apositive_test_query249_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query13", + "ranked_ids": [ + "apositive_test_query13_00002", + "negative_test_query13_00001", + "apositive_test_query13_00006", + "apositive_test_query13_00001", + "negative_test_query13_00002", + "negative_test_query207_00013", + "apositive_test_query13_00000", + "apositive_test_query13_00004", + "negative_test_query13_00003", + "negative_test_query13_00000", + "apositive_test_query13_00008", + "apositive_test_query13_00012", + "apositive_test_query13_00007", + "apositive_test_query13_00009", + "apositive_test_query13_00010", + "apositive_test_query13_00003", + "negative_test_query13_00004", + "negative_test_query13_00006", + "negative_test_query13_00005", + "negative_test_query245_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query130", + "ranked_ids": [ + "negative_test_query130_00001", + "apositive_test_query130_00005", + "negative_test_query345_00002", + "negative_test_query130_00008", + "negative_test_query198_00011", + "apositive_test_query130_00002", + "negative_test_query198_00003", + "negative_test_query90_00014", + "apositive_test_query284_00003", + "apositive_test_query284_00001", + "negative_test_query130_00006", + "negative_test_query284_00001", + "apositive_test_query90_00001", + "apositive_test_query323_00004", + "negative_test_query284_00013", + "negative_test_query284_00009", + "apositive_test_query30_00002", + "negative_test_query289_00004", + "apositive_test_query275_00000", + "apositive_test_query233_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query131", + "ranked_ids": [ + "apositive_test_query131_00000", + "negative_test_query186_00012", + "negative_test_query131_00001", + "apositive_test_query131_00001", + "negative_test_query247_00004", + "negative_test_query313_00008", + "negative_test_query131_00002", + "apositive_test_query106_00000", + "negative_test_query284_00002", + "apositive_test_query344_00003", + "negative_test_query284_00006", + "negative_test_query357_00001", + "apositive_test_query50_00008", + "negative_test_query168_00013", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query158_00001", + "negative_test_query184_00012", + "negative_test_query345_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "negative_test_query132_00006", + "negative_test_query123_00012", + "negative_test_query81_00001", + "negative_test_query81_00005", + "negative_test_query132_00018", + "negative_test_query132_00000", + "negative_test_query132_00004", + "negative_test_query181_00008", + "negative_test_query132_00010", + "negative_test_query132_00012", + "negative_test_query11_00006", + "negative_test_query304_00000", + "apositive_test_query161_00007", + "negative_test_query240_00011", + "negative_test_query132_00016", + "negative_test_query184_00015", + "apositive_test_query3_00004", + "negative_test_query148_00011", + "negative_test_query246_00016", + "apositive_test_query187_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.2 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query287_00001", + "apositive_test_query133_00000", + "apositive_test_query133_00001", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query133_00014", + "apositive_test_query129_00004", + "apositive_test_query133_00002", + "apositive_test_query91_00002", + "negative_test_query310_00002", + "negative_test_query206_00017", + "negative_test_query59_00008", + "apositive_test_query129_00006", + "apositive_test_query174_00011", + "negative_test_query12_00014", + "apositive_test_query129_00002", + "negative_test_query260_00006", + "apositive_test_query129_00007", + "apositive_test_query129_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query134", + "ranked_ids": [ + "negative_test_query134_00010", + "negative_test_query359_00011", + "negative_test_query134_00008", + "negative_test_query218_00009", + "apositive_test_query134_00002", + "negative_test_query134_00006", + "negative_test_query107_00003", + "apositive_test_query213_00001", + "negative_test_query108_00003", + "apositive_test_query357_00000", + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query142_00001", + "negative_test_query238_00001", + "negative_test_query46_00004", + "negative_test_query46_00011", + "apositive_test_query259_00011", + "apositive_test_query238_00001", + "negative_test_query134_00000", + "apositive_test_query276_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00003", + "negative_test_query135_00017", + "negative_test_query135_00007", + "apositive_test_query135_00000", + "negative_test_query135_00011", + "apositive_test_query135_00001", + "apositive_test_query133_00011", + "negative_test_query135_00001", + "negative_test_query135_00005", + "negative_test_query287_00009", + "negative_test_query135_00015", + "negative_test_query135_00014", + "negative_test_query135_00013", + "negative_test_query135_00004", + "negative_test_query135_00009", + "negative_test_query297_00017", + "negative_test_query325_00012", + "negative_test_query135_00008", + "negative_test_query135_00010", + "apositive_test_query28_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query136", + "ranked_ids": [ + "negative_test_query128_00011", + "apositive_test_query136_00002", + "apositive_test_query136_00004", + "apositive_test_query136_00014", + "apositive_test_query136_00011", + "apositive_test_query136_00000", + "apositive_test_query136_00010", + "apositive_test_query136_00007", + "apositive_test_query136_00001", + "apositive_test_query136_00009", + "apositive_test_query136_00013", + "apositive_test_query136_00005", + "negative_test_query301_00001", + "negative_test_query136_00003", + "negative_test_query149_00007", + "negative_test_query129_00001", + "apositive_test_query60_00014", + "negative_test_query60_00002", + "negative_test_query8_00010", + "negative_test_query245_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query137", + "ranked_ids": [ + "apositive_test_query253_00000", + "negative_test_query253_00007", + "negative_test_query137_00014", + "apositive_test_query13_00001", + "negative_test_query137_00003", + "apositive_test_query137_00000", + "negative_test_query349_00006", + "negative_test_query95_00004", + "negative_test_query13_00002", + "negative_test_query207_00006", + "negative_test_query207_00013", + "negative_test_query137_00012", + "negative_test_query74_00001", + "apositive_test_query137_00002", + "apositive_test_query137_00003", + "negative_test_query262_00009", + "negative_test_query137_00000", + "negative_test_query149_00013", + "negative_test_query74_00003", + "negative_test_query74_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query138", + "ranked_ids": [ + "apositive_test_query138_00000", + "apositive_test_query138_00001", + "negative_test_query138_00006", + "negative_test_query138_00003", + "negative_test_query138_00014", + "negative_test_query138_00013", + "negative_test_query138_00004", + "negative_test_query138_00015", + "negative_test_query138_00008", + "apositive_test_query138_00002", + "negative_test_query183_00004", + "negative_test_query138_00001", + "negative_test_query138_00005", + "apositive_test_query356_00003", + "apositive_test_query23_00003", + "apositive_test_query244_00005", + "negative_test_query240_00000", + "negative_test_query132_00009", + "negative_test_query213_00006", + "negative_test_query184_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query35_00001", + "apositive_test_query139_00001", + "apositive_test_query35_00000", + "apositive_test_query139_00000", + "negative_test_query35_00010", + "apositive_test_query146_00007", + "negative_test_query8_00009", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query123_00006", + "negative_test_query298_00008", + "negative_test_query101_00006", + "negative_test_query269_00006", + "negative_test_query231_00012", + "apositive_test_query354_00001", + "negative_test_query262_00002", + "negative_test_query320_00007", + "negative_test_query81_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query14", + "ranked_ids": [ + "apositive_test_query77_00000", + "negative_test_query14_00001", + "negative_test_query155_00009", + "apositive_test_query14_00003", + "negative_test_query14_00007", + "negative_test_query14_00005", + "negative_test_query260_00014", + "apositive_test_query296_00005", + "negative_test_query260_00000", + "negative_test_query53_00003", + "negative_test_query14_00006", + "negative_test_query155_00001", + "negative_test_query179_00001", + "apositive_test_query216_00002", + "negative_test_query14_00004", + "apositive_test_query64_00000", + "negative_test_query199_00009", + "negative_test_query102_00011", + "apositive_test_query26_00006", + "negative_test_query350_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00005", + "negative_test_query140_00016", + "negative_test_query140_00001", + "apositive_test_query281_00013", + "negative_test_query140_00003", + "negative_test_query215_00000", + "apositive_test_query273_00005", + "negative_test_query140_00013", + "negative_test_query140_00010", + "negative_test_query140_00014", + "negative_test_query140_00018", + "negative_test_query140_00002", + "negative_test_query257_00012", + "negative_test_query257_00003", + "apositive_test_query18_00003", + "apositive_test_query18_00001", + "negative_test_query140_00009", + "negative_test_query18_00006", + "negative_test_query18_00013", + "negative_test_query140_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query141", + "ranked_ids": [ + "apositive_test_query288_00003", + "apositive_test_query141_00006", + "negative_test_query105_00014", + "apositive_test_query141_00005", + "negative_test_query245_00016", + "negative_test_query245_00008", + "negative_test_query57_00002", + "apositive_test_query141_00000", + "negative_test_query308_00005", + "negative_test_query141_00006", + "negative_test_query176_00003", + "negative_test_query25_00006", + "apositive_test_query288_00005", + "negative_test_query325_00008", + "negative_test_query220_00011", + "negative_test_query344_00013", + "apositive_test_query288_00007", + "negative_test_query245_00013", + "negative_test_query141_00010", + "apositive_test_query246_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query142", + "ranked_ids": [ + "apositive_test_query142_00001", + "negative_test_query238_00001", + "negative_test_query46_00001", + "negative_test_query102_00001", + "apositive_test_query142_00002", + "negative_test_query142_00003", + "negative_test_query46_00002", + "apositive_test_query142_00003", + "negative_test_query223_00005", + "negative_test_query142_00009", + "negative_test_query46_00012", + "negative_test_query105_00006", + "negative_test_query203_00003", + "negative_test_query142_00007", + "negative_test_query39_00008", + "negative_test_query359_00005", + "negative_test_query105_00008", + "negative_test_query200_00014", + "negative_test_query142_00004", + "negative_test_query337_00016" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query143", + "ranked_ids": [ + "apositive_test_query143_00000", + "negative_test_query143_00006", + "negative_test_query143_00011", + "negative_test_query143_00004", + "negative_test_query143_00009", + "negative_test_query45_00009", + "negative_test_query143_00005", + "apositive_test_query25_00006", + "negative_test_query143_00000", + "negative_test_query143_00012", + "negative_test_query90_00014", + "negative_test_query143_00010", + "negative_test_query143_00007", + "negative_test_query143_00014", + "negative_test_query143_00013", + "negative_test_query143_00015", + "negative_test_query143_00016", + "apositive_test_query25_00005", + "negative_test_query56_00005", + "apositive_test_query308_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query144_00010", + "negative_test_query41_00001", + "negative_test_query144_00012", + "negative_test_query144_00011", + "negative_test_query41_00009", + "negative_test_query315_00011", + "negative_test_query144_00018", + "negative_test_query331_00015", + "negative_test_query166_00001", + "negative_test_query144_00016", + "negative_test_query144_00003", + "negative_test_query144_00001", + "negative_test_query41_00011", + "negative_test_query198_00016", + "negative_test_query144_00017", + "negative_test_query144_00006", + "apositive_test_query41_00000", + "negative_test_query154_00009", + "negative_test_query226_00011", + "negative_test_query144_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query145", + "ranked_ids": [ + "negative_test_query53_00012", + "apositive_test_query225_00002", + "negative_test_query230_00002", + "apositive_test_query213_00004", + "apositive_test_query160_00010", + "negative_test_query118_00000", + "apositive_test_query28_00008", + "negative_test_query79_00005", + "apositive_test_query107_00001", + "negative_test_query112_00007", + "apositive_test_query305_00002", + "negative_test_query171_00003", + "apositive_test_query207_00004", + "apositive_test_query145_00002", + "apositive_test_query307_00000", + "apositive_test_query217_00007", + "negative_test_query305_00002", + "negative_test_query187_00014", + "negative_test_query211_00013", + "negative_test_query190_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.1875 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query153_00008", + "apositive_test_query146_00004", + "apositive_test_query146_00003", + "apositive_test_query146_00005", + "negative_test_query358_00008", + "apositive_test_query3_00010", + "apositive_test_query276_00001", + "negative_test_query146_00002", + "apositive_test_query146_00010", + "apositive_test_query314_00009", + "negative_test_query146_00001", + "apositive_test_query314_00004", + "negative_test_query144_00008", + "apositive_test_query314_00001", + "apositive_test_query161_00008", + "apositive_test_query161_00007", + "apositive_test_query314_00003", + "negative_test_query65_00006", + "negative_test_query99_00001", + "apositive_test_query161_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.4 + }, + "query_id": "test_query147", + "ranked_ids": [ + "apositive_test_query161_00004", + "negative_test_query147_00013", + "apositive_test_query147_00002", + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query147_00008", + "negative_test_query309_00014", + "negative_test_query11_00009", + "negative_test_query258_00008", + "negative_test_query53_00009", + "negative_test_query354_00007", + "apositive_test_query76_00001", + "negative_test_query309_00016", + "negative_test_query200_00011", + "apositive_test_query76_00004", + "apositive_test_query147_00003", + "negative_test_query181_00008", + "negative_test_query147_00005", + "negative_test_query17_00009", + "negative_test_query180_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query148_00005", + "negative_test_query124_00010", + "negative_test_query148_00012", + "negative_test_query210_00013", + "negative_test_query89_00007", + "negative_test_query148_00018", + "negative_test_query148_00017", + "negative_test_query210_00008", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query112_00013", + "negative_test_query148_00008", + "negative_test_query196_00004", + "negative_test_query210_00006", + "negative_test_query315_00002", + "negative_test_query321_00013", + "negative_test_query80_00007", + "apositive_test_query304_00001", + "negative_test_query148_00014", + "negative_test_query210_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "negative_test_query149_00001", + "negative_test_query149_00008", + "negative_test_query149_00012", + "negative_test_query149_00011", + "apositive_test_query225_00004", + "negative_test_query149_00010", + "negative_test_query332_00008", + "negative_test_query149_00002", + "apositive_test_query145_00001", + "negative_test_query149_00017", + "apositive_test_query149_00000", + "apositive_test_query149_00001", + "negative_test_query225_00006", + "negative_test_query149_00015", + "negative_test_query63_00009", + "negative_test_query149_00013", + "negative_test_query85_00009", + "apositive_test_query113_00003", + "negative_test_query74_00007", + "negative_test_query332_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00010", + "negative_test_query15_00014", + "negative_test_query15_00012", + "negative_test_query15_00008", + "negative_test_query15_00007", + "negative_test_query266_00008", + "negative_test_query15_00013", + "negative_test_query176_00011", + "apositive_test_query325_00000", + "apositive_test_query15_00001", + "negative_test_query15_00004", + "negative_test_query15_00015", + "negative_test_query325_00011", + "negative_test_query15_00011", + "negative_test_query15_00005", + "negative_test_query15_00017", + "negative_test_query266_00013", + "negative_test_query266_00006", + "apositive_test_query242_00000", + "apositive_test_query291_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query150", + "ranked_ids": [ + "negative_test_query250_00013", + "negative_test_query150_00010", + "negative_test_query306_00004", + "apositive_test_query150_00003", + "negative_test_query139_00005", + "negative_test_query1_00012", + "apositive_test_query112_00000", + "apositive_test_query89_00000", + "apositive_test_query20_00010", + "negative_test_query62_00009", + "apositive_test_query20_00007", + "negative_test_query301_00007", + "negative_test_query213_00007", + "negative_test_query346_00008", + "negative_test_query63_00004", + "negative_test_query267_00006", + "negative_test_query254_00003", + "negative_test_query350_00018", + "apositive_test_query158_00001", + "negative_test_query313_00017" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "apositive_test_query126_00000", + "negative_test_query190_00003", + "negative_test_query151_00003", + "negative_test_query151_00006", + "negative_test_query190_00005", + "apositive_test_query300_00001", + "negative_test_query190_00014", + "negative_test_query293_00008", + "negative_test_query44_00002", + "negative_test_query151_00015", + "negative_test_query190_00015", + "negative_test_query201_00008", + "negative_test_query151_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.4 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query237_00011", + "apositive_test_query152_00005", + "apositive_test_query152_00009", + "apositive_test_query152_00001", + "apositive_test_query152_00004", + "negative_test_query312_00014", + "negative_test_query237_00000", + "negative_test_query157_00010", + "negative_test_query237_00004", + "negative_test_query124_00010", + "negative_test_query237_00009", + "negative_test_query349_00001", + "negative_test_query262_00010", + "negative_test_query349_00005", + "negative_test_query342_00005", + "negative_test_query95_00017", + "negative_test_query182_00004", + "negative_test_query253_00004", + "apositive_test_query194_00008", + "negative_test_query250_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3125 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00003", + "apositive_test_query153_00001", + "apositive_test_query153_00015", + "apositive_test_query153_00002", + "apositive_test_query153_00004", + "negative_test_query112_00007", + "negative_test_query283_00015", + "negative_test_query278_00006", + "negative_test_query359_00003", + "apositive_test_query153_00000", + "apositive_test_query153_00007", + "negative_test_query278_00007", + "negative_test_query250_00006", + "negative_test_query112_00015", + "apositive_test_query20_00006", + "negative_test_query228_00002", + "negative_test_query2_00001", + "apositive_test_query213_00004", + "apositive_test_query176_00000", + "negative_test_query322_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query154", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query20_00003", + "apositive_test_query356_00005", + "apositive_test_query358_00008", + "apositive_test_query23_00004", + "apositive_test_query20_00007", + "apositive_test_query33_00000", + "apositive_test_query154_00006", + "negative_test_query41_00007", + "negative_test_query331_00012", + "negative_test_query334_00000", + "negative_test_query133_00002", + "apositive_test_query20_00010", + "negative_test_query245_00006", + "apositive_test_query154_00005", + "negative_test_query356_00004", + "negative_test_query311_00005", + "apositive_test_query66_00000", + "negative_test_query68_00015", + "negative_test_query102_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00009", + "negative_test_query155_00003", + "negative_test_query155_00006", + "negative_test_query155_00005", + "negative_test_query155_00001", + "negative_test_query155_00012", + "apositive_test_query155_00005", + "negative_test_query155_00004", + "apositive_test_query355_00007", + "negative_test_query155_00010", + "apositive_test_query3_00008", + "apositive_test_query155_00001", + "negative_test_query155_00002", + "negative_test_query119_00001", + "negative_test_query111_00016", + "negative_test_query199_00016", + "negative_test_query155_00000", + "apositive_test_query40_00004", + "negative_test_query123_00014", + "negative_test_query99_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "apositive_test_query210_00001", + "apositive_test_query156_00000", + "negative_test_query92_00010", + "negative_test_query1_00012", + "negative_test_query283_00001", + "negative_test_query92_00003", + "negative_test_query283_00009", + "negative_test_query49_00005", + "apositive_test_query330_00002", + "negative_test_query330_00006", + "apositive_test_query39_00002", + "apositive_test_query154_00005", + "negative_test_query246_00016", + "negative_test_query92_00005", + "negative_test_query299_00014", + "negative_test_query156_00001", + "apositive_test_query25_00009", + "negative_test_query32_00004", + "negative_test_query140_00008", + "negative_test_query15_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "negative_test_query237_00004", + "apositive_test_query152_00005", + "apositive_test_query152_00001", + "negative_test_query237_00009", + "apositive_test_query237_00000", + "negative_test_query157_00005", + "negative_test_query157_00010", + "apositive_test_query152_00004", + "negative_test_query152_00003", + "negative_test_query152_00008", + "negative_test_query237_00002", + "apositive_test_query157_00001", + "negative_test_query237_00000", + "apositive_test_query157_00002", + "apositive_test_query182_00000", + "apositive_test_query237_00004", + "negative_test_query271_00003", + "apositive_test_query237_00002", + "negative_test_query312_00017", + "apositive_test_query152_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2 + }, + "query_id": "test_query158", + "ranked_ids": [ + "negative_test_query102_00015", + "negative_test_query334_00016", + "apositive_test_query158_00003", + "apositive_test_query158_00000", + "negative_test_query223_00007", + "negative_test_query105_00013", + "negative_test_query26_00004", + "apositive_test_query186_00003", + "negative_test_query223_00002", + "negative_test_query26_00005", + "negative_test_query158_00006", + "negative_test_query247_00010", + "negative_test_query102_00003", + "negative_test_query213_00002", + "negative_test_query26_00003", + "negative_test_query164_00004", + "negative_test_query26_00000", + "negative_test_query334_00009", + "apositive_test_query158_00007", + "negative_test_query26_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query159", + "ranked_ids": [ + "apositive_test_query159_00001", + "negative_test_query159_00005", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query264_00013", + "apositive_test_query212_00002", + "negative_test_query159_00016", + "apositive_test_query212_00001", + "negative_test_query38_00010", + "apositive_test_query326_00007", + "apositive_test_query50_00004", + "negative_test_query264_00003", + "apositive_test_query38_00003", + "negative_test_query7_00005", + "apositive_test_query212_00003", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query66_00003", + "negative_test_query122_00002", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query16", + "ranked_ids": [ + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query308_00004", + "apositive_test_query16_00004", + "negative_test_query16_00008", + "negative_test_query16_00001", + "negative_test_query16_00000", + "negative_test_query16_00010", + "negative_test_query16_00009", + "apositive_test_query16_00002", + "negative_test_query16_00002", + "negative_test_query280_00007", + "negative_test_query16_00012", + "negative_test_query16_00011", + "negative_test_query240_00003", + "negative_test_query237_00010", + "negative_test_query16_00003", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query208_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query160", + "ranked_ids": [ + "apositive_test_query160_00009", + "negative_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query149_00007", + "negative_test_query214_00013", + "negative_test_query330_00005", + "negative_test_query160_00004", + "apositive_test_query160_00004", + "negative_test_query211_00009", + "negative_test_query64_00009", + "apositive_test_query64_00000", + "negative_test_query64_00015", + "apositive_test_query355_00007", + "apositive_test_query120_00002", + "negative_test_query271_00010", + "negative_test_query304_00007", + "negative_test_query106_00004", + "negative_test_query176_00016", + "negative_test_query215_00010", + "apositive_test_query260_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query161", + "ranked_ids": [ + "apositive_test_query161_00006", + "negative_test_query155_00012", + "negative_test_query199_00004", + "apositive_test_query161_00005", + "apositive_test_query3_00006", + "apositive_test_query276_00010", + "apositive_test_query161_00008", + "apositive_test_query305_00002", + "negative_test_query260_00004", + "apositive_test_query301_00003", + "apositive_test_query37_00004", + "negative_test_query22_00015", + "negative_test_query57_00001", + "negative_test_query71_00006", + "apositive_test_query314_00006", + "apositive_test_query314_00004", + "negative_test_query358_00001", + "negative_test_query301_00015", + "apositive_test_query301_00000", + "apositive_test_query261_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query162", + "ranked_ids": [ + "negative_test_query177_00002", + "apositive_test_query162_00010", + "apositive_test_query162_00009", + "negative_test_query162_00004", + "negative_test_query162_00001", + "negative_test_query95_00006", + "apositive_test_query162_00000", + "apositive_test_query162_00003", + "apositive_test_query162_00005", + "apositive_test_query162_00006", + "apositive_test_query194_00006", + "negative_test_query343_00012", + "negative_test_query240_00008", + "apositive_test_query162_00004", + "apositive_test_query114_00001", + "apositive_test_query194_00003", + "negative_test_query95_00013", + "negative_test_query177_00005", + "apositive_test_query194_00009", + "negative_test_query230_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query197_00002", + "negative_test_query163_00003", + "negative_test_query163_00008", + "negative_test_query90_00011", + "negative_test_query58_00015", + "negative_test_query328_00006", + "negative_test_query47_00001", + "negative_test_query9_00005", + "negative_test_query265_00015", + "negative_test_query202_00002", + "apositive_test_query172_00006", + "apositive_test_query357_00004", + "negative_test_query16_00013", + "negative_test_query245_00001", + "negative_test_query163_00011", + "negative_test_query163_00013", + "negative_test_query197_00001", + "negative_test_query224_00010", + "negative_test_query350_00017", + "apositive_test_query152_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query164_00000", + "apositive_test_query146_00014", + "negative_test_query164_00015", + "negative_test_query158_00008", + "negative_test_query131_00014", + "negative_test_query2_00000", + "negative_test_query332_00001", + "apositive_test_query13_00000", + "negative_test_query85_00010", + "negative_test_query101_00005", + "negative_test_query165_00003", + "negative_test_query1_00003", + "negative_test_query208_00014", + "negative_test_query360_00012", + "negative_test_query100_00013", + "negative_test_query344_00009", + "negative_test_query245_00005", + "negative_test_query321_00011", + "negative_test_query309_00012", + "apositive_test_query297_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query86_00006", + "negative_test_query149_00012", + "negative_test_query122_00003", + "apositive_test_query165_00001", + "negative_test_query165_00013", + "negative_test_query86_00010", + "negative_test_query165_00007", + "negative_test_query165_00000", + "negative_test_query285_00007", + "negative_test_query348_00001", + "negative_test_query285_00004", + "negative_test_query264_00016", + "negative_test_query215_00013", + "negative_test_query299_00011", + "negative_test_query327_00016", + "negative_test_query327_00010", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query38_00010", + "apositive_test_query219_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query166", + "ranked_ids": [ + "apositive_test_query166_00005", + "negative_test_query166_00012", + "apositive_test_query166_00000", + "negative_test_query166_00006", + "negative_test_query166_00013", + "negative_test_query166_00000", + "apositive_test_query166_00003", + "apositive_test_query166_00002", + "apositive_test_query166_00001", + "negative_test_query166_00005", + "negative_test_query166_00004", + "negative_test_query166_00007", + "negative_test_query165_00015", + "negative_test_query166_00008", + "negative_test_query166_00001", + "negative_test_query166_00003", + "negative_test_query224_00014", + "negative_test_query249_00000", + "negative_test_query255_00003", + "apositive_test_query224_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query167_00012", + "negative_test_query167_00002", + "negative_test_query167_00016", + "negative_test_query167_00014", + "negative_test_query167_00004", + "negative_test_query167_00011", + "negative_test_query167_00010", + "negative_test_query167_00005", + "negative_test_query278_00010", + "negative_test_query278_00016", + "negative_test_query167_00003", + "negative_test_query167_00015", + "negative_test_query167_00001", + "negative_test_query167_00013", + "negative_test_query167_00008", + "negative_test_query167_00000", + "negative_test_query167_00007", + "negative_test_query167_00009", + "apositive_test_query288_00005", + "negative_test_query141_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.44149241373678083, + "recall@5": 0.5 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query83_00003", + "apositive_test_query168_00000", + "apositive_test_query168_00001", + "apositive_test_query190_00001", + "negative_test_query168_00014", + "apositive_test_query168_00002", + "apositive_test_query286_00007", + "negative_test_query6_00001", + "negative_test_query6_00005", + "negative_test_query168_00000", + "negative_test_query344_00009", + "negative_test_query6_00000", + "apositive_test_query340_00000", + "negative_test_query260_00011", + "negative_test_query270_00003", + "negative_test_query83_00003", + "negative_test_query85_00010", + "apositive_test_query83_00004", + "negative_test_query168_00007", + "negative_test_query168_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "negative_test_query49_00017", + "negative_test_query104_00001", + "apositive_test_query80_00003", + "negative_test_query10_00005", + "negative_test_query89_00006", + "negative_test_query72_00013", + "apositive_test_query69_00000", + "negative_test_query72_00001", + "apositive_test_query286_00008", + "negative_test_query72_00016", + "apositive_test_query180_00001", + "negative_test_query343_00001", + "apositive_test_query330_00001", + "negative_test_query260_00015", + "apositive_test_query242_00006", + "apositive_test_query291_00009", + "negative_test_query10_00013", + "negative_test_query180_00004", + "negative_test_query169_00001", + "apositive_test_query266_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00000", + "negative_test_query17_00016", + "negative_test_query130_00007", + "negative_test_query17_00005", + "negative_test_query17_00002", + "negative_test_query17_00014", + "negative_test_query302_00011", + "negative_test_query10_00005", + "apositive_test_query225_00007", + "negative_test_query34_00003", + "negative_test_query148_00001", + "negative_test_query17_00015", + "negative_test_query312_00004", + "negative_test_query17_00003", + "negative_test_query17_00012", + "negative_test_query305_00004", + "negative_test_query164_00006", + "apositive_test_query355_00007", + "apositive_test_query153_00013", + "negative_test_query292_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5555555555555556 + }, + "query_id": "test_query170", + "ranked_ids": [ + "apositive_test_query170_00006", + "apositive_test_query170_00005", + "apositive_test_query170_00004", + "apositive_test_query170_00002", + "apositive_test_query170_00000", + "negative_test_query170_00001", + "negative_test_query0_00006", + "apositive_test_query170_00007", + "negative_test_query170_00008", + "negative_test_query170_00005", + "negative_test_query170_00007", + "apositive_test_query297_00000", + "negative_test_query170_00000", + "negative_test_query348_00001", + "negative_test_query297_00005", + "apositive_test_query170_00001", + "negative_test_query170_00004", + "negative_test_query227_00002", + "apositive_test_query70_00003", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query171", + "ranked_ids": [ + "negative_test_query171_00013", + "negative_test_query171_00005", + "apositive_test_query171_00001", + "negative_test_query171_00004", + "negative_test_query171_00011", + "negative_test_query171_00000", + "negative_test_query219_00010", + "negative_test_query171_00003", + "negative_test_query24_00007", + "negative_test_query277_00007", + "negative_test_query261_00000", + "negative_test_query171_00015", + "negative_test_query171_00001", + "apositive_test_query224_00000", + "negative_test_query126_00013", + "negative_test_query171_00017", + "negative_test_query171_00007", + "negative_test_query191_00006", + "negative_test_query346_00005", + "negative_test_query219_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.375 + }, + "query_id": "test_query172", + "ranked_ids": [ + "apositive_test_query172_00003", + "negative_test_query172_00007", + "apositive_test_query172_00002", + "apositive_test_query254_00003", + "apositive_test_query172_00004", + "negative_test_query350_00009", + "negative_test_query192_00010", + "apositive_test_query136_00011", + "apositive_test_query136_00001", + "negative_test_query208_00015", + "negative_test_query173_00006", + "negative_test_query8_00000", + "negative_test_query64_00006", + "apositive_test_query136_00009", + "negative_test_query192_00000", + "negative_test_query208_00007", + "negative_test_query349_00001", + "apositive_test_query124_00002", + "apositive_test_query13_00003", + "apositive_test_query172_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query64_00017", + "apositive_test_query173_00004", + "negative_test_query173_00006", + "apositive_test_query173_00002", + "negative_test_query173_00003", + "negative_test_query173_00011", + "apositive_test_query173_00003", + "negative_test_query64_00006", + "negative_test_query350_00002", + "negative_test_query173_00013", + "apositive_test_query173_00000", + "negative_test_query173_00000", + "negative_test_query339_00006", + "apositive_test_query134_00000", + "negative_test_query64_00004", + "apositive_test_query173_00001", + "negative_test_query64_00013", + "apositive_test_query101_00004", + "negative_test_query303_00005", + "negative_test_query222_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00000", + "apositive_test_query174_00004", + "apositive_test_query174_00012", + "apositive_test_query174_00001", + "negative_test_query19_00012", + "apositive_test_query174_00005", + "apositive_test_query4_00012", + "negative_test_query262_00005", + "apositive_test_query174_00002", + "apositive_test_query174_00003", + "apositive_test_query174_00006", + "apositive_test_query174_00009", + "apositive_test_query174_00008", + "apositive_test_query174_00010", + "apositive_test_query174_00007", + "apositive_test_query174_00011", + "apositive_test_query174_00014", + "apositive_test_query174_00015", + "apositive_test_query174_00013", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00004", + "apositive_test_query175_00003", + "apositive_test_query175_00000", + "negative_test_query175_00010", + "apositive_test_query175_00005", + "negative_test_query175_00013", + "negative_test_query204_00001", + "negative_test_query204_00000", + "apositive_test_query50_00015", + "negative_test_query105_00012", + "negative_test_query38_00006", + "apositive_test_query353_00005", + "apositive_test_query204_00010", + "negative_test_query97_00007", + "negative_test_query204_00002", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "negative_test_query105_00016", + "negative_test_query7_00000", + "negative_test_query175_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query176_00015", + "negative_test_query195_00006", + "apositive_test_query128_00007", + "negative_test_query284_00006", + "negative_test_query9_00003", + "negative_test_query221_00007", + "negative_test_query224_00003", + "negative_test_query195_00012", + "negative_test_query191_00008", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query284_00002", + "negative_test_query24_00014", + "apositive_test_query344_00003", + "apositive_test_query92_00002", + "negative_test_query144_00018", + "negative_test_query35_00005", + "negative_test_query93_00000", + "apositive_test_query56_00006", + "negative_test_query9_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query172_00008", + "negative_test_query177_00008", + "negative_test_query124_00012", + "negative_test_query152_00005", + "negative_test_query177_00004", + "apositive_test_query177_00004", + "apositive_test_query177_00006", + "negative_test_query147_00005", + "negative_test_query177_00006", + "apositive_test_query162_00009", + "negative_test_query147_00002", + "negative_test_query124_00001", + "apositive_test_query162_00010", + "apositive_test_query230_00005", + "negative_test_query147_00011", + "apositive_test_query177_00008", + "negative_test_query147_00009", + "apositive_test_query229_00008", + "negative_test_query147_00001", + "negative_test_query162_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query178", + "ranked_ids": [ + "negative_test_query178_00007", + "apositive_test_query178_00005", + "negative_test_query178_00001", + "negative_test_query285_00009", + "apositive_test_query285_00006", + "negative_test_query165_00001", + "apositive_test_query178_00004", + "negative_test_query274_00001", + "apositive_test_query352_00001", + "negative_test_query214_00002", + "negative_test_query43_00001", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query178_00010", + "negative_test_query178_00003", + "negative_test_query178_00012", + "negative_test_query321_00017", + "negative_test_query327_00004", + "negative_test_query285_00008", + "negative_test_query178_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "negative_test_query179_00014", + "apositive_test_query173_00003", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "apositive_test_query173_00000", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query303_00005", + "negative_test_query14_00001", + "negative_test_query316_00001", + "negative_test_query297_00000", + "apositive_test_query316_00001", + "negative_test_query279_00010", + "negative_test_query206_00009", + "negative_test_query173_00013", + "negative_test_query191_00003", + "negative_test_query179_00001", + "negative_test_query64_00003", + "negative_test_query191_00011", + "negative_test_query260_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.6 + }, + "query_id": "test_query18", + "ranked_ids": [ + "apositive_test_query18_00001", + "apositive_test_query18_00003", + "negative_test_query18_00011", + "negative_test_query18_00006", + "apositive_test_query18_00000", + "negative_test_query140_00003", + "apositive_test_query18_00004", + "apositive_test_query18_00002", + "apositive_test_query281_00013", + "negative_test_query18_00012", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query18_00013", + "negative_test_query257_00012", + "negative_test_query287_00004", + "negative_test_query88_00002", + "negative_test_query156_00018", + "apositive_test_query288_00004", + "negative_test_query300_00015", + "negative_test_query88_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query180", + "ranked_ids": [ + "apositive_test_query180_00000", + "apositive_test_query180_00002", + "apositive_test_query180_00001", + "negative_test_query180_00004", + "negative_test_query180_00002", + "apositive_test_query262_00003", + "negative_test_query312_00018", + "apositive_test_query262_00000", + "negative_test_query180_00011", + "negative_test_query207_00007", + "apositive_test_query262_00002", + "negative_test_query180_00007", + "negative_test_query180_00016", + "negative_test_query246_00003", + "negative_test_query180_00006", + "negative_test_query112_00014", + "negative_test_query114_00003", + "negative_test_query169_00015", + "negative_test_query95_00014", + "negative_test_query176_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00006", + "negative_test_query181_00005", + "negative_test_query181_00012", + "negative_test_query181_00017", + "negative_test_query181_00010", + "negative_test_query181_00002", + "negative_test_query181_00014", + "negative_test_query181_00001", + "negative_test_query181_00004", + "negative_test_query181_00007", + "negative_test_query181_00018", + "negative_test_query181_00016", + "negative_test_query348_00007", + "apositive_test_query256_00003", + "negative_test_query181_00008", + "negative_test_query96_00006", + "negative_test_query320_00007", + "negative_test_query181_00009", + "apositive_test_query119_00011", + "negative_test_query183_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "negative_test_query182_00017", + "apositive_test_query182_00000", + "negative_test_query182_00004", + "negative_test_query182_00011", + "negative_test_query237_00004", + "negative_test_query337_00012", + "negative_test_query240_00001", + "negative_test_query182_00018", + "negative_test_query182_00002", + "apositive_test_query152_00009", + "negative_test_query237_00006", + "negative_test_query230_00008", + "negative_test_query296_00010", + "negative_test_query237_00001", + "negative_test_query237_00007", + "negative_test_query342_00001", + "negative_test_query342_00003", + "negative_test_query342_00004", + "negative_test_query342_00009", + "negative_test_query77_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00009", + "negative_test_query183_00012", + "negative_test_query183_00014", + "negative_test_query7_00007", + "negative_test_query183_00006", + "apositive_test_query7_00001", + "negative_test_query183_00007", + "negative_test_query7_00008", + "negative_test_query183_00013", + "negative_test_query238_00007", + "negative_test_query183_00015", + "negative_test_query183_00008", + "negative_test_query7_00009", + "negative_test_query183_00018", + "negative_test_query112_00017", + "negative_test_query183_00010", + "negative_test_query183_00000", + "negative_test_query7_00014", + "negative_test_query7_00010", + "apositive_test_query178_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "negative_test_query216_00008", + "negative_test_query216_00014", + "negative_test_query216_00007", + "apositive_test_query72_00000", + "negative_test_query158_00008", + "apositive_test_query216_00004", + "apositive_test_query216_00001", + "apositive_test_query316_00000", + "apositive_test_query33_00006", + "apositive_test_query216_00000", + "negative_test_query156_00001", + "negative_test_query101_00007", + "apositive_test_query33_00004", + "negative_test_query344_00010", + "negative_test_query228_00011", + "negative_test_query216_00002", + "negative_test_query322_00000", + "negative_test_query216_00006", + "negative_test_query41_00013", + "apositive_test_query20_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query185", + "ranked_ids": [ + "apositive_test_query307_00008", + "negative_test_query103_00001", + "apositive_test_query185_00003", + "negative_test_query185_00002", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query72_00009", + "negative_test_query185_00000", + "negative_test_query103_00000", + "negative_test_query350_00007", + "negative_test_query72_00011", + "negative_test_query72_00017", + "negative_test_query185_00005", + "negative_test_query48_00009", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query139_00014", + "apositive_test_query43_00006", + "negative_test_query48_00014", + "negative_test_query72_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "negative_test_query247_00012", + "negative_test_query247_00007", + "negative_test_query186_00005", + "negative_test_query313_00015", + "negative_test_query46_00001", + "apositive_test_query306_00003", + "negative_test_query47_00009", + "apositive_test_query79_00005", + "negative_test_query336_00008", + "negative_test_query142_00009", + "negative_test_query29_00003", + "negative_test_query313_00014", + "apositive_test_query50_00010", + "negative_test_query313_00005", + "apositive_test_query50_00000", + "negative_test_query105_00001", + "negative_test_query334_00017", + "negative_test_query259_00003", + "apositive_test_query50_00001", + "negative_test_query57_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query306_00004", + "negative_test_query245_00006", + "negative_test_query279_00002", + "apositive_test_query311_00002", + "negative_test_query176_00008", + "apositive_test_query176_00000", + "apositive_test_query127_00014", + "negative_test_query230_00000", + "apositive_test_query358_00008", + "negative_test_query62_00009", + "apositive_test_query154_00005", + "negative_test_query168_00002", + "negative_test_query184_00015", + "negative_test_query331_00002", + "negative_test_query13_00005", + "negative_test_query356_00004", + "apositive_test_query20_00010", + "negative_test_query245_00015", + "apositive_test_query66_00000", + "apositive_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "negative_test_query188_00008", + "negative_test_query20_00006", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "apositive_test_query188_00000", + "negative_test_query188_00010", + "apositive_test_query20_00006", + "negative_test_query220_00011", + "negative_test_query344_00007", + "apositive_test_query20_00009", + "negative_test_query188_00009", + "negative_test_query112_00017", + "negative_test_query188_00003", + "negative_test_query188_00013", + "negative_test_query344_00013", + "negative_test_query188_00011", + "negative_test_query188_00004", + "negative_test_query188_00014", + "apositive_test_query220_00004", + "apositive_test_query220_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00004", + "apositive_test_query189_00003", + "negative_test_query189_00013", + "negative_test_query189_00001", + "negative_test_query189_00007", + "apositive_test_query189_00002", + "negative_test_query140_00016", + "negative_test_query189_00006", + "negative_test_query189_00000", + "negative_test_query85_00007", + "apositive_test_query189_00001", + "apositive_test_query85_00002", + "negative_test_query44_00005", + "negative_test_query260_00007", + "negative_test_query55_00003", + "negative_test_query18_00006", + "negative_test_query189_00002", + "negative_test_query190_00001", + "negative_test_query287_00000", + "apositive_test_query254_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query19", + "ranked_ids": [ + "apositive_test_query19_00000", + "negative_test_query19_00003", + "negative_test_query4_00001", + "apositive_test_query4_00000", + "negative_test_query15_00000", + "negative_test_query19_00006", + "negative_test_query229_00002", + "negative_test_query19_00008", + "apositive_test_query174_00014", + "negative_test_query266_00010", + "apositive_test_query4_00010", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query106_00004", + "apositive_test_query174_00007", + "negative_test_query169_00014", + "negative_test_query141_00000", + "negative_test_query174_00001", + "negative_test_query141_00001", + "negative_test_query167_00018" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query271_00005", + "apositive_test_query260_00003", + "negative_test_query109_00006", + "apositive_test_query311_00000", + "negative_test_query6_00007", + "negative_test_query260_00007", + "negative_test_query85_00006", + "negative_test_query345_00006", + "negative_test_query181_00003", + "apositive_test_query234_00001", + "negative_test_query246_00002", + "negative_test_query190_00013", + "negative_test_query205_00016", + "negative_test_query264_00008", + "apositive_test_query85_00000", + "negative_test_query233_00008", + "negative_test_query265_00004", + "negative_test_query85_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "apositive_test_query191_00001", + "negative_test_query191_00010", + "negative_test_query60_00000", + "negative_test_query96_00006", + "negative_test_query297_00002", + "negative_test_query332_00012", + "apositive_test_query63_00003", + "apositive_test_query191_00002", + "negative_test_query181_00013", + "negative_test_query191_00012", + "apositive_test_query341_00003", + "apositive_test_query341_00005", + "negative_test_query241_00007", + "negative_test_query256_00009", + "negative_test_query191_00011", + "negative_test_query64_00003", + "negative_test_query191_00003", + "negative_test_query191_00002", + "negative_test_query274_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00002", + "negative_test_query192_00009", + "negative_test_query192_00010", + "negative_test_query120_00004", + "negative_test_query192_00001", + "negative_test_query192_00002", + "negative_test_query192_00013", + "apositive_test_query192_00000", + "negative_test_query172_00005", + "negative_test_query192_00008", + "negative_test_query350_00010", + "negative_test_query350_00013", + "negative_test_query120_00007", + "negative_test_query192_00004", + "negative_test_query322_00008", + "apositive_test_query192_00004", + "apositive_test_query254_00007", + "negative_test_query192_00012", + "negative_test_query192_00014", + "negative_test_query192_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query193", + "ranked_ids": [ + "apositive_test_query99_00003", + "apositive_test_query193_00000", + "apositive_test_query193_00009", + "apositive_test_query193_00001", + "apositive_test_query193_00008", + "apositive_test_query193_00011", + "apositive_test_query193_00004", + "apositive_test_query153_00012", + "apositive_test_query193_00005", + "negative_test_query99_00009", + "apositive_test_query99_00002", + "apositive_test_query193_00002", + "negative_test_query146_00001", + "negative_test_query193_00002", + "negative_test_query193_00006", + "negative_test_query193_00003", + "apositive_test_query146_00008", + "apositive_test_query276_00006", + "negative_test_query193_00005", + "negative_test_query146_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query194_00002", + "apositive_test_query194_00007", + "apositive_test_query162_00006", + "apositive_test_query124_00000", + "negative_test_query162_00000", + "negative_test_query177_00007", + "apositive_test_query194_00006", + "negative_test_query95_00016", + "negative_test_query148_00009", + "apositive_test_query254_00006", + "negative_test_query262_00011", + "negative_test_query62_00014", + "apositive_test_query194_00000", + "apositive_test_query194_00001", + "apositive_test_query42_00001", + "negative_test_query137_00011", + "negative_test_query194_00003", + "negative_test_query89_00007", + "negative_test_query262_00007", + "apositive_test_query114_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query195", + "ranked_ids": [ + "negative_test_query195_00002", + "apositive_test_query195_00002", + "negative_test_query285_00000", + "negative_test_query195_00013", + "negative_test_query222_00003", + "apositive_test_query195_00001", + "apositive_test_query195_00005", + "negative_test_query195_00008", + "negative_test_query195_00001", + "negative_test_query109_00008", + "negative_test_query149_00012", + "negative_test_query195_00004", + "negative_test_query321_00005", + "apositive_test_query251_00000", + "negative_test_query166_00009", + "negative_test_query321_00008", + "apositive_test_query249_00012", + "negative_test_query205_00006", + "negative_test_query199_00003", + "negative_test_query195_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query196_00003", + "apositive_test_query32_00011", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "negative_test_query64_00005", + "apositive_test_query32_00006", + "negative_test_query14_00001", + "negative_test_query235_00008", + "negative_test_query32_00002", + "apositive_test_query32_00004", + "apositive_test_query32_00002", + "negative_test_query173_00012", + "negative_test_query179_00001", + "negative_test_query139_00011", + "negative_test_query196_00001", + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query341_00008", + "negative_test_query325_00004", + "negative_test_query96_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4367467095119258, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query197", + "ranked_ids": [ + "negative_test_query163_00001", + "negative_test_query197_00010", + "apositive_test_query197_00000", + "apositive_test_query197_00002", + "negative_test_query197_00006", + "negative_test_query197_00016", + "negative_test_query163_00012", + "apositive_test_query189_00001", + "apositive_test_query197_00001", + "negative_test_query197_00013", + "negative_test_query34_00012", + "negative_test_query178_00008", + "negative_test_query197_00004", + "apositive_test_query34_00000", + "negative_test_query28_00001", + "negative_test_query321_00006", + "negative_test_query163_00002", + "apositive_test_query163_00001", + "apositive_test_query163_00000", + "negative_test_query163_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query198_00007", + "negative_test_query239_00011", + "negative_test_query239_00009", + "negative_test_query239_00001", + "negative_test_query198_00000", + "negative_test_query294_00002", + "negative_test_query239_00010", + "negative_test_query112_00005", + "negative_test_query239_00013", + "negative_test_query198_00010", + "negative_test_query198_00002", + "negative_test_query239_00007", + "negative_test_query343_00002", + "negative_test_query62_00016", + "negative_test_query198_00008", + "negative_test_query293_00010", + "negative_test_query293_00011", + "negative_test_query352_00002", + "negative_test_query219_00007", + "negative_test_query198_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query199_00015", + "negative_test_query140_00014", + "negative_test_query199_00014", + "negative_test_query305_00007", + "negative_test_query54_00003", + "negative_test_query250_00006", + "negative_test_query139_00014", + "negative_test_query199_00005", + "negative_test_query199_00013", + "negative_test_query199_00006", + "negative_test_query185_00002", + "apositive_test_query43_00006", + "negative_test_query199_00002", + "negative_test_query350_00007", + "negative_test_query134_00006", + "apositive_test_query244_00002", + "negative_test_query199_00001", + "negative_test_query221_00008", + "negative_test_query213_00011", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query2_00013", + "negative_test_query190_00016", + "negative_test_query2_00009", + "negative_test_query15_00000", + "negative_test_query44_00008", + "negative_test_query2_00012", + "negative_test_query2_00002", + "apositive_test_query217_00006", + "negative_test_query2_00011", + "negative_test_query352_00003", + "negative_test_query2_00010", + "negative_test_query125_00012", + "negative_test_query343_00004", + "negative_test_query197_00004", + "negative_test_query2_00016", + "negative_test_query85_00007", + "negative_test_query126_00014", + "negative_test_query348_00005", + "negative_test_query104_00007", + "apositive_test_query57_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query154_00006", + "apositive_test_query20_00010", + "apositive_test_query154_00005", + "apositive_test_query20_00007", + "apositive_test_query23_00004", + "apositive_test_query20_00003", + "apositive_test_query356_00005", + "negative_test_query154_00002", + "negative_test_query66_00002", + "negative_test_query356_00004", + "apositive_test_query66_00000", + "apositive_test_query20_00009", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "apositive_test_query33_00008", + "negative_test_query188_00000", + "apositive_test_query154_00004", + "negative_test_query33_00008", + "negative_test_query220_00005", + "apositive_test_query33_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query264_00006", + "negative_test_query200_00000", + "negative_test_query38_00005", + "negative_test_query200_00009", + "negative_test_query200_00015", + "negative_test_query337_00004", + "negative_test_query337_00009", + "negative_test_query200_00006", + "negative_test_query200_00012", + "negative_test_query200_00004", + "negative_test_query200_00010", + "negative_test_query200_00013", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query337_00007", + "negative_test_query200_00007", + "negative_test_query8_00008", + "negative_test_query264_00007", + "negative_test_query88_00013", + "negative_test_query303_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query201", + "ranked_ids": [ + "apositive_test_query30_00005", + "negative_test_query231_00012", + "apositive_test_query201_00002", + "negative_test_query47_00012", + "negative_test_query259_00002", + "negative_test_query71_00013", + "apositive_test_query201_00006", + "negative_test_query0_00004", + "apositive_test_query0_00011", + "negative_test_query71_00000", + "negative_test_query71_00005", + "negative_test_query71_00002", + "negative_test_query65_00002", + "apositive_test_query71_00001", + "negative_test_query71_00009", + "negative_test_query209_00008", + "apositive_test_query30_00001", + "apositive_test_query20_00005", + "negative_test_query47_00013", + "negative_test_query159_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query235_00001", + "negative_test_query202_00004", + "negative_test_query202_00001", + "negative_test_query202_00000", + "negative_test_query312_00012", + "negative_test_query202_00006", + "apositive_test_query206_00000", + "negative_test_query202_00009", + "apositive_test_query302_00006", + "negative_test_query202_00008", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query202_00016", + "negative_test_query176_00011", + "negative_test_query202_00010", + "negative_test_query321_00018", + "negative_test_query15_00010", + "apositive_test_query309_00000", + "negative_test_query188_00011", + "apositive_test_query61_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2 + }, + "query_id": "test_query203", + "ranked_ids": [ + "negative_test_query203_00006", + "negative_test_query203_00008", + "apositive_test_query203_00007", + "apositive_test_query203_00009", + "negative_test_query203_00005", + "negative_test_query250_00005", + "negative_test_query209_00000", + "apositive_test_query203_00004", + "apositive_test_query110_00003", + "apositive_test_query284_00003", + "apositive_test_query130_00002", + "negative_test_query284_00014", + "negative_test_query250_00009", + "negative_test_query130_00001", + "apositive_test_query203_00000", + "apositive_test_query203_00003", + "apositive_test_query30_00005", + "negative_test_query250_00011", + "negative_test_query250_00016", + "negative_test_query35_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.058823529411764705 + }, + "query_id": "test_query204", + "ranked_ids": [ + "apositive_test_query353_00005", + "apositive_test_query353_00002", + "negative_test_query204_00001", + "apositive_test_query204_00000", + "negative_test_query204_00002", + "apositive_test_query353_00000", + "negative_test_query97_00007", + "negative_test_query353_00003", + "apositive_test_query204_00002", + "apositive_test_query175_00004", + "apositive_test_query204_00009", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "apositive_test_query353_00011", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00007", + "negative_test_query353_00004", + "apositive_test_query97_00001", + "negative_test_query204_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query205_00015", + "negative_test_query256_00000", + "negative_test_query96_00011", + "negative_test_query205_00007", + "negative_test_query228_00016", + "negative_test_query31_00015", + "negative_test_query256_00015", + "negative_test_query341_00003", + "negative_test_query256_00007", + "negative_test_query256_00006", + "negative_test_query73_00000", + "negative_test_query73_00011", + "negative_test_query205_00008", + "negative_test_query35_00008", + "apositive_test_query205_00000", + "negative_test_query35_00007", + "apositive_test_query256_00002", + "negative_test_query115_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query299_00001", + "negative_test_query303_00015", + "negative_test_query206_00002", + "negative_test_query206_00007", + "negative_test_query206_00015", + "negative_test_query206_00014", + "negative_test_query206_00000", + "negative_test_query206_00001", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00005", + "negative_test_query303_00003", + "negative_test_query206_00003", + "apositive_test_query348_00001", + "negative_test_query206_00013", + "negative_test_query70_00011", + "negative_test_query2_00016", + "negative_test_query64_00016", + "negative_test_query206_00018", + "negative_test_query299_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.5 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00000", + "negative_test_query207_00002", + "apositive_test_query207_00002", + "negative_test_query207_00012", + "apositive_test_query207_00004", + "negative_test_query207_00006", + "apositive_test_query207_00005", + "negative_test_query207_00011", + "negative_test_query207_00005", + "negative_test_query207_00009", + "apositive_test_query207_00001", + "negative_test_query207_00008", + "negative_test_query207_00001", + "negative_test_query207_00000", + "negative_test_query207_00003", + "negative_test_query207_00007", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query37_00004", + "negative_test_query207_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "apositive_test_query208_00000", + "negative_test_query208_00018", + "negative_test_query128_00003", + "negative_test_query208_00017", + "negative_test_query208_00016", + "negative_test_query208_00010", + "negative_test_query208_00000", + "negative_test_query73_00008", + "apositive_test_query128_00002", + "negative_test_query208_00003", + "negative_test_query256_00011", + "negative_test_query208_00008", + "negative_test_query128_00001", + "apositive_test_query128_00007", + "negative_test_query208_00004", + "negative_test_query208_00012", + "negative_test_query61_00012", + "negative_test_query211_00014", + "negative_test_query208_00006", + "negative_test_query208_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query209", + "ranked_ids": [ + "apositive_test_query209_00001", + "negative_test_query223_00003", + "negative_test_query231_00006", + "negative_test_query209_00007", + "apositive_test_query37_00001", + "negative_test_query102_00010", + "apositive_test_query26_00007", + "apositive_test_query158_00004", + "apositive_test_query158_00002", + "negative_test_query347_00004", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query102_00016", + "negative_test_query26_00002", + "apositive_test_query158_00009", + "apositive_test_query119_00008", + "negative_test_query223_00007", + "negative_test_query209_00001", + "negative_test_query26_00004", + "negative_test_query313_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "apositive_test_query128_00003", + "negative_test_query128_00004", + "negative_test_query224_00015", + "negative_test_query234_00007", + "negative_test_query10_00008", + "apositive_test_query128_00002", + "negative_test_query21_00005", + "negative_test_query21_00010", + "negative_test_query8_00006", + "negative_test_query208_00000", + "apositive_test_query21_00000", + "negative_test_query128_00006", + "negative_test_query21_00009", + "negative_test_query330_00000", + "apositive_test_query128_00004", + "negative_test_query208_00017", + "negative_test_query21_00012", + "negative_test_query234_00006", + "negative_test_query21_00001", + "negative_test_query350_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query210_00013", + "negative_test_query210_00015", + "negative_test_query148_00014", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query202_00014", + "negative_test_query156_00015", + "negative_test_query210_00008", + "negative_test_query148_00008", + "negative_test_query300_00000", + "negative_test_query288_00007", + "negative_test_query17_00002", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query210_00007", + "negative_test_query335_00004", + "negative_test_query148_00007", + "negative_test_query199_00015", + "negative_test_query210_00010", + "negative_test_query288_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query211", + "ranked_ids": [ + "apositive_test_query211_00001", + "negative_test_query211_00015", + "negative_test_query137_00012", + "negative_test_query159_00006", + "negative_test_query211_00000", + "negative_test_query239_00014", + "apositive_test_query176_00000", + "apositive_test_query198_00000", + "negative_test_query32_00005", + "negative_test_query189_00008", + "negative_test_query100_00012", + "negative_test_query195_00007", + "apositive_test_query162_00010", + "negative_test_query167_00015", + "negative_test_query211_00012", + "negative_test_query293_00013", + "apositive_test_query101_00000", + "negative_test_query202_00006", + "negative_test_query83_00010", + "apositive_test_query163_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query212", + "ranked_ids": [ + "negative_test_query167_00017", + "apositive_test_query326_00004", + "apositive_test_query326_00000", + "apositive_test_query212_00004", + "negative_test_query183_00009", + "negative_test_query183_00012", + "apositive_test_query98_00000", + "negative_test_query167_00016", + "apositive_test_query244_00007", + "apositive_test_query66_00005", + "negative_test_query282_00010", + "negative_test_query346_00006", + "negative_test_query344_00008", + "negative_test_query282_00009", + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query89_00004", + "apositive_test_query71_00001", + "apositive_test_query11_00007", + "negative_test_query77_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00000", + "apositive_test_query213_00004", + "apositive_test_query213_00002", + "negative_test_query213_00001", + "apositive_test_query259_00001", + "negative_test_query213_00011", + "negative_test_query137_00015", + "negative_test_query213_00005", + "apositive_test_query213_00003", + "negative_test_query224_00015", + "negative_test_query138_00016", + "negative_test_query213_00004", + "apositive_test_query305_00003", + "negative_test_query112_00003", + "apositive_test_query158_00001", + "apositive_test_query36_00002", + "apositive_test_query259_00000", + "negative_test_query138_00001", + "negative_test_query313_00017", + "negative_test_query357_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "apositive_test_query214_00000", + "apositive_test_query234_00001", + "negative_test_query214_00013", + "negative_test_query214_00001", + "negative_test_query214_00003", + "apositive_test_query323_00003", + "negative_test_query214_00004", + "negative_test_query214_00015", + "negative_test_query214_00016", + "negative_test_query214_00010", + "negative_test_query17_00013", + "apositive_test_query89_00000", + "apositive_test_query323_00002", + "negative_test_query324_00002", + "negative_test_query9_00008", + "negative_test_query324_00012", + "apositive_test_query160_00009", + "apositive_test_query75_00003", + "apositive_test_query17_00001", + "negative_test_query103_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00010", + "negative_test_query215_00012", + "apositive_test_query215_00000", + "negative_test_query215_00006", + "negative_test_query215_00018", + "negative_test_query215_00008", + "negative_test_query215_00007", + "negative_test_query215_00009", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query257_00001", + "negative_test_query215_00011", + "negative_test_query215_00015", + "negative_test_query215_00001", + "negative_test_query263_00001", + "negative_test_query293_00013", + "negative_test_query295_00004", + "negative_test_query215_00017", + "apositive_test_query80_00005", + "negative_test_query215_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.4 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query216_00000", + "negative_test_query216_00008", + "negative_test_query216_00007", + "negative_test_query216_00002", + "apositive_test_query216_00004", + "negative_test_query216_00000", + "apositive_test_query154_00005", + "negative_test_query246_00016", + "negative_test_query216_00006", + "negative_test_query216_00014", + "negative_test_query112_00002", + "apositive_test_query216_00001", + "apositive_test_query20_00001", + "apositive_test_query216_00002", + "negative_test_query216_00009", + "negative_test_query44_00002", + "apositive_test_query330_00002", + "negative_test_query216_00005", + "apositive_test_query182_00000", + "apositive_test_query33_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.25 + }, + "query_id": "test_query217", + "ranked_ids": [ + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "apositive_test_query217_00000", + "negative_test_query217_00011", + "negative_test_query185_00003", + "negative_test_query217_00004", + "apositive_test_query307_00010", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "negative_test_query28_00002", + "apositive_test_query213_00004", + "negative_test_query230_00002", + "negative_test_query112_00007", + "negative_test_query53_00012", + "negative_test_query76_00003", + "negative_test_query307_00004", + "negative_test_query72_00005", + "apositive_test_query160_00010", + "apositive_test_query107_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query218", + "ranked_ids": [ + "negative_test_query134_00003", + "apositive_test_query218_00004", + "apositive_test_query218_00005", + "negative_test_query218_00009", + "negative_test_query218_00012", + "negative_test_query218_00000", + "apositive_test_query218_00003", + "apositive_test_query134_00000", + "negative_test_query218_00002", + "negative_test_query218_00003", + "negative_test_query43_00006", + "negative_test_query274_00008", + "negative_test_query43_00010", + "negative_test_query304_00005", + "apositive_test_query314_00015", + "apositive_test_query60_00012", + "negative_test_query58_00010", + "negative_test_query134_00009", + "apositive_test_query218_00000", + "negative_test_query165_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "negative_test_query219_00016", + "apositive_test_query219_00001", + "apositive_test_query219_00000", + "negative_test_query219_00014", + "negative_test_query187_00015", + "negative_test_query219_00008", + "negative_test_query165_00010", + "negative_test_query308_00006", + "negative_test_query59_00000", + "negative_test_query219_00012", + "negative_test_query227_00007", + "negative_test_query219_00011", + "negative_test_query219_00015", + "apositive_test_query177_00004", + "negative_test_query219_00017", + "negative_test_query219_00005", + "negative_test_query171_00005", + "negative_test_query305_00005", + "negative_test_query24_00007", + "negative_test_query183_00016" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "apositive_test_query22_00000", + "negative_test_query22_00003", + "negative_test_query22_00015", + "negative_test_query22_00004", + "negative_test_query22_00017", + "negative_test_query22_00008", + "negative_test_query22_00000", + "negative_test_query22_00005", + "negative_test_query22_00009", + "negative_test_query22_00007", + "negative_test_query22_00002", + "negative_test_query22_00018", + "negative_test_query22_00012", + "negative_test_query22_00013", + "negative_test_query22_00001", + "negative_test_query22_00016", + "negative_test_query66_00004", + "negative_test_query22_00006", + "apositive_test_query162_00007", + "apositive_test_query177_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.5 + }, + "query_id": "test_query220", + "ranked_ids": [ + "apositive_test_query220_00003", + "negative_test_query344_00007", + "apositive_test_query220_00000", + "apositive_test_query220_00001", + "negative_test_query154_00009", + "negative_test_query220_00011", + "negative_test_query188_00013", + "negative_test_query344_00013", + "apositive_test_query220_00004", + "apositive_test_query188_00000", + "negative_test_query188_00008", + "negative_test_query188_00014", + "negative_test_query188_00004", + "negative_test_query188_00009", + "negative_test_query20_00002", + "negative_test_query220_00004", + "negative_test_query188_00007", + "apositive_test_query20_00009", + "apositive_test_query146_00009", + "negative_test_query161_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query142_00004", + "negative_test_query65_00007", + "negative_test_query277_00010", + "apositive_test_query174_00006", + "apositive_test_query180_00002", + "negative_test_query278_00004", + "apositive_test_query306_00006", + "negative_test_query15_00005", + "negative_test_query30_00010", + "negative_test_query65_00013", + "apositive_test_query203_00002", + "negative_test_query184_00001", + "negative_test_query12_00005", + "negative_test_query171_00008", + "apositive_test_query244_00008", + "apositive_test_query162_00005", + "negative_test_query180_00011", + "negative_test_query244_00006", + "apositive_test_query92_00003", + "negative_test_query39_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query222", + "ranked_ids": [ + "apositive_test_query289_00002", + "apositive_test_query222_00001", + "apositive_test_query222_00000", + "negative_test_query222_00014", + "negative_test_query222_00011", + "negative_test_query222_00006", + "apositive_test_query275_00008", + "apositive_test_query275_00000", + "apositive_test_query275_00001", + "negative_test_query222_00017", + "negative_test_query289_00006", + "negative_test_query290_00004", + "apositive_test_query275_00004", + "apositive_test_query275_00006", + "negative_test_query271_00004", + "negative_test_query271_00015", + "negative_test_query275_00002", + "apositive_test_query275_00005", + "negative_test_query275_00004", + "apositive_test_query302_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query359_00008", + "negative_test_query47_00001", + "negative_test_query359_00014", + "negative_test_query323_00005", + "apositive_test_query259_00011", + "apositive_test_query201_00004", + "apositive_test_query259_00008", + "negative_test_query164_00011", + "negative_test_query259_00006", + "apositive_test_query259_00003", + "negative_test_query345_00002", + "negative_test_query47_00007", + "apositive_test_query201_00006", + "negative_test_query47_00002", + "apositive_test_query107_00005", + "negative_test_query51_00003", + "negative_test_query107_00004", + "apositive_test_query260_00003", + "negative_test_query109_00006", + "negative_test_query6_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query224", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query224_00012", + "apositive_test_query256_00001", + "negative_test_query224_00014", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query205_00009", + "apositive_test_query224_00003", + "apositive_test_query179_00002", + "apositive_test_query32_00002", + "negative_test_query350_00016", + "apositive_test_query83_00000", + "negative_test_query98_00018", + "negative_test_query224_00008", + "negative_test_query350_00015", + "negative_test_query224_00011", + "negative_test_query149_00008", + "negative_test_query212_00013", + "negative_test_query196_00001", + "negative_test_query304_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00000", + "negative_test_query145_00000", + "apositive_test_query225_00003", + "apositive_test_query225_00001", + "apositive_test_query225_00005", + "apositive_test_query225_00004", + "apositive_test_query145_00001", + "apositive_test_query225_00007", + "apositive_test_query225_00009", + "negative_test_query149_00011", + "negative_test_query225_00006", + "apositive_test_query145_00005", + "negative_test_query343_00009", + "negative_test_query183_00015", + "negative_test_query225_00008", + "apositive_test_query225_00002", + "negative_test_query225_00000", + "negative_test_query161_00004", + "negative_test_query54_00002", + "apositive_test_query169_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.5 + }, + "query_id": "test_query226", + "ranked_ids": [ + "apositive_test_query226_00002", + "apositive_test_query226_00004", + "negative_test_query226_00009", + "apositive_test_query226_00005", + "negative_test_query226_00001", + "negative_test_query226_00007", + "negative_test_query226_00012", + "negative_test_query226_00003", + "negative_test_query226_00010", + "negative_test_query226_00013", + "negative_test_query226_00005", + "negative_test_query226_00004", + "apositive_test_query226_00001", + "negative_test_query226_00006", + "negative_test_query226_00000", + "negative_test_query226_00011", + "apositive_test_query226_00003", + "apositive_test_query226_00000", + "apositive_test_query153_00013", + "negative_test_query360_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query227", + "ranked_ids": [ + "apositive_test_query227_00000", + "negative_test_query227_00006", + "negative_test_query227_00014", + "negative_test_query227_00002", + "negative_test_query227_00003", + "negative_test_query227_00012", + "negative_test_query227_00004", + "negative_test_query227_00000", + "apositive_test_query70_00001", + "negative_test_query227_00010", + "negative_test_query227_00005", + "apositive_test_query227_00001", + "negative_test_query70_00011", + "negative_test_query227_00011", + "negative_test_query235_00006", + "negative_test_query321_00004", + "negative_test_query115_00007", + "negative_test_query59_00004", + "negative_test_query227_00013", + "apositive_test_query348_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query228", + "ranked_ids": [ + "negative_test_query228_00012", + "negative_test_query165_00011", + "apositive_test_query228_00000", + "negative_test_query228_00004", + "negative_test_query228_00000", + "negative_test_query73_00016", + "negative_test_query228_00002", + "negative_test_query2_00001", + "negative_test_query228_00014", + "negative_test_query228_00003", + "apositive_test_query254_00000", + "negative_test_query216_00013", + "negative_test_query322_00001", + "negative_test_query228_00011", + "apositive_test_query73_00001", + "negative_test_query179_00010", + "apositive_test_query318_00001", + "apositive_test_query322_00003", + "apositive_test_query322_00007", + "apositive_test_query333_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query229_00002", + "apositive_test_query229_00003", + "apositive_test_query229_00000", + "apositive_test_query229_00007", + "negative_test_query229_00008", + "negative_test_query229_00005", + "negative_test_query229_00006", + "apositive_test_query158_00006", + "negative_test_query229_00009", + "negative_test_query229_00000", + "apositive_test_query158_00002", + "negative_test_query229_00007", + "apositive_test_query69_00009", + "apositive_test_query18_00004", + "negative_test_query355_00001", + "negative_test_query44_00005", + "apositive_test_query146_00004", + "negative_test_query0_00001", + "apositive_test_query94_00007", + "negative_test_query94_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query23", + "ranked_ids": [ + "apositive_test_query23_00001", + "apositive_test_query23_00000", + "negative_test_query23_00014", + "apositive_test_query23_00002", + "negative_test_query356_00008", + "negative_test_query259_00007", + "apositive_test_query39_00002", + "apositive_test_query52_00004", + "apositive_test_query146_00015", + "negative_test_query239_00012", + "apositive_test_query52_00002", + "apositive_test_query54_00004", + "apositive_test_query54_00007", + "apositive_test_query225_00000", + "negative_test_query66_00010", + "negative_test_query133_00003", + "negative_test_query307_00002", + "negative_test_query292_00013", + "apositive_test_query52_00001", + "apositive_test_query52_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query230_00007", + "negative_test_query230_00010", + "negative_test_query230_00000", + "apositive_test_query230_00003", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "apositive_test_query3_00001", + "negative_test_query329_00000", + "negative_test_query269_00010", + "negative_test_query23_00012", + "apositive_test_query3_00006", + "negative_test_query99_00003", + "negative_test_query3_00005", + "negative_test_query3_00006", + "apositive_test_query3_00009", + "negative_test_query3_00004", + "negative_test_query3_00000", + "negative_test_query280_00006", + "apositive_test_query280_00002", + "negative_test_query280_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query231", + "ranked_ids": [ + "negative_test_query9_00017", + "negative_test_query345_00011", + "apositive_test_query130_00000", + "negative_test_query75_00000", + "apositive_test_query0_00001", + "apositive_test_query231_00003", + "apositive_test_query201_00002", + "apositive_test_query201_00006", + "negative_test_query231_00001", + "apositive_test_query130_00004", + "negative_test_query47_00006", + "negative_test_query306_00006", + "negative_test_query221_00006", + "negative_test_query130_00010", + "negative_test_query213_00004", + "negative_test_query130_00000", + "negative_test_query359_00008", + "apositive_test_query119_00006", + "negative_test_query47_00005", + "negative_test_query201_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "apositive_test_query232_00000", + "negative_test_query232_00000", + "negative_test_query232_00011", + "negative_test_query232_00014", + "negative_test_query232_00013", + "negative_test_query232_00003", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query232_00008", + "negative_test_query232_00015", + "negative_test_query232_00018", + "negative_test_query232_00006", + "negative_test_query232_00002", + "negative_test_query232_00004", + "negative_test_query232_00010", + "negative_test_query232_00001", + "negative_test_query232_00012", + "negative_test_query232_00016" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query233", + "ranked_ids": [ + "apositive_test_query233_00002", + "apositive_test_query233_00000", + "negative_test_query2_00010", + "negative_test_query233_00001", + "negative_test_query233_00010", + "negative_test_query233_00004", + "negative_test_query190_00007", + "negative_test_query233_00005", + "negative_test_query233_00015", + "negative_test_query233_00006", + "negative_test_query233_00009", + "negative_test_query246_00001", + "negative_test_query263_00009", + "negative_test_query130_00001", + "negative_test_query219_00015", + "apositive_test_query2_00000", + "apositive_test_query275_00003", + "negative_test_query130_00008", + "negative_test_query2_00014", + "negative_test_query176_00017" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00015", + "apositive_test_query304_00001", + "apositive_test_query234_00003", + "apositive_test_query304_00003", + "negative_test_query234_00005", + "negative_test_query234_00011", + "negative_test_query304_00001", + "negative_test_query304_00005", + "negative_test_query214_00008", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query214_00006", + "negative_test_query234_00010", + "negative_test_query304_00008", + "negative_test_query304_00012", + "apositive_test_query43_00005", + "apositive_test_query304_00002", + "negative_test_query10_00008", + "negative_test_query304_00013", + "apositive_test_query218_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00002", + "apositive_test_query32_00006", + "apositive_test_query32_00000", + "negative_test_query235_00014", + "negative_test_query235_00009", + "negative_test_query235_00008", + "negative_test_query235_00004", + "negative_test_query61_00001", + "negative_test_query196_00000", + "negative_test_query235_00000", + "apositive_test_query235_00001", + "apositive_test_query235_00000", + "negative_test_query235_00015", + "negative_test_query235_00011", + "apositive_test_query32_00004", + "negative_test_query173_00005", + "apositive_test_query235_00003", + "negative_test_query235_00013", + "negative_test_query24_00017", + "negative_test_query219_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "negative_test_query236_00004", + "apositive_test_query279_00002", + "negative_test_query236_00000", + "negative_test_query236_00014", + "negative_test_query236_00009", + "negative_test_query236_00012", + "negative_test_query279_00012", + "negative_test_query236_00006", + "negative_test_query236_00013", + "negative_test_query279_00005", + "negative_test_query236_00011", + "negative_test_query279_00003", + "negative_test_query236_00018", + "negative_test_query236_00005", + "negative_test_query22_00016", + "negative_test_query236_00007", + "negative_test_query236_00015", + "negative_test_query222_00000", + "negative_test_query236_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.4 + }, + "query_id": "test_query237", + "ranked_ids": [ + "apositive_test_query237_00001", + "negative_test_query237_00006", + "negative_test_query152_00008", + "apositive_test_query237_00000", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query237_00002", + "negative_test_query237_00007", + "negative_test_query237_00001", + "negative_test_query237_00002", + "negative_test_query261_00012", + "negative_test_query237_00009", + "negative_test_query237_00004", + "apositive_test_query152_00004", + "negative_test_query237_00005", + "negative_test_query290_00010", + "negative_test_query237_00000", + "apositive_test_query157_00001", + "negative_test_query152_00003", + "negative_test_query312_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "apositive_test_query306_00003", + "negative_test_query123_00001", + "apositive_test_query146_00013", + "negative_test_query56_00000", + "apositive_test_query276_00008", + "negative_test_query46_00012", + "negative_test_query111_00013", + "apositive_test_query29_00007", + "negative_test_query102_00007", + "negative_test_query56_00008", + "apositive_test_query29_00001", + "negative_test_query238_00011", + "apositive_test_query142_00003", + "negative_test_query29_00006", + "negative_test_query108_00007", + "negative_test_query13_00003", + "negative_test_query142_00011", + "negative_test_query111_00008", + "negative_test_query223_00005", + "negative_test_query203_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "negative_test_query239_00015", + "negative_test_query239_00003", + "negative_test_query239_00000", + "apositive_test_query239_00000", + "negative_test_query239_00001", + "negative_test_query239_00011", + "negative_test_query239_00013", + "negative_test_query198_00002", + "negative_test_query239_00007", + "negative_test_query239_00014", + "negative_test_query198_00007", + "negative_test_query163_00004", + "negative_test_query198_00008", + "negative_test_query268_00005", + "negative_test_query239_00009", + "negative_test_query239_00010", + "negative_test_query352_00002", + "negative_test_query61_00004", + "negative_test_query183_00015", + "negative_test_query167_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "negative_test_query24_00008", + "negative_test_query24_00004", + "negative_test_query24_00000", + "apositive_test_query226_00004", + "apositive_test_query106_00002", + "negative_test_query15_00000", + "apositive_test_query24_00000", + "apositive_test_query57_00004", + "negative_test_query24_00002", + "negative_test_query24_00009", + "negative_test_query250_00008", + "negative_test_query52_00012", + "negative_test_query128_00002", + "negative_test_query252_00005", + "negative_test_query356_00003", + "negative_test_query24_00012", + "negative_test_query198_00006", + "negative_test_query323_00000", + "negative_test_query190_00016", + "negative_test_query24_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "negative_test_query240_00012", + "apositive_test_query240_00000", + "negative_test_query240_00006", + "negative_test_query240_00002", + "negative_test_query240_00004", + "negative_test_query240_00015", + "negative_test_query240_00010", + "negative_test_query240_00018", + "negative_test_query240_00007", + "negative_test_query240_00017", + "negative_test_query240_00016", + "negative_test_query246_00002", + "negative_test_query240_00011", + "negative_test_query240_00003", + "apositive_test_query311_00000", + "negative_test_query27_00010", + "negative_test_query282_00013", + "negative_test_query293_00004", + "negative_test_query6_00000", + "negative_test_query297_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "apositive_test_query241_00000", + "negative_test_query241_00010", + "negative_test_query241_00006", + "negative_test_query241_00005", + "negative_test_query241_00002", + "negative_test_query241_00016", + "negative_test_query241_00017", + "negative_test_query256_00000", + "negative_test_query173_00013", + "negative_test_query241_00008", + "negative_test_query206_00004", + "negative_test_query206_00011", + "negative_test_query139_00007", + "negative_test_query206_00009", + "negative_test_query258_00000", + "negative_test_query53_00000", + "negative_test_query205_00003", + "negative_test_query298_00014", + "negative_test_query191_00014", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query291_00000", + "negative_test_query242_00000", + "apositive_test_query291_00001", + "negative_test_query291_00000", + "apositive_test_query242_00002", + "apositive_test_query242_00001", + "apositive_test_query242_00005", + "apositive_test_query247_00004", + "negative_test_query242_00004", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "apositive_test_query291_00008", + "negative_test_query291_00004", + "apositive_test_query242_00010", + "negative_test_query291_00003", + "apositive_test_query242_00008", + "apositive_test_query247_00005", + "negative_test_query291_00005", + "apositive_test_query291_00005", + "apositive_test_query242_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query243_00000", + "apositive_test_query243_00002", + "apositive_test_query243_00012", + "apositive_test_query243_00005", + "negative_test_query274_00014", + "negative_test_query289_00000", + "negative_test_query243_00000", + "apositive_test_query243_00003", + "negative_test_query205_00007", + "negative_test_query243_00002", + "apositive_test_query160_00003", + "negative_test_query250_00013", + "negative_test_query303_00005", + "negative_test_query192_00010", + "negative_test_query64_00003", + "negative_test_query192_00007", + "apositive_test_query173_00004", + "apositive_test_query243_00011", + "negative_test_query184_00004", + "negative_test_query223_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query244", + "ranked_ids": [ + "apositive_test_query244_00000", + "apositive_test_query244_00002", + "apositive_test_query244_00001", + "negative_test_query244_00005", + "negative_test_query244_00001", + "negative_test_query244_00008", + "apositive_test_query244_00004", + "negative_test_query244_00004", + "negative_test_query244_00009", + "negative_test_query244_00003", + "negative_test_query244_00000", + "negative_test_query244_00010", + "apositive_test_query244_00007", + "apositive_test_query244_00005", + "negative_test_query244_00002", + "apositive_test_query244_00006", + "apositive_test_query356_00003", + "apositive_test_query23_00003", + "apositive_test_query42_00002", + "negative_test_query126_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query245_00003", + "negative_test_query245_00010", + "negative_test_query64_00013", + "negative_test_query135_00013", + "negative_test_query245_00000", + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query245_00000", + "negative_test_query13_00005", + "negative_test_query202_00000", + "negative_test_query245_00015", + "apositive_test_query245_00001", + "negative_test_query121_00002", + "negative_test_query293_00003", + "negative_test_query122_00002", + "negative_test_query159_00003", + "apositive_test_query141_00006", + "apositive_test_query141_00002", + "negative_test_query336_00002", + "negative_test_query202_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query246_00016", + "negative_test_query246_00002", + "negative_test_query246_00001", + "negative_test_query246_00012", + "negative_test_query252_00003", + "negative_test_query6_00014", + "negative_test_query190_00003", + "negative_test_query135_00017", + "negative_test_query311_00005", + "negative_test_query198_00012", + "negative_test_query44_00002", + "negative_test_query317_00014", + "apositive_test_query0_00001", + "negative_test_query6_00006", + "negative_test_query134_00001", + "apositive_test_query68_00002", + "apositive_test_query216_00000", + "negative_test_query293_00000", + "negative_test_query218_00004", + "negative_test_query58_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "negative_test_query291_00005", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query242_00001", + "apositive_test_query291_00001", + "negative_test_query291_00003", + "negative_test_query247_00005", + "apositive_test_query247_00005", + "negative_test_query242_00000", + "negative_test_query291_00000", + "negative_test_query186_00010", + "apositive_test_query291_00002", + "apositive_test_query242_00000", + "apositive_test_query291_00000", + "negative_test_query291_00001", + "negative_test_query105_00012", + "apositive_test_query242_00010", + "negative_test_query247_00011", + "negative_test_query247_00003", + "apositive_test_query242_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.375 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00008", + "negative_test_query248_00010", + "apositive_test_query248_00005", + "apositive_test_query248_00002", + "apositive_test_query248_00007", + "negative_test_query248_00002", + "negative_test_query248_00011", + "negative_test_query251_00017", + "apositive_test_query248_00003", + "negative_test_query248_00004", + "negative_test_query29_00001", + "negative_test_query248_00006", + "apositive_test_query29_00004", + "apositive_test_query29_00000", + "apositive_test_query29_00005", + "negative_test_query248_00009", + "apositive_test_query248_00001", + "negative_test_query205_00017", + "negative_test_query248_00000", + "negative_test_query248_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query249", + "ranked_ids": [ + "negative_test_query71_00006", + "apositive_test_query249_00000", + "negative_test_query249_00001", + "negative_test_query156_00002", + "negative_test_query122_00016", + "negative_test_query193_00001", + "apositive_test_query112_00000", + "negative_test_query325_00015", + "negative_test_query210_00010", + "apositive_test_query177_00004", + "apositive_test_query281_00001", + "apositive_test_query249_00004", + "negative_test_query326_00005", + "negative_test_query302_00007", + "negative_test_query278_00016", + "negative_test_query301_00013", + "apositive_test_query204_00003", + "negative_test_query156_00007", + "apositive_test_query353_00004", + "negative_test_query72_00017" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query25", + "ranked_ids": [ + "negative_test_query25_00007", + "apositive_test_query25_00008", + "apositive_test_query25_00000", + "negative_test_query45_00014", + "apositive_test_query25_00006", + "apositive_test_query25_00005", + "apositive_test_query25_00007", + "negative_test_query25_00001", + "negative_test_query143_00000", + "negative_test_query25_00002", + "apositive_test_query25_00003", + "apositive_test_query25_00004", + "apositive_test_query25_00010", + "negative_test_query25_00005", + "apositive_test_query25_00001", + "apositive_test_query25_00009", + "negative_test_query30_00003", + "negative_test_query346_00006", + "negative_test_query9_00001", + "negative_test_query328_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query250", + "ranked_ids": [ + "negative_test_query250_00009", + "apositive_test_query250_00000", + "apositive_test_query203_00000", + "apositive_test_query203_00006", + "negative_test_query186_00007", + "negative_test_query250_00011", + "negative_test_query250_00013", + "negative_test_query111_00015", + "negative_test_query79_00013", + "negative_test_query57_00009", + "negative_test_query102_00010", + "negative_test_query250_00001", + "negative_test_query159_00014", + "negative_test_query192_00009", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query159_00004", + "negative_test_query250_00003", + "negative_test_query184_00015", + "negative_test_query64_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query251", + "ranked_ids": [ + "negative_test_query298_00003", + "negative_test_query189_00011", + "apositive_test_query134_00003", + "negative_test_query56_00005", + "negative_test_query322_00002", + "negative_test_query234_00006", + "apositive_test_query251_00001", + "negative_test_query251_00003", + "negative_test_query251_00004", + "negative_test_query160_00001", + "negative_test_query143_00001", + "apositive_test_query29_00002", + "negative_test_query8_00009", + "negative_test_query326_00000", + "negative_test_query268_00011", + "apositive_test_query347_00000", + "negative_test_query14_00000", + "negative_test_query170_00008", + "negative_test_query269_00007", + "negative_test_query347_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.4 + }, + "query_id": "test_query252", + "ranked_ids": [ + "negative_test_query252_00009", + "apositive_test_query252_00000", + "negative_test_query151_00000", + "negative_test_query252_00012", + "apositive_test_query252_00003", + "negative_test_query126_00001", + "negative_test_query252_00007", + "apositive_test_query125_00001", + "negative_test_query330_00006", + "negative_test_query212_00000", + "negative_test_query252_00011", + "negative_test_query318_00006", + "negative_test_query252_00001", + "apositive_test_query252_00002", + "negative_test_query44_00011", + "apositive_test_query110_00001", + "negative_test_query14_00006", + "negative_test_query321_00004", + "negative_test_query16_00006", + "negative_test_query252_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.3 + }, + "query_id": "test_query253", + "ranked_ids": [ + "apositive_test_query253_00003", + "apositive_test_query13_00012", + "apositive_test_query253_00004", + "apositive_test_query253_00007", + "apositive_test_query194_00008", + "apositive_test_query162_00008", + "negative_test_query177_00000", + "negative_test_query253_00005", + "apositive_test_query349_00000", + "apositive_test_query74_00002", + "negative_test_query194_00001", + "apositive_test_query186_00004", + "negative_test_query253_00004", + "apositive_test_query74_00003", + "negative_test_query226_00013", + "negative_test_query124_00001", + "negative_test_query253_00003", + "negative_test_query172_00008", + "negative_test_query124_00002", + "negative_test_query95_00017" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query254", + "ranked_ids": [ + "apositive_test_query254_00004", + "apositive_test_query254_00001", + "apositive_test_query254_00000", + "negative_test_query350_00000", + "apositive_test_query254_00005", + "negative_test_query270_00010", + "apositive_test_query254_00006", + "negative_test_query254_00008", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00003", + "apositive_test_query254_00010", + "apositive_test_query254_00002", + "negative_test_query136_00003", + "apositive_test_query333_00004", + "negative_test_query254_00004", + "negative_test_query228_00016", + "negative_test_query21_00009", + "negative_test_query35_00015", + "negative_test_query316_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "apositive_test_query255_00000", + "negative_test_query255_00005", + "apositive_test_query322_00001", + "negative_test_query255_00010", + "negative_test_query255_00017", + "negative_test_query326_00006", + "negative_test_query255_00006", + "negative_test_query246_00003", + "negative_test_query255_00018", + "negative_test_query255_00000", + "negative_test_query89_00004", + "apositive_test_query25_00004", + "negative_test_query208_00008", + "negative_test_query278_00015", + "negative_test_query255_00013", + "apositive_test_query313_00000", + "negative_test_query255_00004", + "apositive_test_query278_00000", + "negative_test_query278_00000", + "negative_test_query110_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query256", + "ranked_ids": [ + "apositive_test_query256_00002", + "negative_test_query256_00003", + "apositive_test_query256_00001", + "negative_test_query279_00000", + "negative_test_query256_00011", + "apositive_test_query119_00000", + "negative_test_query183_00008", + "apositive_test_query256_00000", + "negative_test_query347_00007", + "negative_test_query7_00009", + "apositive_test_query43_00001", + "apositive_test_query256_00003", + "negative_test_query256_00008", + "apositive_test_query216_00002", + "negative_test_query319_00001", + "negative_test_query82_00001", + "negative_test_query5_00010", + "negative_test_query135_00016", + "apositive_test_query142_00002", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.5 + }, + "query_id": "test_query257", + "ranked_ids": [ + "negative_test_query257_00005", + "negative_test_query215_00000", + "apositive_test_query257_00000", + "apositive_test_query257_00001", + "apositive_test_query257_00002", + "negative_test_query257_00003", + "apositive_test_query257_00005", + "negative_test_query52_00005", + "apositive_test_query286_00014", + "apositive_test_query257_00003", + "negative_test_query257_00007", + "apositive_test_query69_00007", + "negative_test_query52_00007", + "apositive_test_query257_00004", + "apositive_test_query52_00001", + "apositive_test_query54_00003", + "negative_test_query54_00005", + "negative_test_query52_00004", + "apositive_test_query151_00000", + "negative_test_query52_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query320_00002", + "negative_test_query319_00000", + "negative_test_query228_00005", + "negative_test_query162_00004", + "negative_test_query206_00016", + "negative_test_query311_00002", + "negative_test_query82_00000", + "negative_test_query167_00007", + "negative_test_query181_00002", + "negative_test_query259_00001", + "negative_test_query38_00002", + "apositive_test_query301_00002", + "negative_test_query144_00010", + "negative_test_query101_00001", + "negative_test_query169_00016", + "negative_test_query171_00002", + "negative_test_query173_00010", + "negative_test_query339_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.25 + }, + "query_id": "test_query259", + "ranked_ids": [ + "apositive_test_query259_00008", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "negative_test_query259_00002", + "apositive_test_query259_00007", + "apositive_test_query259_00006", + "apositive_test_query259_00003", + "negative_test_query259_00006", + "apositive_test_query231_00001", + "apositive_test_query345_00000", + "negative_test_query259_00001", + "apositive_test_query54_00011", + "negative_test_query250_00015", + "negative_test_query111_00010", + "negative_test_query359_00003", + "negative_test_query47_00004", + "negative_test_query359_00012", + "negative_test_query30_00009", + "negative_test_query213_00004", + "apositive_test_query0_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "apositive_test_query26_00005", + "negative_test_query26_00008", + "negative_test_query105_00013", + "apositive_test_query186_00003", + "apositive_test_query158_00003", + "negative_test_query223_00002", + "negative_test_query26_00005", + "apositive_test_query102_00000", + "negative_test_query334_00016", + "apositive_test_query26_00004", + "negative_test_query158_00006", + "negative_test_query247_00010", + "negative_test_query26_00003", + "negative_test_query164_00004", + "negative_test_query102_00015", + "negative_test_query238_00002", + "negative_test_query119_00006", + "negative_test_query158_00009", + "negative_test_query26_00009", + "negative_test_query26_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "apositive_test_query286_00011", + "negative_test_query6_00015", + "apositive_test_query6_00002", + "negative_test_query260_00011", + "negative_test_query260_00009", + "apositive_test_query6_00000", + "negative_test_query6_00004", + "apositive_test_query6_00001", + "negative_test_query252_00004", + "negative_test_query6_00005", + "negative_test_query6_00011", + "negative_test_query6_00009", + "negative_test_query125_00001", + "negative_test_query6_00002", + "negative_test_query260_00002", + "negative_test_query2_00004", + "negative_test_query6_00012", + "apositive_test_query186_00001", + "apositive_test_query109_00000", + "apositive_test_query109_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query261", + "ranked_ids": [ + "negative_test_query261_00006", + "negative_test_query116_00004", + "negative_test_query261_00005", + "negative_test_query261_00008", + "negative_test_query261_00002", + "negative_test_query261_00010", + "negative_test_query261_00013", + "negative_test_query261_00001", + "negative_test_query261_00000", + "negative_test_query261_00007", + "apositive_test_query261_00002", + "negative_test_query261_00004", + "negative_test_query261_00011", + "negative_test_query261_00003", + "apositive_test_query261_00000", + "negative_test_query261_00009", + "negative_test_query118_00006", + "negative_test_query2_00015", + "apositive_test_query261_00003", + "apositive_test_query29_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query262", + "ranked_ids": [ + "apositive_test_query262_00003", + "negative_test_query312_00018", + "negative_test_query106_00005", + "negative_test_query301_00005", + "negative_test_query266_00003", + "apositive_test_query262_00000", + "negative_test_query262_00000", + "negative_test_query312_00000", + "negative_test_query312_00003", + "apositive_test_query106_00003", + "negative_test_query124_00012", + "apositive_test_query194_00002", + "negative_test_query106_00010", + "negative_test_query152_00006", + "apositive_test_query42_00006", + "apositive_test_query177_00008", + "negative_test_query172_00000", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query326_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query263", + "ranked_ids": [ + "negative_test_query135_00004", + "negative_test_query190_00009", + "negative_test_query169_00014", + "apositive_test_query242_00011", + "apositive_test_query291_00010", + "negative_test_query125_00012", + "negative_test_query263_00005", + "apositive_test_query155_00003", + "negative_test_query253_00009", + "apositive_test_query37_00002", + "apositive_test_query231_00001", + "negative_test_query307_00005", + "negative_test_query71_00006", + "negative_test_query88_00003", + "apositive_test_query263_00002", + "apositive_test_query292_00002", + "negative_test_query201_00010", + "negative_test_query213_00001", + "negative_test_query213_00005", + "negative_test_query357_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "negative_test_query264_00011", + "negative_test_query200_00011", + "negative_test_query264_00004", + "negative_test_query200_00017", + "negative_test_query200_00002", + "negative_test_query264_00014", + "negative_test_query264_00002", + "negative_test_query264_00018", + "apositive_test_query264_00000", + "negative_test_query159_00013", + "apositive_test_query38_00003", + "negative_test_query264_00006", + "negative_test_query159_00011", + "negative_test_query337_00002", + "negative_test_query264_00012", + "negative_test_query264_00007", + "negative_test_query264_00008", + "negative_test_query337_00012", + "negative_test_query264_00016", + "negative_test_query159_00017" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query265", + "ranked_ids": [ + "apositive_test_query265_00000", + "apositive_test_query265_00001", + "negative_test_query274_00008", + "negative_test_query265_00000", + "negative_test_query265_00002", + "apositive_test_query156_00000", + "negative_test_query165_00007", + "negative_test_query86_00008", + "negative_test_query265_00009", + "negative_test_query165_00016", + "negative_test_query92_00012", + "apositive_test_query234_00001", + "negative_test_query271_00005", + "negative_test_query265_00007", + "negative_test_query52_00000", + "negative_test_query139_00000", + "apositive_test_query78_00002", + "negative_test_query86_00007", + "negative_test_query190_00013", + "negative_test_query66_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "negative_test_query266_00008", + "negative_test_query266_00018", + "negative_test_query176_00018", + "apositive_test_query266_00000", + "negative_test_query266_00001", + "apositive_test_query19_00002", + "negative_test_query141_00000", + "negative_test_query106_00004", + "negative_test_query266_00009", + "apositive_test_query54_00009", + "negative_test_query266_00014", + "negative_test_query266_00000", + "negative_test_query266_00017", + "negative_test_query266_00005", + "negative_test_query106_00001", + "negative_test_query266_00002", + "negative_test_query174_00001", + "negative_test_query266_00010", + "negative_test_query169_00014", + "negative_test_query266_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "negative_test_query79_00004", + "negative_test_query267_00002", + "negative_test_query98_00013", + "negative_test_query359_00005", + "negative_test_query313_00016", + "apositive_test_query157_00008", + "apositive_test_query295_00001", + "negative_test_query295_00002", + "negative_test_query345_00013", + "negative_test_query5_00012", + "apositive_test_query353_00002", + "negative_test_query79_00002", + "apositive_test_query204_00000", + "apositive_test_query250_00000", + "negative_test_query204_00001", + "negative_test_query186_00006", + "negative_test_query30_00000", + "negative_test_query115_00007", + "apositive_test_query113_00002", + "apositive_test_query50_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query268", + "ranked_ids": [ + "negative_test_query268_00000", + "apositive_test_query268_00000", + "negative_test_query47_00017", + "apositive_test_query268_00004", + "negative_test_query268_00013", + "negative_test_query268_00012", + "apositive_test_query51_00003", + "apositive_test_query201_00003", + "apositive_test_query268_00002", + "negative_test_query268_00004", + "negative_test_query268_00007", + "negative_test_query268_00003", + "negative_test_query268_00008", + "negative_test_query195_00009", + "negative_test_query323_00002", + "negative_test_query335_00011", + "negative_test_query335_00002", + "negative_test_query228_00007", + "negative_test_query210_00007", + "negative_test_query268_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query269", + "ranked_ids": [ + "negative_test_query329_00014", + "negative_test_query269_00000", + "apositive_test_query269_00000", + "negative_test_query269_00001", + "negative_test_query269_00003", + "negative_test_query269_00009", + "negative_test_query269_00007", + "negative_test_query269_00015", + "apositive_test_query296_00005", + "negative_test_query269_00004", + "negative_test_query269_00012", + "negative_test_query329_00004", + "negative_test_query269_00002", + "negative_test_query269_00008", + "negative_test_query296_00003", + "negative_test_query269_00010", + "apositive_test_query269_00003", + "negative_test_query269_00005", + "apositive_test_query296_00003", + "negative_test_query269_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "negative_test_query27_00009", + "negative_test_query76_00009", + "negative_test_query147_00000", + "negative_test_query164_00007", + "apositive_test_query59_00000", + "negative_test_query259_00006", + "negative_test_query27_00004", + "negative_test_query138_00006", + "negative_test_query315_00001", + "negative_test_query358_00003", + "negative_test_query33_00004", + "apositive_test_query29_00004", + "negative_test_query30_00009", + "negative_test_query130_00006", + "apositive_test_query81_00007", + "negative_test_query27_00014", + "apositive_test_query127_00010", + "apositive_test_query154_00000", + "negative_test_query11_00008", + "apositive_test_query16_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query270", + "ranked_ids": [ + "negative_test_query270_00012", + "apositive_test_query270_00001", + "apositive_test_query270_00000", + "negative_test_query270_00000", + "negative_test_query270_00004", + "negative_test_query270_00009", + "negative_test_query270_00008", + "negative_test_query270_00014", + "negative_test_query270_00011", + "negative_test_query270_00002", + "negative_test_query270_00006", + "negative_test_query270_00003", + "negative_test_query270_00015", + "negative_test_query270_00010", + "apositive_test_query270_00002", + "apositive_test_query81_00000", + "negative_test_query270_00005", + "negative_test_query270_00016", + "negative_test_query76_00005", + "negative_test_query81_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query271", + "ranked_ids": [ + "negative_test_query78_00010", + "apositive_test_query271_00000", + "apositive_test_query271_00001", + "negative_test_query327_00010", + "negative_test_query330_00008", + "negative_test_query120_00003", + "apositive_test_query10_00002", + "negative_test_query241_00018", + "negative_test_query227_00009", + "negative_test_query84_00014", + "negative_test_query271_00007", + "apositive_test_query314_00007", + "negative_test_query169_00009", + "negative_test_query198_00015", + "negative_test_query221_00012", + "apositive_test_query189_00000", + "apositive_test_query109_00001", + "negative_test_query90_00001", + "apositive_test_query67_00000", + "negative_test_query112_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00003", + "negative_test_query272_00016", + "negative_test_query272_00010", + "negative_test_query144_00000", + "negative_test_query272_00008", + "negative_test_query272_00015", + "negative_test_query272_00001", + "negative_test_query41_00008", + "negative_test_query272_00002", + "apositive_test_query272_00000", + "negative_test_query328_00000", + "negative_test_query272_00014", + "negative_test_query272_00009", + "negative_test_query272_00000", + "negative_test_query272_00017", + "negative_test_query18_00014", + "negative_test_query272_00012", + "negative_test_query272_00013", + "negative_test_query332_00013", + "apositive_test_query251_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.2 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00008", + "apositive_test_query18_00003", + "negative_test_query18_00011", + "apositive_test_query273_00005", + "apositive_test_query273_00000", + "negative_test_query18_00006", + "negative_test_query273_00006", + "negative_test_query273_00000", + "apositive_test_query281_00008", + "apositive_test_query18_00001", + "negative_test_query273_00009", + "apositive_test_query273_00004", + "apositive_test_query18_00000", + "apositive_test_query18_00004", + "negative_test_query18_00012", + "negative_test_query273_00004", + "negative_test_query300_00015", + "negative_test_query166_00009", + "negative_test_query140_00013", + "apositive_test_query36_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query274", + "ranked_ids": [ + "apositive_test_query274_00000", + "negative_test_query274_00000", + "apositive_test_query274_00003", + "negative_test_query274_00014", + "negative_test_query34_00014", + "negative_test_query352_00012", + "negative_test_query274_00011", + "negative_test_query274_00012", + "negative_test_query274_00005", + "negative_test_query218_00011", + "apositive_test_query274_00001", + "negative_test_query200_00009", + "negative_test_query274_00007", + "negative_test_query256_00015", + "negative_test_query274_00009", + "apositive_test_query327_00002", + "negative_test_query352_00009", + "negative_test_query78_00007", + "negative_test_query34_00007", + "negative_test_query274_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query275_00000", + "negative_test_query263_00009", + "apositive_test_query275_00006", + "apositive_test_query275_00007", + "apositive_test_query275_00005", + "apositive_test_query275_00004", + "apositive_test_query275_00008", + "negative_test_query275_00000", + "negative_test_query275_00004", + "negative_test_query202_00005", + "apositive_test_query275_00003", + "negative_test_query289_00013", + "negative_test_query149_00016", + "negative_test_query275_00007", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query16_00001", + "apositive_test_query110_00001", + "negative_test_query87_00017", + "apositive_test_query93_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query22_00004", + "negative_test_query199_00012", + "apositive_test_query153_00006", + "apositive_test_query161_00002", + "apositive_test_query276_00005", + "apositive_test_query276_00004", + "negative_test_query276_00001", + "negative_test_query280_00002", + "apositive_test_query314_00006", + "negative_test_query153_00001", + "negative_test_query113_00011", + "apositive_test_query314_00007", + "negative_test_query146_00001", + "negative_test_query251_00012", + "apositive_test_query3_00010", + "negative_test_query146_00000", + "apositive_test_query276_00006", + "apositive_test_query314_00005", + "negative_test_query146_00003", + "apositive_test_query146_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query277", + "ranked_ids": [ + "apositive_test_query277_00001", + "apositive_test_query277_00000", + "apositive_test_query277_00003", + "negative_test_query277_00008", + "negative_test_query277_00007", + "negative_test_query277_00004", + "negative_test_query277_00014", + "negative_test_query277_00003", + "negative_test_query277_00015", + "apositive_test_query107_00007", + "negative_test_query313_00009", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "apositive_test_query277_00002", + "negative_test_query345_00000", + "negative_test_query277_00006", + "negative_test_query277_00000", + "negative_test_query277_00013", + "negative_test_query277_00010", + "negative_test_query277_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query140_00006", + "negative_test_query278_00002", + "apositive_test_query263_00005", + "negative_test_query278_00011", + "apositive_test_query4_00013", + "negative_test_query260_00009", + "apositive_test_query36_00001", + "negative_test_query260_00002", + "negative_test_query324_00011", + "negative_test_query158_00003", + "negative_test_query260_00011", + "negative_test_query15_00007", + "negative_test_query177_00001", + "apositive_test_query186_00001", + "negative_test_query64_00007", + "negative_test_query278_00009", + "negative_test_query255_00016", + "negative_test_query255_00018", + "negative_test_query110_00011", + "negative_test_query27_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.6 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query279_00000", + "apositive_test_query279_00001", + "apositive_test_query177_00004", + "negative_test_query279_00014", + "apositive_test_query279_00003", + "negative_test_query279_00009", + "apositive_test_query279_00004", + "apositive_test_query348_00007", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "apositive_test_query3_00000", + "negative_test_query309_00011", + "negative_test_query279_00013", + "negative_test_query181_00008", + "apositive_test_query76_00005", + "negative_test_query279_00007", + "apositive_test_query76_00002", + "apositive_test_query358_00007", + "apositive_test_query162_00009", + "negative_test_query132_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query28", + "ranked_ids": [ + "negative_test_query72_00004", + "apositive_test_query28_00008", + "apositive_test_query217_00000", + "apositive_test_query217_00001", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query28_00003", + "apositive_test_query28_00009", + "apositive_test_query217_00003", + "negative_test_query72_00018", + "negative_test_query48_00009", + "negative_test_query217_00004", + "apositive_test_query28_00007", + "negative_test_query307_00003", + "negative_test_query28_00002", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query185_00001", + "negative_test_query217_00011", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "apositive_test_query280_00000", + "negative_test_query280_00004", + "negative_test_query280_00005", + "negative_test_query280_00008", + "apositive_test_query280_00003", + "negative_test_query106_00006", + "apositive_test_query280_00004", + "negative_test_query280_00010", + "negative_test_query9_00012", + "negative_test_query280_00006", + "apositive_test_query253_00009", + "negative_test_query280_00012", + "negative_test_query280_00013", + "negative_test_query280_00002", + "negative_test_query129_00008", + "apositive_test_query74_00001", + "negative_test_query236_00018", + "negative_test_query280_00003", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query281_00004", + "apositive_test_query273_00002", + "apositive_test_query281_00010", + "apositive_test_query281_00012", + "apositive_test_query281_00007", + "apositive_test_query281_00003", + "apositive_test_query281_00001", + "apositive_test_query281_00008", + "apositive_test_query281_00016", + "apositive_test_query273_00000", + "negative_test_query273_00003", + "apositive_test_query281_00014", + "negative_test_query257_00012", + "apositive_test_query281_00011", + "negative_test_query273_00004", + "apositive_test_query273_00008", + "negative_test_query273_00007", + "negative_test_query249_00000", + "apositive_test_query273_00003", + "negative_test_query125_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query282", + "ranked_ids": [ + "negative_test_query344_00001", + "apositive_test_query282_00000", + "negative_test_query282_00000", + "negative_test_query282_00004", + "negative_test_query282_00007", + "apositive_test_query282_00001", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query344_00000", + "negative_test_query282_00016", + "negative_test_query282_00010", + "negative_test_query344_00008", + "negative_test_query282_00012", + "negative_test_query282_00009", + "apositive_test_query20_00004", + "negative_test_query282_00001", + "apositive_test_query220_00002", + "negative_test_query282_00003", + "negative_test_query282_00006", + "negative_test_query344_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query138_00005", + "negative_test_query283_00016", + "negative_test_query283_00004", + "negative_test_query93_00004", + "negative_test_query93_00002", + "negative_test_query283_00007", + "negative_test_query93_00010", + "negative_test_query283_00003", + "negative_test_query138_00009", + "negative_test_query270_00006", + "negative_test_query93_00009", + "negative_test_query31_00010", + "negative_test_query265_00013", + "negative_test_query93_00006", + "negative_test_query283_00012", + "negative_test_query283_00014", + "negative_test_query283_00000", + "apositive_test_query93_00002", + "negative_test_query92_00010", + "negative_test_query299_00017" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.6 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "apositive_test_query284_00003", + "negative_test_query284_00009", + "negative_test_query284_00001", + "apositive_test_query284_00001", + "apositive_test_query198_00002", + "negative_test_query14_00014", + "apositive_test_query162_00005", + "negative_test_query284_00012", + "apositive_test_query259_00009", + "negative_test_query284_00004", + "apositive_test_query163_00003", + "negative_test_query114_00006", + "apositive_test_query141_00002", + "apositive_test_query141_00006", + "apositive_test_query163_00002", + "negative_test_query109_00009", + "apositive_test_query79_00004", + "negative_test_query288_00011", + "negative_test_query141_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.375 + }, + "query_id": "test_query285", + "ranked_ids": [ + "apositive_test_query285_00003", + "apositive_test_query285_00004", + "apositive_test_query285_00001", + "negative_test_query285_00004", + "negative_test_query285_00002", + "negative_test_query304_00003", + "apositive_test_query285_00002", + "negative_test_query285_00008", + "apositive_test_query285_00007", + "negative_test_query285_00005", + "apositive_test_query285_00005", + "apositive_test_query285_00006", + "negative_test_query285_00000", + "negative_test_query285_00007", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query285_00010", + "negative_test_query285_00009", + "negative_test_query86_00001", + "negative_test_query285_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.2 + }, + "query_id": "test_query286", + "ranked_ids": [ + "negative_test_query36_00001", + "negative_test_query286_00000", + "apositive_test_query286_00010", + "apositive_test_query286_00013", + "apositive_test_query286_00012", + "negative_test_query257_00005", + "negative_test_query257_00001", + "apositive_test_query286_00006", + "apositive_test_query286_00014", + "negative_test_query52_00001", + "apositive_test_query80_00000", + "negative_test_query286_00003", + "negative_test_query54_00004", + "apositive_test_query257_00000", + "apositive_test_query286_00005", + "negative_test_query36_00013", + "apositive_test_query257_00001", + "negative_test_query125_00012", + "negative_test_query52_00005", + "negative_test_query139_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query287_00001", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query133_00001", + "apositive_test_query133_00002", + "apositive_test_query133_00004", + "apositive_test_query133_00013", + "apositive_test_query133_00014", + "apositive_test_query133_00006", + "negative_test_query301_00010", + "apositive_test_query287_00004", + "apositive_test_query133_00012", + "negative_test_query133_00002", + "negative_test_query359_00011", + "negative_test_query153_00003", + "apositive_test_query153_00003", + "apositive_test_query4_00009", + "negative_test_query134_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query288", + "ranked_ids": [ + "apositive_test_query141_00006", + "apositive_test_query141_00002", + "negative_test_query288_00006", + "apositive_test_query288_00002", + "negative_test_query288_00002", + "apositive_test_query210_00002", + "apositive_test_query210_00001", + "apositive_test_query304_00001", + "negative_test_query148_00014", + "negative_test_query210_00005", + "negative_test_query288_00001", + "negative_test_query148_00004", + "negative_test_query213_00007", + "negative_test_query288_00010", + "negative_test_query199_00015", + "negative_test_query17_00002", + "negative_test_query288_00007", + "negative_test_query210_00013", + "negative_test_query199_00010", + "negative_test_query93_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00006", + "negative_test_query7_00006", + "apositive_test_query289_00002", + "negative_test_query289_00013", + "negative_test_query289_00001", + "negative_test_query334_00014", + "apositive_test_query275_00005", + "apositive_test_query289_00000", + "negative_test_query289_00000", + "negative_test_query305_00011", + "apositive_test_query289_00004", + "negative_test_query267_00005", + "negative_test_query47_00009", + "apositive_test_query157_00008", + "negative_test_query250_00016", + "negative_test_query247_00001", + "negative_test_query107_00004", + "negative_test_query113_00009", + "negative_test_query289_00003", + "negative_test_query289_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5 + }, + "query_id": "test_query29", + "ranked_ids": [ + "apositive_test_query29_00004", + "apositive_test_query29_00000", + "apositive_test_query29_00007", + "apositive_test_query29_00005", + "negative_test_query29_00009", + "negative_test_query29_00008", + "negative_test_query248_00009", + "apositive_test_query29_00001", + "apositive_test_query29_00003", + "negative_test_query39_00011", + "apositive_test_query29_00002", + "negative_test_query248_00007", + "negative_test_query29_00007", + "negative_test_query292_00015", + "negative_test_query29_00010", + "negative_test_query29_00006", + "negative_test_query213_00012", + "apositive_test_query29_00006", + "negative_test_query29_00000", + "negative_test_query29_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query290", + "ranked_ids": [ + "apositive_test_query290_00001", + "apositive_test_query290_00003", + "negative_test_query290_00005", + "negative_test_query98_00003", + "negative_test_query290_00000", + "apositive_test_query290_00002", + "negative_test_query237_00010", + "negative_test_query342_00005", + "negative_test_query290_00014", + "negative_test_query290_00009", + "apositive_test_query237_00002", + "negative_test_query290_00003", + "negative_test_query59_00002", + "negative_test_query116_00002", + "negative_test_query290_00006", + "negative_test_query290_00008", + "negative_test_query290_00002", + "negative_test_query157_00005", + "negative_test_query237_00013", + "negative_test_query312_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query242_00001", + "apositive_test_query291_00000", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "negative_test_query291_00005", + "apositive_test_query291_00008", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "apositive_test_query242_00005", + "apositive_test_query242_00010", + "apositive_test_query247_00005", + "apositive_test_query291_00005", + "negative_test_query291_00004", + "negative_test_query291_00003", + "apositive_test_query242_00013", + "negative_test_query186_00000", + "negative_test_query291_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query292", + "ranked_ids": [ + "apositive_test_query292_00001", + "apositive_test_query292_00000", + "negative_test_query292_00010", + "negative_test_query106_00007", + "negative_test_query292_00014", + "negative_test_query292_00004", + "negative_test_query292_00000", + "negative_test_query292_00012", + "negative_test_query292_00015", + "negative_test_query292_00005", + "negative_test_query292_00007", + "negative_test_query292_00008", + "negative_test_query292_00016", + "negative_test_query298_00000", + "negative_test_query292_00006", + "negative_test_query61_00011", + "negative_test_query139_00001", + "negative_test_query205_00000", + "negative_test_query99_00004", + "negative_test_query151_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9469024295259745, + "recall@5": 1.0 + }, + "query_id": "test_query293", + "ranked_ids": [ + "apositive_test_query293_00002", + "apositive_test_query293_00001", + "negative_test_query293_00006", + "apositive_test_query178_00003", + "apositive_test_query293_00000", + "negative_test_query293_00015", + "negative_test_query27_00008", + "negative_test_query293_00009", + "negative_test_query27_00011", + "negative_test_query27_00002", + "negative_test_query293_00003", + "apositive_test_query44_00001", + "negative_test_query89_00002", + "negative_test_query293_00000", + "negative_test_query293_00011", + "apositive_test_query52_00005", + "negative_test_query293_00016", + "negative_test_query293_00010", + "negative_test_query220_00007", + "negative_test_query167_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00005", + "negative_test_query294_00003", + "negative_test_query294_00008", + "negative_test_query294_00011", + "apositive_test_query294_00001", + "negative_test_query294_00013", + "negative_test_query294_00002", + "negative_test_query294_00006", + "negative_test_query294_00012", + "negative_test_query314_00001", + "negative_test_query294_00016", + "negative_test_query294_00004", + "negative_test_query294_00001", + "negative_test_query294_00007", + "negative_test_query277_00005", + "negative_test_query58_00003", + "negative_test_query111_00015", + "apositive_test_query170_00005", + "apositive_test_query177_00004", + "negative_test_query156_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.38364913903790027, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query295", + "ranked_ids": [ + "negative_test_query26_00002", + "apositive_test_query57_00000", + "negative_test_query295_00008", + "apositive_test_query295_00001", + "apositive_test_query295_00000", + "negative_test_query295_00002", + "apositive_test_query295_00002", + "negative_test_query295_00013", + "apositive_test_query223_00001", + "negative_test_query295_00011", + "negative_test_query295_00016", + "negative_test_query57_00009", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query168_00005", + "negative_test_query295_00000", + "negative_test_query295_00001", + "negative_test_query115_00005", + "negative_test_query295_00004", + "negative_test_query295_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query296", + "ranked_ids": [ + "negative_test_query340_00009", + "apositive_test_query296_00002", + "apositive_test_query296_00001", + "apositive_test_query296_00005", + "apositive_test_query296_00004", + "negative_test_query296_00005", + "negative_test_query329_00011", + "negative_test_query296_00001", + "negative_test_query112_00012", + "apositive_test_query296_00000", + "apositive_test_query296_00003", + "negative_test_query296_00009", + "negative_test_query340_00014", + "negative_test_query296_00003", + "negative_test_query296_00013", + "negative_test_query296_00008", + "negative_test_query296_00000", + "negative_test_query0_00004", + "negative_test_query296_00012", + "negative_test_query296_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query191_00003", + "negative_test_query297_00012", + "negative_test_query297_00004", + "negative_test_query297_00014", + "negative_test_query297_00013", + "negative_test_query64_00003", + "apositive_test_query139_00001", + "negative_test_query297_00001", + "negative_test_query297_00007", + "negative_test_query297_00003", + "negative_test_query297_00008", + "negative_test_query297_00002", + "negative_test_query191_00011", + "negative_test_query70_00005", + "negative_test_query64_00005", + "apositive_test_query297_00000", + "negative_test_query191_00010", + "negative_test_query329_00007", + "negative_test_query297_00010", + "negative_test_query117_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "apositive_test_query298_00000", + "apositive_test_query298_00002", + "negative_test_query216_00010", + "apositive_test_query347_00001", + "negative_test_query298_00006", + "negative_test_query347_00014", + "negative_test_query240_00001", + "negative_test_query202_00011", + "apositive_test_query298_00001", + "negative_test_query182_00011", + "negative_test_query298_00004", + "negative_test_query298_00003", + "negative_test_query298_00001", + "negative_test_query182_00010", + "negative_test_query347_00000", + "apositive_test_query56_00002", + "negative_test_query175_00009", + "negative_test_query193_00006", + "negative_test_query347_00016", + "apositive_test_query347_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query299_00005", + "negative_test_query299_00006", + "apositive_test_query265_00002", + "apositive_test_query346_00001", + "negative_test_query351_00009", + "apositive_test_query141_00002", + "apositive_test_query325_00002", + "negative_test_query300_00000", + "apositive_test_query101_00003", + "negative_test_query125_00005", + "negative_test_query288_00007", + "negative_test_query305_00007", + "negative_test_query134_00000", + "negative_test_query92_00010", + "negative_test_query288_00000", + "negative_test_query210_00008", + "negative_test_query89_00008", + "apositive_test_query227_00003", + "negative_test_query3_00003", + "negative_test_query58_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query3", + "ranked_ids": [ + "apositive_test_query3_00001", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "negative_test_query147_00014", + "negative_test_query3_00000", + "negative_test_query83_00007", + "negative_test_query230_00000", + "negative_test_query230_00007", + "negative_test_query3_00007", + "negative_test_query329_00000", + "negative_test_query147_00011", + "negative_test_query23_00012", + "negative_test_query3_00005", + "negative_test_query230_00010", + "negative_test_query279_00000", + "apositive_test_query177_00006", + "apositive_test_query3_00007", + "apositive_test_query3_00002", + "apositive_test_query230_00003", + "apositive_test_query3_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query30", + "ranked_ids": [ + "apositive_test_query71_00000", + "apositive_test_query71_00002", + "apositive_test_query306_00002", + "apositive_test_query30_00005", + "apositive_test_query30_00001", + "apositive_test_query30_00004", + "negative_test_query231_00002", + "negative_test_query71_00005", + "negative_test_query231_00008", + "negative_test_query71_00010", + "apositive_test_query209_00005", + "negative_test_query259_00002", + "apositive_test_query30_00000", + "negative_test_query182_00014", + "apositive_test_query30_00003", + "negative_test_query231_00000", + "negative_test_query201_00005", + "negative_test_query357_00002", + "negative_test_query356_00010", + "negative_test_query97_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query300_00005", + "negative_test_query266_00010", + "negative_test_query15_00000", + "negative_test_query300_00009", + "apositive_test_query300_00001", + "negative_test_query300_00013", + "negative_test_query252_00005", + "negative_test_query312_00011", + "negative_test_query190_00004", + "apositive_test_query300_00002", + "negative_test_query185_00001", + "negative_test_query300_00008", + "negative_test_query300_00006", + "negative_test_query167_00018", + "negative_test_query300_00011", + "negative_test_query300_00010", + "negative_test_query141_00000", + "negative_test_query190_00016", + "negative_test_query141_00001", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.787702056960637, + "recall@5": 0.75 + }, + "query_id": "test_query301", + "ranked_ids": [ + "apositive_test_query301_00000", + "apositive_test_query301_00001", + "negative_test_query301_00011", + "negative_test_query301_00002", + "apositive_test_query301_00002", + "negative_test_query301_00006", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query2_00008", + "negative_test_query301_00001", + "negative_test_query301_00000", + "negative_test_query301_00013", + "negative_test_query301_00015", + "negative_test_query148_00009", + "negative_test_query301_00007", + "apositive_test_query86_00000", + "apositive_test_query86_00002", + "apositive_test_query171_00000", + "negative_test_query297_00015", + "negative_test_query21_00018" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query302", + "ranked_ids": [ + "apositive_test_query308_00008", + "apositive_test_query302_00000", + "apositive_test_query302_00003", + "negative_test_query302_00010", + "apositive_test_query302_00002", + "negative_test_query302_00001", + "apositive_test_query302_00001", + "apositive_test_query308_00009", + "negative_test_query302_00012", + "apositive_test_query302_00005", + "apositive_test_query302_00004", + "negative_test_query308_00008", + "apositive_test_query324_00002", + "negative_test_query302_00000", + "apositive_test_query308_00001", + "negative_test_query308_00003", + "negative_test_query308_00002", + "apositive_test_query324_00001", + "negative_test_query308_00005", + "negative_test_query308_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query35_00003", + "negative_test_query303_00017", + "apositive_test_query303_00000", + "negative_test_query173_00001", + "negative_test_query316_00009", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00004", + "negative_test_query196_00007", + "negative_test_query241_00006", + "negative_test_query206_00011", + "negative_test_query243_00000", + "negative_test_query206_00009", + "negative_test_query303_00015", + "negative_test_query307_00006", + "negative_test_query61_00011", + "negative_test_query303_00005", + "negative_test_query274_00014", + "negative_test_query8_00013", + "apositive_test_query86_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query304_00001", + "apositive_test_query234_00003", + "negative_test_query43_00001", + "negative_test_query214_00002", + "negative_test_query304_00005", + "apositive_test_query304_00002", + "negative_test_query234_00010", + "negative_test_query304_00013", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query304_00012", + "apositive_test_query304_00003", + "apositive_test_query285_00001", + "negative_test_query304_00008", + "apositive_test_query285_00005", + "negative_test_query304_00003", + "negative_test_query304_00004", + "negative_test_query304_00010", + "negative_test_query43_00006", + "negative_test_query304_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query305", + "ranked_ids": [ + "negative_test_query305_00014", + "negative_test_query323_00003", + "apositive_test_query305_00001", + "negative_test_query305_00005", + "apositive_test_query357_00000", + "apositive_test_query127_00005", + "negative_test_query0_00007", + "negative_test_query203_00006", + "negative_test_query0_00003", + "negative_test_query0_00004", + "apositive_test_query259_00011", + "negative_test_query56_00004", + "negative_test_query164_00011", + "negative_test_query310_00000", + "negative_test_query134_00006", + "negative_test_query224_00015", + "apositive_test_query71_00003", + "negative_test_query65_00013", + "negative_test_query51_00003", + "negative_test_query47_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query306", + "ranked_ids": [ + "negative_test_query71_00013", + "apositive_test_query71_00002", + "apositive_test_query306_00002", + "negative_test_query305_00014", + "apositive_test_query209_00005", + "negative_test_query204_00001", + "apositive_test_query71_00001", + "negative_test_query71_00012", + "negative_test_query259_00002", + "negative_test_query159_00009", + "apositive_test_query30_00002", + "apositive_test_query119_00006", + "apositive_test_query71_00003", + "negative_test_query71_00011", + "apositive_test_query30_00001", + "apositive_test_query289_00003", + "negative_test_query182_00014", + "apositive_test_query127_00007", + "negative_test_query142_00002", + "negative_test_query209_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "apositive_test_query279_00002", + "apositive_test_query3_00003", + "negative_test_query251_00001", + "negative_test_query70_00010", + "negative_test_query144_00004", + "negative_test_query187_00011", + "negative_test_query346_00017", + "apositive_test_query13_00005", + "negative_test_query207_00004", + "apositive_test_query170_00005", + "negative_test_query18_00005", + "apositive_test_query279_00003", + "negative_test_query103_00012", + "negative_test_query48_00018", + "negative_test_query163_00010", + "negative_test_query307_00006", + "apositive_test_query177_00004", + "negative_test_query217_00003", + "negative_test_query360_00003", + "negative_test_query132_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query308_00008", + "apositive_test_query308_00000", + "apositive_test_query308_00002", + "apositive_test_query308_00009", + "apositive_test_query308_00001", + "apositive_test_query324_00000", + "negative_test_query159_00013", + "negative_test_query324_00012", + "apositive_test_query324_00002", + "negative_test_query308_00008", + "apositive_test_query324_00001", + "negative_test_query324_00014", + "negative_test_query308_00003", + "apositive_test_query308_00003", + "negative_test_query308_00009", + "negative_test_query94_00011", + "negative_test_query308_00007", + "apositive_test_query7_00000", + "apositive_test_query310_00008", + "apositive_test_query341_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "negative_test_query123_00007", + "negative_test_query309_00005", + "negative_test_query354_00002", + "negative_test_query11_00000", + "apositive_test_query358_00004", + "apositive_test_query161_00008", + "negative_test_query309_00003", + "negative_test_query309_00015", + "apositive_test_query309_00000", + "negative_test_query309_00012", + "apositive_test_query358_00003", + "apositive_test_query358_00007", + "negative_test_query309_00006", + "negative_test_query354_00008", + "apositive_test_query358_00006", + "negative_test_query113_00013", + "negative_test_query81_00002", + "negative_test_query113_00011", + "apositive_test_query76_00002", + "negative_test_query42_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query31_00003", + "negative_test_query31_00011", + "negative_test_query31_00013", + "apositive_test_query31_00000", + "negative_test_query313_00007", + "negative_test_query31_00015", + "negative_test_query31_00001", + "negative_test_query129_00000", + "negative_test_query31_00008", + "negative_test_query31_00006", + "negative_test_query31_00002", + "negative_test_query31_00007", + "negative_test_query31_00010", + "apositive_test_query249_00009", + "negative_test_query31_00000", + "apositive_test_query60_00011", + "negative_test_query38_00003", + "apositive_test_query213_00005", + "negative_test_query31_00004", + "negative_test_query326_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.4 + }, + "query_id": "test_query310", + "ranked_ids": [ + "negative_test_query36_00015", + "apositive_test_query310_00002", + "apositive_test_query310_00008", + "apositive_test_query310_00000", + "apositive_test_query310_00001", + "apositive_test_query310_00009", + "negative_test_query148_00010", + "apositive_test_query310_00003", + "apositive_test_query218_00005", + "negative_test_query329_00003", + "negative_test_query241_00001", + "negative_test_query191_00010", + "negative_test_query179_00013", + "negative_test_query74_00002", + "apositive_test_query310_00006", + "negative_test_query330_00007", + "negative_test_query292_00007", + "negative_test_query148_00003", + "negative_test_query310_00001", + "negative_test_query59_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query311_00008", + "negative_test_query311_00005", + "negative_test_query311_00012", + "apositive_test_query311_00000", + "apositive_test_query330_00002", + "apositive_test_query330_00001", + "negative_test_query240_00008", + "negative_test_query240_00014", + "negative_test_query246_00006", + "negative_test_query202_00017", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query330_00009", + "apositive_test_query5_00002", + "negative_test_query58_00001", + "negative_test_query12_00008", + "negative_test_query288_00004", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "negative_test_query237_00000", + "apositive_test_query212_00004", + "apositive_test_query152_00004", + "apositive_test_query326_00004", + "apositive_test_query312_00000", + "negative_test_query312_00006", + "negative_test_query343_00016", + "apositive_test_query98_00000", + "negative_test_query157_00002", + "negative_test_query182_00002", + "negative_test_query212_00003", + "negative_test_query246_00015", + "negative_test_query206_00010", + "negative_test_query191_00000", + "negative_test_query242_00001", + "negative_test_query294_00015", + "negative_test_query283_00016", + "apositive_test_query220_00005", + "negative_test_query305_00015", + "apositive_test_query237_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query46_00003", + "negative_test_query313_00010", + "negative_test_query46_00005", + "negative_test_query334_00001", + "negative_test_query46_00011", + "negative_test_query359_00005", + "negative_test_query111_00013", + "negative_test_query57_00001", + "negative_test_query56_00001", + "apositive_test_query142_00005", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "negative_test_query336_00001", + "negative_test_query119_00005", + "apositive_test_query108_00001", + "negative_test_query313_00002", + "apositive_test_query155_00003", + "negative_test_query158_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query314_00016", + "apositive_test_query314_00007", + "apositive_test_query314_00003", + "apositive_test_query3_00009", + "apositive_test_query276_00007", + "negative_test_query314_00001", + "apositive_test_query314_00005", + "apositive_test_query314_00006", + "apositive_test_query314_00002", + "apositive_test_query314_00011", + "apositive_test_query314_00010", + "apositive_test_query314_00004", + "apositive_test_query314_00012", + "negative_test_query37_00006", + "apositive_test_query314_00013", + "negative_test_query314_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query315_00000", + "apositive_test_query315_00002", + "negative_test_query269_00004", + "apositive_test_query319_00005", + "apositive_test_query82_00005", + "negative_test_query333_00007", + "negative_test_query85_00013", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query345_00006", + "apositive_test_query85_00003", + "negative_test_query85_00014", + "apositive_test_query319_00001", + "negative_test_query85_00005", + "apositive_test_query315_00001", + "negative_test_query315_00009", + "apositive_test_query82_00001", + "negative_test_query319_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query316", + "ranked_ids": [ + "apositive_test_query316_00000", + "apositive_test_query316_00001", + "negative_test_query316_00002", + "negative_test_query316_00011", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query316_00000", + "negative_test_query316_00004", + "negative_test_query316_00005", + "negative_test_query316_00008", + "negative_test_query316_00001", + "negative_test_query316_00016", + "negative_test_query301_00005", + "apositive_test_query63_00007", + "negative_test_query101_00007", + "apositive_test_query216_00000", + "negative_test_query158_00008", + "apositive_test_query72_00000", + "negative_test_query322_00000", + "negative_test_query228_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00004", + "negative_test_query317_00005", + "negative_test_query317_00013", + "negative_test_query317_00014", + "negative_test_query317_00008", + "apositive_test_query317_00000", + "negative_test_query317_00000", + "apositive_test_query317_00001", + "negative_test_query317_00010", + "negative_test_query317_00012", + "negative_test_query317_00006", + "negative_test_query317_00016", + "negative_test_query317_00017", + "negative_test_query116_00014", + "negative_test_query317_00015", + "negative_test_query55_00014", + "negative_test_query271_00002", + "negative_test_query116_00012", + "negative_test_query331_00009", + "negative_test_query283_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00000", + "apositive_test_query318_00002", + "apositive_test_query318_00001", + "negative_test_query318_00015", + "negative_test_query318_00012", + "negative_test_query73_00012", + "apositive_test_query318_00003", + "negative_test_query318_00010", + "negative_test_query318_00005", + "negative_test_query318_00002", + "negative_test_query318_00003", + "negative_test_query318_00000", + "negative_test_query318_00004", + "negative_test_query73_00004", + "negative_test_query73_00005", + "negative_test_query318_00008", + "negative_test_query318_00009", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query73_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00001", + "apositive_test_query319_00003", + "apositive_test_query82_00003", + "apositive_test_query319_00004", + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query82_00004", + "apositive_test_query82_00001", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query319_00008", + "negative_test_query319_00009", + "negative_test_query82_00009", + "negative_test_query340_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query196_00000", + "apositive_test_query136_00003", + "negative_test_query196_00001", + "apositive_test_query32_00002", + "negative_test_query189_00014", + "apositive_test_query357_00000", + "negative_test_query55_00011", + "negative_test_query321_00001", + "apositive_test_query17_00000", + "negative_test_query78_00007", + "apositive_test_query32_00013", + "negative_test_query55_00014", + "apositive_test_query192_00000", + "negative_test_query211_00005", + "negative_test_query341_00010", + "negative_test_query211_00016", + "negative_test_query68_00009", + "negative_test_query83_00001", + "apositive_test_query336_00001", + "negative_test_query339_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query320", + "ranked_ids": [ + "negative_test_query320_00004", + "negative_test_query344_00010", + "negative_test_query318_00011", + "negative_test_query100_00001", + "apositive_test_query238_00000", + "negative_test_query212_00006", + "apositive_test_query320_00003", + "negative_test_query320_00005", + "negative_test_query131_00009", + "negative_test_query91_00007", + "negative_test_query230_00001", + "negative_test_query193_00006", + "negative_test_query258_00012", + "negative_test_query53_00014", + "negative_test_query304_00005", + "negative_test_query43_00000", + "negative_test_query216_00014", + "apositive_test_query267_00001", + "negative_test_query320_00008", + "negative_test_query290_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "apositive_test_query321_00000", + "negative_test_query321_00000", + "negative_test_query321_00010", + "negative_test_query321_00002", + "negative_test_query34_00012", + "negative_test_query352_00005", + "apositive_test_query185_00009", + "negative_test_query103_00011", + "negative_test_query126_00005", + "negative_test_query141_00009", + "negative_test_query136_00004", + "apositive_test_query217_00006", + "negative_test_query197_00004", + "negative_test_query86_00014", + "negative_test_query226_00002", + "negative_test_query322_00002", + "negative_test_query126_00011", + "negative_test_query15_00013", + "negative_test_query210_00016", + "negative_test_query103_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query322", + "ranked_ids": [ + "apositive_test_query322_00000", + "apositive_test_query322_00005", + "apositive_test_query322_00002", + "negative_test_query322_00003", + "apositive_test_query322_00001", + "negative_test_query322_00006", + "apositive_test_query322_00006", + "negative_test_query255_00010", + "apositive_test_query322_00004", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query322_00002", + "apositive_test_query322_00003", + "negative_test_query318_00002", + "negative_test_query73_00010", + "negative_test_query322_00005", + "apositive_test_query228_00001", + "apositive_test_query254_00006", + "negative_test_query73_00008", + "negative_test_query322_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query323", + "ranked_ids": [ + "negative_test_query323_00003", + "apositive_test_query323_00000", + "apositive_test_query130_00001", + "apositive_test_query323_00002", + "apositive_test_query323_00004", + "negative_test_query323_00012", + "apositive_test_query323_00001", + "apositive_test_query323_00006", + "negative_test_query323_00000", + "negative_test_query323_00007", + "negative_test_query323_00001", + "negative_test_query75_00002", + "negative_test_query345_00011", + "negative_test_query209_00004", + "negative_test_query201_00009", + "negative_test_query255_00002", + "negative_test_query323_00006", + "negative_test_query75_00007", + "negative_test_query75_00004", + "negative_test_query15_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8854598815714874, + "recall@5": 1.0 + }, + "query_id": "test_query324", + "ranked_ids": [ + "apositive_test_query324_00002", + "negative_test_query324_00005", + "apositive_test_query324_00001", + "negative_test_query308_00003", + "apositive_test_query324_00000", + "negative_test_query324_00003", + "negative_test_query324_00004", + "apositive_test_query308_00001", + "negative_test_query308_00007", + "negative_test_query324_00015", + "negative_test_query324_00006", + "apositive_test_query308_00000", + "apositive_test_query308_00003", + "negative_test_query324_00016", + "negative_test_query324_00014", + "negative_test_query308_00009", + "negative_test_query218_00003", + "apositive_test_query308_00009", + "negative_test_query324_00007", + "apositive_test_query308_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "apositive_test_query293_00002", + "negative_test_query176_00003", + "negative_test_query325_00007", + "negative_test_query15_00015", + "negative_test_query15_00017", + "negative_test_query325_00002", + "negative_test_query293_00015", + "negative_test_query15_00005", + "negative_test_query15_00016", + "apositive_test_query19_00002", + "negative_test_query325_00008", + "negative_test_query266_00014", + "negative_test_query325_00013", + "apositive_test_query178_00003", + "negative_test_query293_00000", + "apositive_test_query246_00001", + "negative_test_query19_00008", + "negative_test_query191_00011", + "negative_test_query297_00008", + "negative_test_query169_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query326", + "ranked_ids": [ + "negative_test_query107_00011", + "apositive_test_query326_00000", + "apositive_test_query212_00004", + "negative_test_query17_00011", + "negative_test_query178_00012", + "negative_test_query156_00013", + "apositive_test_query326_00003", + "apositive_test_query326_00004", + "apositive_test_query52_00002", + "apositive_test_query54_00002", + "negative_test_query265_00007", + "apositive_test_query326_00001", + "apositive_test_query355_00002", + "negative_test_query183_00001", + "apositive_test_query294_00001", + "apositive_test_query23_00000", + "negative_test_query81_00003", + "negative_test_query181_00007", + "negative_test_query265_00005", + "negative_test_query45_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query327", + "ranked_ids": [ + "apositive_test_query327_00000", + "negative_test_query321_00014", + "apositive_test_query327_00001", + "negative_test_query327_00014", + "negative_test_query327_00000", + "negative_test_query327_00013", + "negative_test_query327_00012", + "negative_test_query327_00003", + "negative_test_query195_00007", + "negative_test_query61_00006", + "negative_test_query303_00010", + "negative_test_query327_00010", + "negative_test_query61_00003", + "apositive_test_query327_00002", + "apositive_test_query271_00000", + "apositive_test_query271_00001", + "negative_test_query227_00009", + "negative_test_query299_00010", + "negative_test_query271_00007", + "negative_test_query335_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query328", + "ranked_ids": [ + "apositive_test_query328_00002", + "negative_test_query328_00003", + "negative_test_query328_00002", + "apositive_test_query328_00001", + "negative_test_query328_00000", + "negative_test_query24_00006", + "negative_test_query92_00009", + "negative_test_query328_00007", + "apositive_test_query40_00003", + "negative_test_query328_00006", + "apositive_test_query328_00007", + "negative_test_query138_00013", + "negative_test_query59_00017", + "negative_test_query144_00005", + "negative_test_query222_00016", + "negative_test_query261_00009", + "apositive_test_query328_00004", + "apositive_test_query227_00001", + "negative_test_query217_00005", + "negative_test_query129_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query329", + "ranked_ids": [ + "apositive_test_query329_00001", + "negative_test_query329_00006", + "negative_test_query329_00010", + "negative_test_query329_00001", + "negative_test_query269_00012", + "negative_test_query329_00009", + "negative_test_query269_00000", + "negative_test_query329_00004", + "negative_test_query329_00011", + "negative_test_query62_00017", + "apositive_test_query329_00002", + "negative_test_query329_00003", + "negative_test_query298_00015", + "negative_test_query329_00013", + "negative_test_query329_00002", + "negative_test_query35_00000", + "negative_test_query183_00006", + "apositive_test_query7_00001", + "negative_test_query202_00016", + "negative_test_query38_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00004", + "apositive_test_query33_00006", + "apositive_test_query33_00004", + "negative_test_query33_00006", + "apositive_test_query154_00007", + "apositive_test_query20_00010", + "apositive_test_query33_00003", + "apositive_test_query284_00004", + "negative_test_query344_00010", + "negative_test_query154_00006", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "apositive_test_query33_00009", + "negative_test_query220_00010", + "negative_test_query216_00004", + "negative_test_query356_00006", + "negative_test_query161_00008", + "apositive_test_query220_00003", + "negative_test_query216_00014", + "apositive_test_query220_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query330", + "ranked_ids": [ + "negative_test_query330_00010", + "apositive_test_query330_00002", + "negative_test_query330_00009", + "negative_test_query330_00008", + "negative_test_query27_00013", + "negative_test_query252_00007", + "negative_test_query27_00018", + "negative_test_query330_00001", + "apositive_test_query60_00014", + "negative_test_query288_00006", + "negative_test_query311_00009", + "negative_test_query330_00000", + "apositive_test_query281_00012", + "negative_test_query330_00012", + "apositive_test_query284_00003", + "apositive_test_query288_00002", + "negative_test_query129_00003", + "negative_test_query180_00000", + "negative_test_query27_00011", + "negative_test_query325_00016" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query331", + "ranked_ids": [ + "negative_test_query331_00013", + "apositive_test_query331_00000", + "apositive_test_query331_00001", + "negative_test_query331_00000", + "negative_test_query250_00016", + "negative_test_query331_00008", + "negative_test_query202_00001", + "apositive_test_query331_00002", + "apositive_test_query223_00001", + "negative_test_query303_00001", + "negative_test_query113_00000", + "negative_test_query334_00014", + "negative_test_query154_00000", + "negative_test_query312_00012", + "negative_test_query331_00012", + "negative_test_query110_00007", + "negative_test_query214_00015", + "negative_test_query251_00009", + "negative_test_query3_00008", + "negative_test_query331_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.6 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00001", + "apositive_test_query332_00002", + "apositive_test_query332_00003", + "negative_test_query202_00010", + "negative_test_query12_00005", + "apositive_test_query142_00004", + "negative_test_query171_00008", + "negative_test_query65_00007", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "negative_test_query180_00011", + "negative_test_query245_00001", + "apositive_test_query348_00008", + "negative_test_query115_00008", + "apositive_test_query162_00005", + "negative_test_query332_00009", + "negative_test_query244_00006", + "negative_test_query145_00010", + "negative_test_query202_00002", + "negative_test_query332_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00002", + "negative_test_query246_00000", + "negative_test_query322_00009", + "apositive_test_query322_00003", + "negative_test_query333_00008", + "negative_test_query216_00013", + "negative_test_query318_00005", + "negative_test_query73_00008", + "negative_test_query67_00000", + "negative_test_query333_00005", + "negative_test_query73_00005", + "apositive_test_query322_00000", + "negative_test_query73_00014", + "negative_test_query318_00010", + "apositive_test_query228_00000", + "apositive_test_query333_00000", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00005", + "negative_test_query136_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query111_00005", + "negative_test_query155_00010", + "negative_test_query334_00010", + "negative_test_query37_00000", + "apositive_test_query276_00009", + "negative_test_query334_00012", + "negative_test_query209_00003", + "negative_test_query155_00013", + "negative_test_query213_00009", + "negative_test_query111_00017", + "negative_test_query107_00001", + "apositive_test_query57_00006", + "negative_test_query111_00006", + "apositive_test_query26_00001", + "negative_test_query111_00010", + "negative_test_query170_00004", + "negative_test_query102_00015", + "negative_test_query334_00016", + "negative_test_query164_00005", + "negative_test_query223_00016" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query335", + "ranked_ids": [ + "apositive_test_query335_00000", + "apositive_test_query335_00003", + "negative_test_query335_00006", + "negative_test_query335_00013", + "negative_test_query335_00000", + "negative_test_query335_00003", + "apositive_test_query335_00002", + "apositive_test_query335_00001", + "negative_test_query335_00010", + "negative_test_query335_00004", + "apositive_test_query335_00004", + "negative_test_query335_00007", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query335_00001", + "negative_test_query335_00009", + "negative_test_query92_00002", + "negative_test_query335_00011", + "negative_test_query45_00010", + "negative_test_query210_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "apositive_test_query50_00005", + "negative_test_query336_00004", + "negative_test_query336_00014", + "negative_test_query336_00009", + "apositive_test_query38_00005", + "apositive_test_query50_00015", + "negative_test_query336_00007", + "apositive_test_query186_00002", + "apositive_test_query50_00012", + "negative_test_query108_00005", + "negative_test_query247_00006", + "apositive_test_query113_00005", + "negative_test_query247_00000", + "negative_test_query247_00008", + "apositive_test_query186_00000", + "apositive_test_query242_00012", + "negative_test_query336_00015", + "negative_test_query247_00003", + "negative_test_query105_00011", + "negative_test_query186_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00010", + "apositive_test_query38_00003", + "negative_test_query200_00001", + "negative_test_query200_00014", + "apositive_test_query337_00000", + "negative_test_query337_00007", + "negative_test_query337_00016", + "negative_test_query200_00007", + "negative_test_query200_00000", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query267_00001", + "negative_test_query200_00015", + "negative_test_query200_00009", + "negative_test_query337_00000", + "negative_test_query159_00013", + "apositive_test_query38_00000", + "negative_test_query337_00004", + "negative_test_query38_00008", + "negative_test_query337_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query338", + "ranked_ids": [ + "apositive_test_query338_00002", + "apositive_test_query338_00005", + "apositive_test_query338_00008", + "apositive_test_query338_00006", + "apositive_test_query338_00000", + "apositive_test_query338_00001", + "apositive_test_query338_00004", + "negative_test_query114_00004", + "negative_test_query114_00007", + "apositive_test_query338_00003", + "negative_test_query81_00004", + "negative_test_query95_00004", + "negative_test_query81_00006", + "negative_test_query119_00003", + "negative_test_query338_00003", + "apositive_test_query81_00000", + "negative_test_query76_00005", + "apositive_test_query114_00005", + "apositive_test_query114_00006", + "negative_test_query53_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8503449055347546, + "recall@5": 1.0 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query339_00001", + "negative_test_query339_00004", + "negative_test_query339_00009", + "negative_test_query339_00008", + "apositive_test_query339_00000", + "negative_test_query339_00012", + "negative_test_query192_00011", + "apositive_test_query96_00001", + "negative_test_query339_00016", + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00001", + "negative_test_query96_00016", + "negative_test_query339_00007", + "negative_test_query96_00004", + "negative_test_query96_00003", + "negative_test_query339_00017", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query339_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query34", + "ranked_ids": [ + "negative_test_query34_00002", + "apositive_test_query352_00000", + "apositive_test_query352_00002", + "apositive_test_query34_00004", + "apositive_test_query352_00003", + "negative_test_query352_00004", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "negative_test_query34_00008", + "negative_test_query352_00010", + "negative_test_query34_00000", + "apositive_test_query352_00001", + "negative_test_query352_00000", + "apositive_test_query34_00001", + "negative_test_query34_00014", + "negative_test_query352_00012", + "negative_test_query352_00013", + "negative_test_query352_00008", + "apositive_test_query34_00000", + "negative_test_query34_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query340", + "ranked_ids": [ + "negative_test_query88_00006", + "apositive_test_query340_00004", + "negative_test_query340_00008", + "negative_test_query340_00014", + "negative_test_query340_00012", + "negative_test_query88_00000", + "apositive_test_query340_00000", + "negative_test_query340_00007", + "negative_test_query340_00011", + "negative_test_query88_00013", + "apositive_test_query340_00002", + "negative_test_query88_00007", + "apositive_test_query340_00001", + "negative_test_query340_00001", + "negative_test_query340_00006", + "apositive_test_query174_00002", + "negative_test_query88_00001", + "apositive_test_query340_00003", + "apositive_test_query174_00012", + "negative_test_query23_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.25 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query96_00005", + "negative_test_query214_00005", + "apositive_test_query341_00003", + "negative_test_query32_00003", + "apositive_test_query341_00001", + "negative_test_query341_00003", + "negative_test_query196_00009", + "negative_test_query316_00008", + "negative_test_query341_00009", + "negative_test_query341_00002", + "apositive_test_query341_00005", + "apositive_test_query341_00002", + "negative_test_query341_00000", + "negative_test_query73_00014", + "negative_test_query341_00005", + "negative_test_query101_00003", + "negative_test_query96_00007", + "negative_test_query96_00013", + "apositive_test_query32_00007", + "negative_test_query196_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query342_00009", + "negative_test_query342_00003", + "negative_test_query342_00015", + "negative_test_query212_00009", + "negative_test_query77_00004", + "negative_test_query134_00013", + "negative_test_query68_00005", + "negative_test_query200_00009", + "negative_test_query84_00007", + "negative_test_query342_00016", + "apositive_test_query342_00000", + "negative_test_query49_00002", + "negative_test_query78_00015", + "negative_test_query136_00003", + "negative_test_query77_00002", + "negative_test_query98_00017", + "negative_test_query98_00011", + "negative_test_query245_00001", + "negative_test_query324_00015", + "negative_test_query325_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query343", + "ranked_ids": [ + "negative_test_query103_00010", + "negative_test_query103_00008", + "apositive_test_query343_00000", + "negative_test_query307_00006", + "apositive_test_query279_00003", + "negative_test_query225_00008", + "apositive_test_query343_00001", + "apositive_test_query307_00008", + "apositive_test_query136_00004", + "apositive_test_query133_00005", + "apositive_test_query287_00003", + "negative_test_query360_00017", + "negative_test_query343_00008", + "apositive_test_query185_00001", + "apositive_test_query217_00002", + "apositive_test_query217_00004", + "apositive_test_query279_00002", + "negative_test_query227_00010", + "apositive_test_query185_00008", + "apositive_test_query28_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "apositive_test_query83_00006", + "negative_test_query282_00008", + "negative_test_query282_00004", + "negative_test_query85_00010", + "negative_test_query344_00009", + "negative_test_query344_00005", + "negative_test_query344_00002", + "negative_test_query282_00001", + "apositive_test_query221_00002", + "negative_test_query344_00004", + "negative_test_query223_00016", + "negative_test_query83_00005", + "negative_test_query83_00008", + "negative_test_query344_00012", + "negative_test_query345_00012", + "negative_test_query231_00015", + "negative_test_query344_00006", + "negative_test_query188_00000", + "negative_test_query282_00000", + "negative_test_query344_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query345_00011", + "negative_test_query345_00003", + "negative_test_query184_00009", + "negative_test_query9_00002", + "negative_test_query90_00003", + "apositive_test_query345_00004", + "negative_test_query345_00007", + "apositive_test_query0_00008", + "negative_test_query345_00001", + "negative_test_query47_00008", + "apositive_test_query9_00000", + "negative_test_query47_00003", + "negative_test_query130_00004", + "negative_test_query345_00010", + "negative_test_query259_00007", + "negative_test_query221_00002", + "apositive_test_query345_00001", + "apositive_test_query203_00005", + "apositive_test_query345_00000", + "apositive_test_query90_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5437713091520254, + "recall@5": 1.0 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query346_00001", + "negative_test_query346_00009", + "apositive_test_query346_00001", + "negative_test_query346_00002", + "apositive_test_query346_00000", + "negative_test_query157_00005", + "negative_test_query346_00010", + "negative_test_query49_00007", + "negative_test_query346_00012", + "negative_test_query346_00003", + "negative_test_query346_00014", + "negative_test_query346_00017", + "negative_test_query346_00006", + "negative_test_query346_00008", + "negative_test_query174_00001", + "negative_test_query19_00012", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "negative_test_query15_00000", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query347", + "ranked_ids": [ + "negative_test_query347_00007", + "apositive_test_query347_00000", + "apositive_test_query347_00001", + "negative_test_query31_00003", + "negative_test_query347_00000", + "negative_test_query347_00001", + "negative_test_query298_00003", + "negative_test_query347_00015", + "negative_test_query347_00002", + "negative_test_query347_00005", + "negative_test_query347_00014", + "negative_test_query347_00008", + "negative_test_query298_00006", + "negative_test_query347_00006", + "negative_test_query347_00009", + "negative_test_query347_00010", + "negative_test_query347_00016", + "negative_test_query347_00012", + "negative_test_query251_00007", + "negative_test_query347_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.3 + }, + "query_id": "test_query348", + "ranked_ids": [ + "negative_test_query348_00006", + "apositive_test_query348_00004", + "apositive_test_query348_00001", + "apositive_test_query348_00006", + "negative_test_query115_00012", + "apositive_test_query348_00005", + "apositive_test_query348_00009", + "negative_test_query348_00008", + "apositive_test_query115_00002", + "negative_test_query115_00006", + "apositive_test_query115_00004", + "apositive_test_query227_00002", + "negative_test_query115_00002", + "apositive_test_query348_00000", + "negative_test_query359_00014", + "negative_test_query297_00017", + "apositive_test_query115_00001", + "negative_test_query341_00008", + "negative_test_query351_00010", + "negative_test_query31_00017" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "negative_test_query253_00007", + "negative_test_query74_00001", + "apositive_test_query253_00005", + "negative_test_query349_00015", + "negative_test_query253_00000", + "negative_test_query312_00016", + "apositive_test_query74_00003", + "negative_test_query349_00003", + "apositive_test_query137_00002", + "apositive_test_query253_00008", + "negative_test_query124_00013", + "apositive_test_query253_00009", + "apositive_test_query42_00004", + "apositive_test_query355_00007", + "negative_test_query13_00002", + "negative_test_query253_00009", + "apositive_test_query74_00002", + "negative_test_query157_00009", + "negative_test_query349_00012", + "negative_test_query194_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query139_00000", + "apositive_test_query35_00000", + "negative_test_query205_00005", + "apositive_test_query35_00001", + "negative_test_query35_00002", + "apositive_test_query35_00003", + "apositive_test_query146_00007", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "negative_test_query139_00016", + "negative_test_query35_00012", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query101_00006", + "negative_test_query298_00008", + "negative_test_query231_00012", + "negative_test_query123_00006", + "negative_test_query269_00006", + "negative_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query350_00015", + "negative_test_query350_00014", + "negative_test_query350_00010", + "negative_test_query120_00007", + "negative_test_query14_00001", + "negative_test_query350_00013", + "negative_test_query179_00001", + "negative_test_query149_00005", + "negative_test_query331_00012", + "negative_test_query63_00000", + "negative_test_query192_00004", + "negative_test_query350_00001", + "negative_test_query350_00006", + "negative_test_query192_00005", + "negative_test_query350_00011", + "negative_test_query316_00006", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query192_00013", + "apositive_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query351_00000", + "apositive_test_query299_00000", + "negative_test_query92_00005", + "apositive_test_query60_00003", + "negative_test_query156_00006", + "negative_test_query351_00002", + "apositive_test_query60_00000", + "negative_test_query351_00006", + "negative_test_query351_00014", + "apositive_test_query311_00000", + "apositive_test_query60_00010", + "apositive_test_query113_00001", + "apositive_test_query218_00004", + "apositive_test_query335_00004", + "apositive_test_query60_00012", + "negative_test_query234_00009", + "negative_test_query53_00011", + "negative_test_query8_00003", + "apositive_test_query193_00009", + "apositive_test_query60_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query34_00001", + "apositive_test_query352_00003", + "apositive_test_query34_00004", + "negative_test_query34_00002", + "negative_test_query352_00004", + "apositive_test_query352_00001", + "apositive_test_query352_00004", + "apositive_test_query352_00000", + "negative_test_query34_00008", + "negative_test_query352_00013", + "negative_test_query352_00010", + "apositive_test_query352_00002", + "negative_test_query34_00000", + "negative_test_query352_00000", + "apositive_test_query34_00000", + "negative_test_query34_00012", + "negative_test_query34_00013", + "apositive_test_query34_00002", + "negative_test_query352_00008", + "negative_test_query34_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.2 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query353_00002", + "apositive_test_query204_00000", + "apositive_test_query353_00005", + "apositive_test_query353_00011", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "negative_test_query353_00002", + "apositive_test_query204_00012", + "apositive_test_query204_00002", + "apositive_test_query204_00009", + "apositive_test_query353_00000", + "apositive_test_query353_00009", + "apositive_test_query51_00002", + "negative_test_query353_00003", + "negative_test_query108_00002", + "apositive_test_query204_00007", + "apositive_test_query204_00015", + "apositive_test_query353_00007", + "negative_test_query71_00008", + "negative_test_query175_00012" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "apositive_test_query354_00001", + "apositive_test_query354_00006", + "negative_test_query309_00014", + "apositive_test_query354_00004", + "negative_test_query309_00013", + "negative_test_query354_00004", + "negative_test_query309_00000", + "negative_test_query194_00002", + "negative_test_query358_00003", + "negative_test_query331_00014", + "negative_test_query354_00006", + "negative_test_query280_00007", + "apositive_test_query16_00003", + "apositive_test_query16_00001", + "negative_test_query148_00009", + "apositive_test_query320_00006", + "apositive_test_query354_00005", + "apositive_test_query147_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query355_00001", + "negative_test_query355_00009", + "negative_test_query355_00007", + "apositive_test_query355_00002", + "negative_test_query355_00008", + "negative_test_query355_00005", + "negative_test_query355_00003", + "apositive_test_query355_00005", + "negative_test_query355_00002", + "negative_test_query159_00017", + "apositive_test_query355_00003", + "negative_test_query264_00017", + "apositive_test_query355_00006", + "negative_test_query68_00000", + "negative_test_query87_00004", + "negative_test_query215_00001", + "negative_test_query264_00003", + "negative_test_query104_00001", + "apositive_test_query38_00006", + "apositive_test_query100_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query66_00003", + "apositive_test_query356_00004", + "apositive_test_query356_00000", + "negative_test_query253_00001", + "apositive_test_query356_00002", + "negative_test_query15_00006", + "negative_test_query47_00011", + "negative_test_query102_00014", + "apositive_test_query30_00005", + "apositive_test_query356_00008", + "negative_test_query102_00010", + "apositive_test_query20_00001", + "apositive_test_query33_00000", + "negative_test_query280_00004", + "apositive_test_query203_00006", + "negative_test_query334_00000", + "apositive_test_query295_00000", + "apositive_test_query127_00007", + "apositive_test_query88_00000", + "apositive_test_query154_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00002", + "apositive_test_query117_00004", + "apositive_test_query314_00000", + "apositive_test_query221_00000", + "apositive_test_query221_00004", + "apositive_test_query119_00005", + "negative_test_query196_00001", + "apositive_test_query221_00002", + "apositive_test_query119_00001", + "negative_test_query300_00007", + "apositive_test_query119_00000", + "negative_test_query58_00009", + "apositive_test_query119_00010", + "apositive_test_query32_00002", + "apositive_test_query119_00012", + "negative_test_query58_00008", + "apositive_test_query357_00003", + "apositive_test_query357_00004", + "apositive_test_query119_00009", + "negative_test_query46_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.3 + }, + "query_id": "test_query358", + "ranked_ids": [ + "negative_test_query309_00005", + "apositive_test_query358_00006", + "negative_test_query123_00007", + "apositive_test_query358_00007", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "apositive_test_query11_00007", + "apositive_test_query309_00000", + "negative_test_query309_00006", + "apositive_test_query358_00003", + "negative_test_query132_00001", + "apositive_test_query161_00000", + "apositive_test_query358_00000", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "apositive_test_query358_00009", + "negative_test_query354_00008", + "negative_test_query279_00010", + "negative_test_query309_00015", + "negative_test_query42_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00006", + "apositive_test_query0_00003", + "negative_test_query359_00002", + "apositive_test_query0_00004", + "negative_test_query305_00000", + "negative_test_query184_00016", + "apositive_test_query0_00010", + "negative_test_query0_00001", + "negative_test_query0_00007", + "apositive_test_query0_00000", + "negative_test_query359_00007", + "negative_test_query268_00014", + "apositive_test_query259_00008", + "apositive_test_query0_00002", + "apositive_test_query0_00007", + "negative_test_query259_00006", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "negative_test_query130_00000", + "apositive_test_query259_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query36", + "ranked_ids": [ + "negative_test_query112_00004", + "apositive_test_query36_00003", + "apositive_test_query286_00006", + "negative_test_query112_00016", + "negative_test_query112_00015", + "negative_test_query36_00014", + "apositive_test_query286_00005", + "negative_test_query286_00000", + "negative_test_query112_00010", + "negative_test_query216_00002", + "negative_test_query36_00001", + "apositive_test_query106_00003", + "negative_test_query260_00009", + "negative_test_query296_00013", + "negative_test_query36_00010", + "negative_test_query6_00003", + "negative_test_query260_00010", + "apositive_test_query136_00013", + "negative_test_query112_00013", + "negative_test_query80_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00001", + "apositive_test_query360_00000", + "apositive_test_query310_00006", + "apositive_test_query69_00014", + "negative_test_query360_00005", + "negative_test_query189_00013", + "negative_test_query240_00010", + "apositive_test_query152_00002", + "apositive_test_query153_00011", + "apositive_test_query142_00004", + "negative_test_query281_00000", + "negative_test_query189_00014", + "negative_test_query360_00004", + "negative_test_query148_00018", + "negative_test_query175_00009", + "negative_test_query216_00011", + "negative_test_query312_00004", + "negative_test_query246_00001", + "negative_test_query360_00009", + "negative_test_query5_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query231_00006", + "apositive_test_query158_00002", + "apositive_test_query177_00000", + "negative_test_query209_00001", + "negative_test_query123_00001", + "negative_test_query131_00011", + "negative_test_query223_00010", + "apositive_test_query37_00001", + "negative_test_query155_00001", + "negative_test_query123_00011", + "apositive_test_query158_00007", + "apositive_test_query213_00006", + "negative_test_query209_00000", + "negative_test_query20_00003", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "negative_test_query152_00008", + "apositive_test_query209_00002", + "apositive_test_query108_00000", + "negative_test_query57_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query38", + "ranked_ids": [ + "apositive_test_query38_00003", + "negative_test_query159_00010", + "negative_test_query267_00016", + "negative_test_query38_00010", + "negative_test_query267_00015", + "negative_test_query47_00016", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query337_00000", + "apositive_test_query38_00007", + "apositive_test_query337_00000", + "apositive_test_query247_00005", + "apositive_test_query38_00002", + "negative_test_query337_00017", + "negative_test_query38_00006", + "negative_test_query159_00011", + "negative_test_query79_00002", + "negative_test_query264_00012", + "negative_test_query175_00005", + "negative_test_query94_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query39", + "ranked_ids": [ + "apositive_test_query39_00003", + "apositive_test_query39_00001", + "apositive_test_query39_00004", + "negative_test_query18_00004", + "apositive_test_query39_00002", + "negative_test_query39_00000", + "negative_test_query39_00006", + "negative_test_query39_00012", + "negative_test_query39_00009", + "apositive_test_query39_00000", + "negative_test_query39_00007", + "negative_test_query331_00010", + "negative_test_query232_00007", + "negative_test_query39_00003", + "negative_test_query66_00008", + "negative_test_query39_00004", + "negative_test_query232_00005", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query29_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.35714285714285715 + }, + "query_id": "test_query4", + "ranked_ids": [ + "apositive_test_query4_00003", + "apositive_test_query4_00000", + "apositive_test_query4_00010", + "apositive_test_query4_00001", + "apositive_test_query4_00004", + "apositive_test_query4_00012", + "negative_test_query4_00001", + "negative_test_query266_00009", + "negative_test_query4_00002", + "negative_test_query174_00001", + "apositive_test_query174_00000", + "apositive_test_query19_00000", + "apositive_test_query4_00005", + "negative_test_query312_00011", + "apositive_test_query174_00001", + "negative_test_query266_00004", + "negative_test_query149_00015", + "negative_test_query15_00000", + "negative_test_query266_00012", + "negative_test_query19_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00004", + "negative_test_query40_00010", + "negative_test_query40_00007", + "negative_test_query40_00006", + "negative_test_query40_00013", + "negative_test_query40_00001", + "negative_test_query40_00000", + "negative_test_query40_00005", + "apositive_test_query40_00002", + "negative_test_query40_00003", + "negative_test_query40_00011", + "negative_test_query347_00003", + "negative_test_query66_00005", + "negative_test_query214_00015", + "negative_test_query40_00009", + "negative_test_query40_00012", + "negative_test_query328_00006", + "negative_test_query106_00009", + "apositive_test_query223_00001", + "negative_test_query324_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query41", + "ranked_ids": [ + "apositive_test_query41_00002", + "negative_test_query41_00000", + "negative_test_query41_00007", + "apositive_test_query41_00001", + "negative_test_query282_00003", + "negative_test_query41_00004", + "negative_test_query41_00010", + "negative_test_query41_00002", + "apositive_test_query41_00000", + "negative_test_query41_00012", + "apositive_test_query41_00003", + "negative_test_query41_00014", + "apositive_test_query249_00012", + "negative_test_query41_00006", + "negative_test_query144_00016", + "negative_test_query226_00011", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query98_00000", + "negative_test_query41_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query42_00002", + "apositive_test_query42_00003", + "negative_test_query42_00003", + "negative_test_query42_00000", + "negative_test_query42_00008", + "apositive_test_query42_00005", + "negative_test_query137_00005", + "apositive_test_query42_00001", + "apositive_test_query42_00000", + "negative_test_query42_00001", + "apositive_test_query42_00002", + "negative_test_query42_00010", + "negative_test_query42_00005", + "negative_test_query42_00007", + "apositive_test_query42_00004", + "negative_test_query42_00006", + "negative_test_query250_00010", + "apositive_test_query276_00012", + "apositive_test_query162_00008", + "negative_test_query42_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query43_00000", + "negative_test_query43_00009", + "apositive_test_query43_00002", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "apositive_test_query146_00004", + "apositive_test_query146_00003", + "apositive_test_query314_00009", + "apositive_test_query276_00001", + "negative_test_query146_00002", + "negative_test_query333_00001", + "apositive_test_query218_00002", + "apositive_test_query161_00002", + "negative_test_query179_00006", + "negative_test_query43_00000", + "apositive_test_query153_00008", + "apositive_test_query153_00014", + "apositive_test_query161_00005", + "negative_test_query319_00002", + "negative_test_query82_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query44", + "ranked_ids": [ + "apositive_test_query44_00001", + "negative_test_query27_00011", + "negative_test_query27_00002", + "negative_test_query330_00009", + "negative_test_query141_00007", + "apositive_test_query20_00003", + "negative_test_query293_00003", + "negative_test_query293_00015", + "negative_test_query27_00008", + "negative_test_query293_00009", + "negative_test_query44_00002", + "apositive_test_query141_00005", + "negative_test_query293_00000", + "apositive_test_query293_00000", + "negative_test_query10_00012", + "apositive_test_query245_00000", + "negative_test_query21_00009", + "negative_test_query330_00012", + "negative_test_query190_00011", + "negative_test_query180_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query45_00013", + "apositive_test_query45_00000", + "negative_test_query45_00001", + "apositive_test_query25_00000", + "negative_test_query45_00007", + "negative_test_query45_00004", + "negative_test_query182_00016", + "negative_test_query328_00005", + "negative_test_query45_00005", + "negative_test_query45_00003", + "negative_test_query143_00011", + "negative_test_query143_00009", + "negative_test_query45_00006", + "apositive_test_query25_00005", + "negative_test_query342_00005", + "negative_test_query143_00014", + "negative_test_query45_00016", + "negative_test_query45_00009", + "negative_test_query143_00004", + "apositive_test_query165_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "negative_test_query108_00003", + "negative_test_query39_00011", + "negative_test_query46_00001", + "negative_test_query56_00001", + "apositive_test_query142_00001", + "apositive_test_query79_00005", + "negative_test_query105_00006", + "negative_test_query46_00003", + "negative_test_query359_00005", + "negative_test_query213_00012", + "negative_test_query313_00010", + "negative_test_query238_00001", + "apositive_test_query146_00013", + "negative_test_query46_00011", + "negative_test_query168_00003", + "negative_test_query79_00012", + "negative_test_query46_00012", + "negative_test_query334_00004", + "apositive_test_query276_00008", + "apositive_test_query186_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query331_00002", + "apositive_test_query47_00001", + "apositive_test_query201_00004", + "negative_test_query47_00013", + "negative_test_query47_00006", + "apositive_test_query201_00002", + "negative_test_query144_00018", + "negative_test_query325_00007", + "negative_test_query47_00002", + "negative_test_query41_00009", + "negative_test_query47_00012", + "negative_test_query195_00005", + "negative_test_query331_00014", + "negative_test_query331_00005", + "negative_test_query47_00011", + "apositive_test_query201_00006", + "negative_test_query331_00013", + "apositive_test_query20_00005", + "negative_test_query345_00002", + "negative_test_query328_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "negative_test_query48_00007", + "apositive_test_query48_00000", + "negative_test_query307_00005", + "apositive_test_query185_00012", + "apositive_test_query28_00013", + "negative_test_query48_00016", + "apositive_test_query28_00010", + "negative_test_query48_00009", + "apositive_test_query343_00001", + "negative_test_query48_00012", + "apositive_test_query185_00004", + "negative_test_query48_00011", + "negative_test_query103_00012", + "negative_test_query48_00018", + "negative_test_query307_00003", + "negative_test_query48_00004", + "negative_test_query307_00001", + "negative_test_query49_00001", + "negative_test_query48_00000", + "apositive_test_query307_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query49", + "ranked_ids": [ + "apositive_test_query49_00001", + "negative_test_query176_00016", + "apositive_test_query29_00006", + "negative_test_query169_00000", + "negative_test_query49_00001", + "negative_test_query342_00005", + "negative_test_query106_00004", + "apositive_test_query54_00009", + "negative_test_query49_00002", + "negative_test_query328_00009", + "apositive_test_query19_00001", + "negative_test_query49_00007", + "negative_test_query52_00011", + "negative_test_query15_00008", + "negative_test_query156_00007", + "apositive_test_query204_00003", + "negative_test_query72_00017", + "negative_test_query156_00010", + "apositive_test_query353_00004", + "negative_test_query122_00016" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query5", + "ranked_ids": [ + "apositive_test_query5_00000", + "negative_test_query246_00013", + "negative_test_query5_00005", + "apositive_test_query5_00001", + "negative_test_query67_00010", + "apositive_test_query5_00002", + "negative_test_query5_00000", + "negative_test_query246_00002", + "negative_test_query5_00006", + "negative_test_query5_00004", + "negative_test_query275_00001", + "negative_test_query5_00003", + "negative_test_query5_00007", + "negative_test_query5_00001", + "negative_test_query5_00010", + "negative_test_query67_00003", + "negative_test_query5_00013", + "negative_test_query246_00008", + "negative_test_query5_00014", + "negative_test_query301_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00014", + "apositive_test_query186_00005", + "apositive_test_query50_00003", + "apositive_test_query50_00013", + "apositive_test_query50_00009", + "negative_test_query336_00011", + "apositive_test_query50_00008", + "apositive_test_query50_00015", + "apositive_test_query50_00006", + "apositive_test_query50_00001", + "apositive_test_query50_00005", + "negative_test_query336_00004", + "apositive_test_query50_00002", + "apositive_test_query50_00012", + "negative_test_query50_00001", + "apositive_test_query131_00000", + "negative_test_query131_00001", + "negative_test_query66_00002", + "negative_test_query119_00002", + "negative_test_query223_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.3 + }, + "query_id": "test_query51", + "ranked_ids": [ + "negative_test_query51_00006", + "apositive_test_query51_00000", + "apositive_test_query51_00006", + "apositive_test_query51_00009", + "negative_test_query97_00011", + "apositive_test_query51_00007", + "negative_test_query51_00000", + "negative_test_query97_00002", + "negative_test_query97_00001", + "apositive_test_query51_00001", + "apositive_test_query51_00005", + "apositive_test_query51_00008", + "negative_test_query97_00007", + "negative_test_query97_00006", + "negative_test_query204_00001", + "negative_test_query267_00007", + "negative_test_query51_00004", + "negative_test_query209_00008", + "negative_test_query231_00012", + "apositive_test_query267_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query257_00005", + "negative_test_query52_00007", + "negative_test_query54_00004", + "negative_test_query215_00000", + "apositive_test_query286_00014", + "negative_test_query257_00009", + "negative_test_query257_00003", + "apositive_test_query257_00001", + "apositive_test_query286_00013", + "apositive_test_query286_00006", + "apositive_test_query257_00002", + "apositive_test_query257_00000", + "apositive_test_query257_00004", + "apositive_test_query257_00003", + "negative_test_query52_00005", + "apositive_test_query69_00007", + "apositive_test_query257_00005", + "negative_test_query257_00007", + "negative_test_query52_00004", + "apositive_test_query52_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00016", + "negative_test_query161_00007", + "negative_test_query150_00015", + "negative_test_query258_00018", + "negative_test_query354_00001", + "negative_test_query354_00011", + "negative_test_query258_00000", + "negative_test_query258_00013", + "negative_test_query53_00000", + "negative_test_query3_00008", + "negative_test_query258_00008", + "apositive_test_query217_00004", + "apositive_test_query28_00006", + "negative_test_query169_00003", + "negative_test_query180_00012", + "negative_test_query348_00001", + "negative_test_query150_00014", + "negative_test_query230_00009", + "apositive_test_query248_00005", + "negative_test_query132_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query54_00006", + "negative_test_query44_00014", + "negative_test_query140_00006", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query56_00011", + "negative_test_query200_00006", + "negative_test_query51_00006", + "negative_test_query44_00007", + "apositive_test_query86_00000", + "negative_test_query301_00006", + "negative_test_query97_00002", + "apositive_test_query51_00008", + "apositive_test_query51_00006", + "apositive_test_query51_00007", + "apositive_test_query323_00006", + "apositive_test_query156_00000", + "negative_test_query136_00004", + "apositive_test_query335_00002", + "apositive_test_query204_00002" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query55", + "ranked_ids": [ + "apositive_test_query55_00000", + "negative_test_query55_00003", + "negative_test_query55_00002", + "apositive_test_query55_00001", + "negative_test_query55_00005", + "negative_test_query55_00016", + "negative_test_query55_00013", + "negative_test_query55_00008", + "negative_test_query55_00015", + "negative_test_query55_00007", + "apositive_test_query55_00002", + "negative_test_query139_00015", + "negative_test_query55_00000", + "negative_test_query55_00010", + "negative_test_query55_00012", + "negative_test_query61_00008", + "negative_test_query55_00011", + "negative_test_query234_00011", + "negative_test_query55_00009", + "negative_test_query55_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query56", + "ranked_ids": [ + "apositive_test_query56_00005", + "negative_test_query105_00006", + "negative_test_query359_00005", + "negative_test_query46_00014", + "negative_test_query111_00006", + "negative_test_query119_00005", + "apositive_test_query56_00006", + "negative_test_query56_00005", + "negative_test_query46_00011", + "negative_test_query56_00002", + "apositive_test_query238_00001", + "negative_test_query56_00006", + "negative_test_query186_00007", + "apositive_test_query46_00001", + "negative_test_query56_00000", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query56_00004", + "negative_test_query56_00003", + "negative_test_query102_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query57", + "ranked_ids": [ + "negative_test_query295_00003", + "apositive_test_query57_00005", + "apositive_test_query57_00003", + "apositive_test_query57_00008", + "apositive_test_query57_00006", + "apositive_test_query57_00000", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query46_00006", + "negative_test_query57_00001", + "negative_test_query111_00010", + "negative_test_query334_00001", + "apositive_test_query259_00006", + "negative_test_query198_00016", + "negative_test_query65_00010", + "apositive_test_query338_00002", + "apositive_test_query203_00000", + "apositive_test_query146_00000", + "negative_test_query57_00010", + "apositive_test_query74_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "negative_test_query58_00010", + "negative_test_query240_00016", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00015", + "negative_test_query58_00001", + "negative_test_query240_00008", + "negative_test_query308_00004", + "negative_test_query240_00002", + "negative_test_query346_00000", + "apositive_test_query311_00004", + "apositive_test_query311_00003", + "negative_test_query11_00004", + "negative_test_query27_00017", + "negative_test_query311_00012", + "negative_test_query27_00016", + "negative_test_query240_00004", + "negative_test_query57_00004", + "negative_test_query240_00005", + "negative_test_query240_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "apositive_test_query59_00000", + "negative_test_query59_00010", + "negative_test_query59_00012", + "negative_test_query116_00003", + "negative_test_query116_00013", + "negative_test_query116_00000", + "negative_test_query59_00006", + "negative_test_query59_00011", + "negative_test_query59_00018", + "negative_test_query59_00014", + "negative_test_query227_00008", + "negative_test_query59_00009", + "negative_test_query59_00017", + "apositive_test_query104_00000", + "negative_test_query59_00015", + "negative_test_query66_00008", + "negative_test_query59_00005", + "negative_test_query81_00003", + "negative_test_query280_00003", + "negative_test_query202_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query6", + "ranked_ids": [ + "apositive_test_query6_00000", + "apositive_test_query6_00002", + "apositive_test_query6_00001", + "negative_test_query6_00002", + "apositive_test_query305_00003", + "negative_test_query6_00004", + "negative_test_query6_00012", + "negative_test_query179_00007", + "negative_test_query6_00001", + "apositive_test_query109_00001", + "apositive_test_query189_00000", + "apositive_test_query242_00004", + "negative_test_query6_00011", + "negative_test_query296_00013", + "negative_test_query252_00004", + "negative_test_query296_00006", + "negative_test_query58_00007", + "negative_test_query112_00010", + "negative_test_query112_00004", + "negative_test_query6_00016" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.25 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00007", + "apositive_test_query60_00003", + "apositive_test_query60_00001", + "apositive_test_query60_00013", + "negative_test_query31_00004", + "negative_test_query243_00004", + "negative_test_query60_00001", + "apositive_test_query60_00000", + "apositive_test_query60_00009", + "apositive_test_query218_00004", + "apositive_test_query285_00007", + "apositive_test_query285_00001", + "negative_test_query218_00007", + "negative_test_query304_00002", + "negative_test_query60_00003", + "negative_test_query304_00003", + "apositive_test_query60_00005", + "apositive_test_query128_00001", + "apositive_test_query60_00011", + "apositive_test_query60_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query61", + "ranked_ids": [ + "apositive_test_query61_00003", + "apositive_test_query61_00001", + "negative_test_query173_00003", + "negative_test_query61_00014", + "negative_test_query129_00005", + "apositive_test_query61_00002", + "negative_test_query61_00010", + "negative_test_query61_00006", + "negative_test_query61_00003", + "negative_test_query61_00000", + "negative_test_query61_00005", + "negative_test_query201_00004", + "apositive_test_query101_00002", + "negative_test_query61_00015", + "negative_test_query211_00005", + "negative_test_query210_00012", + "negative_test_query61_00004", + "negative_test_query61_00002", + "negative_test_query327_00016", + "apositive_test_query173_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00000", + "negative_test_query62_00004", + "negative_test_query62_00001", + "negative_test_query62_00002", + "negative_test_query62_00010", + "negative_test_query62_00003", + "negative_test_query62_00011", + "apositive_test_query62_00001", + "negative_test_query62_00013", + "negative_test_query62_00008", + "negative_test_query62_00006", + "negative_test_query352_00005", + "negative_test_query250_00003", + "negative_test_query16_00001", + "negative_test_query15_00011", + "negative_test_query287_00007", + "negative_test_query219_00006", + "negative_test_query88_00015", + "negative_test_query222_00007", + "negative_test_query38_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00001", + "negative_test_query297_00006", + "apositive_test_query63_00004", + "negative_test_query63_00003", + "negative_test_query224_00007", + "negative_test_query63_00005", + "negative_test_query63_00002", + "negative_test_query63_00001", + "negative_test_query321_00002", + "apositive_test_query288_00004", + "negative_test_query63_00010", + "negative_test_query331_00010", + "apositive_test_query63_00006", + "negative_test_query78_00006", + "negative_test_query63_00004", + "negative_test_query264_00015", + "negative_test_query241_00011", + "apositive_test_query141_00001", + "apositive_test_query63_00008", + "negative_test_query63_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query211_00009", + "negative_test_query64_00015", + "negative_test_query205_00009", + "negative_test_query64_00013", + "negative_test_query64_00010", + "negative_test_query149_00017", + "negative_test_query120_00003", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query64_00016", + "negative_test_query120_00000", + "negative_test_query63_00009", + "apositive_test_query160_00003", + "negative_test_query310_00005", + "negative_test_query149_00001", + "apositive_test_query160_00000", + "negative_test_query63_00000", + "negative_test_query234_00010", + "negative_test_query64_00009", + "negative_test_query214_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "apositive_test_query65_00001", + "negative_test_query65_00004", + "apositive_test_query65_00000", + "apositive_test_query134_00001", + "apositive_test_query356_00004", + "negative_test_query30_00011", + "negative_test_query47_00005", + "apositive_test_query260_00000", + "negative_test_query260_00013", + "negative_test_query6_00008", + "apositive_test_query62_00000", + "negative_test_query230_00001", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query23_00006", + "negative_test_query6_00003", + "negative_test_query268_00012", + "negative_test_query65_00000", + "negative_test_query260_00012", + "negative_test_query261_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.5 + }, + "query_id": "test_query66", + "ranked_ids": [ + "apositive_test_query66_00004", + "negative_test_query66_00011", + "apositive_test_query66_00003", + "apositive_test_query66_00005", + "negative_test_query188_00014", + "apositive_test_query66_00002", + "apositive_test_query20_00005", + "apositive_test_query20_00001", + "negative_test_query220_00010", + "negative_test_query220_00001", + "negative_test_query161_00008", + "apositive_test_query220_00003", + "negative_test_query188_00006", + "negative_test_query154_00009", + "apositive_test_query188_00000", + "negative_test_query161_00003", + "apositive_test_query188_00003", + "negative_test_query9_00012", + "negative_test_query344_00007", + "apositive_test_query220_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query67", + "ranked_ids": [ + "negative_test_query67_00011", + "negative_test_query67_00005", + "negative_test_query67_00001", + "negative_test_query67_00010", + "negative_test_query246_00016", + "negative_test_query67_00008", + "apositive_test_query286_00010", + "negative_test_query67_00007", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "apositive_test_query67_00000", + "negative_test_query246_00009", + "negative_test_query67_00015", + "negative_test_query123_00008", + "apositive_test_query286_00004", + "negative_test_query260_00013", + "apositive_test_query286_00007", + "negative_test_query6_00008", + "negative_test_query65_00008", + "negative_test_query55_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query68", + "ranked_ids": [ + "apositive_test_query68_00002", + "negative_test_query68_00001", + "negative_test_query126_00008", + "apositive_test_query68_00000", + "negative_test_query68_00013", + "apositive_test_query326_00005", + "negative_test_query151_00008", + "negative_test_query330_00009", + "negative_test_query68_00014", + "apositive_test_query252_00001", + "negative_test_query68_00016", + "apositive_test_query330_00001", + "apositive_test_query68_00001", + "negative_test_query68_00007", + "apositive_test_query89_00003", + "negative_test_query212_00000", + "negative_test_query311_00005", + "apositive_test_query5_00002", + "negative_test_query288_00004", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00002", + "apositive_test_query69_00011", + "apositive_test_query257_00003", + "apositive_test_query69_00000", + "apositive_test_query69_00009", + "apositive_test_query69_00013", + "negative_test_query169_00002", + "apositive_test_query69_00008", + "apositive_test_query355_00001", + "negative_test_query18_00010", + "apositive_test_query69_00010", + "apositive_test_query69_00005", + "apositive_test_query69_00012", + "apositive_test_query281_00016", + "apositive_test_query174_00016", + "apositive_test_query107_00007", + "apositive_test_query52_00001", + "negative_test_query277_00001", + "apositive_test_query54_00003", + "apositive_test_query225_00001" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "negative_test_query7_00005", + "apositive_test_query119_00006", + "apositive_test_query326_00007", + "apositive_test_query212_00002", + "apositive_test_query212_00001", + "negative_test_query159_00009", + "negative_test_query7_00001", + "negative_test_query306_00008", + "negative_test_query38_00010", + "negative_test_query84_00007", + "apositive_test_query159_00001", + "negative_test_query264_00013", + "negative_test_query79_00002", + "negative_test_query231_00011", + "apositive_test_query38_00003", + "negative_test_query327_00011", + "negative_test_query159_00004", + "negative_test_query184_00006", + "negative_test_query290_00006", + "negative_test_query159_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00004", + "negative_test_query70_00011", + "apositive_test_query70_00002", + "negative_test_query227_00013", + "negative_test_query227_00006", + "apositive_test_query70_00001", + "negative_test_query70_00006", + "negative_test_query70_00003", + "negative_test_query70_00012", + "negative_test_query70_00000", + "negative_test_query227_00003", + "negative_test_query227_00000", + "negative_test_query227_00012", + "negative_test_query348_00007", + "negative_test_query227_00014", + "negative_test_query227_00010", + "apositive_test_query227_00000", + "apositive_test_query170_00001", + "negative_test_query59_00004", + "negative_test_query283_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query71", + "ranked_ids": [ + "apositive_test_query71_00003", + "negative_test_query71_00009", + "negative_test_query71_00001", + "negative_test_query71_00013", + "apositive_test_query71_00000", + "negative_test_query259_00002", + "negative_test_query65_00002", + "apositive_test_query71_00001", + "negative_test_query71_00011", + "negative_test_query71_00005", + "apositive_test_query30_00005", + "negative_test_query71_00000", + "apositive_test_query71_00002", + "apositive_test_query209_00005", + "negative_test_query231_00014", + "negative_test_query71_00002", + "negative_test_query186_00000", + "negative_test_query247_00002", + "negative_test_query313_00001", + "negative_test_query71_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "apositive_test_query72_00000", + "negative_test_query72_00001", + "negative_test_query72_00008", + "negative_test_query72_00003", + "negative_test_query72_00016", + "negative_test_query185_00005", + "negative_test_query72_00015", + "negative_test_query72_00007", + "negative_test_query185_00000", + "negative_test_query72_00006", + "negative_test_query72_00013", + "negative_test_query17_00009", + "negative_test_query55_00005", + "negative_test_query92_00001", + "apositive_test_query175_00001", + "negative_test_query228_00000", + "negative_test_query285_00009", + "negative_test_query265_00003", + "negative_test_query72_00011", + "negative_test_query233_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query73", + "ranked_ids": [ + "negative_test_query322_00009", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00004", + "apositive_test_query322_00003", + "negative_test_query73_00001", + "negative_test_query73_00006", + "negative_test_query73_00008", + "negative_test_query216_00013", + "apositive_test_query73_00000", + "negative_test_query73_00013", + "apositive_test_query333_00000", + "apositive_test_query322_00008", + "negative_test_query233_00015", + "apositive_test_query333_00005", + "negative_test_query228_00013", + "negative_test_query73_00010", + "apositive_test_query333_00002", + "negative_test_query246_00000", + "negative_test_query333_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.75369761125927, + "recall@5": 0.75 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query74_00000", + "negative_test_query74_00000", + "apositive_test_query74_00001", + "apositive_test_query74_00003", + "negative_test_query74_00001", + "negative_test_query74_00014", + "negative_test_query74_00015", + "negative_test_query74_00013", + "apositive_test_query74_00002", + "negative_test_query44_00006", + "apositive_test_query114_00001", + "negative_test_query95_00009", + "negative_test_query85_00005", + "apositive_test_query113_00001", + "apositive_test_query60_00006", + "negative_test_query129_00008", + "apositive_test_query238_00003", + "negative_test_query238_00013", + "negative_test_query74_00003", + "negative_test_query203_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query75", + "ranked_ids": [ + "apositive_test_query75_00005", + "negative_test_query130_00012", + "apositive_test_query75_00001", + "apositive_test_query75_00000", + "negative_test_query89_00000", + "negative_test_query319_00004", + "negative_test_query75_00006", + "negative_test_query75_00004", + "apositive_test_query75_00003", + "negative_test_query75_00003", + "negative_test_query82_00004", + "negative_test_query75_00012", + "negative_test_query112_00018", + "negative_test_query209_00004", + "negative_test_query323_00011", + "apositive_test_query75_00006", + "negative_test_query99_00006", + "negative_test_query67_00017", + "negative_test_query260_00013", + "negative_test_query6_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query76", + "ranked_ids": [ + "apositive_test_query76_00006", + "negative_test_query76_00010", + "apositive_test_query76_00004", + "negative_test_query76_00007", + "negative_test_query91_00004", + "apositive_test_query76_00002", + "apositive_test_query76_00001", + "negative_test_query147_00007", + "apositive_test_query76_00000", + "negative_test_query181_00008", + "negative_test_query53_00009", + "negative_test_query76_00009", + "negative_test_query76_00002", + "negative_test_query53_00001", + "negative_test_query147_00005", + "negative_test_query76_00008", + "negative_test_query279_00009", + "negative_test_query76_00011", + "negative_test_query309_00011", + "negative_test_query258_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query216_00006", + "negative_test_query98_00014", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query148_00013", + "negative_test_query342_00007", + "negative_test_query77_00013", + "negative_test_query329_00015", + "negative_test_query144_00003", + "negative_test_query263_00002", + "negative_test_query150_00011", + "negative_test_query14_00010", + "negative_test_query215_00004", + "negative_test_query301_00003", + "negative_test_query287_00012", + "negative_test_query216_00012", + "negative_test_query87_00016", + "negative_test_query315_00003", + "negative_test_query326_00009", + "negative_test_query47_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query78", + "ranked_ids": [ + "negative_test_query214_00010", + "apositive_test_query78_00001", + "negative_test_query290_00006", + "negative_test_query290_00013", + "negative_test_query78_00016", + "negative_test_query78_00007", + "apositive_test_query38_00003", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query72_00005", + "apositive_test_query78_00000", + "apositive_test_query326_00000", + "negative_test_query200_00009", + "negative_test_query7_00005", + "negative_test_query144_00000", + "negative_test_query236_00018", + "negative_test_query274_00014", + "negative_test_query329_00002", + "negative_test_query272_00001", + "negative_test_query192_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "negative_test_query79_00013", + "negative_test_query79_00002", + "apositive_test_query221_00004", + "apositive_test_query79_00004", + "negative_test_query158_00000", + "apositive_test_query79_00000", + "apositive_test_query306_00010", + "negative_test_query303_00016", + "apositive_test_query119_00005", + "negative_test_query79_00004", + "negative_test_query56_00002", + "apositive_test_query295_00001", + "negative_test_query38_00007", + "negative_test_query53_00015", + "apositive_test_query113_00002", + "apositive_test_query157_00008", + "negative_test_query157_00000", + "negative_test_query313_00013", + "negative_test_query79_00009", + "negative_test_query105_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query8_00001", + "negative_test_query8_00004", + "negative_test_query8_00014", + "negative_test_query8_00011", + "negative_test_query205_00007", + "apositive_test_query8_00000", + "negative_test_query192_00006", + "negative_test_query8_00015", + "apositive_test_query192_00003", + "negative_test_query192_00003", + "negative_test_query8_00012", + "apositive_test_query243_00005", + "negative_test_query8_00008", + "apositive_test_query192_00002", + "apositive_test_query243_00010", + "negative_test_query329_00005", + "negative_test_query8_00000", + "negative_test_query96_00010" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query80_00004", + "negative_test_query179_00005", + "apositive_test_query80_00002", + "apositive_test_query179_00001", + "negative_test_query80_00003", + "negative_test_query80_00001", + "apositive_test_query189_00004", + "negative_test_query80_00012", + "negative_test_query36_00010", + "negative_test_query80_00009", + "apositive_test_query179_00003", + "negative_test_query102_00009", + "negative_test_query273_00005", + "apositive_test_query136_00013", + "negative_test_query287_00005", + "negative_test_query85_00007", + "negative_test_query289_00005", + "apositive_test_query36_00000", + "negative_test_query270_00003", + "negative_test_query179_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query81", + "ranked_ids": [ + "apositive_test_query81_00000", + "apositive_test_query81_00004", + "negative_test_query81_00006", + "negative_test_query81_00005", + "negative_test_query81_00000", + "apositive_test_query81_00001", + "apositive_test_query81_00005", + "apositive_test_query81_00006", + "negative_test_query81_00010", + "negative_test_query81_00009", + "apositive_test_query81_00008", + "negative_test_query81_00008", + "negative_test_query81_00007", + "negative_test_query81_00004", + "negative_test_query81_00001", + "apositive_test_query81_00003", + "negative_test_query207_00008", + "negative_test_query246_00017", + "negative_test_query27_00006", + "negative_test_query145_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query319_00003", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query82_00009", + "negative_test_query319_00009", + "negative_test_query319_00008", + "negative_test_query340_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query83_00002", + "apositive_test_query83_00004", + "apositive_test_query229_00001", + "apositive_test_query168_00001", + "apositive_test_query83_00003", + "negative_test_query168_00007", + "apositive_test_query95_00000", + "negative_test_query348_00006", + "apositive_test_query168_00000", + "apositive_test_query146_00014", + "apositive_test_query164_00000", + "apositive_test_query83_00000", + "negative_test_query28_00005", + "negative_test_query309_00016", + "negative_test_query279_00012", + "negative_test_query48_00006", + "negative_test_query0_00006", + "apositive_test_query217_00006", + "negative_test_query357_00002", + "negative_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query84", + "ranked_ids": [ + "apositive_test_query84_00001", + "negative_test_query84_00006", + "negative_test_query84_00012", + "apositive_test_query84_00000", + "negative_test_query84_00017", + "negative_test_query84_00002", + "negative_test_query84_00007", + "negative_test_query84_00000", + "negative_test_query84_00008", + "negative_test_query84_00005", + "negative_test_query43_00002", + "apositive_test_query249_00013", + "negative_test_query84_00001", + "negative_test_query84_00016", + "apositive_test_query155_00002", + "negative_test_query84_00009", + "negative_test_query84_00013", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query268_00009" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query85_00000", + "negative_test_query85_00012", + "apositive_test_query85_00002", + "negative_test_query85_00000", + "apositive_test_query109_00004", + "negative_test_query85_00003", + "apositive_test_query85_00001", + "apositive_test_query85_00003", + "negative_test_query85_00009", + "negative_test_query85_00011", + "negative_test_query112_00008", + "negative_test_query106_00003", + "negative_test_query85_00001", + "apositive_test_query284_00004", + "apositive_test_query286_00004", + "negative_test_query44_00008", + "apositive_test_query286_00006", + "apositive_test_query81_00008", + "apositive_test_query133_00013", + "apositive_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query86", + "ranked_ids": [ + "negative_test_query86_00001", + "apositive_test_query86_00000", + "negative_test_query86_00013", + "apositive_test_query86_00003", + "negative_test_query310_00007", + "apositive_test_query86_00001", + "apositive_test_query86_00002", + "negative_test_query65_00013", + "negative_test_query86_00012", + "negative_test_query299_00007", + "negative_test_query86_00014", + "negative_test_query86_00000", + "negative_test_query86_00008", + "negative_test_query86_00007", + "negative_test_query86_00002", + "negative_test_query86_00011", + "negative_test_query210_00012", + "negative_test_query86_00003", + "negative_test_query245_00012", + "negative_test_query188_00013" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query87", + "ranked_ids": [ + "negative_test_query87_00013", + "apositive_test_query87_00000", + "apositive_test_query87_00001", + "negative_test_query87_00004", + "negative_test_query87_00000", + "negative_test_query87_00007", + "negative_test_query87_00008", + "negative_test_query87_00014", + "negative_test_query87_00015", + "negative_test_query87_00011", + "negative_test_query87_00010", + "negative_test_query87_00006", + "negative_test_query87_00016", + "negative_test_query275_00006", + "negative_test_query263_00002", + "negative_test_query182_00017", + "negative_test_query87_00002", + "negative_test_query315_00006", + "apositive_test_query302_00001", + "negative_test_query200_00011" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "negative_test_query88_00012", + "negative_test_query88_00013", + "negative_test_query88_00005", + "apositive_test_query88_00001", + "negative_test_query88_00016", + "negative_test_query325_00006", + "apositive_test_query147_00002", + "apositive_test_query250_00002", + "negative_test_query88_00002", + "negative_test_query88_00003", + "negative_test_query88_00017", + "negative_test_query105_00014", + "negative_test_query88_00010", + "negative_test_query340_00012", + "negative_test_query88_00008", + "negative_test_query184_00013", + "apositive_test_query223_00000", + "negative_test_query334_00006", + "negative_test_query240_00009", + "negative_test_query203_00000" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query110_00011", + "apositive_test_query89_00004", + "negative_test_query89_00004", + "negative_test_query326_00006", + "negative_test_query156_00013", + "negative_test_query89_00008", + "negative_test_query89_00000", + "negative_test_query212_00000", + "negative_test_query68_00004", + "negative_test_query130_00003", + "apositive_test_query255_00000", + "negative_test_query89_00005", + "negative_test_query278_00009", + "negative_test_query89_00007", + "negative_test_query278_00000", + "apositive_test_query75_00006", + "apositive_test_query68_00001", + "negative_test_query325_00012", + "apositive_test_query89_00000", + "negative_test_query233_00005" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00008", + "negative_test_query9_00006", + "negative_test_query9_00001", + "negative_test_query90_00004", + "apositive_test_query9_00000", + "negative_test_query221_00002", + "negative_test_query9_00018", + "negative_test_query9_00004", + "negative_test_query90_00009", + "negative_test_query9_00016", + "negative_test_query9_00005", + "negative_test_query305_00009", + "negative_test_query9_00011", + "negative_test_query9_00015", + "negative_test_query9_00013", + "apositive_test_query90_00001", + "negative_test_query221_00011", + "negative_test_query90_00007", + "negative_test_query221_00007", + "negative_test_query9_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query231_00002", + "apositive_test_query90_00000", + "negative_test_query90_00010", + "negative_test_query310_00007", + "negative_test_query231_00014", + "negative_test_query90_00000", + "negative_test_query90_00011", + "negative_test_query90_00017", + "negative_test_query90_00002", + "negative_test_query227_00012", + "negative_test_query90_00012", + "negative_test_query127_00001", + "negative_test_query227_00001", + "negative_test_query261_00002", + "negative_test_query90_00015", + "apositive_test_query304_00000", + "negative_test_query90_00006", + "apositive_test_query14_00002", + "negative_test_query227_00006", + "negative_test_query9_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query91", + "ranked_ids": [ + "negative_test_query292_00009", + "apositive_test_query91_00000", + "apositive_test_query91_00002", + "negative_test_query91_00010", + "negative_test_query279_00011", + "negative_test_query319_00002", + "negative_test_query358_00004", + "apositive_test_query342_00001", + "negative_test_query82_00002", + "negative_test_query315_00013", + "apositive_test_query91_00006", + "negative_test_query350_00004", + "apositive_test_query11_00006", + "negative_test_query130_00010", + "apositive_test_query157_00001", + "negative_test_query118_00015", + "apositive_test_query157_00000", + "negative_test_query102_00002", + "negative_test_query303_00011", + "negative_test_query59_00006" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.5 + }, + "query_id": "test_query92", + "ranked_ids": [ + "negative_test_query92_00003", + "apositive_test_query92_00003", + "apositive_test_query92_00004", + "apositive_test_query198_00001", + "apositive_test_query92_00005", + "negative_test_query283_00001", + "apositive_test_query92_00000", + "apositive_test_query92_00001", + "negative_test_query105_00016", + "apositive_test_query92_00002", + "apositive_test_query100_00001", + "negative_test_query122_00018", + "negative_test_query283_00009", + "negative_test_query1_00012", + "apositive_test_query99_00003", + "negative_test_query92_00013", + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query100_00012", + "negative_test_query100_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query93_00004", + "apositive_test_query93_00000", + "apositive_test_query93_00003", + "negative_test_query93_00003", + "apositive_test_query243_00006", + "negative_test_query219_00000", + "negative_test_query195_00006", + "negative_test_query335_00003", + "negative_test_query280_00012", + "negative_test_query92_00012", + "negative_test_query108_00003", + "negative_test_query195_00001", + "negative_test_query277_00003", + "negative_test_query321_00001", + "negative_test_query335_00011", + "negative_test_query5_00015", + "apositive_test_query323_00003", + "apositive_test_query335_00003", + "apositive_test_query160_00007", + "negative_test_query129_00003" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.25 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00001", + "apositive_test_query94_00001", + "apositive_test_query127_00006", + "apositive_test_query127_00011", + "apositive_test_query94_00002", + "apositive_test_query94_00005", + "negative_test_query94_00003", + "apositive_test_query94_00004", + "negative_test_query94_00007", + "negative_test_query94_00009", + "apositive_test_query94_00000", + "negative_test_query94_00002", + "apositive_test_query127_00010", + "apositive_test_query94_00006", + "negative_test_query94_00005", + "apositive_test_query127_00004", + "apositive_test_query94_00007", + "negative_test_query155_00008", + "apositive_test_query127_00008", + "apositive_test_query127_00015" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "apositive_test_query95_00001", + "apositive_test_query95_00000", + "negative_test_query95_00015", + "negative_test_query95_00005", + "negative_test_query95_00010", + "apositive_test_query253_00007", + "negative_test_query95_00013", + "negative_test_query95_00002", + "negative_test_query12_00015", + "negative_test_query95_00012", + "negative_test_query95_00003", + "negative_test_query95_00008", + "negative_test_query12_00017", + "negative_test_query95_00014", + "negative_test_query95_00006", + "apositive_test_query177_00000", + "negative_test_query244_00010", + "negative_test_query334_00012", + "apositive_test_query292_00001", + "negative_test_query57_00007" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.41618115554873086, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query339_00000", + "negative_test_query192_00011", + "apositive_test_query96_00001", + "negative_test_query339_00001", + "apositive_test_query96_00002", + "negative_test_query339_00007", + "negative_test_query339_00005", + "negative_test_query339_00016", + "negative_test_query339_00017", + "negative_test_query339_00008", + "negative_test_query96_00004", + "negative_test_query96_00016", + "negative_test_query96_00013", + "negative_test_query96_00003", + "apositive_test_query339_00001", + "negative_test_query339_00009", + "negative_test_query339_00006", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query339_00004" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.5 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "apositive_test_query97_00003", + "apositive_test_query97_00002", + "negative_test_query97_00012", + "apositive_test_query97_00005", + "apositive_test_query97_00001", + "negative_test_query97_00009", + "negative_test_query113_00006", + "negative_test_query97_00006", + "negative_test_query84_00008", + "apositive_test_query51_00000", + "negative_test_query16_00014", + "negative_test_query97_00011", + "apositive_test_query289_00003", + "negative_test_query113_00009", + "negative_test_query209_00008", + "apositive_test_query51_00002", + "apositive_test_query267_00000", + "apositive_test_query201_00006", + "negative_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00007", + "negative_test_query98_00011", + "negative_test_query98_00004", + "negative_test_query98_00016", + "negative_test_query98_00018", + "apositive_test_query342_00000", + "negative_test_query231_00014", + "negative_test_query98_00017", + "negative_test_query98_00005", + "apositive_test_query14_00003", + "negative_test_query197_00007", + "apositive_test_query14_00002", + "negative_test_query23_00013", + "negative_test_query129_00003", + "negative_test_query212_00006", + "negative_test_query39_00005", + "apositive_test_query201_00004", + "negative_test_query300_00000", + "negative_test_query345_00002", + "negative_test_query316_00014" + ], + "valid": true + }, + { + "algorithm": "rankgpt_sw_w5", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query99", + "ranked_ids": [ + "apositive_test_query193_00008", + "apositive_test_query193_00010", + "apositive_test_query193_00012", + "negative_test_query193_00002", + "negative_test_query296_00013", + "negative_test_query330_00008", + "negative_test_query174_00002", + "negative_test_query193_00001", + "negative_test_query129_00001", + "apositive_test_query253_00003", + "negative_test_query27_00018", + "negative_test_query193_00005", + "apositive_test_query99_00001", + "apositive_test_query153_00012", + "apositive_test_query333_00003", + "apositive_test_query99_00007", + "apositive_test_query347_00002", + "negative_test_query256_00002", + "negative_test_query129_00003", + "negative_test_query27_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00006", + "apositive_test_query0_00003", + "negative_test_query359_00002", + "apositive_test_query0_00004", + "negative_test_query9_00017", + "apositive_test_query107_00004", + "apositive_test_query0_00002", + "negative_test_query0_00000", + "negative_test_query36_00011", + "negative_test_query221_00006", + "apositive_test_query130_00000", + "apositive_test_query153_00004", + "negative_test_query0_00007", + "negative_test_query130_00002", + "negative_test_query201_00000", + "negative_test_query259_00006", + "negative_test_query357_00000", + "negative_test_query0_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query1", + "ranked_ids": [ + "apositive_test_query1_00000", + "negative_test_query1_00005", + "apositive_test_query348_00003", + "negative_test_query61_00013", + "negative_test_query1_00009", + "negative_test_query1_00001", + "negative_test_query61_00002", + "negative_test_query157_00007", + "negative_test_query61_00014", + "negative_test_query61_00003", + "negative_test_query120_00006", + "negative_test_query297_00011", + "negative_test_query256_00005", + "negative_test_query27_00017", + "apositive_test_query188_00002", + "negative_test_query245_00006", + "negative_test_query184_00015", + "negative_test_query202_00001", + "negative_test_query110_00007", + "negative_test_query331_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "error": "ranking must be a permutation of 1..20.", + "error_type": "RerankParseError", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query10", + "ranked_ids": [], + "valid": false + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.4 + }, + "query_id": "test_query100", + "ranked_ids": [ + "apositive_test_query100_00000", + "negative_test_query100_00013", + "negative_test_query100_00001", + "apositive_test_query100_00004", + "negative_test_query100_00014", + "apositive_test_query100_00002", + "negative_test_query271_00006", + "negative_test_query15_00000", + "negative_test_query290_00015", + "apositive_test_query100_00003", + "negative_test_query100_00009", + "apositive_test_query338_00002", + "negative_test_query100_00010", + "negative_test_query252_00005", + "negative_test_query323_00000", + "apositive_test_query338_00005", + "apositive_test_query181_00000", + "negative_test_query258_00017", + "negative_test_query147_00000", + "negative_test_query237_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query101_00005", + "apositive_test_query101_00002", + "apositive_test_query73_00002", + "negative_test_query241_00005", + "negative_test_query61_00013", + "negative_test_query61_00014", + "negative_test_query61_00002", + "negative_test_query35_00004", + "negative_test_query191_00014", + "apositive_test_query196_00000", + "negative_test_query189_00014", + "negative_test_query61_00000", + "negative_test_query173_00013", + "negative_test_query103_00009", + "apositive_test_query291_00000", + "negative_test_query258_00000", + "negative_test_query134_00008", + "negative_test_query53_00000", + "apositive_test_query336_00001", + "negative_test_query68_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query102_00014", + "negative_test_query108_00009", + "negative_test_query131_00009", + "apositive_test_query238_00000", + "apositive_test_query20_00001", + "negative_test_query313_00016", + "negative_test_query183_00012", + "negative_test_query183_00009", + "negative_test_query203_00007", + "apositive_test_query30_00005", + "negative_test_query209_00006", + "negative_test_query7_00001", + "apositive_test_query356_00008", + "apositive_test_query47_00001", + "negative_test_query47_00011", + "negative_test_query47_00010", + "negative_test_query205_00016", + "negative_test_query232_00010", + "negative_test_query219_00004", + "negative_test_query90_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "negative_test_query103_00003", + "negative_test_query185_00005", + "apositive_test_query185_00003", + "apositive_test_query185_00002", + "negative_test_query103_00011", + "negative_test_query48_00017", + "negative_test_query185_00003", + "negative_test_query307_00008", + "apositive_test_query103_00001", + "negative_test_query307_00004", + "apositive_test_query103_00000", + "negative_test_query217_00005", + "negative_test_query48_00003", + "apositive_test_query103_00005", + "negative_test_query72_00012", + "negative_test_query222_00009", + "negative_test_query103_00006", + "negative_test_query48_00014", + "negative_test_query103_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "negative_test_query104_00002", + "negative_test_query104_00018", + "negative_test_query104_00003", + "negative_test_query104_00015", + "apositive_test_query104_00000", + "negative_test_query104_00013", + "negative_test_query104_00011", + "negative_test_query104_00005", + "negative_test_query104_00010", + "negative_test_query104_00012", + "negative_test_query104_00014", + "negative_test_query104_00001", + "negative_test_query104_00009", + "negative_test_query104_00007", + "negative_test_query48_00010", + "apositive_test_query28_00011", + "negative_test_query343_00013", + "apositive_test_query48_00000", + "negative_test_query104_00016", + "negative_test_query104_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query105", + "ranked_ids": [ + "apositive_test_query105_00000", + "negative_test_query105_00011", + "negative_test_query79_00002", + "negative_test_query97_00007", + "negative_test_query105_00006", + "negative_test_query79_00004", + "negative_test_query119_00005", + "apositive_test_query157_00008", + "negative_test_query295_00005", + "negative_test_query313_00013", + "negative_test_query105_00005", + "negative_test_query359_00008", + "apositive_test_query83_00004", + "negative_test_query142_00010", + "negative_test_query267_00011", + "apositive_test_query158_00002", + "negative_test_query168_00005", + "negative_test_query56_00002", + "negative_test_query131_00011", + "negative_test_query223_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query106", + "ranked_ids": [ + "apositive_test_query106_00003", + "negative_test_query36_00001", + "negative_test_query286_00000", + "apositive_test_query106_00005", + "apositive_test_query286_00005", + "negative_test_query326_00008", + "apositive_test_query106_00000", + "apositive_test_query177_00008", + "negative_test_query106_00012", + "negative_test_query106_00000", + "negative_test_query147_00001", + "apositive_test_query106_00001", + "negative_test_query162_00007", + "negative_test_query212_00002", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query106_00005", + "negative_test_query106_00006", + "apositive_test_query42_00006", + "negative_test_query114_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "negative_test_query46_00012", + "apositive_test_query357_00003", + "apositive_test_query221_00001", + "negative_test_query142_00000", + "negative_test_query160_00001", + "negative_test_query105_00015", + "apositive_test_query36_00002", + "negative_test_query31_00013", + "negative_test_query142_00004", + "negative_test_query105_00008", + "apositive_test_query142_00002", + "negative_test_query305_00015", + "apositive_test_query211_00001", + "apositive_test_query142_00004", + "negative_test_query146_00000", + "apositive_test_query276_00003", + "negative_test_query296_00001", + "negative_test_query296_00008", + "negative_test_query124_00005", + "negative_test_query333_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5 + }, + "query_id": "test_query108", + "ranked_ids": [ + "apositive_test_query108_00000", + "apositive_test_query108_00002", + "apositive_test_query108_00003", + "negative_test_query311_00006", + "apositive_test_query108_00007", + "negative_test_query108_00011", + "negative_test_query108_00008", + "apositive_test_query108_00005", + "negative_test_query168_00001", + "negative_test_query57_00009", + "negative_test_query102_00012", + "negative_test_query108_00010", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query108_00006", + "negative_test_query102_00007", + "negative_test_query250_00009", + "negative_test_query108_00007", + "negative_test_query250_00011", + "negative_test_query108_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.6 + }, + "query_id": "test_query109", + "ranked_ids": [ + "apositive_test_query109_00000", + "negative_test_query190_00003", + "apositive_test_query109_00001", + "apositive_test_query109_00003", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query109_00003", + "negative_test_query6_00006", + "negative_test_query109_00013", + "negative_test_query109_00011", + "negative_test_query109_00014", + "negative_test_query51_00004", + "negative_test_query109_00012", + "negative_test_query109_00000", + "negative_test_query268_00010", + "negative_test_query38_00006", + "apositive_test_query248_00006", + "negative_test_query109_00004", + "negative_test_query109_00009", + "negative_test_query109_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.25 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00001", + "apositive_test_query11_00000", + "negative_test_query309_00002", + "negative_test_query132_00003", + "negative_test_query358_00005", + "negative_test_query11_00002", + "negative_test_query226_00010", + "apositive_test_query11_00004", + "apositive_test_query11_00006", + "negative_test_query11_00004", + "negative_test_query309_00001", + "apositive_test_query358_00006", + "apositive_test_query11_00007", + "negative_test_query309_00004", + "apositive_test_query301_00003", + "negative_test_query123_00007", + "negative_test_query358_00001", + "apositive_test_query11_00003", + "negative_test_query132_00013", + "apositive_test_query358_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "negative_test_query89_00006", + "negative_test_query110_00013", + "negative_test_query110_00005", + "negative_test_query110_00002", + "negative_test_query110_00014", + "negative_test_query343_00001", + "apositive_test_query344_00002", + "negative_test_query110_00006", + "negative_test_query17_00008", + "apositive_test_query138_00000", + "apositive_test_query180_00001", + "negative_test_query110_00007", + "negative_test_query10_00005", + "apositive_test_query330_00001", + "apositive_test_query11_00002", + "negative_test_query110_00004", + "negative_test_query311_00011", + "apositive_test_query291_00009", + "negative_test_query260_00015", + "apositive_test_query242_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query111_00015", + "negative_test_query223_00000", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query134_00006", + "negative_test_query111_00018", + "negative_test_query213_00011", + "negative_test_query137_00010", + "negative_test_query65_00013", + "apositive_test_query162_00005", + "apositive_test_query306_00002", + "negative_test_query111_00008", + "negative_test_query58_00013", + "negative_test_query65_00007", + "apositive_test_query174_00008", + "negative_test_query292_00010", + "negative_test_query243_00001", + "negative_test_query14_00015", + "negative_test_query245_00001", + "apositive_test_query308_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "negative_test_query36_00014", + "negative_test_query112_00010", + "apositive_test_query286_00005", + "apositive_test_query106_00003", + "negative_test_query112_00016", + "negative_test_query246_00016", + "apositive_test_query36_00003", + "negative_test_query216_00002", + "apositive_test_query286_00006", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "negative_test_query67_00003", + "negative_test_query112_00004", + "negative_test_query112_00015", + "negative_test_query112_00013", + "negative_test_query36_00010", + "negative_test_query80_00007", + "negative_test_query85_00002", + "negative_test_query6_00003", + "apositive_test_query246_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.5 + }, + "query_id": "test_query113", + "ranked_ids": [ + "apositive_test_query113_00001", + "apositive_test_query113_00000", + "negative_test_query67_00014", + "negative_test_query7_00016", + "apositive_test_query113_00002", + "negative_test_query201_00008", + "negative_test_query113_00003", + "negative_test_query38_00003", + "negative_test_query6_00016", + "negative_test_query223_00003", + "apositive_test_query186_00005", + "apositive_test_query67_00000", + "negative_test_query67_00002", + "apositive_test_query79_00001", + "negative_test_query216_00011", + "negative_test_query105_00007", + "negative_test_query188_00005", + "negative_test_query47_00005", + "negative_test_query105_00000", + "apositive_test_query188_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query114", + "ranked_ids": [ + "apositive_test_query114_00000", + "apositive_test_query114_00003", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "apositive_test_query114_00001", + "negative_test_query106_00009", + "negative_test_query262_00007", + "negative_test_query349_00003", + "apositive_test_query124_00004", + "apositive_test_query106_00003", + "negative_test_query114_00002", + "apositive_test_query114_00006", + "negative_test_query114_00003", + "negative_test_query81_00009", + "negative_test_query114_00007", + "negative_test_query114_00008", + "negative_test_query114_00004", + "negative_test_query114_00010", + "negative_test_query114_00005", + "negative_test_query114_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query115", + "ranked_ids": [ + "apositive_test_query115_00002", + "apositive_test_query115_00004", + "apositive_test_query115_00001", + "negative_test_query115_00003", + "apositive_test_query115_00000", + "apositive_test_query115_00003", + "negative_test_query115_00006", + "negative_test_query115_00002", + "negative_test_query115_00007", + "apositive_test_query348_00000", + "negative_test_query115_00012", + "apositive_test_query348_00009", + "negative_test_query276_00006", + "negative_test_query139_00000", + "negative_test_query115_00004", + "negative_test_query205_00004", + "negative_test_query115_00010", + "apositive_test_query133_00010", + "negative_test_query121_00008", + "apositive_test_query121_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "apositive_test_query116_00000", + "apositive_test_query49_00000", + "negative_test_query116_00015", + "negative_test_query59_00013", + "negative_test_query116_00013", + "negative_test_query116_00003", + "negative_test_query98_00005", + "negative_test_query116_00002", + "negative_test_query116_00012", + "negative_test_query116_00008", + "negative_test_query116_00000", + "negative_test_query116_00011", + "negative_test_query59_00007", + "negative_test_query116_00004", + "negative_test_query116_00007", + "negative_test_query116_00009", + "negative_test_query261_00013", + "negative_test_query116_00010", + "negative_test_query116_00014", + "negative_test_query116_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.25 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00003", + "negative_test_query117_00010", + "negative_test_query117_00009", + "apositive_test_query354_00003", + "apositive_test_query117_00006", + "apositive_test_query117_00004", + "negative_test_query48_00000", + "negative_test_query117_00007", + "negative_test_query63_00001", + "negative_test_query117_00001", + "apositive_test_query134_00004", + "apositive_test_query117_00007", + "negative_test_query194_00002", + "apositive_test_query117_00000", + "apositive_test_query354_00000", + "negative_test_query117_00004", + "negative_test_query202_00006", + "negative_test_query117_00003", + "negative_test_query175_00013", + "negative_test_query117_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "apositive_test_query155_00005", + "negative_test_query340_00007", + "negative_test_query23_00005", + "negative_test_query296_00007", + "apositive_test_query174_00010", + "apositive_test_query174_00003", + "apositive_test_query174_00004", + "apositive_test_query174_00005", + "apositive_test_query174_00000", + "apositive_test_query340_00002", + "apositive_test_query174_00001", + "negative_test_query88_00007", + "apositive_test_query174_00006", + "negative_test_query171_00005", + "negative_test_query171_00016", + "apositive_test_query171_00001", + "apositive_test_query310_00001", + "negative_test_query296_00010", + "negative_test_query84_00012", + "apositive_test_query217_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query119", + "ranked_ids": [ + "apositive_test_query221_00004", + "apositive_test_query119_00005", + "negative_test_query119_00005", + "negative_test_query158_00001", + "apositive_test_query119_00001", + "negative_test_query157_00001", + "apositive_test_query119_00012", + "negative_test_query152_00004", + "negative_test_query56_00002", + "negative_test_query119_00002", + "negative_test_query223_00011", + "apositive_test_query26_00000", + "apositive_test_query306_00010", + "apositive_test_query105_00000", + "negative_test_query158_00000", + "apositive_test_query79_00004", + "negative_test_query79_00002", + "negative_test_query157_00007", + "apositive_test_query113_00002", + "negative_test_query359_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query12_00016", + "negative_test_query12_00014", + "negative_test_query12_00006", + "negative_test_query12_00004", + "apositive_test_query12_00000", + "negative_test_query12_00013", + "negative_test_query12_00017", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query12_00007", + "negative_test_query12_00003", + "negative_test_query12_00008", + "negative_test_query12_00001", + "negative_test_query12_00015", + "negative_test_query12_00011", + "negative_test_query231_00013", + "negative_test_query234_00014", + "apositive_test_query185_00000", + "apositive_test_query217_00005", + "negative_test_query175_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00008", + "negative_test_query120_00002", + "negative_test_query1_00009", + "negative_test_query121_00002", + "negative_test_query120_00012", + "apositive_test_query120_00002", + "negative_test_query120_00010", + "negative_test_query352_00013", + "apositive_test_query120_00003", + "apositive_test_query32_00007", + "negative_test_query196_00006", + "negative_test_query148_00014", + "negative_test_query120_00006", + "negative_test_query120_00001", + "apositive_test_query352_00000", + "apositive_test_query120_00000", + "negative_test_query303_00018", + "negative_test_query121_00012", + "apositive_test_query120_00001", + "apositive_test_query117_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.8333333333333334 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00002", + "apositive_test_query121_00000", + "apositive_test_query121_00005", + "apositive_test_query121_00001", + "apositive_test_query121_00004", + "negative_test_query121_00008", + "negative_test_query121_00013", + "apositive_test_query121_00003", + "negative_test_query121_00000", + "negative_test_query121_00012", + "negative_test_query121_00003", + "apositive_test_query307_00010", + "negative_test_query44_00012", + "negative_test_query121_00007", + "negative_test_query313_00006", + "negative_test_query53_00003", + "negative_test_query256_00003", + "negative_test_query308_00009", + "negative_test_query115_00009", + "negative_test_query133_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query61_00015", + "apositive_test_query254_00002", + "negative_test_query17_00012", + "negative_test_query86_00006", + "negative_test_query195_00008", + "apositive_test_query173_00000", + "negative_test_query86_00001", + "apositive_test_query134_00000", + "negative_test_query64_00017", + "negative_test_query297_00000", + "apositive_test_query122_00000", + "negative_test_query173_00003", + "negative_test_query179_00002", + "negative_test_query5_00001", + "negative_test_query179_00014", + "negative_test_query358_00000", + "negative_test_query210_00003", + "apositive_test_query175_00004", + "negative_test_query134_00007", + "apositive_test_query173_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query123", + "ranked_ids": [ + "negative_test_query123_00013", + "apositive_test_query123_00002", + "negative_test_query123_00005", + "negative_test_query123_00008", + "negative_test_query123_00010", + "negative_test_query161_00009", + "apositive_test_query123_00003", + "negative_test_query123_00004", + "negative_test_query123_00001", + "apositive_test_query123_00004", + "apositive_test_query123_00000", + "negative_test_query155_00001", + "negative_test_query123_00000", + "negative_test_query123_00002", + "negative_test_query155_00009", + "negative_test_query340_00012", + "negative_test_query309_00008", + "negative_test_query123_00006", + "negative_test_query123_00007", + "negative_test_query269_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "apositive_test_query13_00004", + "apositive_test_query194_00000", + "negative_test_query347_00010", + "apositive_test_query13_00000", + "negative_test_query124_00003", + "apositive_test_query162_00002", + "negative_test_query349_00017", + "negative_test_query42_00005", + "negative_test_query187_00003", + "negative_test_query42_00006", + "negative_test_query262_00011", + "apositive_test_query162_00001", + "negative_test_query187_00006", + "negative_test_query123_00009", + "negative_test_query171_00002", + "negative_test_query93_00010", + "negative_test_query144_00010", + "negative_test_query76_00004", + "negative_test_query309_00012", + "negative_test_query339_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.3191471544989227, + "recall@5": 0.5 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query125_00014", + "negative_test_query140_00017", + "negative_test_query360_00016", + "apositive_test_query125_00000", + "apositive_test_query125_00003", + "negative_test_query125_00013", + "negative_test_query125_00003", + "negative_test_query125_00004", + "negative_test_query125_00000", + "apositive_test_query273_00003", + "negative_test_query125_00011", + "negative_test_query3_00000", + "negative_test_query125_00006", + "negative_test_query12_00009", + "negative_test_query301_00012", + "apositive_test_query286_00009", + "negative_test_query125_00009", + "negative_test_query6_00003", + "apositive_test_query286_00006", + "negative_test_query260_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query126", + "ranked_ids": [ + "apositive_test_query126_00000", + "negative_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00017", + "negative_test_query190_00017", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query151_00006", + "negative_test_query190_00005", + "negative_test_query27_00013", + "negative_test_query311_00005", + "apositive_test_query330_00002", + "negative_test_query131_00009", + "apositive_test_query330_00001", + "apositive_test_query5_00002", + "negative_test_query220_00000", + "negative_test_query12_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00004", + "apositive_test_query127_00007", + "apositive_test_query127_00012", + "apositive_test_query127_00016", + "apositive_test_query127_00008", + "apositive_test_query127_00014", + "apositive_test_query127_00009", + "negative_test_query267_00009", + "apositive_test_query127_00000", + "apositive_test_query127_00011", + "apositive_test_query127_00003", + "negative_test_query94_00007", + "negative_test_query94_00003", + "apositive_test_query127_00006", + "negative_test_query94_00009", + "apositive_test_query127_00013", + "apositive_test_query94_00005", + "apositive_test_query163_00000", + "apositive_test_query311_00003", + "negative_test_query198_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.25 + }, + "query_id": "test_query128", + "ranked_ids": [ + "apositive_test_query128_00001", + "apositive_test_query128_00000", + "negative_test_query256_00011", + "negative_test_query21_00017", + "negative_test_query283_00006", + "negative_test_query310_00009", + "apositive_test_query216_00002", + "negative_test_query21_00000", + "apositive_test_query80_00005", + "apositive_test_query142_00000", + "negative_test_query8_00017", + "negative_test_query128_00010", + "negative_test_query128_00006", + "negative_test_query128_00011", + "apositive_test_query21_00000", + "negative_test_query75_00012", + "apositive_test_query8_00000", + "negative_test_query192_00006", + "apositive_test_query196_00004", + "apositive_test_query32_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query129", + "ranked_ids": [ + "apositive_test_query129_00000", + "negative_test_query211_00012", + "negative_test_query156_00017", + "negative_test_query63_00002", + "negative_test_query63_00006", + "apositive_test_query129_00001", + "negative_test_query117_00006", + "apositive_test_query244_00001", + "negative_test_query165_00012", + "apositive_test_query356_00006", + "negative_test_query129_00010", + "negative_test_query84_00008", + "negative_test_query245_00014", + "apositive_test_query308_00004", + "apositive_test_query150_00000", + "negative_test_query16_00007", + "negative_test_query187_00012", + "negative_test_query342_00013", + "apositive_test_query155_00002", + "apositive_test_query249_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query13", + "ranked_ids": [ + "apositive_test_query13_00002", + "negative_test_query13_00001", + "apositive_test_query13_00006", + "apositive_test_query13_00000", + "negative_test_query207_00013", + "apositive_test_query13_00001", + "apositive_test_query13_00004", + "apositive_test_query13_00008", + "apositive_test_query13_00010", + "apositive_test_query13_00003", + "negative_test_query13_00002", + "negative_test_query13_00004", + "apositive_test_query13_00012", + "negative_test_query13_00006", + "apositive_test_query13_00009", + "apositive_test_query13_00007", + "negative_test_query13_00005", + "negative_test_query245_00015", + "negative_test_query13_00000", + "negative_test_query13_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query130", + "ranked_ids": [ + "apositive_test_query130_00005", + "negative_test_query130_00001", + "negative_test_query130_00008", + "negative_test_query198_00011", + "negative_test_query345_00002", + "negative_test_query284_00001", + "negative_test_query198_00003", + "apositive_test_query90_00001", + "negative_test_query130_00006", + "apositive_test_query284_00003", + "apositive_test_query130_00002", + "apositive_test_query284_00001", + "negative_test_query284_00013", + "apositive_test_query323_00004", + "negative_test_query284_00009", + "apositive_test_query233_00000", + "apositive_test_query275_00000", + "negative_test_query90_00014", + "negative_test_query289_00004", + "apositive_test_query30_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query131", + "ranked_ids": [ + "apositive_test_query131_00000", + "apositive_test_query131_00001", + "negative_test_query131_00001", + "negative_test_query186_00012", + "negative_test_query313_00008", + "negative_test_query131_00002", + "negative_test_query247_00004", + "apositive_test_query106_00000", + "apositive_test_query50_00008", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query158_00001", + "apositive_test_query158_00002", + "negative_test_query168_00013", + "negative_test_query184_00012", + "negative_test_query345_00009", + "negative_test_query357_00001", + "negative_test_query284_00002", + "negative_test_query284_00006", + "apositive_test_query344_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "negative_test_query132_00006", + "negative_test_query123_00012", + "negative_test_query81_00005", + "negative_test_query132_00018", + "negative_test_query81_00001", + "negative_test_query132_00000", + "negative_test_query132_00010", + "negative_test_query132_00012", + "negative_test_query181_00008", + "negative_test_query132_00004", + "negative_test_query132_00016", + "negative_test_query11_00006", + "negative_test_query240_00011", + "negative_test_query246_00016", + "apositive_test_query161_00007", + "negative_test_query184_00015", + "apositive_test_query3_00004", + "negative_test_query148_00011", + "negative_test_query304_00000", + "apositive_test_query187_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.2 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query287_00001", + "apositive_test_query133_00000", + "apositive_test_query133_00001", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query133_00014", + "apositive_test_query129_00004", + "apositive_test_query133_00002", + "apositive_test_query129_00002", + "negative_test_query12_00014", + "apositive_test_query129_00007", + "apositive_test_query174_00011", + "apositive_test_query129_00006", + "apositive_test_query129_00005", + "negative_test_query260_00006", + "negative_test_query310_00002", + "negative_test_query206_00017", + "apositive_test_query91_00002", + "negative_test_query59_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query134", + "ranked_ids": [ + "negative_test_query134_00010", + "negative_test_query134_00006", + "apositive_test_query213_00001", + "negative_test_query359_00011", + "apositive_test_query276_00004", + "apositive_test_query134_00002", + "negative_test_query134_00000", + "negative_test_query107_00003", + "apositive_test_query259_00011", + "negative_test_query46_00004", + "negative_test_query218_00009", + "negative_test_query134_00008", + "apositive_test_query357_00000", + "negative_test_query46_00011", + "negative_test_query313_00012", + "apositive_test_query142_00001", + "negative_test_query108_00003", + "negative_test_query334_00013", + "apositive_test_query238_00001", + "negative_test_query238_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00003", + "negative_test_query135_00017", + "negative_test_query135_00011", + "apositive_test_query135_00001", + "apositive_test_query133_00011", + "negative_test_query135_00005", + "negative_test_query135_00007", + "apositive_test_query135_00000", + "negative_test_query287_00009", + "negative_test_query135_00004", + "negative_test_query135_00001", + "negative_test_query135_00015", + "negative_test_query135_00013", + "negative_test_query135_00014", + "negative_test_query325_00012", + "negative_test_query135_00008", + "negative_test_query135_00010", + "negative_test_query297_00017", + "apositive_test_query28_00008", + "negative_test_query135_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query136", + "ranked_ids": [ + "apositive_test_query136_00014", + "apositive_test_query136_00011", + "apositive_test_query136_00004", + "apositive_test_query136_00000", + "apositive_test_query136_00002", + "apositive_test_query136_00010", + "negative_test_query128_00011", + "apositive_test_query136_00007", + "apositive_test_query136_00001", + "negative_test_query301_00001", + "apositive_test_query136_00005", + "apositive_test_query136_00009", + "apositive_test_query136_00013", + "negative_test_query136_00003", + "negative_test_query129_00001", + "apositive_test_query60_00014", + "negative_test_query245_00003", + "negative_test_query149_00007", + "negative_test_query60_00002", + "negative_test_query8_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query137", + "ranked_ids": [ + "apositive_test_query137_00000", + "negative_test_query137_00014", + "apositive_test_query253_00000", + "negative_test_query137_00003", + "negative_test_query253_00007", + "negative_test_query95_00004", + "negative_test_query349_00006", + "negative_test_query74_00001", + "apositive_test_query137_00002", + "apositive_test_query137_00003", + "negative_test_query137_00012", + "apositive_test_query13_00001", + "negative_test_query262_00009", + "negative_test_query207_00013", + "negative_test_query13_00002", + "negative_test_query74_00003", + "negative_test_query137_00000", + "negative_test_query74_00012", + "negative_test_query207_00006", + "negative_test_query149_00013" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query138", + "ranked_ids": [ + "apositive_test_query138_00000", + "negative_test_query138_00008", + "negative_test_query138_00006", + "apositive_test_query138_00001", + "negative_test_query138_00003", + "negative_test_query138_00013", + "negative_test_query138_00001", + "negative_test_query183_00004", + "negative_test_query138_00014", + "apositive_test_query356_00003", + "negative_test_query240_00000", + "negative_test_query138_00015", + "negative_test_query138_00004", + "negative_test_query213_00006", + "negative_test_query184_00005", + "negative_test_query132_00009", + "negative_test_query138_00005", + "apositive_test_query23_00003", + "apositive_test_query244_00005", + "apositive_test_query138_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query35_00001", + "apositive_test_query139_00000", + "apositive_test_query35_00000", + "apositive_test_query139_00001", + "apositive_test_query146_00007", + "negative_test_query101_00006", + "negative_test_query35_00010", + "negative_test_query124_00007", + "negative_test_query320_00007", + "negative_test_query81_00002", + "apositive_test_query354_00001", + "negative_test_query262_00002", + "negative_test_query198_00011", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query8_00009", + "negative_test_query123_00006", + "negative_test_query269_00006", + "negative_test_query231_00012", + "negative_test_query350_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query14", + "ranked_ids": [ + "negative_test_query155_00009", + "apositive_test_query14_00003", + "apositive_test_query77_00000", + "apositive_test_query216_00002", + "negative_test_query260_00014", + "negative_test_query14_00001", + "negative_test_query179_00001", + "negative_test_query53_00003", + "negative_test_query155_00001", + "apositive_test_query296_00005", + "negative_test_query14_00006", + "apositive_test_query64_00000", + "negative_test_query14_00007", + "negative_test_query199_00009", + "negative_test_query350_00015", + "negative_test_query14_00005", + "negative_test_query14_00004", + "negative_test_query102_00011", + "negative_test_query260_00000", + "apositive_test_query26_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00005", + "negative_test_query140_00003", + "apositive_test_query281_00013", + "negative_test_query215_00000", + "negative_test_query140_00001", + "negative_test_query257_00003", + "negative_test_query140_00016", + "negative_test_query140_00014", + "negative_test_query140_00010", + "negative_test_query140_00013", + "negative_test_query140_00018", + "apositive_test_query273_00005", + "negative_test_query257_00012", + "apositive_test_query18_00003", + "negative_test_query18_00013", + "negative_test_query140_00009", + "apositive_test_query18_00001", + "negative_test_query140_00007", + "negative_test_query18_00006", + "negative_test_query140_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query141", + "ranked_ids": [ + "apositive_test_query288_00003", + "negative_test_query245_00008", + "apositive_test_query141_00006", + "apositive_test_query141_00005", + "negative_test_query245_00016", + "negative_test_query57_00002", + "apositive_test_query141_00000", + "apositive_test_query288_00007", + "apositive_test_query288_00005", + "negative_test_query344_00013", + "negative_test_query141_00010", + "negative_test_query220_00011", + "negative_test_query245_00013", + "negative_test_query176_00003", + "apositive_test_query246_00001", + "negative_test_query141_00006", + "negative_test_query25_00006", + "negative_test_query325_00008", + "negative_test_query308_00005", + "negative_test_query105_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query142", + "ranked_ids": [ + "negative_test_query238_00001", + "negative_test_query102_00001", + "apositive_test_query142_00001", + "negative_test_query105_00008", + "apositive_test_query142_00002", + "negative_test_query142_00007", + "negative_test_query203_00003", + "negative_test_query46_00001", + "negative_test_query46_00012", + "negative_test_query46_00002", + "negative_test_query223_00005", + "negative_test_query142_00003", + "negative_test_query142_00009", + "apositive_test_query142_00003", + "negative_test_query359_00005", + "negative_test_query39_00008", + "negative_test_query142_00004", + "negative_test_query105_00006", + "negative_test_query200_00014", + "negative_test_query337_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query143", + "ranked_ids": [ + "apositive_test_query143_00000", + "negative_test_query143_00006", + "negative_test_query143_00011", + "negative_test_query143_00009", + "negative_test_query143_00005", + "negative_test_query143_00015", + "apositive_test_query25_00006", + "negative_test_query90_00014", + "negative_test_query143_00000", + "negative_test_query143_00004", + "negative_test_query143_00012", + "negative_test_query45_00009", + "negative_test_query143_00010", + "negative_test_query143_00013", + "negative_test_query143_00007", + "negative_test_query143_00014", + "apositive_test_query308_00007", + "negative_test_query143_00016", + "apositive_test_query25_00005", + "negative_test_query56_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query144_00012", + "negative_test_query144_00010", + "negative_test_query41_00001", + "negative_test_query41_00009", + "negative_test_query331_00015", + "negative_test_query144_00018", + "negative_test_query144_00006", + "negative_test_query144_00003", + "negative_test_query144_00001", + "negative_test_query144_00011", + "negative_test_query144_00016", + "negative_test_query198_00016", + "negative_test_query154_00009", + "negative_test_query166_00001", + "apositive_test_query41_00000", + "negative_test_query315_00011", + "negative_test_query41_00011", + "negative_test_query226_00011", + "negative_test_query144_00009", + "negative_test_query144_00017" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query145", + "ranked_ids": [ + "negative_test_query230_00002", + "negative_test_query53_00012", + "apositive_test_query160_00010", + "apositive_test_query225_00002", + "apositive_test_query213_00004", + "apositive_test_query145_00002", + "negative_test_query79_00005", + "apositive_test_query305_00002", + "negative_test_query305_00002", + "negative_test_query112_00007", + "apositive_test_query107_00001", + "apositive_test_query207_00004", + "negative_test_query118_00000", + "negative_test_query171_00003", + "negative_test_query187_00014", + "negative_test_query190_00012", + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "negative_test_query211_00013", + "apositive_test_query307_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.125 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query153_00008", + "apositive_test_query146_00004", + "apositive_test_query146_00010", + "apositive_test_query3_00010", + "negative_test_query358_00008", + "apositive_test_query314_00004", + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query146_00003", + "negative_test_query65_00006", + "apositive_test_query146_00005", + "negative_test_query146_00001", + "negative_test_query99_00001", + "negative_test_query144_00008", + "apositive_test_query161_00005", + "apositive_test_query161_00008", + "apositive_test_query161_00007", + "apositive_test_query314_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.4 + }, + "query_id": "test_query147", + "ranked_ids": [ + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query147_00013", + "apositive_test_query147_00002", + "apositive_test_query161_00004", + "negative_test_query258_00008", + "negative_test_query53_00009", + "apositive_test_query76_00001", + "negative_test_query354_00007", + "apositive_test_query147_00003", + "negative_test_query181_00008", + "negative_test_query17_00009", + "apositive_test_query76_00004", + "negative_test_query309_00014", + "negative_test_query309_00016", + "negative_test_query147_00008", + "negative_test_query180_00012", + "negative_test_query147_00005", + "negative_test_query11_00009", + "negative_test_query200_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query148_00005", + "negative_test_query124_00010", + "negative_test_query148_00018", + "negative_test_query148_00017", + "negative_test_query210_00013", + "negative_test_query148_00012", + "negative_test_query210_00008", + "negative_test_query148_00014", + "negative_test_query80_00007", + "apositive_test_query304_00002", + "negative_test_query315_00002", + "negative_test_query321_00013", + "apositive_test_query304_00001", + "negative_test_query210_00006", + "negative_test_query196_00004", + "apositive_test_query234_00003", + "negative_test_query112_00013", + "negative_test_query89_00007", + "negative_test_query148_00008", + "negative_test_query210_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5706417189553201, + "recall@5": 1.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "negative_test_query149_00001", + "negative_test_query149_00012", + "apositive_test_query149_00001", + "apositive_test_query149_00000", + "negative_test_query63_00009", + "negative_test_query149_00002", + "negative_test_query149_00015", + "negative_test_query149_00008", + "negative_test_query332_00004", + "negative_test_query149_00011", + "negative_test_query149_00010", + "negative_test_query149_00017", + "apositive_test_query145_00001", + "apositive_test_query225_00004", + "negative_test_query225_00006", + "negative_test_query85_00009", + "negative_test_query332_00008", + "negative_test_query149_00013", + "apositive_test_query113_00003", + "negative_test_query74_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00010", + "negative_test_query15_00012", + "negative_test_query15_00005", + "negative_test_query15_00007", + "apositive_test_query325_00000", + "negative_test_query15_00004", + "negative_test_query325_00011", + "negative_test_query15_00011", + "negative_test_query15_00017", + "apositive_test_query15_00001", + "negative_test_query15_00013", + "negative_test_query15_00014", + "negative_test_query266_00008", + "negative_test_query266_00013", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query266_00006", + "negative_test_query176_00011", + "negative_test_query15_00015", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query150", + "ranked_ids": [ + "apositive_test_query150_00003", + "apositive_test_query20_00007", + "negative_test_query1_00012", + "negative_test_query150_00010", + "negative_test_query250_00013", + "negative_test_query306_00004", + "apositive_test_query89_00000", + "apositive_test_query20_00010", + "negative_test_query62_00009", + "negative_test_query254_00003", + "negative_test_query139_00005", + "negative_test_query267_00006", + "negative_test_query350_00018", + "apositive_test_query158_00001", + "negative_test_query301_00007", + "negative_test_query213_00007", + "negative_test_query313_00017", + "negative_test_query346_00008", + "apositive_test_query112_00000", + "negative_test_query63_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00000", + "negative_test_query126_00006", + "apositive_test_query151_00003", + "negative_test_query190_00003", + "apositive_test_query126_00000", + "negative_test_query151_00006", + "negative_test_query190_00005", + "negative_test_query151_00003", + "apositive_test_query300_00001", + "negative_test_query44_00002", + "negative_test_query151_00015", + "negative_test_query190_00015", + "negative_test_query293_00008", + "negative_test_query201_00008", + "negative_test_query151_00004", + "negative_test_query190_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query237_00011", + "negative_test_query237_00000", + "apositive_test_query152_00009", + "negative_test_query312_00014", + "apositive_test_query152_00004", + "negative_test_query157_00010", + "negative_test_query124_00010", + "apositive_test_query152_00001", + "negative_test_query237_00004", + "apositive_test_query152_00005", + "negative_test_query349_00001", + "negative_test_query250_00002", + "negative_test_query237_00009", + "negative_test_query182_00004", + "apositive_test_query194_00008", + "negative_test_query253_00004", + "negative_test_query95_00017", + "negative_test_query342_00005", + "negative_test_query349_00005", + "negative_test_query262_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.25 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00003", + "apositive_test_query153_00002", + "apositive_test_query153_00015", + "apositive_test_query153_00001", + "negative_test_query112_00007", + "negative_test_query250_00006", + "apositive_test_query153_00004", + "apositive_test_query153_00007", + "negative_test_query359_00003", + "apositive_test_query153_00000", + "apositive_test_query213_00004", + "negative_test_query283_00015", + "apositive_test_query176_00000", + "negative_test_query278_00006", + "negative_test_query112_00015", + "apositive_test_query20_00006", + "negative_test_query2_00001", + "negative_test_query322_00001", + "negative_test_query278_00007", + "negative_test_query228_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1 + }, + "query_id": "test_query154", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query20_00003", + "apositive_test_query356_00005", + "apositive_test_query154_00006", + "apositive_test_query23_00004", + "apositive_test_query33_00000", + "negative_test_query331_00012", + "negative_test_query133_00002", + "apositive_test_query358_00008", + "apositive_test_query154_00005", + "apositive_test_query20_00007", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "negative_test_query334_00000", + "apositive_test_query20_00010", + "negative_test_query311_00005", + "negative_test_query41_00007", + "negative_test_query245_00006", + "negative_test_query68_00015", + "negative_test_query102_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00009", + "negative_test_query155_00006", + "negative_test_query155_00003", + "negative_test_query155_00005", + "apositive_test_query155_00005", + "apositive_test_query3_00008", + "negative_test_query123_00014", + "apositive_test_query155_00001", + "apositive_test_query355_00007", + "negative_test_query155_00004", + "negative_test_query155_00012", + "negative_test_query155_00001", + "negative_test_query155_00002", + "negative_test_query99_00001", + "negative_test_query155_00010", + "negative_test_query119_00001", + "apositive_test_query40_00004", + "negative_test_query155_00000", + "negative_test_query111_00016", + "negative_test_query199_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "apositive_test_query156_00000", + "apositive_test_query210_00001", + "negative_test_query330_00006", + "negative_test_query92_00010", + "negative_test_query156_00001", + "negative_test_query140_00008", + "negative_test_query1_00012", + "apositive_test_query330_00002", + "negative_test_query246_00016", + "apositive_test_query25_00009", + "negative_test_query92_00005", + "apositive_test_query154_00005", + "negative_test_query283_00001", + "negative_test_query15_00010", + "negative_test_query92_00003", + "negative_test_query299_00014", + "negative_test_query32_00004", + "negative_test_query283_00009", + "negative_test_query49_00005", + "apositive_test_query39_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "negative_test_query237_00004", + "apositive_test_query152_00001", + "apositive_test_query152_00005", + "negative_test_query157_00010", + "negative_test_query152_00008", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query152_00003", + "apositive_test_query152_00000", + "apositive_test_query157_00002", + "apositive_test_query157_00001", + "apositive_test_query237_00000", + "negative_test_query237_00002", + "negative_test_query271_00003", + "negative_test_query237_00009", + "apositive_test_query237_00004", + "apositive_test_query237_00002", + "negative_test_query312_00017", + "negative_test_query157_00005", + "apositive_test_query182_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query158", + "ranked_ids": [ + "negative_test_query334_00016", + "negative_test_query223_00007", + "negative_test_query105_00013", + "negative_test_query158_00006", + "negative_test_query164_00004", + "negative_test_query102_00015", + "apositive_test_query186_00003", + "negative_test_query102_00003", + "negative_test_query213_00002", + "negative_test_query247_00010", + "negative_test_query26_00003", + "negative_test_query26_00000", + "negative_test_query26_00004", + "negative_test_query334_00009", + "negative_test_query26_00010", + "negative_test_query26_00005", + "apositive_test_query158_00000", + "apositive_test_query158_00007", + "negative_test_query223_00002", + "apositive_test_query158_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query159", + "ranked_ids": [ + "negative_test_query264_00013", + "apositive_test_query159_00001", + "negative_test_query159_00011", + "negative_test_query159_00005", + "apositive_test_query50_00004", + "negative_test_query264_00012", + "apositive_test_query212_00002", + "apositive_test_query326_00007", + "apositive_test_query38_00003", + "negative_test_query38_00010", + "negative_test_query264_00003", + "apositive_test_query212_00001", + "negative_test_query159_00016", + "apositive_test_query212_00003", + "negative_test_query7_00005", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query66_00003", + "negative_test_query122_00002", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query16", + "ranked_ids": [ + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query308_00004", + "apositive_test_query16_00004", + "negative_test_query16_00009", + "negative_test_query16_00008", + "negative_test_query16_00000", + "negative_test_query16_00001", + "negative_test_query16_00002", + "negative_test_query16_00010", + "negative_test_query240_00003", + "apositive_test_query16_00002", + "negative_test_query280_00007", + "negative_test_query16_00003", + "negative_test_query237_00010", + "apositive_test_query286_00011", + "negative_test_query16_00011", + "negative_test_query6_00015", + "negative_test_query208_00012", + "negative_test_query16_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query160", + "ranked_ids": [ + "negative_test_query160_00003", + "apositive_test_query160_00009", + "apositive_test_query160_00000", + "negative_test_query330_00005", + "negative_test_query160_00004", + "negative_test_query149_00007", + "apositive_test_query64_00000", + "negative_test_query214_00013", + "apositive_test_query355_00007", + "apositive_test_query160_00004", + "negative_test_query215_00010", + "apositive_test_query260_00000", + "negative_test_query64_00015", + "negative_test_query211_00009", + "apositive_test_query120_00002", + "negative_test_query64_00009", + "negative_test_query106_00004", + "negative_test_query176_00016", + "negative_test_query271_00010", + "negative_test_query304_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query161", + "ranked_ids": [ + "apositive_test_query161_00006", + "negative_test_query155_00012", + "apositive_test_query3_00006", + "apositive_test_query276_00010", + "apositive_test_query161_00005", + "apositive_test_query305_00002", + "negative_test_query199_00004", + "apositive_test_query161_00008", + "negative_test_query22_00015", + "apositive_test_query37_00004", + "negative_test_query71_00006", + "negative_test_query260_00004", + "apositive_test_query301_00003", + "apositive_test_query314_00006", + "negative_test_query358_00001", + "negative_test_query301_00015", + "negative_test_query57_00001", + "apositive_test_query301_00000", + "apositive_test_query261_00004", + "apositive_test_query314_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.25 + }, + "query_id": "test_query162", + "ranked_ids": [ + "negative_test_query177_00002", + "apositive_test_query162_00010", + "apositive_test_query162_00009", + "negative_test_query162_00004", + "apositive_test_query162_00003", + "apositive_test_query162_00000", + "negative_test_query162_00001", + "apositive_test_query162_00006", + "negative_test_query95_00006", + "apositive_test_query162_00005", + "negative_test_query240_00008", + "apositive_test_query194_00006", + "negative_test_query177_00005", + "apositive_test_query162_00004", + "apositive_test_query194_00009", + "negative_test_query343_00012", + "apositive_test_query114_00001", + "negative_test_query95_00013", + "apositive_test_query194_00003", + "negative_test_query230_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query197_00002", + "negative_test_query163_00008", + "negative_test_query163_00011", + "negative_test_query90_00011", + "negative_test_query163_00003", + "negative_test_query197_00001", + "negative_test_query163_00013", + "apositive_test_query357_00004", + "negative_test_query202_00002", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query16_00013", + "negative_test_query328_00006", + "negative_test_query47_00001", + "negative_test_query265_00015", + "negative_test_query58_00015", + "apositive_test_query172_00006", + "negative_test_query9_00005", + "negative_test_query245_00001", + "apositive_test_query152_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query164_00000", + "negative_test_query131_00014", + "negative_test_query158_00008", + "apositive_test_query146_00014", + "negative_test_query2_00000", + "negative_test_query164_00015", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query332_00001", + "negative_test_query165_00003", + "negative_test_query101_00005", + "negative_test_query360_00012", + "apositive_test_query13_00000", + "apositive_test_query297_00000", + "negative_test_query245_00005", + "negative_test_query321_00011", + "negative_test_query1_00003", + "negative_test_query208_00014", + "negative_test_query100_00013", + "negative_test_query309_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query165_00013", + "negative_test_query122_00003", + "negative_test_query86_00006", + "negative_test_query86_00010", + "negative_test_query165_00000", + "negative_test_query165_00007", + "negative_test_query149_00012", + "negative_test_query285_00007", + "apositive_test_query165_00001", + "negative_test_query285_00004", + "negative_test_query348_00001", + "negative_test_query299_00011", + "negative_test_query327_00010", + "apositive_test_query219_00000", + "negative_test_query327_00016", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query264_00016", + "negative_test_query38_00010", + "negative_test_query215_00013" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query166", + "ranked_ids": [ + "apositive_test_query166_00005", + "negative_test_query166_00012", + "negative_test_query166_00000", + "apositive_test_query166_00003", + "negative_test_query166_00013", + "negative_test_query166_00006", + "apositive_test_query166_00001", + "negative_test_query166_00004", + "negative_test_query166_00005", + "apositive_test_query166_00002", + "negative_test_query165_00015", + "apositive_test_query166_00000", + "negative_test_query166_00008", + "negative_test_query249_00000", + "negative_test_query166_00007", + "negative_test_query255_00003", + "negative_test_query166_00001", + "apositive_test_query224_00000", + "negative_test_query224_00014", + "negative_test_query166_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query167_00012", + "negative_test_query167_00014", + "negative_test_query167_00002", + "negative_test_query278_00016", + "negative_test_query167_00015", + "negative_test_query167_00011", + "negative_test_query167_00016", + "negative_test_query167_00004", + "negative_test_query278_00010", + "negative_test_query167_00003", + "negative_test_query167_00009", + "apositive_test_query288_00005", + "negative_test_query167_00010", + "negative_test_query167_00005", + "negative_test_query141_00010", + "negative_test_query167_00000", + "negative_test_query167_00007", + "negative_test_query167_00001", + "negative_test_query167_00013", + "negative_test_query167_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6096199500078984, + "recall@5": 0.75 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query83_00003", + "apositive_test_query168_00000", + "apositive_test_query168_00001", + "apositive_test_query168_00002", + "apositive_test_query190_00001", + "negative_test_query168_00014", + "negative_test_query6_00001", + "negative_test_query168_00000", + "negative_test_query6_00005", + "negative_test_query6_00000", + "apositive_test_query340_00000", + "negative_test_query270_00003", + "negative_test_query168_00007", + "apositive_test_query83_00004", + "negative_test_query83_00003", + "apositive_test_query286_00007", + "negative_test_query168_00009", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "apositive_test_query69_00000", + "negative_test_query72_00013", + "negative_test_query104_00001", + "negative_test_query49_00017", + "apositive_test_query286_00008", + "apositive_test_query80_00003", + "negative_test_query72_00016", + "apositive_test_query180_00001", + "negative_test_query72_00001", + "negative_test_query89_00006", + "negative_test_query180_00004", + "negative_test_query343_00001", + "apositive_test_query330_00001", + "apositive_test_query242_00006", + "negative_test_query260_00015", + "apositive_test_query291_00009", + "negative_test_query169_00001", + "negative_test_query10_00005", + "negative_test_query10_00013", + "apositive_test_query266_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00000", + "negative_test_query17_00002", + "negative_test_query17_00005", + "negative_test_query17_00014", + "negative_test_query130_00007", + "negative_test_query292_00010", + "negative_test_query312_00004", + "apositive_test_query153_00013", + "negative_test_query164_00006", + "negative_test_query17_00012", + "negative_test_query10_00005", + "negative_test_query302_00011", + "apositive_test_query355_00007", + "negative_test_query17_00016", + "negative_test_query34_00003", + "apositive_test_query225_00007", + "negative_test_query305_00004", + "negative_test_query17_00003", + "negative_test_query17_00015", + "negative_test_query148_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5555555555555556 + }, + "query_id": "test_query170", + "ranked_ids": [ + "apositive_test_query170_00005", + "apositive_test_query170_00000", + "apositive_test_query170_00006", + "apositive_test_query170_00002", + "apositive_test_query170_00004", + "negative_test_query170_00001", + "apositive_test_query170_00007", + "negative_test_query0_00006", + "negative_test_query170_00007", + "negative_test_query170_00008", + "negative_test_query170_00000", + "negative_test_query170_00005", + "negative_test_query170_00004", + "apositive_test_query297_00000", + "apositive_test_query170_00001", + "negative_test_query348_00001", + "negative_test_query227_00010", + "negative_test_query227_00002", + "negative_test_query297_00005", + "apositive_test_query70_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query171", + "ranked_ids": [ + "negative_test_query171_00005", + "apositive_test_query171_00001", + "negative_test_query171_00013", + "negative_test_query171_00004", + "negative_test_query171_00011", + "negative_test_query219_00010", + "negative_test_query171_00000", + "negative_test_query277_00007", + "negative_test_query24_00007", + "negative_test_query261_00000", + "negative_test_query171_00007", + "negative_test_query171_00003", + "negative_test_query219_00002", + "negative_test_query171_00017", + "negative_test_query346_00005", + "negative_test_query171_00001", + "negative_test_query191_00006", + "negative_test_query171_00015", + "negative_test_query126_00013", + "apositive_test_query224_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query172", + "ranked_ids": [ + "apositive_test_query172_00003", + "negative_test_query172_00007", + "apositive_test_query172_00004", + "apositive_test_query254_00003", + "negative_test_query350_00009", + "apositive_test_query124_00002", + "negative_test_query192_00010", + "apositive_test_query136_00011", + "apositive_test_query136_00001", + "apositive_test_query136_00009", + "negative_test_query8_00000", + "negative_test_query192_00000", + "negative_test_query349_00001", + "apositive_test_query172_00002", + "apositive_test_query13_00003", + "apositive_test_query172_00006", + "negative_test_query64_00006", + "negative_test_query208_00007", + "negative_test_query173_00006", + "negative_test_query208_00015" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query64_00017", + "negative_test_query173_00006", + "negative_test_query173_00011", + "apositive_test_query173_00004", + "apositive_test_query173_00002", + "negative_test_query173_00003", + "apositive_test_query173_00000", + "apositive_test_query173_00003", + "apositive_test_query173_00001", + "apositive_test_query134_00000", + "negative_test_query64_00006", + "negative_test_query64_00013", + "negative_test_query350_00002", + "negative_test_query222_00010", + "negative_test_query64_00004", + "negative_test_query303_00005", + "apositive_test_query101_00004", + "negative_test_query173_00000", + "negative_test_query173_00013", + "negative_test_query339_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00000", + "apositive_test_query174_00005", + "apositive_test_query174_00004", + "apositive_test_query174_00001", + "apositive_test_query174_00012", + "negative_test_query19_00012", + "apositive_test_query174_00006", + "apositive_test_query174_00003", + "apositive_test_query174_00002", + "apositive_test_query174_00008", + "apositive_test_query174_00009", + "apositive_test_query174_00011", + "apositive_test_query174_00010", + "apositive_test_query174_00007", + "apositive_test_query174_00015", + "apositive_test_query174_00013", + "apositive_test_query4_00012", + "apositive_test_query174_00014", + "negative_test_query174_00001", + "negative_test_query262_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00000", + "apositive_test_query175_00003", + "apositive_test_query175_00004", + "negative_test_query175_00010", + "apositive_test_query175_00005", + "negative_test_query204_00002", + "negative_test_query97_00007", + "apositive_test_query204_00004", + "negative_test_query175_00013", + "apositive_test_query353_00010", + "apositive_test_query353_00005", + "negative_test_query204_00001", + "negative_test_query105_00012", + "negative_test_query38_00006", + "apositive_test_query204_00010", + "apositive_test_query50_00015", + "negative_test_query105_00016", + "negative_test_query7_00000", + "negative_test_query175_00000", + "negative_test_query204_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query176_00015", + "negative_test_query195_00006", + "negative_test_query195_00012", + "apositive_test_query128_00007", + "negative_test_query224_00003", + "negative_test_query221_00007", + "negative_test_query9_00003", + "negative_test_query24_00014", + "negative_test_query191_00008", + "negative_test_query284_00006", + "negative_test_query9_00007", + "negative_test_query284_00002", + "apositive_test_query344_00003", + "negative_test_query337_00005", + "negative_test_query200_00005", + "negative_test_query144_00018", + "negative_test_query35_00005", + "negative_test_query93_00000", + "apositive_test_query56_00006", + "apositive_test_query92_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query172_00008", + "apositive_test_query177_00004", + "negative_test_query177_00004", + "apositive_test_query177_00006", + "negative_test_query124_00012", + "negative_test_query177_00006", + "negative_test_query177_00008", + "negative_test_query147_00011", + "negative_test_query152_00005", + "apositive_test_query230_00005", + "apositive_test_query177_00008", + "negative_test_query124_00001", + "negative_test_query147_00005", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query147_00009", + "apositive_test_query229_00008", + "negative_test_query147_00002", + "apositive_test_query162_00009", + "apositive_test_query162_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query178", + "ranked_ids": [ + "negative_test_query178_00007", + "apositive_test_query178_00004", + "apositive_test_query178_00005", + "negative_test_query178_00000", + "negative_test_query178_00001", + "negative_test_query178_00012", + "negative_test_query178_00010", + "negative_test_query178_00003", + "negative_test_query285_00009", + "apositive_test_query285_00006", + "negative_test_query285_00008", + "negative_test_query43_00001", + "negative_test_query321_00017", + "negative_test_query214_00002", + "negative_test_query327_00004", + "apositive_test_query60_00011", + "negative_test_query165_00001", + "negative_test_query31_00004", + "apositive_test_query352_00001", + "negative_test_query274_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "negative_test_query179_00014", + "apositive_test_query173_00000", + "apositive_test_query173_00003", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query303_00005", + "negative_test_query297_00000", + "apositive_test_query316_00001", + "negative_test_query316_00001", + "negative_test_query279_00010", + "negative_test_query206_00009", + "negative_test_query14_00001", + "negative_test_query173_00013", + "negative_test_query179_00001", + "negative_test_query191_00003", + "negative_test_query260_00012", + "negative_test_query64_00003", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.8 + }, + "query_id": "test_query18", + "ranked_ids": [ + "apositive_test_query18_00000", + "apositive_test_query18_00001", + "apositive_test_query18_00003", + "apositive_test_query18_00002", + "negative_test_query140_00003", + "negative_test_query18_00011", + "negative_test_query18_00006", + "apositive_test_query281_00013", + "negative_test_query18_00012", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query257_00012", + "negative_test_query287_00004", + "apositive_test_query18_00004", + "negative_test_query88_00002", + "negative_test_query88_00012", + "negative_test_query300_00015", + "negative_test_query18_00013", + "negative_test_query156_00018", + "apositive_test_query288_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query180", + "ranked_ids": [ + "apositive_test_query180_00000", + "apositive_test_query180_00001", + "negative_test_query180_00002", + "negative_test_query180_00004", + "negative_test_query180_00007", + "negative_test_query180_00016", + "apositive_test_query180_00002", + "apositive_test_query262_00003", + "negative_test_query312_00018", + "negative_test_query114_00003", + "negative_test_query180_00011", + "apositive_test_query262_00000", + "negative_test_query207_00007", + "apositive_test_query262_00002", + "negative_test_query176_00001", + "negative_test_query180_00006", + "negative_test_query112_00014", + "negative_test_query246_00003", + "negative_test_query95_00014", + "negative_test_query169_00015" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00006", + "negative_test_query181_00017", + "negative_test_query181_00012", + "negative_test_query181_00005", + "negative_test_query181_00002", + "negative_test_query181_00010", + "negative_test_query181_00014", + "negative_test_query181_00004", + "negative_test_query181_00007", + "negative_test_query348_00007", + "negative_test_query181_00016", + "negative_test_query320_00007", + "negative_test_query181_00008", + "negative_test_query181_00009", + "apositive_test_query119_00011", + "negative_test_query96_00006", + "negative_test_query181_00001", + "apositive_test_query256_00003", + "negative_test_query181_00018", + "negative_test_query183_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "apositive_test_query182_00000", + "negative_test_query182_00017", + "negative_test_query182_00011", + "negative_test_query182_00004", + "negative_test_query182_00002", + "apositive_test_query152_00009", + "negative_test_query237_00004", + "negative_test_query237_00006", + "negative_test_query342_00009", + "negative_test_query342_00003", + "negative_test_query342_00001", + "negative_test_query237_00001", + "negative_test_query77_00002", + "negative_test_query237_00007", + "negative_test_query342_00004", + "negative_test_query182_00018", + "negative_test_query240_00001", + "negative_test_query230_00008", + "negative_test_query337_00012", + "negative_test_query296_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00009", + "negative_test_query183_00014", + "negative_test_query7_00007", + "negative_test_query183_00012", + "negative_test_query183_00015", + "negative_test_query183_00006", + "apositive_test_query7_00001", + "negative_test_query183_00007", + "negative_test_query7_00008", + "negative_test_query183_00008", + "negative_test_query7_00014", + "negative_test_query7_00009", + "negative_test_query183_00018", + "negative_test_query183_00013", + "negative_test_query183_00000", + "negative_test_query183_00010", + "negative_test_query7_00010", + "negative_test_query112_00017", + "negative_test_query238_00007", + "apositive_test_query178_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "negative_test_query216_00014", + "apositive_test_query216_00001", + "negative_test_query344_00010", + "negative_test_query228_00011", + "apositive_test_query216_00004", + "negative_test_query216_00008", + "negative_test_query158_00008", + "negative_test_query216_00007", + "apositive_test_query33_00004", + "negative_test_query101_00007", + "apositive_test_query216_00000", + "negative_test_query216_00002", + "apositive_test_query33_00006", + "apositive_test_query72_00000", + "negative_test_query156_00001", + "apositive_test_query20_00010", + "negative_test_query41_00013", + "negative_test_query216_00006", + "negative_test_query322_00000", + "apositive_test_query316_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query185", + "ranked_ids": [ + "apositive_test_query307_00008", + "negative_test_query72_00009", + "apositive_test_query185_00002", + "negative_test_query103_00001", + "negative_test_query185_00002", + "negative_test_query48_00017", + "apositive_test_query185_00003", + "negative_test_query72_00017", + "negative_test_query185_00005", + "negative_test_query48_00009", + "negative_test_query185_00000", + "negative_test_query72_00011", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query48_00014", + "negative_test_query350_00007", + "negative_test_query139_00014", + "negative_test_query72_00012", + "negative_test_query103_00000", + "apositive_test_query43_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "negative_test_query247_00012", + "negative_test_query46_00001", + "apositive_test_query306_00003", + "negative_test_query186_00005", + "negative_test_query247_00007", + "negative_test_query313_00015", + "apositive_test_query50_00000", + "negative_test_query47_00009", + "apositive_test_query50_00001", + "apositive_test_query79_00005", + "apositive_test_query50_00010", + "negative_test_query313_00014", + "negative_test_query57_00007", + "negative_test_query259_00003", + "negative_test_query142_00009", + "negative_test_query29_00003", + "negative_test_query334_00017", + "negative_test_query313_00005", + "negative_test_query336_00008", + "negative_test_query105_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query306_00004", + "negative_test_query245_00006", + "negative_test_query230_00000", + "negative_test_query176_00008", + "negative_test_query279_00002", + "apositive_test_query358_00008", + "apositive_test_query311_00002", + "negative_test_query62_00009", + "apositive_test_query127_00014", + "apositive_test_query176_00000", + "negative_test_query168_00002", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query154_00005", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "negative_test_query331_00002", + "apositive_test_query20_00010", + "apositive_test_query33_00008", + "negative_test_query184_00015" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "negative_test_query188_00008", + "apositive_test_query188_00000", + "negative_test_query188_00014", + "negative_test_query188_00009", + "apositive_test_query20_00006", + "apositive_test_query20_00009", + "negative_test_query161_00010", + "negative_test_query344_00007", + "negative_test_query112_00017", + "negative_test_query20_00006", + "negative_test_query220_00011", + "negative_test_query188_00003", + "apositive_test_query146_00009", + "negative_test_query188_00011", + "negative_test_query344_00013", + "negative_test_query188_00013", + "negative_test_query188_00010", + "negative_test_query188_00004", + "apositive_test_query220_00004", + "apositive_test_query220_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.4 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00007", + "negative_test_query189_00001", + "negative_test_query189_00004", + "apositive_test_query189_00003", + "apositive_test_query189_00001", + "apositive_test_query189_00002", + "negative_test_query189_00013", + "negative_test_query140_00016", + "negative_test_query18_00006", + "negative_test_query44_00005", + "apositive_test_query85_00002", + "negative_test_query287_00000", + "negative_test_query189_00006", + "negative_test_query189_00000", + "negative_test_query260_00007", + "negative_test_query85_00007", + "negative_test_query189_00002", + "negative_test_query55_00003", + "apositive_test_query254_00006", + "negative_test_query190_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query19", + "ranked_ids": [ + "negative_test_query19_00003", + "apositive_test_query19_00000", + "negative_test_query4_00001", + "negative_test_query229_00002", + "negative_test_query106_00004", + "apositive_test_query4_00000", + "apositive_test_query174_00014", + "apositive_test_query4_00010", + "negative_test_query169_00014", + "apositive_test_query174_00007", + "negative_test_query19_00006", + "negative_test_query266_00010", + "negative_test_query15_00000", + "negative_test_query106_00007", + "negative_test_query19_00008", + "negative_test_query292_00004", + "negative_test_query174_00001", + "negative_test_query141_00001", + "negative_test_query167_00018", + "negative_test_query141_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query271_00005", + "negative_test_query85_00006", + "apositive_test_query311_00000", + "apositive_test_query260_00003", + "negative_test_query260_00007", + "negative_test_query109_00006", + "negative_test_query6_00007", + "negative_test_query345_00006", + "negative_test_query190_00013", + "negative_test_query181_00003", + "negative_test_query233_00008", + "negative_test_query265_00004", + "negative_test_query85_00014", + "negative_test_query264_00008", + "negative_test_query205_00016", + "apositive_test_query234_00001", + "apositive_test_query85_00000", + "negative_test_query246_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8048099750039491, + "recall@5": 0.75 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "apositive_test_query191_00001", + "negative_test_query191_00010", + "apositive_test_query191_00002", + "negative_test_query181_00013", + "negative_test_query60_00000", + "negative_test_query297_00002", + "negative_test_query256_00009", + "apositive_test_query63_00003", + "negative_test_query191_00011", + "negative_test_query191_00012", + "negative_test_query241_00007", + "apositive_test_query341_00003", + "negative_test_query332_00012", + "apositive_test_query341_00005", + "negative_test_query191_00003", + "negative_test_query274_00002", + "negative_test_query191_00002", + "negative_test_query96_00006", + "negative_test_query64_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00000", + "apositive_test_query192_00002", + "negative_test_query120_00004", + "negative_test_query192_00001", + "negative_test_query192_00013", + "negative_test_query192_00002", + "negative_test_query172_00005", + "negative_test_query192_00009", + "negative_test_query192_00008", + "negative_test_query192_00010", + "apositive_test_query192_00004", + "negative_test_query350_00010", + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query192_00012", + "negative_test_query192_00014", + "negative_test_query322_00008", + "apositive_test_query254_00007", + "negative_test_query192_00005", + "negative_test_query192_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query193", + "ranked_ids": [ + "apositive_test_query99_00003", + "apositive_test_query193_00004", + "apositive_test_query193_00009", + "apositive_test_query193_00001", + "apositive_test_query193_00000", + "apositive_test_query193_00008", + "apositive_test_query99_00002", + "apositive_test_query193_00011", + "apositive_test_query193_00005", + "negative_test_query193_00002", + "negative_test_query99_00009", + "apositive_test_query153_00012", + "negative_test_query146_00001", + "negative_test_query146_00003", + "negative_test_query193_00003", + "apositive_test_query193_00002", + "apositive_test_query276_00006", + "negative_test_query193_00006", + "apositive_test_query146_00008", + "negative_test_query193_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query194_00002", + "apositive_test_query124_00000", + "apositive_test_query162_00006", + "apositive_test_query194_00007", + "apositive_test_query194_00000", + "apositive_test_query42_00001", + "apositive_test_query114_00001", + "negative_test_query262_00011", + "apositive_test_query194_00001", + "apositive_test_query194_00006", + "negative_test_query262_00007", + "negative_test_query137_00011", + "negative_test_query162_00000", + "negative_test_query194_00003", + "negative_test_query62_00014", + "negative_test_query177_00007", + "negative_test_query95_00016", + "apositive_test_query254_00006", + "negative_test_query148_00009", + "negative_test_query89_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query195", + "ranked_ids": [ + "negative_test_query222_00003", + "apositive_test_query195_00002", + "negative_test_query195_00002", + "apositive_test_query195_00001", + "negative_test_query195_00013", + "apositive_test_query195_00005", + "negative_test_query285_00000", + "negative_test_query195_00001", + "negative_test_query109_00008", + "negative_test_query195_00008", + "negative_test_query195_00005", + "negative_test_query149_00012", + "apositive_test_query249_00012", + "negative_test_query321_00008", + "negative_test_query321_00005", + "apositive_test_query251_00000", + "negative_test_query205_00006", + "negative_test_query166_00009", + "negative_test_query195_00004", + "negative_test_query199_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query196_00003", + "apositive_test_query32_00011", + "apositive_test_query32_00012", + "apositive_test_query32_00004", + "negative_test_query235_00008", + "apositive_test_query32_00006", + "negative_test_query196_00011", + "negative_test_query64_00005", + "negative_test_query32_00002", + "negative_test_query14_00001", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "negative_test_query196_00007", + "negative_test_query173_00012", + "negative_test_query8_00013", + "negative_test_query341_00008", + "negative_test_query325_00004", + "negative_test_query96_00013", + "negative_test_query139_00011", + "negative_test_query179_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4367467095119258, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query197", + "ranked_ids": [ + "negative_test_query163_00001", + "negative_test_query197_00010", + "apositive_test_query197_00000", + "apositive_test_query197_00002", + "negative_test_query197_00016", + "negative_test_query197_00006", + "negative_test_query197_00013", + "negative_test_query163_00002", + "negative_test_query197_00004", + "apositive_test_query189_00001", + "apositive_test_query163_00000", + "apositive_test_query34_00000", + "apositive_test_query163_00001", + "negative_test_query178_00008", + "apositive_test_query197_00001", + "negative_test_query321_00006", + "negative_test_query28_00001", + "negative_test_query34_00012", + "negative_test_query163_00009", + "negative_test_query163_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query198_00000", + "negative_test_query239_00011", + "negative_test_query239_00013", + "negative_test_query239_00001", + "negative_test_query239_00010", + "negative_test_query198_00002", + "negative_test_query239_00007", + "negative_test_query198_00008", + "negative_test_query198_00010", + "negative_test_query294_00002", + "negative_test_query352_00002", + "negative_test_query343_00002", + "negative_test_query198_00004", + "negative_test_query112_00005", + "negative_test_query293_00011", + "negative_test_query293_00010", + "negative_test_query62_00016", + "negative_test_query219_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query54_00003", + "negative_test_query140_00014", + "negative_test_query199_00014", + "negative_test_query139_00014", + "negative_test_query305_00007", + "negative_test_query199_00015", + "negative_test_query350_00007", + "negative_test_query134_00006", + "negative_test_query185_00002", + "negative_test_query199_00006", + "negative_test_query199_00001", + "negative_test_query213_00011", + "negative_test_query250_00006", + "negative_test_query199_00002", + "apositive_test_query43_00006", + "negative_test_query199_00013", + "negative_test_query199_00005", + "negative_test_query221_00008", + "negative_test_query72_00005", + "apositive_test_query244_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query44_00008", + "negative_test_query2_00002", + "negative_test_query2_00012", + "negative_test_query190_00016", + "negative_test_query2_00009", + "negative_test_query2_00013", + "negative_test_query2_00016", + "negative_test_query15_00000", + "negative_test_query125_00012", + "negative_test_query85_00007", + "negative_test_query2_00011", + "apositive_test_query217_00006", + "negative_test_query352_00003", + "negative_test_query104_00007", + "negative_test_query126_00014", + "negative_test_query197_00004", + "apositive_test_query57_00004", + "negative_test_query348_00005", + "negative_test_query343_00004", + "negative_test_query2_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query154_00006", + "apositive_test_query20_00010", + "apositive_test_query20_00007", + "apositive_test_query154_00005", + "apositive_test_query154_00002", + "apositive_test_query20_00003", + "negative_test_query33_00007", + "apositive_test_query66_00000", + "negative_test_query154_00002", + "negative_test_query356_00004", + "apositive_test_query356_00005", + "apositive_test_query20_00009", + "negative_test_query66_00002", + "apositive_test_query23_00004", + "apositive_test_query33_00008", + "negative_test_query188_00000", + "apositive_test_query33_00000", + "negative_test_query33_00008", + "apositive_test_query154_00004", + "negative_test_query220_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query200_00000", + "negative_test_query200_00009", + "negative_test_query200_00004", + "negative_test_query337_00009", + "negative_test_query200_00015", + "negative_test_query264_00006", + "negative_test_query200_00013", + "negative_test_query38_00005", + "negative_test_query200_00012", + "negative_test_query337_00004", + "negative_test_query200_00010", + "negative_test_query337_00007", + "negative_test_query200_00007", + "negative_test_query337_00005", + "negative_test_query88_00013", + "negative_test_query200_00005", + "negative_test_query8_00008", + "negative_test_query200_00006", + "negative_test_query303_00009", + "negative_test_query264_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query201", + "ranked_ids": [ + "negative_test_query47_00012", + "apositive_test_query201_00002", + "apositive_test_query30_00005", + "negative_test_query231_00012", + "negative_test_query259_00002", + "apositive_test_query0_00011", + "negative_test_query47_00013", + "negative_test_query71_00009", + "negative_test_query71_00013", + "negative_test_query209_00008", + "apositive_test_query201_00006", + "negative_test_query0_00004", + "negative_test_query71_00002", + "negative_test_query71_00005", + "apositive_test_query20_00005", + "negative_test_query71_00000", + "negative_test_query159_00009", + "negative_test_query65_00002", + "apositive_test_query30_00001", + "apositive_test_query71_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query202_00004", + "negative_test_query202_00000", + "negative_test_query202_00001", + "negative_test_query235_00001", + "apositive_test_query34_00003", + "negative_test_query202_00010", + "negative_test_query352_00001", + "negative_test_query202_00006", + "negative_test_query202_00008", + "negative_test_query202_00016", + "negative_test_query176_00011", + "apositive_test_query206_00000", + "negative_test_query312_00012", + "negative_test_query15_00010", + "apositive_test_query302_00006", + "negative_test_query202_00009", + "apositive_test_query309_00000", + "apositive_test_query61_00003", + "negative_test_query188_00011", + "negative_test_query321_00018" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query203", + "ranked_ids": [ + "negative_test_query203_00006", + "negative_test_query203_00008", + "negative_test_query203_00005", + "negative_test_query250_00005", + "apositive_test_query110_00003", + "apositive_test_query203_00004", + "apositive_test_query203_00009", + "apositive_test_query284_00003", + "negative_test_query130_00001", + "negative_test_query209_00000", + "apositive_test_query30_00005", + "apositive_test_query203_00003", + "negative_test_query284_00014", + "apositive_test_query203_00007", + "apositive_test_query130_00002", + "negative_test_query250_00009", + "apositive_test_query203_00000", + "negative_test_query250_00016", + "negative_test_query250_00011", + "negative_test_query35_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.058823529411764705 + }, + "query_id": "test_query204", + "ranked_ids": [ + "apositive_test_query353_00005", + "apositive_test_query204_00000", + "negative_test_query204_00002", + "apositive_test_query353_00000", + "negative_test_query353_00003", + "apositive_test_query353_00002", + "apositive_test_query204_00009", + "negative_test_query204_00001", + "apositive_test_query204_00002", + "apositive_test_query204_00007", + "negative_test_query353_00001", + "apositive_test_query204_00008", + "apositive_test_query97_00001", + "negative_test_query97_00007", + "apositive_test_query353_00010", + "apositive_test_query353_00011", + "negative_test_query204_00000", + "apositive_test_query204_00004", + "apositive_test_query175_00004", + "negative_test_query353_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query205_00015", + "negative_test_query256_00000", + "apositive_test_query205_00000", + "negative_test_query228_00016", + "negative_test_query35_00007", + "negative_test_query205_00007", + "negative_test_query31_00015", + "negative_test_query35_00008", + "negative_test_query256_00015", + "negative_test_query96_00011", + "negative_test_query256_00007", + "apositive_test_query256_00002", + "negative_test_query115_00000", + "negative_test_query341_00003", + "negative_test_query205_00008", + "negative_test_query73_00000", + "negative_test_query73_00011", + "negative_test_query256_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query206_00007", + "negative_test_query206_00002", + "negative_test_query206_00014", + "negative_test_query206_00015", + "negative_test_query303_00015", + "negative_test_query206_00013", + "negative_test_query206_00000", + "negative_test_query206_00001", + "negative_test_query206_00006", + "negative_test_query206_00003", + "negative_test_query70_00011", + "negative_test_query299_00001", + "negative_test_query303_00002", + "negative_test_query206_00005", + "negative_test_query2_00016", + "negative_test_query303_00003", + "negative_test_query206_00018", + "negative_test_query299_00002", + "apositive_test_query348_00001", + "negative_test_query64_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.5 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00004", + "apositive_test_query207_00000", + "negative_test_query207_00002", + "apositive_test_query207_00005", + "negative_test_query207_00011", + "apositive_test_query207_00002", + "negative_test_query207_00012", + "negative_test_query207_00005", + "apositive_test_query207_00001", + "negative_test_query207_00008", + "negative_test_query207_00006", + "negative_test_query207_00009", + "negative_test_query207_00000", + "negative_test_query207_00007", + "apositive_test_query13_00001", + "negative_test_query207_00001", + "negative_test_query207_00013", + "negative_test_query37_00004", + "negative_test_query207_00003", + "negative_test_query207_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "apositive_test_query208_00000", + "negative_test_query128_00003", + "negative_test_query208_00017", + "negative_test_query208_00016", + "negative_test_query208_00010", + "negative_test_query208_00000", + "negative_test_query208_00018", + "apositive_test_query128_00002", + "negative_test_query211_00014", + "apositive_test_query128_00007", + "negative_test_query256_00011", + "negative_test_query128_00001", + "negative_test_query73_00008", + "negative_test_query208_00008", + "negative_test_query208_00003", + "negative_test_query61_00012", + "negative_test_query208_00004", + "negative_test_query208_00006", + "negative_test_query208_00009", + "negative_test_query208_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query209", + "ranked_ids": [ + "apositive_test_query209_00001", + "apositive_test_query37_00001", + "negative_test_query209_00007", + "negative_test_query231_00006", + "negative_test_query102_00010", + "apositive_test_query26_00007", + "negative_test_query223_00003", + "negative_test_query26_00002", + "apositive_test_query158_00009", + "apositive_test_query158_00004", + "negative_test_query347_00004", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "apositive_test_query119_00008", + "negative_test_query102_00016", + "negative_test_query223_00010", + "negative_test_query209_00001", + "negative_test_query223_00007", + "negative_test_query26_00004", + "negative_test_query313_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "apositive_test_query128_00003", + "apositive_test_query21_00000", + "negative_test_query128_00006", + "negative_test_query21_00010", + "negative_test_query234_00007", + "apositive_test_query128_00002", + "negative_test_query128_00004", + "negative_test_query21_00005", + "negative_test_query21_00009", + "negative_test_query330_00000", + "negative_test_query224_00015", + "negative_test_query10_00008", + "apositive_test_query128_00004", + "negative_test_query208_00000", + "negative_test_query21_00012", + "negative_test_query208_00017", + "negative_test_query8_00006", + "negative_test_query234_00006", + "negative_test_query21_00001", + "negative_test_query350_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query210_00013", + "negative_test_query210_00015", + "negative_test_query202_00014", + "negative_test_query199_00015", + "negative_test_query148_00008", + "negative_test_query321_00013", + "negative_test_query17_00002", + "negative_test_query288_00007", + "negative_test_query335_00004", + "negative_test_query112_00013", + "negative_test_query148_00014", + "negative_test_query156_00015", + "negative_test_query300_00000", + "negative_test_query148_00007", + "negative_test_query210_00007", + "negative_test_query80_00007", + "negative_test_query210_00006", + "negative_test_query210_00008", + "negative_test_query210_00010", + "negative_test_query288_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query211", + "ranked_ids": [ + "apositive_test_query211_00001", + "negative_test_query159_00006", + "negative_test_query137_00012", + "negative_test_query211_00015", + "negative_test_query211_00012", + "apositive_test_query101_00000", + "negative_test_query211_00000", + "negative_test_query32_00005", + "negative_test_query195_00007", + "negative_test_query293_00013", + "negative_test_query239_00014", + "negative_test_query100_00012", + "apositive_test_query163_00004", + "apositive_test_query198_00000", + "negative_test_query189_00008", + "negative_test_query202_00006", + "apositive_test_query176_00000", + "negative_test_query83_00010", + "negative_test_query167_00015", + "apositive_test_query162_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query212", + "ranked_ids": [ + "apositive_test_query212_00004", + "apositive_test_query326_00000", + "apositive_test_query326_00004", + "negative_test_query167_00016", + "apositive_test_query89_00004", + "negative_test_query282_00010", + "negative_test_query183_00009", + "negative_test_query77_00004", + "negative_test_query167_00017", + "negative_test_query282_00009", + "apositive_test_query71_00001", + "apositive_test_query98_00000", + "apositive_test_query11_00007", + "negative_test_query344_00008", + "negative_test_query183_00012", + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query66_00005", + "apositive_test_query244_00007", + "negative_test_query346_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00000", + "apositive_test_query213_00004", + "apositive_test_query213_00002", + "negative_test_query213_00001", + "negative_test_query213_00011", + "apositive_test_query213_00003", + "apositive_test_query259_00001", + "negative_test_query213_00005", + "apositive_test_query158_00001", + "apositive_test_query305_00003", + "negative_test_query313_00017", + "negative_test_query138_00016", + "apositive_test_query36_00002", + "negative_test_query137_00015", + "negative_test_query112_00003", + "negative_test_query357_00007", + "negative_test_query213_00004", + "negative_test_query138_00001", + "apositive_test_query259_00000", + "negative_test_query224_00015" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "apositive_test_query214_00000", + "negative_test_query214_00013", + "apositive_test_query234_00001", + "negative_test_query214_00001", + "negative_test_query214_00003", + "negative_test_query214_00004", + "negative_test_query214_00010", + "negative_test_query214_00015", + "apositive_test_query323_00003", + "apositive_test_query160_00009", + "negative_test_query214_00016", + "negative_test_query17_00013", + "apositive_test_query323_00002", + "negative_test_query324_00012", + "negative_test_query9_00008", + "apositive_test_query75_00003", + "apositive_test_query17_00001", + "negative_test_query324_00002", + "apositive_test_query89_00000", + "negative_test_query103_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00012", + "apositive_test_query215_00000", + "negative_test_query215_00000", + "negative_test_query215_00010", + "negative_test_query257_00003", + "negative_test_query215_00011", + "negative_test_query215_00008", + "negative_test_query215_00004", + "apositive_test_query80_00005", + "negative_test_query257_00001", + "negative_test_query215_00009", + "negative_test_query215_00015", + "negative_test_query215_00017", + "negative_test_query215_00006", + "negative_test_query215_00001", + "negative_test_query295_00004", + "negative_test_query215_00018", + "negative_test_query215_00007", + "negative_test_query263_00001", + "negative_test_query293_00013" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query216_00000", + "negative_test_query216_00002", + "apositive_test_query216_00002", + "negative_test_query216_00007", + "negative_test_query216_00000", + "apositive_test_query216_00004", + "apositive_test_query216_00001", + "negative_test_query216_00014", + "negative_test_query216_00006", + "negative_test_query216_00008", + "negative_test_query44_00002", + "apositive_test_query154_00005", + "negative_test_query112_00002", + "negative_test_query216_00009", + "negative_test_query216_00005", + "apositive_test_query20_00001", + "negative_test_query246_00016", + "apositive_test_query33_00004", + "apositive_test_query330_00002", + "apositive_test_query182_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "negative_test_query217_00004", + "apositive_test_query185_00006", + "apositive_test_query217_00000", + "negative_test_query222_00005", + "negative_test_query217_00011", + "apositive_test_query307_00010", + "negative_test_query28_00002", + "negative_test_query307_00004", + "negative_test_query72_00005", + "negative_test_query185_00003", + "negative_test_query112_00007", + "negative_test_query230_00002", + "negative_test_query53_00012", + "apositive_test_query213_00004", + "negative_test_query76_00003", + "apositive_test_query107_00001", + "apositive_test_query160_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query218", + "ranked_ids": [ + "negative_test_query134_00003", + "apositive_test_query218_00004", + "negative_test_query218_00000", + "apositive_test_query218_00005", + "apositive_test_query218_00003", + "negative_test_query218_00009", + "negative_test_query218_00012", + "apositive_test_query134_00000", + "negative_test_query218_00003", + "negative_test_query58_00010", + "apositive_test_query314_00015", + "negative_test_query134_00009", + "negative_test_query43_00006", + "negative_test_query218_00002", + "negative_test_query43_00010", + "apositive_test_query218_00000", + "negative_test_query165_00013", + "negative_test_query274_00008", + "apositive_test_query60_00012", + "negative_test_query304_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "apositive_test_query219_00000", + "negative_test_query219_00014", + "negative_test_query219_00016", + "apositive_test_query219_00001", + "negative_test_query187_00015", + "negative_test_query219_00008", + "negative_test_query219_00012", + "negative_test_query59_00000", + "negative_test_query219_00015", + "apositive_test_query177_00004", + "negative_test_query219_00017", + "negative_test_query305_00005", + "negative_test_query219_00011", + "negative_test_query227_00007", + "negative_test_query308_00006", + "negative_test_query171_00005", + "negative_test_query24_00007", + "negative_test_query183_00016", + "negative_test_query165_00010", + "negative_test_query219_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "apositive_test_query22_00000", + "negative_test_query22_00008", + "negative_test_query22_00002", + "negative_test_query22_00015", + "negative_test_query22_00003", + "negative_test_query22_00004", + "negative_test_query22_00018", + "negative_test_query22_00001", + "negative_test_query22_00012", + "negative_test_query22_00017", + "negative_test_query22_00000", + "negative_test_query22_00007", + "negative_test_query22_00005", + "negative_test_query22_00006", + "negative_test_query22_00009", + "negative_test_query22_00016", + "negative_test_query22_00013", + "negative_test_query66_00004", + "apositive_test_query177_00003", + "apositive_test_query162_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.5 + }, + "query_id": "test_query220", + "ranked_ids": [ + "negative_test_query344_00007", + "apositive_test_query220_00003", + "apositive_test_query220_00001", + "negative_test_query188_00013", + "apositive_test_query220_00004", + "apositive_test_query220_00000", + "negative_test_query154_00009", + "negative_test_query188_00004", + "negative_test_query220_00011", + "negative_test_query188_00008", + "negative_test_query188_00009", + "negative_test_query344_00013", + "apositive_test_query188_00000", + "negative_test_query188_00014", + "negative_test_query188_00007", + "negative_test_query20_00002", + "apositive_test_query20_00009", + "apositive_test_query146_00009", + "negative_test_query220_00004", + "negative_test_query161_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query142_00004", + "apositive_test_query306_00006", + "negative_test_query277_00010", + "negative_test_query65_00013", + "negative_test_query65_00007", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "negative_test_query184_00001", + "negative_test_query12_00005", + "negative_test_query15_00005", + "apositive_test_query174_00006", + "apositive_test_query180_00002", + "negative_test_query278_00004", + "negative_test_query180_00011", + "apositive_test_query244_00008", + "negative_test_query244_00006", + "apositive_test_query92_00003", + "apositive_test_query162_00005", + "negative_test_query39_00007", + "negative_test_query171_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query222", + "ranked_ids": [ + "apositive_test_query275_00001", + "apositive_test_query222_00000", + "apositive_test_query289_00002", + "negative_test_query289_00006", + "negative_test_query271_00015", + "negative_test_query290_00004", + "apositive_test_query275_00006", + "negative_test_query271_00004", + "negative_test_query222_00017", + "apositive_test_query275_00000", + "apositive_test_query275_00008", + "negative_test_query275_00002", + "apositive_test_query302_00000", + "apositive_test_query275_00005", + "negative_test_query275_00004", + "apositive_test_query222_00001", + "apositive_test_query275_00004", + "negative_test_query222_00014", + "negative_test_query222_00006", + "negative_test_query222_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query359_00008", + "negative_test_query47_00001", + "apositive_test_query259_00011", + "negative_test_query359_00014", + "negative_test_query259_00006", + "negative_test_query323_00005", + "apositive_test_query201_00004", + "negative_test_query107_00004", + "apositive_test_query260_00003", + "apositive_test_query259_00003", + "negative_test_query47_00007", + "negative_test_query47_00002", + "negative_test_query6_00007", + "negative_test_query109_00006", + "negative_test_query51_00003", + "apositive_test_query259_00008", + "negative_test_query345_00002", + "apositive_test_query201_00006", + "apositive_test_query107_00005", + "negative_test_query164_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query224", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query224_00012", + "negative_test_query350_00016", + "negative_test_query224_00014", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query98_00018", + "negative_test_query149_00008", + "apositive_test_query224_00003", + "apositive_test_query256_00001", + "negative_test_query224_00011", + "apositive_test_query179_00002", + "negative_test_query205_00009", + "negative_test_query304_00001", + "apositive_test_query83_00000", + "negative_test_query350_00015", + "negative_test_query212_00013", + "negative_test_query196_00001", + "negative_test_query224_00008", + "apositive_test_query32_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00001", + "negative_test_query145_00000", + "apositive_test_query225_00000", + "apositive_test_query225_00003", + "apositive_test_query225_00007", + "apositive_test_query225_00009", + "negative_test_query225_00006", + "apositive_test_query225_00004", + "apositive_test_query225_00005", + "apositive_test_query145_00001", + "negative_test_query149_00011", + "negative_test_query343_00009", + "apositive_test_query145_00005", + "negative_test_query225_00008", + "apositive_test_query225_00002", + "negative_test_query161_00004", + "negative_test_query183_00015", + "negative_test_query225_00000", + "negative_test_query54_00002", + "apositive_test_query169_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "error": "ranking must be a permutation of 1..20.", + "error_type": "RerankParseError", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query226", + "ranked_ids": [], + "valid": false + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00003", + "negative_test_query227_00004", + "negative_test_query227_00006", + "negative_test_query70_00011", + "negative_test_query227_00012", + "negative_test_query227_00002", + "apositive_test_query227_00000", + "apositive_test_query70_00001", + "negative_test_query227_00000", + "negative_test_query227_00013", + "apositive_test_query227_00001", + "negative_test_query227_00014", + "negative_test_query321_00004", + "negative_test_query59_00004", + "negative_test_query235_00006", + "negative_test_query227_00010", + "negative_test_query115_00007", + "negative_test_query227_00011", + "negative_test_query227_00005", + "apositive_test_query348_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query228", + "ranked_ids": [ + "negative_test_query228_00012", + "negative_test_query165_00011", + "apositive_test_query228_00000", + "negative_test_query228_00004", + "negative_test_query228_00002", + "negative_test_query228_00014", + "negative_test_query228_00000", + "negative_test_query228_00003", + "negative_test_query73_00016", + "negative_test_query2_00001", + "negative_test_query322_00001", + "negative_test_query228_00011", + "negative_test_query216_00013", + "apositive_test_query254_00000", + "apositive_test_query318_00001", + "apositive_test_query322_00003", + "apositive_test_query322_00007", + "apositive_test_query333_00001", + "apositive_test_query73_00001", + "negative_test_query179_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query229_00002", + "apositive_test_query229_00003", + "apositive_test_query229_00000", + "negative_test_query229_00008", + "negative_test_query229_00005", + "apositive_test_query229_00007", + "negative_test_query229_00007", + "negative_test_query229_00006", + "negative_test_query229_00000", + "negative_test_query229_00009", + "apositive_test_query158_00006", + "apositive_test_query146_00004", + "apositive_test_query158_00002", + "negative_test_query0_00001", + "negative_test_query355_00001", + "apositive_test_query94_00007", + "negative_test_query94_00010", + "apositive_test_query18_00004", + "negative_test_query44_00005", + "apositive_test_query69_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.6 + }, + "query_id": "test_query23", + "ranked_ids": [ + "apositive_test_query23_00000", + "apositive_test_query23_00002", + "apositive_test_query23_00001", + "negative_test_query259_00007", + "negative_test_query356_00008", + "negative_test_query23_00014", + "negative_test_query66_00010", + "negative_test_query239_00012", + "apositive_test_query54_00007", + "negative_test_query292_00013", + "apositive_test_query39_00002", + "apositive_test_query54_00004", + "apositive_test_query52_00000", + "apositive_test_query52_00001", + "negative_test_query307_00002", + "apositive_test_query52_00002", + "negative_test_query133_00003", + "apositive_test_query52_00004", + "apositive_test_query146_00015", + "apositive_test_query225_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query230_00007", + "negative_test_query230_00010", + "negative_test_query230_00000", + "apositive_test_query230_00003", + "apositive_test_query3_00001", + "negative_test_query329_00000", + "negative_test_query269_00010", + "negative_test_query23_00012", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "apositive_test_query3_00006", + "negative_test_query99_00003", + "apositive_test_query3_00009", + "negative_test_query3_00004", + "negative_test_query3_00005", + "negative_test_query280_00006", + "apositive_test_query280_00002", + "negative_test_query280_00004", + "negative_test_query3_00000", + "negative_test_query3_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query231", + "ranked_ids": [ + "apositive_test_query130_00000", + "negative_test_query345_00011", + "apositive_test_query0_00001", + "negative_test_query9_00017", + "negative_test_query75_00000", + "negative_test_query221_00006", + "apositive_test_query231_00003", + "apositive_test_query201_00002", + "negative_test_query213_00004", + "negative_test_query130_00000", + "negative_test_query231_00001", + "negative_test_query201_00011", + "negative_test_query359_00008", + "negative_test_query47_00005", + "apositive_test_query130_00004", + "negative_test_query47_00006", + "negative_test_query130_00010", + "apositive_test_query201_00006", + "negative_test_query306_00006", + "apositive_test_query119_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "apositive_test_query232_00000", + "negative_test_query232_00013", + "negative_test_query232_00014", + "negative_test_query232_00000", + "negative_test_query232_00011", + "negative_test_query232_00003", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query232_00008", + "negative_test_query232_00015", + "negative_test_query232_00018", + "negative_test_query232_00002", + "negative_test_query232_00004", + "negative_test_query232_00012", + "negative_test_query232_00006", + "negative_test_query232_00016", + "negative_test_query232_00010", + "negative_test_query232_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query233", + "ranked_ids": [ + "apositive_test_query233_00000", + "apositive_test_query233_00002", + "negative_test_query233_00001", + "negative_test_query233_00010", + "negative_test_query2_00010", + "negative_test_query233_00006", + "negative_test_query190_00007", + "negative_test_query233_00004", + "apositive_test_query2_00000", + "negative_test_query246_00001", + "negative_test_query2_00014", + "negative_test_query176_00017", + "negative_test_query233_00015", + "apositive_test_query275_00003", + "negative_test_query233_00005", + "negative_test_query219_00015", + "negative_test_query263_00009", + "negative_test_query130_00001", + "negative_test_query130_00008", + "negative_test_query233_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00015", + "negative_test_query234_00005", + "negative_test_query304_00001", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "apositive_test_query304_00001", + "negative_test_query304_00005", + "apositive_test_query304_00003", + "negative_test_query214_00008", + "negative_test_query304_00012", + "negative_test_query234_00010", + "negative_test_query214_00006", + "apositive_test_query43_00005", + "negative_test_query304_00008", + "negative_test_query304_00013", + "negative_test_query285_00011", + "apositive_test_query218_00002", + "negative_test_query304_00009", + "negative_test_query234_00011", + "negative_test_query10_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00002", + "apositive_test_query32_00006", + "negative_test_query235_00008", + "apositive_test_query32_00000", + "negative_test_query196_00000", + "negative_test_query235_00014", + "negative_test_query235_00000", + "apositive_test_query32_00004", + "apositive_test_query235_00003", + "negative_test_query235_00009", + "apositive_test_query235_00000", + "negative_test_query235_00011", + "negative_test_query219_00013", + "apositive_test_query235_00001", + "negative_test_query235_00004", + "negative_test_query24_00017", + "negative_test_query235_00013", + "negative_test_query61_00001", + "negative_test_query235_00015", + "negative_test_query173_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "negative_test_query236_00000", + "negative_test_query236_00009", + "apositive_test_query279_00002", + "negative_test_query236_00014", + "negative_test_query279_00012", + "negative_test_query236_00006", + "negative_test_query236_00012", + "negative_test_query236_00004", + "negative_test_query279_00005", + "negative_test_query236_00011", + "negative_test_query236_00013", + "negative_test_query22_00016", + "negative_test_query236_00005", + "negative_test_query279_00003", + "negative_test_query236_00007", + "negative_test_query222_00000", + "negative_test_query236_00002", + "negative_test_query236_00015", + "negative_test_query236_00018" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "apositive_test_query237_00001", + "negative_test_query152_00008", + "negative_test_query237_00006", + "negative_test_query237_00004", + "negative_test_query237_00005", + "apositive_test_query237_00002", + "negative_test_query152_00003", + "negative_test_query237_00002", + "negative_test_query237_00000", + "apositive_test_query152_00004", + "apositive_test_query237_00000", + "negative_test_query312_00001", + "negative_test_query290_00010", + "negative_test_query237_00007", + "negative_test_query237_00009", + "negative_test_query157_00010", + "apositive_test_query157_00001", + "apositive_test_query152_00001", + "negative_test_query237_00001", + "negative_test_query261_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "negative_test_query102_00007", + "apositive_test_query146_00013", + "negative_test_query108_00007", + "negative_test_query13_00003", + "negative_test_query142_00011", + "negative_test_query56_00008", + "apositive_test_query306_00003", + "negative_test_query46_00012", + "negative_test_query111_00013", + "negative_test_query203_00003", + "negative_test_query111_00008", + "negative_test_query223_00005", + "apositive_test_query142_00003", + "negative_test_query123_00001", + "negative_test_query56_00000", + "apositive_test_query276_00008", + "negative_test_query238_00011", + "apositive_test_query29_00001", + "apositive_test_query29_00007", + "negative_test_query29_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "apositive_test_query239_00000", + "negative_test_query239_00015", + "negative_test_query239_00003", + "negative_test_query239_00001", + "negative_test_query198_00002", + "negative_test_query239_00000", + "negative_test_query239_00007", + "negative_test_query239_00014", + "negative_test_query239_00010", + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query239_00011", + "negative_test_query352_00002", + "negative_test_query198_00008", + "negative_test_query268_00005", + "negative_test_query239_00013", + "negative_test_query61_00004", + "negative_test_query183_00015", + "negative_test_query163_00004", + "negative_test_query167_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "negative_test_query24_00000", + "negative_test_query250_00008", + "negative_test_query24_00009", + "negative_test_query24_00002", + "negative_test_query24_00008", + "apositive_test_query57_00004", + "negative_test_query24_00012", + "apositive_test_query24_00000", + "negative_test_query24_00004", + "negative_test_query128_00002", + "negative_test_query52_00012", + "negative_test_query252_00005", + "apositive_test_query106_00002", + "negative_test_query15_00000", + "negative_test_query198_00006", + "apositive_test_query226_00004", + "negative_test_query24_00001", + "negative_test_query323_00000", + "negative_test_query190_00016", + "negative_test_query356_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "apositive_test_query240_00000", + "negative_test_query240_00012", + "negative_test_query240_00006", + "negative_test_query240_00004", + "negative_test_query240_00018", + "negative_test_query240_00015", + "negative_test_query240_00002", + "negative_test_query240_00007", + "negative_test_query240_00010", + "negative_test_query240_00011", + "negative_test_query240_00016", + "apositive_test_query311_00000", + "negative_test_query246_00002", + "negative_test_query27_00010", + "negative_test_query293_00004", + "negative_test_query6_00000", + "negative_test_query282_00013", + "negative_test_query297_00005", + "negative_test_query240_00017", + "negative_test_query240_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "apositive_test_query241_00000", + "negative_test_query241_00005", + "negative_test_query241_00010", + "negative_test_query241_00016", + "negative_test_query241_00017", + "negative_test_query241_00002", + "negative_test_query241_00008", + "negative_test_query173_00013", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query206_00011", + "negative_test_query256_00000", + "negative_test_query206_00009", + "negative_test_query139_00007", + "negative_test_query258_00000", + "negative_test_query205_00003", + "negative_test_query298_00014", + "negative_test_query53_00000", + "negative_test_query191_00014", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query291_00000", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "negative_test_query242_00000", + "apositive_test_query242_00000", + "apositive_test_query242_00002", + "apositive_test_query242_00013", + "apositive_test_query242_00001", + "apositive_test_query291_00005", + "apositive_test_query247_00005", + "apositive_test_query291_00002", + "negative_test_query242_00004", + "apositive_test_query242_00008", + "negative_test_query291_00000", + "apositive_test_query291_00008", + "negative_test_query291_00004", + "apositive_test_query242_00010", + "apositive_test_query242_00005", + "negative_test_query291_00003", + "negative_test_query291_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query243_00000", + "apositive_test_query243_00002", + "negative_test_query274_00014", + "apositive_test_query243_00005", + "apositive_test_query243_00012", + "apositive_test_query243_00003", + "negative_test_query243_00002", + "apositive_test_query243_00011", + "negative_test_query243_00000", + "apositive_test_query160_00003", + "negative_test_query64_00003", + "negative_test_query205_00007", + "negative_test_query192_00010", + "negative_test_query192_00007", + "negative_test_query303_00005", + "apositive_test_query173_00004", + "negative_test_query250_00013", + "negative_test_query184_00004", + "negative_test_query223_00004", + "negative_test_query289_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query244", + "ranked_ids": [ + "apositive_test_query244_00000", + "negative_test_query244_00001", + "negative_test_query244_00005", + "apositive_test_query244_00001", + "negative_test_query244_00009", + "apositive_test_query244_00002", + "apositive_test_query244_00004", + "negative_test_query244_00008", + "negative_test_query244_00002", + "negative_test_query244_00003", + "negative_test_query244_00010", + "negative_test_query244_00004", + "apositive_test_query244_00006", + "apositive_test_query244_00005", + "apositive_test_query244_00007", + "negative_test_query244_00000", + "apositive_test_query42_00002", + "negative_test_query126_00015", + "apositive_test_query23_00003", + "apositive_test_query356_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query245_00003", + "negative_test_query245_00010", + "negative_test_query64_00013", + "negative_test_query135_00013", + "negative_test_query245_00000", + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query245_00000", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query245_00001", + "negative_test_query202_00000", + "negative_test_query121_00002", + "negative_test_query293_00003", + "negative_test_query202_00004", + "negative_test_query122_00002", + "apositive_test_query141_00002", + "apositive_test_query141_00006", + "negative_test_query159_00003", + "negative_test_query336_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query252_00003", + "negative_test_query246_00002", + "negative_test_query246_00016", + "negative_test_query246_00012", + "negative_test_query246_00001", + "negative_test_query190_00003", + "negative_test_query6_00014", + "negative_test_query135_00017", + "negative_test_query311_00005", + "apositive_test_query68_00002", + "negative_test_query44_00002", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "apositive_test_query216_00000", + "apositive_test_query0_00001", + "negative_test_query6_00006", + "negative_test_query198_00012", + "negative_test_query293_00000", + "negative_test_query317_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "negative_test_query291_00003", + "apositive_test_query247_00004", + "apositive_test_query242_00001", + "apositive_test_query291_00001", + "apositive_test_query242_00002", + "apositive_test_query291_00000", + "negative_test_query247_00005", + "negative_test_query291_00005", + "negative_test_query242_00000", + "apositive_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query291_00002", + "apositive_test_query247_00005", + "apositive_test_query242_00010", + "apositive_test_query242_00012", + "negative_test_query247_00011", + "negative_test_query247_00003", + "negative_test_query291_00001", + "negative_test_query105_00012", + "negative_test_query186_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00008", + "apositive_test_query248_00002", + "apositive_test_query248_00007", + "negative_test_query248_00002", + "negative_test_query248_00011", + "negative_test_query248_00010", + "apositive_test_query248_00005", + "apositive_test_query248_00003", + "negative_test_query251_00017", + "negative_test_query248_00006", + "negative_test_query248_00004", + "apositive_test_query29_00004", + "apositive_test_query29_00000", + "negative_test_query29_00001", + "negative_test_query205_00017", + "negative_test_query248_00000", + "negative_test_query248_00001", + "apositive_test_query29_00005", + "negative_test_query248_00009", + "apositive_test_query248_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query249", + "ranked_ids": [ + "negative_test_query156_00002", + "apositive_test_query249_00000", + "negative_test_query249_00001", + "apositive_test_query249_00004", + "apositive_test_query281_00001", + "negative_test_query122_00016", + "negative_test_query193_00001", + "apositive_test_query204_00003", + "negative_test_query156_00007", + "negative_test_query210_00010", + "negative_test_query302_00007", + "negative_test_query325_00015", + "apositive_test_query353_00004", + "negative_test_query71_00006", + "negative_test_query326_00005", + "apositive_test_query112_00000", + "negative_test_query278_00016", + "negative_test_query72_00017", + "negative_test_query301_00013", + "apositive_test_query177_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query25", + "ranked_ids": [ + "apositive_test_query25_00000", + "apositive_test_query25_00008", + "negative_test_query25_00007", + "apositive_test_query25_00003", + "apositive_test_query25_00005", + "apositive_test_query25_00006", + "negative_test_query25_00005", + "apositive_test_query25_00007", + "negative_test_query143_00000", + "negative_test_query45_00014", + "apositive_test_query25_00009", + "apositive_test_query25_00001", + "negative_test_query25_00001", + "negative_test_query25_00002", + "apositive_test_query25_00010", + "apositive_test_query25_00004", + "negative_test_query30_00003", + "negative_test_query346_00006", + "negative_test_query9_00001", + "negative_test_query328_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query250", + "ranked_ids": [ + "negative_test_query186_00007", + "negative_test_query250_00011", + "negative_test_query250_00009", + "apositive_test_query203_00000", + "apositive_test_query250_00000", + "negative_test_query79_00013", + "negative_test_query111_00015", + "negative_test_query250_00013", + "negative_test_query57_00009", + "negative_test_query102_00010", + "apositive_test_query203_00006", + "negative_test_query250_00001", + "negative_test_query184_00015", + "negative_test_query250_00003", + "negative_test_query192_00009", + "negative_test_query159_00014", + "negative_test_query101_00008", + "negative_test_query159_00004", + "negative_test_query61_00007", + "negative_test_query64_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query251", + "ranked_ids": [ + "negative_test_query189_00011", + "negative_test_query234_00006", + "negative_test_query322_00002", + "negative_test_query56_00005", + "apositive_test_query134_00003", + "negative_test_query251_00003", + "apositive_test_query251_00001", + "apositive_test_query29_00002", + "negative_test_query251_00004", + "negative_test_query298_00003", + "negative_test_query326_00000", + "negative_test_query347_00014", + "negative_test_query14_00000", + "negative_test_query8_00009", + "negative_test_query268_00011", + "negative_test_query160_00001", + "negative_test_query143_00001", + "apositive_test_query347_00000", + "negative_test_query170_00008", + "negative_test_query269_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query252", + "ranked_ids": [ + "apositive_test_query252_00000", + "negative_test_query252_00009", + "negative_test_query252_00012", + "negative_test_query252_00001", + "negative_test_query252_00011", + "negative_test_query318_00006", + "negative_test_query212_00000", + "apositive_test_query125_00001", + "negative_test_query14_00006", + "negative_test_query44_00011", + "negative_test_query126_00001", + "negative_test_query252_00010", + "negative_test_query321_00004", + "negative_test_query151_00000", + "apositive_test_query252_00002", + "negative_test_query330_00006", + "apositive_test_query110_00001", + "apositive_test_query252_00003", + "negative_test_query252_00007", + "negative_test_query16_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.3 + }, + "query_id": "test_query253", + "ranked_ids": [ + "apositive_test_query253_00003", + "apositive_test_query253_00004", + "apositive_test_query194_00008", + "apositive_test_query253_00007", + "apositive_test_query13_00012", + "negative_test_query194_00001", + "apositive_test_query74_00002", + "negative_test_query253_00005", + "apositive_test_query162_00008", + "negative_test_query226_00013", + "apositive_test_query349_00000", + "negative_test_query177_00000", + "negative_test_query253_00004", + "apositive_test_query186_00004", + "negative_test_query172_00008", + "negative_test_query124_00001", + "negative_test_query124_00002", + "apositive_test_query74_00003", + "negative_test_query253_00003", + "negative_test_query95_00017" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query254", + "ranked_ids": [ + "apositive_test_query254_00004", + "apositive_test_query254_00000", + "negative_test_query350_00000", + "negative_test_query270_00010", + "apositive_test_query254_00003", + "apositive_test_query254_00001", + "apositive_test_query254_00002", + "apositive_test_query254_00006", + "apositive_test_query254_00010", + "negative_test_query254_00004", + "negative_test_query254_00008", + "negative_test_query136_00003", + "apositive_test_query254_00005", + "negative_test_query35_00015", + "negative_test_query228_00016", + "negative_test_query21_00009", + "negative_test_query21_00005", + "negative_test_query316_00013", + "apositive_test_query333_00004", + "negative_test_query128_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "apositive_test_query255_00000", + "negative_test_query255_00006", + "negative_test_query278_00015", + "negative_test_query255_00017", + "apositive_test_query322_00001", + "negative_test_query208_00008", + "negative_test_query246_00003", + "negative_test_query278_00000", + "negative_test_query255_00010", + "apositive_test_query25_00004", + "apositive_test_query278_00000", + "negative_test_query255_00018", + "negative_test_query326_00006", + "negative_test_query89_00004", + "negative_test_query255_00000", + "negative_test_query110_00001", + "apositive_test_query313_00000", + "negative_test_query255_00004", + "negative_test_query255_00013", + "negative_test_query255_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5654495432396527, + "recall@5": 0.75 + }, + "query_id": "test_query256", + "ranked_ids": [ + "negative_test_query256_00003", + "apositive_test_query256_00001", + "negative_test_query256_00011", + "apositive_test_query256_00000", + "apositive_test_query256_00002", + "apositive_test_query256_00003", + "apositive_test_query119_00000", + "negative_test_query279_00000", + "negative_test_query256_00008", + "negative_test_query319_00001", + "negative_test_query82_00001", + "apositive_test_query216_00002", + "apositive_test_query142_00002", + "negative_test_query5_00010", + "negative_test_query105_00008", + "negative_test_query347_00007", + "negative_test_query183_00008", + "apositive_test_query43_00001", + "negative_test_query135_00016", + "negative_test_query7_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.5 + }, + "query_id": "test_query257", + "ranked_ids": [ + "negative_test_query257_00003", + "apositive_test_query257_00004", + "apositive_test_query257_00000", + "apositive_test_query257_00001", + "negative_test_query215_00000", + "apositive_test_query257_00002", + "apositive_test_query69_00007", + "apositive_test_query257_00005", + "apositive_test_query257_00003", + "apositive_test_query286_00014", + "negative_test_query257_00007", + "negative_test_query52_00007", + "negative_test_query257_00005", + "negative_test_query52_00005", + "apositive_test_query52_00001", + "apositive_test_query54_00003", + "negative_test_query52_00002", + "negative_test_query54_00005", + "negative_test_query52_00004", + "apositive_test_query151_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query228_00005", + "negative_test_query319_00000", + "negative_test_query320_00002", + "negative_test_query82_00000", + "negative_test_query206_00016", + "negative_test_query311_00002", + "negative_test_query259_00001", + "negative_test_query167_00007", + "negative_test_query181_00002", + "negative_test_query162_00004", + "apositive_test_query301_00002", + "negative_test_query173_00010", + "negative_test_query339_00008", + "negative_test_query38_00002", + "negative_test_query169_00016", + "negative_test_query171_00002", + "negative_test_query144_00010", + "negative_test_query101_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query259", + "ranked_ids": [ + "negative_test_query259_00002", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "apositive_test_query259_00008", + "apositive_test_query231_00001", + "apositive_test_query259_00006", + "apositive_test_query259_00003", + "apositive_test_query54_00011", + "negative_test_query259_00001", + "apositive_test_query345_00000", + "negative_test_query250_00015", + "negative_test_query213_00004", + "negative_test_query359_00012", + "negative_test_query111_00010", + "apositive_test_query259_00007", + "negative_test_query359_00003", + "negative_test_query47_00004", + "apositive_test_query0_00007", + "negative_test_query30_00009", + "negative_test_query259_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "negative_test_query26_00008", + "apositive_test_query26_00005", + "negative_test_query223_00002", + "apositive_test_query158_00003", + "apositive_test_query186_00003", + "apositive_test_query26_00004", + "negative_test_query105_00013", + "negative_test_query26_00003", + "negative_test_query158_00006", + "negative_test_query119_00006", + "negative_test_query164_00004", + "negative_test_query247_00010", + "negative_test_query158_00009", + "negative_test_query26_00009", + "negative_test_query26_00005", + "negative_test_query26_00001", + "negative_test_query102_00015", + "negative_test_query334_00016", + "apositive_test_query102_00000", + "negative_test_query238_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "negative_test_query260_00011", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "apositive_test_query6_00002", + "negative_test_query252_00004", + "negative_test_query6_00011", + "apositive_test_query6_00000", + "apositive_test_query6_00001", + "negative_test_query6_00004", + "negative_test_query125_00001", + "negative_test_query6_00002", + "negative_test_query6_00009", + "negative_test_query260_00009", + "negative_test_query6_00005", + "negative_test_query2_00004", + "apositive_test_query186_00001", + "negative_test_query260_00002", + "apositive_test_query109_00000", + "apositive_test_query109_00001", + "negative_test_query6_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query261", + "ranked_ids": [ + "negative_test_query261_00005", + "negative_test_query116_00004", + "negative_test_query261_00006", + "negative_test_query261_00013", + "negative_test_query261_00008", + "negative_test_query261_00003", + "negative_test_query261_00002", + "negative_test_query261_00007", + "negative_test_query261_00000", + "negative_test_query261_00004", + "negative_test_query261_00010", + "negative_test_query261_00001", + "negative_test_query261_00011", + "apositive_test_query261_00002", + "apositive_test_query29_00001", + "apositive_test_query261_00003", + "negative_test_query261_00009", + "apositive_test_query261_00000", + "negative_test_query2_00015", + "negative_test_query118_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query262", + "ranked_ids": [ + "apositive_test_query262_00003", + "negative_test_query312_00018", + "apositive_test_query262_00000", + "negative_test_query106_00010", + "apositive_test_query194_00002", + "negative_test_query124_00012", + "negative_test_query106_00005", + "negative_test_query172_00000", + "apositive_test_query42_00006", + "apositive_test_query177_00008", + "negative_test_query262_00000", + "negative_test_query147_00001", + "negative_test_query312_00000", + "negative_test_query162_00007", + "negative_test_query312_00003", + "negative_test_query326_00008", + "negative_test_query266_00003", + "negative_test_query152_00006", + "negative_test_query301_00005", + "apositive_test_query106_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query263", + "ranked_ids": [ + "negative_test_query169_00014", + "negative_test_query135_00004", + "negative_test_query190_00009", + "apositive_test_query263_00002", + "negative_test_query263_00005", + "apositive_test_query242_00011", + "apositive_test_query291_00010", + "negative_test_query201_00010", + "apositive_test_query37_00002", + "negative_test_query125_00012", + "negative_test_query88_00003", + "negative_test_query71_00006", + "apositive_test_query292_00002", + "negative_test_query357_00006", + "negative_test_query213_00001", + "negative_test_query307_00005", + "negative_test_query213_00005", + "apositive_test_query231_00001", + "apositive_test_query155_00003", + "negative_test_query253_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "apositive_test_query264_00000", + "negative_test_query264_00011", + "negative_test_query264_00004", + "negative_test_query200_00011", + "negative_test_query264_00006", + "negative_test_query264_00014", + "negative_test_query159_00011", + "negative_test_query200_00017", + "negative_test_query264_00012", + "negative_test_query337_00012", + "apositive_test_query38_00003", + "negative_test_query264_00018", + "negative_test_query200_00002", + "negative_test_query264_00007", + "negative_test_query337_00002", + "negative_test_query264_00002", + "negative_test_query159_00013", + "negative_test_query264_00016", + "negative_test_query264_00008", + "negative_test_query159_00017" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query265", + "ranked_ids": [ + "apositive_test_query265_00000", + "negative_test_query265_00002", + "apositive_test_query265_00001", + "negative_test_query265_00000", + "negative_test_query265_00007", + "apositive_test_query156_00000", + "negative_test_query265_00009", + "apositive_test_query234_00001", + "negative_test_query274_00008", + "negative_test_query52_00000", + "negative_test_query190_00013", + "negative_test_query139_00000", + "negative_test_query86_00008", + "apositive_test_query78_00002", + "negative_test_query165_00016", + "negative_test_query165_00007", + "negative_test_query92_00012", + "negative_test_query66_00010", + "negative_test_query86_00007", + "negative_test_query271_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "negative_test_query266_00018", + "negative_test_query266_00008", + "apositive_test_query266_00000", + "negative_test_query176_00018", + "negative_test_query106_00004", + "negative_test_query266_00014", + "negative_test_query266_00009", + "negative_test_query141_00000", + "negative_test_query266_00001", + "apositive_test_query54_00009", + "negative_test_query169_00014", + "apositive_test_query19_00002", + "negative_test_query266_00017", + "negative_test_query266_00007", + "negative_test_query266_00005", + "negative_test_query266_00010", + "negative_test_query106_00001", + "negative_test_query174_00001", + "negative_test_query266_00002", + "negative_test_query266_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "negative_test_query267_00002", + "negative_test_query79_00004", + "negative_test_query98_00013", + "negative_test_query359_00005", + "negative_test_query79_00002", + "apositive_test_query50_00004", + "apositive_test_query113_00002", + "apositive_test_query295_00001", + "apositive_test_query157_00008", + "negative_test_query186_00006", + "negative_test_query295_00002", + "negative_test_query204_00001", + "negative_test_query115_00007", + "negative_test_query5_00012", + "apositive_test_query204_00000", + "negative_test_query30_00000", + "negative_test_query345_00013", + "negative_test_query313_00016", + "apositive_test_query353_00002", + "apositive_test_query250_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query268", + "ranked_ids": [ + "apositive_test_query268_00000", + "apositive_test_query201_00003", + "apositive_test_query268_00004", + "negative_test_query47_00017", + "negative_test_query268_00000", + "negative_test_query268_00013", + "apositive_test_query51_00003", + "negative_test_query268_00012", + "apositive_test_query268_00002", + "negative_test_query268_00007", + "negative_test_query268_00003", + "negative_test_query268_00004", + "negative_test_query195_00009", + "negative_test_query268_00005", + "negative_test_query335_00011", + "negative_test_query210_00007", + "negative_test_query323_00002", + "negative_test_query335_00002", + "negative_test_query268_00008", + "negative_test_query228_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query269", + "ranked_ids": [ + "negative_test_query329_00014", + "negative_test_query269_00009", + "apositive_test_query269_00000", + "negative_test_query269_00001", + "negative_test_query269_00000", + "negative_test_query269_00015", + "apositive_test_query296_00005", + "negative_test_query329_00004", + "negative_test_query269_00004", + "negative_test_query269_00012", + "negative_test_query269_00003", + "negative_test_query269_00005", + "negative_test_query269_00007", + "negative_test_query269_00002", + "negative_test_query296_00003", + "apositive_test_query269_00003", + "negative_test_query269_00011", + "negative_test_query269_00010", + "apositive_test_query296_00003", + "negative_test_query269_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "negative_test_query27_00009", + "negative_test_query27_00004", + "negative_test_query147_00000", + "negative_test_query76_00009", + "negative_test_query358_00003", + "apositive_test_query154_00000", + "negative_test_query33_00004", + "negative_test_query259_00006", + "negative_test_query11_00008", + "negative_test_query138_00006", + "negative_test_query30_00009", + "negative_test_query164_00007", + "apositive_test_query59_00000", + "negative_test_query130_00006", + "negative_test_query315_00001", + "apositive_test_query29_00004", + "apositive_test_query81_00007", + "apositive_test_query127_00010", + "apositive_test_query16_00002", + "negative_test_query27_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query270", + "ranked_ids": [ + "negative_test_query270_00012", + "negative_test_query270_00000", + "apositive_test_query270_00001", + "negative_test_query270_00004", + "negative_test_query270_00011", + "negative_test_query270_00006", + "apositive_test_query270_00000", + "negative_test_query270_00005", + "negative_test_query270_00009", + "negative_test_query270_00016", + "negative_test_query270_00015", + "negative_test_query270_00002", + "apositive_test_query270_00002", + "negative_test_query270_00003", + "negative_test_query270_00010", + "apositive_test_query81_00000", + "negative_test_query270_00014", + "negative_test_query270_00008", + "negative_test_query76_00005", + "negative_test_query81_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query271", + "ranked_ids": [ + "apositive_test_query271_00001", + "apositive_test_query271_00000", + "negative_test_query330_00008", + "negative_test_query78_00010", + "apositive_test_query10_00002", + "negative_test_query271_00007", + "negative_test_query120_00003", + "negative_test_query327_00010", + "negative_test_query241_00018", + "negative_test_query169_00009", + "negative_test_query227_00009", + "negative_test_query198_00015", + "negative_test_query112_00014", + "apositive_test_query67_00000", + "apositive_test_query109_00001", + "negative_test_query90_00001", + "negative_test_query84_00014", + "apositive_test_query314_00007", + "apositive_test_query189_00000", + "negative_test_query221_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00003", + "negative_test_query272_00016", + "negative_test_query144_00000", + "negative_test_query272_00015", + "negative_test_query272_00010", + "negative_test_query272_00001", + "negative_test_query272_00008", + "negative_test_query272_00002", + "negative_test_query332_00013", + "negative_test_query41_00008", + "negative_test_query272_00017", + "negative_test_query272_00000", + "negative_test_query272_00012", + "negative_test_query272_00014", + "negative_test_query272_00013", + "negative_test_query272_00009", + "apositive_test_query272_00000", + "negative_test_query18_00014", + "negative_test_query328_00000", + "apositive_test_query251_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00008", + "apositive_test_query273_00005", + "apositive_test_query273_00000", + "negative_test_query273_00006", + "apositive_test_query281_00008", + "negative_test_query18_00011", + "apositive_test_query18_00004", + "apositive_test_query18_00003", + "apositive_test_query18_00000", + "negative_test_query273_00000", + "apositive_test_query18_00001", + "negative_test_query18_00006", + "negative_test_query18_00012", + "negative_test_query273_00004", + "negative_test_query300_00015", + "negative_test_query273_00009", + "negative_test_query140_00013", + "apositive_test_query36_00001", + "apositive_test_query273_00004", + "negative_test_query166_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query274", + "ranked_ids": [ + "apositive_test_query274_00000", + "negative_test_query274_00000", + "negative_test_query274_00014", + "apositive_test_query274_00003", + "negative_test_query274_00011", + "negative_test_query274_00005", + "negative_test_query34_00014", + "negative_test_query218_00011", + "negative_test_query352_00012", + "negative_test_query274_00007", + "negative_test_query274_00012", + "negative_test_query274_00009", + "apositive_test_query274_00001", + "negative_test_query256_00015", + "negative_test_query200_00009", + "negative_test_query78_00007", + "apositive_test_query327_00002", + "negative_test_query274_00004", + "negative_test_query352_00009", + "negative_test_query34_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query275_00000", + "apositive_test_query275_00005", + "negative_test_query263_00009", + "apositive_test_query275_00003", + "apositive_test_query275_00006", + "apositive_test_query275_00007", + "negative_test_query275_00004", + "apositive_test_query275_00008", + "apositive_test_query275_00004", + "negative_test_query202_00005", + "negative_test_query289_00013", + "negative_test_query275_00000", + "negative_test_query264_00017", + "negative_test_query159_00017", + "negative_test_query275_00007", + "negative_test_query149_00016", + "apositive_test_query110_00001", + "apositive_test_query93_00003", + "negative_test_query16_00001", + "negative_test_query87_00017" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query22_00004", + "negative_test_query199_00012", + "apositive_test_query276_00005", + "negative_test_query276_00001", + "apositive_test_query153_00006", + "negative_test_query153_00001", + "apositive_test_query161_00002", + "negative_test_query113_00011", + "apositive_test_query314_00005", + "apositive_test_query146_00015", + "negative_test_query146_00000", + "apositive_test_query314_00006", + "apositive_test_query3_00010", + "apositive_test_query276_00006", + "negative_test_query146_00001", + "negative_test_query146_00003", + "apositive_test_query314_00007", + "negative_test_query251_00012", + "apositive_test_query276_00004", + "negative_test_query280_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9828920819566879, + "recall@5": 1.0 + }, + "query_id": "test_query277", + "ranked_ids": [ + "apositive_test_query277_00001", + "apositive_test_query277_00003", + "apositive_test_query277_00000", + "negative_test_query277_00007", + "apositive_test_query277_00002", + "negative_test_query277_00008", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "negative_test_query345_00000", + "negative_test_query277_00010", + "negative_test_query277_00006", + "negative_test_query277_00013", + "negative_test_query277_00004", + "negative_test_query277_00014", + "negative_test_query277_00015", + "apositive_test_query107_00007", + "negative_test_query277_00003", + "negative_test_query277_00000", + "negative_test_query277_00009", + "negative_test_query313_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query140_00006", + "negative_test_query278_00002", + "apositive_test_query263_00005", + "negative_test_query278_00011", + "apositive_test_query4_00013", + "apositive_test_query36_00001", + "negative_test_query260_00002", + "negative_test_query260_00009", + "negative_test_query260_00011", + "negative_test_query15_00007", + "negative_test_query177_00001", + "negative_test_query278_00009", + "negative_test_query255_00018", + "negative_test_query158_00003", + "negative_test_query255_00016", + "apositive_test_query186_00001", + "negative_test_query110_00011", + "negative_test_query27_00005", + "negative_test_query64_00007", + "negative_test_query324_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.8 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query279_00000", + "negative_test_query279_00014", + "apositive_test_query279_00003", + "apositive_test_query279_00004", + "apositive_test_query279_00001", + "negative_test_query279_00009", + "apositive_test_query177_00004", + "apositive_test_query177_00006", + "apositive_test_query348_00007", + "apositive_test_query3_00002", + "negative_test_query279_00013", + "apositive_test_query3_00000", + "apositive_test_query76_00005", + "negative_test_query181_00008", + "negative_test_query279_00007", + "apositive_test_query358_00007", + "negative_test_query309_00011", + "apositive_test_query162_00009", + "negative_test_query132_00013", + "apositive_test_query76_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query28", + "ranked_ids": [ + "apositive_test_query217_00003", + "apositive_test_query217_00000", + "negative_test_query48_00009", + "apositive_test_query28_00008", + "apositive_test_query28_00009", + "negative_test_query72_00004", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00001", + "negative_test_query307_00004", + "negative_test_query217_00011", + "negative_test_query185_00001", + "negative_test_query28_00002", + "apositive_test_query28_00003", + "apositive_test_query28_00007", + "negative_test_query185_00003", + "negative_test_query72_00018", + "negative_test_query217_00004", + "negative_test_query72_00005", + "negative_test_query307_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.5 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "negative_test_query280_00004", + "negative_test_query106_00006", + "apositive_test_query280_00004", + "apositive_test_query280_00000", + "negative_test_query280_00012", + "negative_test_query9_00012", + "negative_test_query280_00010", + "negative_test_query280_00008", + "negative_test_query280_00005", + "negative_test_query280_00003", + "negative_test_query280_00002", + "negative_test_query280_00006", + "negative_test_query280_00013", + "negative_test_query236_00018", + "apositive_test_query280_00003", + "negative_test_query129_00008", + "apositive_test_query253_00009", + "negative_test_query106_00005", + "apositive_test_query74_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query281_00004", + "apositive_test_query273_00002", + "apositive_test_query281_00010", + "apositive_test_query281_00007", + "apositive_test_query281_00012", + "apositive_test_query281_00003", + "apositive_test_query281_00008", + "apositive_test_query281_00016", + "apositive_test_query281_00001", + "apositive_test_query273_00000", + "negative_test_query273_00003", + "negative_test_query273_00004", + "negative_test_query273_00007", + "apositive_test_query273_00008", + "apositive_test_query281_00014", + "negative_test_query257_00012", + "apositive_test_query273_00003", + "negative_test_query249_00000", + "apositive_test_query281_00011", + "negative_test_query125_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query282", + "ranked_ids": [ + "negative_test_query344_00001", + "apositive_test_query282_00000", + "negative_test_query282_00007", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query282_00000", + "negative_test_query282_00004", + "negative_test_query344_00000", + "apositive_test_query282_00001", + "negative_test_query282_00001", + "apositive_test_query20_00004", + "negative_test_query282_00006", + "negative_test_query282_00010", + "negative_test_query344_00004", + "negative_test_query344_00008", + "apositive_test_query220_00002", + "negative_test_query282_00003", + "negative_test_query282_00016", + "negative_test_query282_00012", + "negative_test_query282_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query283_00007", + "negative_test_query93_00002", + "negative_test_query283_00012", + "negative_test_query93_00004", + "negative_test_query138_00005", + "negative_test_query283_00004", + "negative_test_query138_00009", + "negative_test_query270_00006", + "negative_test_query283_00003", + "negative_test_query283_00000", + "negative_test_query283_00014", + "negative_test_query93_00010", + "apositive_test_query93_00002", + "negative_test_query93_00009", + "negative_test_query299_00017", + "negative_test_query93_00006", + "negative_test_query283_00016", + "negative_test_query265_00013", + "negative_test_query31_00010", + "negative_test_query92_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.6 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "apositive_test_query284_00003", + "negative_test_query284_00009", + "negative_test_query284_00001", + "apositive_test_query284_00001", + "negative_test_query284_00012", + "apositive_test_query163_00003", + "negative_test_query284_00004", + "negative_test_query114_00006", + "apositive_test_query79_00004", + "apositive_test_query198_00002", + "negative_test_query109_00009", + "negative_test_query14_00014", + "negative_test_query288_00011", + "apositive_test_query141_00002", + "apositive_test_query141_00006", + "apositive_test_query162_00005", + "apositive_test_query163_00002", + "negative_test_query141_00002", + "apositive_test_query259_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5 + }, + "query_id": "test_query285", + "ranked_ids": [ + "apositive_test_query285_00004", + "apositive_test_query285_00001", + "apositive_test_query285_00007", + "apositive_test_query285_00002", + "negative_test_query304_00003", + "apositive_test_query285_00003", + "apositive_test_query285_00006", + "negative_test_query285_00008", + "negative_test_query285_00002", + "negative_test_query285_00004", + "apositive_test_query285_00005", + "negative_test_query285_00005", + "negative_test_query285_00009", + "negative_test_query285_00000", + "negative_test_query285_00010", + "negative_test_query86_00001", + "negative_test_query285_00001", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query285_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.2 + }, + "query_id": "test_query286", + "ranked_ids": [ + "apositive_test_query286_00013", + "apositive_test_query286_00012", + "negative_test_query286_00000", + "apositive_test_query286_00010", + "negative_test_query257_00005", + "negative_test_query36_00001", + "negative_test_query257_00001", + "apositive_test_query286_00006", + "negative_test_query54_00004", + "apositive_test_query80_00000", + "apositive_test_query257_00000", + "negative_test_query286_00003", + "apositive_test_query257_00001", + "apositive_test_query286_00014", + "negative_test_query52_00001", + "apositive_test_query286_00005", + "negative_test_query139_00008", + "negative_test_query125_00012", + "negative_test_query52_00005", + "negative_test_query36_00013" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query287_00001", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query133_00002", + "apositive_test_query133_00001", + "apositive_test_query133_00006", + "apositive_test_query133_00014", + "apositive_test_query133_00013", + "negative_test_query301_00010", + "apositive_test_query287_00004", + "apositive_test_query133_00004", + "apositive_test_query133_00012", + "negative_test_query133_00002", + "apositive_test_query153_00003", + "apositive_test_query4_00009", + "negative_test_query153_00003", + "negative_test_query134_00000", + "negative_test_query359_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query288", + "ranked_ids": [ + "apositive_test_query141_00002", + "negative_test_query288_00006", + "negative_test_query288_00002", + "apositive_test_query288_00002", + "apositive_test_query210_00001", + "negative_test_query199_00015", + "apositive_test_query141_00006", + "negative_test_query199_00010", + "negative_test_query288_00010", + "negative_test_query288_00007", + "negative_test_query210_00005", + "negative_test_query213_00007", + "negative_test_query93_00000", + "negative_test_query288_00001", + "negative_test_query17_00002", + "negative_test_query148_00014", + "negative_test_query210_00013", + "apositive_test_query304_00001", + "negative_test_query148_00004", + "apositive_test_query210_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00006", + "negative_test_query289_00013", + "apositive_test_query289_00002", + "negative_test_query334_00014", + "negative_test_query7_00006", + "apositive_test_query275_00005", + "apositive_test_query289_00004", + "negative_test_query289_00001", + "negative_test_query47_00009", + "negative_test_query289_00004", + "negative_test_query113_00009", + "negative_test_query250_00016", + "apositive_test_query157_00008", + "negative_test_query289_00000", + "negative_test_query107_00004", + "negative_test_query267_00005", + "apositive_test_query289_00000", + "negative_test_query289_00003", + "negative_test_query247_00001", + "negative_test_query305_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.375 + }, + "query_id": "test_query29", + "ranked_ids": [ + "apositive_test_query29_00004", + "apositive_test_query29_00007", + "apositive_test_query29_00005", + "negative_test_query29_00009", + "negative_test_query248_00009", + "negative_test_query39_00011", + "apositive_test_query29_00000", + "negative_test_query292_00015", + "apositive_test_query29_00001", + "apositive_test_query29_00003", + "negative_test_query248_00007", + "negative_test_query29_00008", + "negative_test_query29_00007", + "apositive_test_query29_00006", + "negative_test_query29_00010", + "negative_test_query29_00006", + "negative_test_query29_00000", + "negative_test_query29_00005", + "negative_test_query213_00012", + "apositive_test_query29_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7365896932159578, + "recall@5": 0.75 + }, + "query_id": "test_query290", + "ranked_ids": [ + "apositive_test_query290_00001", + "negative_test_query290_00005", + "apositive_test_query290_00002", + "negative_test_query98_00003", + "apositive_test_query290_00003", + "negative_test_query290_00009", + "negative_test_query342_00005", + "negative_test_query290_00000", + "negative_test_query290_00003", + "negative_test_query237_00010", + "negative_test_query290_00006", + "apositive_test_query237_00002", + "negative_test_query290_00014", + "negative_test_query312_00001", + "negative_test_query237_00013", + "negative_test_query290_00002", + "negative_test_query290_00008", + "negative_test_query116_00002", + "negative_test_query59_00002", + "negative_test_query157_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query291_00000", + "apositive_test_query242_00001", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "negative_test_query242_00000", + "apositive_test_query242_00005", + "apositive_test_query242_00010", + "apositive_test_query291_00008", + "apositive_test_query291_00005", + "apositive_test_query242_00013", + "negative_test_query291_00005", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query291_00003", + "negative_test_query291_00000", + "apositive_test_query247_00005", + "negative_test_query291_00007", + "negative_test_query291_00004", + "negative_test_query186_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query292", + "ranked_ids": [ + "negative_test_query292_00010", + "apositive_test_query292_00001", + "apositive_test_query292_00000", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query292_00014", + "negative_test_query292_00000", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query292_00012", + "negative_test_query292_00005", + "negative_test_query99_00004", + "negative_test_query292_00007", + "negative_test_query298_00000", + "negative_test_query61_00011", + "negative_test_query292_00006", + "negative_test_query292_00008", + "negative_test_query139_00001", + "negative_test_query151_00006", + "negative_test_query205_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9674679834891693, + "recall@5": 1.0 + }, + "query_id": "test_query293", + "ranked_ids": [ + "apositive_test_query293_00001", + "apositive_test_query293_00000", + "negative_test_query293_00015", + "apositive_test_query293_00002", + "negative_test_query293_00009", + "apositive_test_query178_00003", + "negative_test_query293_00000", + "negative_test_query27_00008", + "negative_test_query293_00003", + "apositive_test_query44_00001", + "negative_test_query27_00011", + "negative_test_query293_00006", + "negative_test_query89_00002", + "negative_test_query293_00011", + "negative_test_query27_00002", + "negative_test_query293_00010", + "negative_test_query167_00013", + "negative_test_query293_00016", + "negative_test_query220_00007", + "apositive_test_query52_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00008", + "negative_test_query294_00002", + "negative_test_query294_00003", + "apositive_test_query294_00001", + "negative_test_query314_00001", + "negative_test_query294_00011", + "negative_test_query294_00013", + "negative_test_query294_00006", + "negative_test_query294_00007", + "negative_test_query294_00016", + "negative_test_query294_00012", + "negative_test_query58_00003", + "negative_test_query294_00005", + "negative_test_query111_00015", + "negative_test_query294_00004", + "apositive_test_query170_00005", + "negative_test_query277_00005", + "negative_test_query294_00001", + "apositive_test_query177_00004", + "negative_test_query156_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query295", + "ranked_ids": [ + "negative_test_query26_00002", + "apositive_test_query57_00000", + "negative_test_query295_00002", + "negative_test_query295_00008", + "apositive_test_query295_00002", + "apositive_test_query295_00000", + "apositive_test_query223_00001", + "apositive_test_query295_00001", + "negative_test_query295_00013", + "negative_test_query295_00011", + "negative_test_query57_00009", + "negative_test_query292_00015", + "negative_test_query295_00016", + "negative_test_query168_00005", + "negative_test_query115_00005", + "negative_test_query295_00001", + "negative_test_query295_00004", + "negative_test_query292_00016", + "negative_test_query295_00012", + "negative_test_query295_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query296", + "ranked_ids": [ + "negative_test_query340_00009", + "apositive_test_query296_00002", + "apositive_test_query296_00001", + "apositive_test_query296_00004", + "apositive_test_query296_00005", + "apositive_test_query296_00003", + "negative_test_query329_00011", + "negative_test_query296_00009", + "negative_test_query340_00014", + "negative_test_query112_00012", + "apositive_test_query296_00000", + "negative_test_query296_00000", + "negative_test_query296_00013", + "negative_test_query296_00001", + "negative_test_query296_00005", + "negative_test_query296_00008", + "negative_test_query0_00004", + "negative_test_query296_00012", + "negative_test_query296_00003", + "negative_test_query296_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query297_00013", + "negative_test_query297_00012", + "negative_test_query297_00014", + "negative_test_query297_00004", + "apositive_test_query139_00001", + "negative_test_query117_00006", + "negative_test_query297_00007", + "negative_test_query191_00010", + "apositive_test_query297_00000", + "negative_test_query297_00002", + "negative_test_query297_00003", + "negative_test_query297_00008", + "negative_test_query70_00005", + "negative_test_query191_00011", + "negative_test_query64_00005", + "negative_test_query64_00003", + "negative_test_query297_00001", + "negative_test_query329_00007", + "negative_test_query297_00010", + "negative_test_query191_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "apositive_test_query298_00002", + "apositive_test_query298_00000", + "negative_test_query216_00010", + "negative_test_query347_00014", + "negative_test_query298_00006", + "apositive_test_query347_00001", + "negative_test_query347_00016", + "negative_test_query298_00003", + "negative_test_query240_00001", + "negative_test_query298_00001", + "negative_test_query298_00004", + "apositive_test_query347_00002", + "negative_test_query347_00000", + "apositive_test_query298_00001", + "negative_test_query182_00010", + "apositive_test_query56_00002", + "negative_test_query182_00011", + "negative_test_query175_00009", + "negative_test_query193_00006", + "negative_test_query202_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query299_00005", + "apositive_test_query346_00001", + "negative_test_query299_00006", + "apositive_test_query325_00002", + "apositive_test_query265_00002", + "negative_test_query351_00009", + "negative_test_query300_00000", + "apositive_test_query141_00002", + "negative_test_query288_00007", + "negative_test_query288_00000", + "negative_test_query210_00008", + "negative_test_query3_00003", + "negative_test_query89_00008", + "negative_test_query125_00005", + "negative_test_query58_00005", + "apositive_test_query227_00003", + "negative_test_query305_00007", + "negative_test_query134_00000", + "apositive_test_query101_00003", + "negative_test_query92_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query3", + "ranked_ids": [ + "apositive_test_query3_00001", + "apositive_test_query3_00000", + "negative_test_query3_00000", + "apositive_test_query3_00003", + "negative_test_query83_00007", + "negative_test_query147_00014", + "negative_test_query230_00000", + "negative_test_query279_00000", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "negative_test_query147_00011", + "negative_test_query3_00007", + "apositive_test_query3_00007", + "apositive_test_query230_00003", + "apositive_test_query3_00008", + "negative_test_query230_00010", + "negative_test_query230_00007", + "negative_test_query3_00005", + "negative_test_query329_00000", + "negative_test_query23_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query30", + "ranked_ids": [ + "apositive_test_query71_00002", + "apositive_test_query30_00004", + "apositive_test_query71_00000", + "apositive_test_query306_00002", + "apositive_test_query30_00001", + "apositive_test_query30_00003", + "negative_test_query231_00008", + "negative_test_query231_00002", + "negative_test_query71_00010", + "negative_test_query259_00002", + "apositive_test_query30_00005", + "negative_test_query71_00005", + "apositive_test_query209_00005", + "negative_test_query356_00010", + "negative_test_query182_00014", + "negative_test_query231_00000", + "negative_test_query357_00002", + "apositive_test_query30_00000", + "negative_test_query97_00011", + "negative_test_query201_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query15_00000", + "negative_test_query300_00009", + "negative_test_query252_00005", + "negative_test_query266_00010", + "apositive_test_query300_00001", + "negative_test_query300_00013", + "negative_test_query312_00011", + "negative_test_query141_00001", + "negative_test_query300_00010", + "negative_test_query190_00004", + "negative_test_query300_00005", + "apositive_test_query300_00002", + "negative_test_query300_00008", + "negative_test_query300_00011", + "negative_test_query174_00001", + "negative_test_query167_00018", + "negative_test_query190_00016", + "negative_test_query300_00006", + "negative_test_query185_00001", + "negative_test_query141_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.592512031964586, + "recall@5": 0.75 + }, + "query_id": "test_query301", + "ranked_ids": [ + "negative_test_query301_00011", + "apositive_test_query301_00002", + "apositive_test_query301_00001", + "negative_test_query301_00006", + "apositive_test_query301_00000", + "negative_test_query301_00002", + "negative_test_query301_00001", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query148_00009", + "negative_test_query2_00008", + "apositive_test_query86_00000", + "negative_test_query301_00007", + "negative_test_query301_00013", + "apositive_test_query86_00002", + "negative_test_query301_00015", + "negative_test_query297_00015", + "negative_test_query21_00018", + "apositive_test_query171_00000", + "negative_test_query301_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query302", + "ranked_ids": [ + "apositive_test_query302_00000", + "apositive_test_query302_00003", + "apositive_test_query324_00002", + "negative_test_query308_00003", + "apositive_test_query302_00002", + "negative_test_query302_00010", + "apositive_test_query308_00001", + "apositive_test_query308_00009", + "apositive_test_query302_00005", + "apositive_test_query302_00004", + "negative_test_query308_00008", + "apositive_test_query324_00001", + "apositive_test_query308_00008", + "apositive_test_query302_00001", + "negative_test_query308_00009", + "negative_test_query302_00012", + "negative_test_query308_00005", + "negative_test_query308_00002", + "negative_test_query302_00000", + "negative_test_query302_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query35_00003", + "negative_test_query303_00017", + "apositive_test_query303_00000", + "negative_test_query173_00001", + "negative_test_query316_00009", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query206_00011", + "negative_test_query303_00015", + "negative_test_query206_00009", + "negative_test_query243_00000", + "negative_test_query274_00014", + "negative_test_query307_00006", + "negative_test_query196_00007", + "negative_test_query61_00011", + "negative_test_query8_00013", + "apositive_test_query86_00001", + "negative_test_query303_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query304_00001", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query214_00002", + "negative_test_query43_00001", + "negative_test_query304_00005", + "apositive_test_query304_00003", + "negative_test_query304_00012", + "negative_test_query304_00010", + "negative_test_query304_00004", + "negative_test_query43_00006", + "negative_test_query304_00013", + "negative_test_query304_00007", + "negative_test_query304_00008", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "negative_test_query285_00011", + "negative_test_query234_00010", + "apositive_test_query285_00005", + "negative_test_query304_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query305", + "ranked_ids": [ + "apositive_test_query305_00001", + "negative_test_query305_00014", + "apositive_test_query357_00000", + "apositive_test_query259_00011", + "negative_test_query134_00006", + "apositive_test_query127_00005", + "negative_test_query305_00005", + "negative_test_query65_00013", + "negative_test_query0_00007", + "negative_test_query323_00003", + "negative_test_query56_00004", + "apositive_test_query71_00003", + "negative_test_query0_00004", + "negative_test_query203_00006", + "negative_test_query51_00003", + "negative_test_query47_00001", + "negative_test_query310_00000", + "negative_test_query0_00003", + "negative_test_query224_00015", + "negative_test_query164_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query306", + "ranked_ids": [ + "apositive_test_query71_00002", + "apositive_test_query306_00002", + "negative_test_query71_00013", + "negative_test_query259_00002", + "apositive_test_query119_00006", + "apositive_test_query209_00005", + "negative_test_query209_00008", + "negative_test_query305_00014", + "apositive_test_query71_00003", + "negative_test_query71_00012", + "apositive_test_query30_00002", + "apositive_test_query30_00001", + "negative_test_query204_00001", + "apositive_test_query71_00001", + "negative_test_query182_00014", + "negative_test_query159_00009", + "apositive_test_query289_00003", + "negative_test_query142_00002", + "apositive_test_query127_00007", + "negative_test_query71_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "negative_test_query48_00018", + "apositive_test_query279_00002", + "apositive_test_query170_00005", + "apositive_test_query279_00003", + "negative_test_query70_00010", + "negative_test_query251_00001", + "apositive_test_query3_00003", + "negative_test_query307_00006", + "negative_test_query207_00004", + "apositive_test_query177_00004", + "negative_test_query217_00003", + "negative_test_query103_00012", + "negative_test_query144_00004", + "negative_test_query187_00011", + "negative_test_query346_00017", + "negative_test_query360_00003", + "negative_test_query163_00010", + "negative_test_query132_00008", + "negative_test_query18_00005", + "apositive_test_query13_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query308_00000", + "apositive_test_query308_00002", + "apositive_test_query308_00001", + "apositive_test_query308_00008", + "apositive_test_query324_00001", + "negative_test_query308_00003", + "negative_test_query324_00012", + "apositive_test_query308_00009", + "apositive_test_query324_00000", + "negative_test_query308_00009", + "negative_test_query94_00011", + "apositive_test_query324_00002", + "negative_test_query324_00014", + "negative_test_query308_00007", + "apositive_test_query308_00003", + "apositive_test_query310_00008", + "negative_test_query159_00013", + "apositive_test_query341_00005", + "apositive_test_query7_00000", + "negative_test_query308_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "apositive_test_query309_00000", + "negative_test_query354_00002", + "negative_test_query123_00007", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "negative_test_query309_00005", + "apositive_test_query358_00007", + "negative_test_query309_00003", + "negative_test_query354_00008", + "apositive_test_query161_00008", + "apositive_test_query358_00006", + "apositive_test_query358_00003", + "negative_test_query113_00011", + "negative_test_query81_00002", + "negative_test_query113_00013", + "negative_test_query309_00006", + "negative_test_query42_00009", + "negative_test_query309_00015", + "apositive_test_query76_00002", + "negative_test_query309_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query31_00003", + "negative_test_query313_00007", + "apositive_test_query213_00005", + "apositive_test_query249_00009", + "apositive_test_query31_00000", + "negative_test_query129_00000", + "negative_test_query31_00011", + "negative_test_query38_00003", + "negative_test_query326_00005", + "negative_test_query31_00006", + "negative_test_query31_00013", + "negative_test_query31_00001", + "negative_test_query31_00010", + "negative_test_query31_00015", + "negative_test_query31_00002", + "negative_test_query31_00008", + "negative_test_query31_00007", + "negative_test_query31_00000", + "negative_test_query31_00004", + "apositive_test_query60_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "error": "ranking must be a permutation of 1..16.", + "error_type": "RerankParseError", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query310", + "ranked_ids": [], + "valid": false + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query311_00008", + "negative_test_query311_00005", + "negative_test_query240_00014", + "apositive_test_query311_00000", + "negative_test_query240_00008", + "negative_test_query311_00012", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "apositive_test_query330_00001", + "apositive_test_query330_00002", + "negative_test_query330_00009", + "apositive_test_query5_00002", + "negative_test_query246_00006", + "negative_test_query12_00008", + "negative_test_query126_00008", + "negative_test_query202_00017", + "negative_test_query288_00004", + "negative_test_query151_00008", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "apositive_test_query212_00004", + "negative_test_query237_00000", + "apositive_test_query152_00004", + "apositive_test_query326_00004", + "negative_test_query312_00006", + "negative_test_query246_00015", + "negative_test_query157_00002", + "negative_test_query182_00002", + "apositive_test_query312_00000", + "negative_test_query343_00016", + "negative_test_query212_00003", + "negative_test_query283_00016", + "apositive_test_query98_00000", + "negative_test_query242_00001", + "apositive_test_query237_00001", + "negative_test_query191_00000", + "negative_test_query294_00015", + "negative_test_query206_00010", + "apositive_test_query220_00005", + "negative_test_query305_00015" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query46_00005", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query46_00011", + "negative_test_query46_00003", + "negative_test_query313_00010", + "negative_test_query158_00000", + "negative_test_query56_00001", + "negative_test_query359_00005", + "apositive_test_query142_00005", + "negative_test_query57_00001", + "negative_test_query334_00001", + "negative_test_query336_00001", + "apositive_test_query108_00001", + "negative_test_query313_00002", + "negative_test_query111_00013", + "apositive_test_query155_00003", + "apositive_test_query146_00009", + "negative_test_query119_00005", + "negative_test_query161_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query314_00016", + "apositive_test_query314_00007", + "apositive_test_query314_00005", + "apositive_test_query276_00007", + "apositive_test_query314_00010", + "apositive_test_query314_00012", + "apositive_test_query314_00003", + "apositive_test_query314_00006", + "apositive_test_query314_00011", + "apositive_test_query314_00013", + "apositive_test_query314_00004", + "apositive_test_query314_00002", + "negative_test_query37_00006", + "apositive_test_query3_00009", + "negative_test_query314_00002", + "negative_test_query314_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query315_00000", + "apositive_test_query315_00002", + "negative_test_query333_00007", + "negative_test_query85_00013", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query315_00001", + "apositive_test_query319_00005", + "negative_test_query269_00004", + "apositive_test_query82_00005", + "negative_test_query315_00009", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query85_00003", + "negative_test_query85_00014", + "apositive_test_query147_00004", + "negative_test_query345_00006", + "negative_test_query319_00001", + "negative_test_query85_00005", + "negative_test_query76_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query316", + "ranked_ids": [ + "apositive_test_query316_00000", + "negative_test_query316_00011", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query316_00004", + "apositive_test_query316_00001", + "negative_test_query316_00001", + "negative_test_query316_00000", + "negative_test_query316_00002", + "negative_test_query316_00005", + "negative_test_query316_00016", + "negative_test_query301_00005", + "negative_test_query158_00008", + "negative_test_query101_00007", + "negative_test_query316_00008", + "apositive_test_query216_00000", + "apositive_test_query63_00007", + "apositive_test_query72_00000", + "negative_test_query228_00011", + "negative_test_query322_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00004", + "negative_test_query317_00005", + "negative_test_query317_00014", + "negative_test_query317_00013", + "negative_test_query317_00008", + "apositive_test_query317_00000", + "negative_test_query317_00010", + "apositive_test_query317_00001", + "negative_test_query317_00000", + "negative_test_query317_00012", + "negative_test_query317_00015", + "negative_test_query116_00014", + "negative_test_query55_00014", + "negative_test_query317_00006", + "negative_test_query317_00017", + "negative_test_query317_00016", + "negative_test_query331_00009", + "negative_test_query283_00013", + "negative_test_query271_00002", + "negative_test_query116_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00000", + "apositive_test_query318_00002", + "apositive_test_query318_00001", + "negative_test_query318_00015", + "negative_test_query318_00012", + "negative_test_query73_00012", + "negative_test_query318_00009", + "negative_test_query318_00003", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query318_00000", + "negative_test_query73_00000", + "negative_test_query318_00010", + "negative_test_query318_00008", + "negative_test_query318_00002", + "negative_test_query318_00005", + "negative_test_query73_00005", + "negative_test_query73_00004", + "negative_test_query318_00004", + "apositive_test_query318_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00003", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query82_00003", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query319_00003", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query340_00013", + "negative_test_query319_00009", + "negative_test_query82_00009", + "negative_test_query319_00008", + "negative_test_query82_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query196_00000", + "apositive_test_query32_00002", + "negative_test_query189_00014", + "apositive_test_query136_00003", + "negative_test_query196_00001", + "apositive_test_query357_00000", + "apositive_test_query192_00000", + "apositive_test_query32_00013", + "negative_test_query211_00016", + "negative_test_query55_00011", + "negative_test_query321_00001", + "negative_test_query341_00010", + "negative_test_query211_00005", + "apositive_test_query17_00000", + "negative_test_query83_00001", + "negative_test_query78_00007", + "negative_test_query68_00009", + "apositive_test_query336_00001", + "negative_test_query339_00006", + "negative_test_query55_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query320", + "ranked_ids": [ + "negative_test_query318_00011", + "apositive_test_query320_00003", + "negative_test_query91_00007", + "negative_test_query320_00005", + "negative_test_query100_00001", + "negative_test_query320_00004", + "negative_test_query320_00008", + "negative_test_query230_00001", + "negative_test_query258_00012", + "negative_test_query53_00014", + "negative_test_query344_00010", + "negative_test_query43_00000", + "negative_test_query304_00005", + "negative_test_query212_00006", + "negative_test_query193_00006", + "negative_test_query131_00009", + "apositive_test_query238_00000", + "negative_test_query290_00007", + "apositive_test_query267_00001", + "negative_test_query216_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "negative_test_query321_00000", + "negative_test_query321_00010", + "apositive_test_query321_00000", + "negative_test_query321_00002", + "negative_test_query34_00012", + "negative_test_query141_00009", + "apositive_test_query185_00009", + "negative_test_query352_00005", + "negative_test_query103_00011", + "negative_test_query126_00005", + "negative_test_query136_00004", + "negative_test_query86_00014", + "negative_test_query226_00002", + "negative_test_query126_00011", + "negative_test_query322_00002", + "apositive_test_query217_00006", + "negative_test_query103_00006", + "negative_test_query197_00004", + "negative_test_query15_00013", + "negative_test_query210_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5555555555555556 + }, + "query_id": "test_query322", + "ranked_ids": [ + "apositive_test_query322_00000", + "apositive_test_query322_00005", + "apositive_test_query322_00006", + "apositive_test_query322_00001", + "apositive_test_query322_00002", + "negative_test_query322_00003", + "negative_test_query73_00010", + "negative_test_query322_00006", + "negative_test_query255_00010", + "apositive_test_query228_00001", + "apositive_test_query322_00003", + "negative_test_query322_00009", + "apositive_test_query322_00004", + "negative_test_query322_00002", + "negative_test_query318_00014", + "negative_test_query322_00005", + "apositive_test_query254_00006", + "negative_test_query318_00002", + "negative_test_query322_00010", + "negative_test_query73_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query323", + "ranked_ids": [ + "apositive_test_query323_00002", + "apositive_test_query323_00001", + "apositive_test_query323_00000", + "apositive_test_query323_00006", + "negative_test_query323_00012", + "apositive_test_query130_00001", + "negative_test_query323_00001", + "negative_test_query75_00002", + "negative_test_query323_00007", + "apositive_test_query323_00004", + "negative_test_query323_00000", + "negative_test_query323_00003", + "negative_test_query323_00006", + "negative_test_query345_00011", + "negative_test_query201_00009", + "negative_test_query75_00007", + "negative_test_query209_00004", + "negative_test_query75_00004", + "negative_test_query15_00006", + "negative_test_query255_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9469024295259745, + "recall@5": 1.0 + }, + "query_id": "test_query324", + "ranked_ids": [ + "apositive_test_query324_00000", + "apositive_test_query324_00001", + "negative_test_query308_00003", + "negative_test_query324_00005", + "apositive_test_query324_00002", + "negative_test_query324_00003", + "negative_test_query324_00004", + "negative_test_query324_00006", + "apositive_test_query308_00009", + "apositive_test_query308_00001", + "apositive_test_query308_00003", + "negative_test_query324_00014", + "apositive_test_query308_00000", + "negative_test_query324_00007", + "negative_test_query308_00007", + "negative_test_query218_00003", + "negative_test_query308_00009", + "negative_test_query324_00015", + "negative_test_query324_00016", + "apositive_test_query308_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query325_00007", + "negative_test_query176_00003", + "negative_test_query15_00015", + "negative_test_query15_00017", + "negative_test_query325_00008", + "negative_test_query293_00015", + "negative_test_query325_00013", + "negative_test_query169_00014", + "apositive_test_query293_00002", + "apositive_test_query178_00003", + "negative_test_query293_00000", + "negative_test_query325_00002", + "negative_test_query266_00014", + "apositive_test_query19_00002", + "negative_test_query191_00011", + "negative_test_query15_00005", + "negative_test_query297_00008", + "apositive_test_query246_00001", + "negative_test_query19_00008", + "negative_test_query15_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query326", + "ranked_ids": [ + "negative_test_query178_00012", + "apositive_test_query326_00000", + "apositive_test_query326_00001", + "apositive_test_query326_00003", + "apositive_test_query212_00004", + "negative_test_query17_00011", + "apositive_test_query326_00004", + "negative_test_query156_00013", + "negative_test_query107_00011", + "apositive_test_query52_00002", + "apositive_test_query54_00002", + "negative_test_query265_00007", + "apositive_test_query355_00002", + "apositive_test_query23_00000", + "apositive_test_query294_00001", + "negative_test_query183_00001", + "negative_test_query181_00007", + "negative_test_query81_00003", + "negative_test_query265_00005", + "negative_test_query45_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query327", + "ranked_ids": [ + "apositive_test_query327_00000", + "apositive_test_query327_00001", + "negative_test_query321_00014", + "negative_test_query327_00014", + "negative_test_query327_00000", + "negative_test_query327_00013", + "negative_test_query327_00003", + "negative_test_query303_00010", + "negative_test_query327_00012", + "negative_test_query61_00006", + "apositive_test_query327_00002", + "negative_test_query327_00010", + "negative_test_query61_00003", + "negative_test_query227_00009", + "apositive_test_query271_00000", + "apositive_test_query271_00001", + "negative_test_query195_00007", + "negative_test_query271_00007", + "negative_test_query299_00010", + "negative_test_query335_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query328", + "ranked_ids": [ + "negative_test_query328_00003", + "negative_test_query24_00006", + "apositive_test_query328_00002", + "negative_test_query328_00000", + "apositive_test_query328_00001", + "negative_test_query328_00002", + "apositive_test_query328_00004", + "negative_test_query59_00017", + "negative_test_query138_00013", + "apositive_test_query40_00003", + "apositive_test_query328_00007", + "negative_test_query217_00005", + "negative_test_query328_00007", + "apositive_test_query227_00001", + "negative_test_query92_00009", + "negative_test_query129_00002", + "negative_test_query144_00005", + "negative_test_query328_00006", + "negative_test_query222_00016", + "negative_test_query261_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query329", + "ranked_ids": [ + "apositive_test_query329_00001", + "negative_test_query329_00006", + "negative_test_query329_00009", + "negative_test_query269_00000", + "negative_test_query329_00004", + "negative_test_query329_00001", + "negative_test_query269_00012", + "negative_test_query329_00010", + "negative_test_query62_00017", + "apositive_test_query329_00002", + "negative_test_query329_00011", + "negative_test_query298_00015", + "apositive_test_query7_00001", + "negative_test_query329_00003", + "negative_test_query183_00006", + "negative_test_query329_00002", + "negative_test_query35_00000", + "negative_test_query329_00013", + "negative_test_query202_00016", + "negative_test_query38_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00004", + "apositive_test_query33_00006", + "apositive_test_query33_00004", + "negative_test_query33_00006", + "apositive_test_query154_00007", + "apositive_test_query33_00009", + "apositive_test_query20_00010", + "negative_test_query154_00006", + "negative_test_query220_00010", + "apositive_test_query284_00004", + "negative_test_query161_00008", + "negative_test_query33_00007", + "negative_test_query344_00010", + "apositive_test_query33_00003", + "negative_test_query216_00004", + "apositive_test_query220_00003", + "apositive_test_query154_00002", + "apositive_test_query220_00000", + "negative_test_query356_00006", + "negative_test_query216_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query330", + "ranked_ids": [ + "apositive_test_query330_00002", + "negative_test_query330_00010", + "negative_test_query330_00009", + "negative_test_query330_00008", + "negative_test_query27_00013", + "negative_test_query252_00007", + "apositive_test_query284_00003", + "negative_test_query330_00001", + "negative_test_query325_00016", + "negative_test_query311_00009", + "negative_test_query330_00000", + "negative_test_query27_00018", + "negative_test_query27_00011", + "negative_test_query180_00000", + "apositive_test_query288_00002", + "negative_test_query129_00003", + "apositive_test_query281_00012", + "apositive_test_query60_00014", + "negative_test_query330_00012", + "negative_test_query288_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query331", + "ranked_ids": [ + "apositive_test_query331_00000", + "negative_test_query331_00013", + "negative_test_query250_00016", + "apositive_test_query331_00002", + "negative_test_query331_00000", + "apositive_test_query331_00001", + "negative_test_query113_00000", + "negative_test_query202_00001", + "apositive_test_query223_00001", + "negative_test_query110_00007", + "negative_test_query214_00015", + "negative_test_query331_00012", + "negative_test_query334_00014", + "negative_test_query331_00008", + "negative_test_query251_00009", + "negative_test_query3_00008", + "negative_test_query154_00000", + "negative_test_query312_00012", + "negative_test_query331_00007", + "negative_test_query303_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.6 + }, + "query_id": "test_query332", + "ranked_ids": [ + "negative_test_query202_00010", + "apositive_test_query332_00003", + "apositive_test_query332_00001", + "apositive_test_query332_00002", + "negative_test_query202_00002", + "apositive_test_query348_00008", + "negative_test_query332_00002", + "negative_test_query115_00008", + "negative_test_query245_00001", + "negative_test_query171_00008", + "negative_test_query12_00005", + "negative_test_query145_00010", + "apositive_test_query142_00004", + "negative_test_query30_00010", + "negative_test_query244_00006", + "negative_test_query180_00011", + "apositive_test_query203_00002", + "apositive_test_query162_00005", + "negative_test_query332_00009", + "negative_test_query65_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00002", + "negative_test_query333_00008", + "negative_test_query67_00000", + "negative_test_query246_00000", + "negative_test_query318_00010", + "negative_test_query73_00005", + "negative_test_query318_00005", + "apositive_test_query322_00000", + "negative_test_query333_00005", + "apositive_test_query322_00003", + "negative_test_query73_00008", + "negative_test_query216_00013", + "negative_test_query73_00014", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00005", + "negative_test_query322_00009", + "negative_test_query136_00003", + "apositive_test_query333_00000", + "apositive_test_query228_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query213_00009", + "negative_test_query107_00001", + "negative_test_query155_00010", + "apositive_test_query57_00006", + "negative_test_query111_00017", + "negative_test_query209_00003", + "negative_test_query111_00006", + "negative_test_query111_00005", + "negative_test_query111_00010", + "negative_test_query334_00012", + "negative_test_query37_00000", + "negative_test_query155_00013", + "negative_test_query334_00010", + "apositive_test_query26_00001", + "negative_test_query334_00016", + "negative_test_query102_00015", + "negative_test_query223_00016", + "negative_test_query164_00005", + "apositive_test_query276_00009", + "negative_test_query170_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.6 + }, + "query_id": "test_query335", + "ranked_ids": [ + "apositive_test_query335_00000", + "negative_test_query335_00006", + "apositive_test_query335_00002", + "apositive_test_query335_00003", + "negative_test_query335_00013", + "apositive_test_query335_00001", + "negative_test_query335_00010", + "apositive_test_query335_00004", + "negative_test_query335_00000", + "negative_test_query335_00003", + "negative_test_query335_00004", + "negative_test_query210_00006", + "negative_test_query335_00007", + "negative_test_query321_00013", + "negative_test_query92_00002", + "negative_test_query335_00009", + "negative_test_query335_00011", + "negative_test_query45_00010", + "negative_test_query210_00005", + "negative_test_query335_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "negative_test_query336_00009", + "apositive_test_query50_00005", + "negative_test_query336_00004", + "apositive_test_query50_00015", + "apositive_test_query38_00005", + "negative_test_query336_00007", + "apositive_test_query50_00012", + "negative_test_query336_00014", + "negative_test_query247_00006", + "apositive_test_query186_00002", + "negative_test_query108_00005", + "negative_test_query247_00000", + "negative_test_query186_00012", + "negative_test_query105_00011", + "negative_test_query336_00015", + "apositive_test_query113_00005", + "negative_test_query247_00008", + "apositive_test_query242_00012", + "apositive_test_query186_00000", + "negative_test_query247_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00010", + "apositive_test_query337_00000", + "negative_test_query200_00001", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "apositive_test_query38_00003", + "negative_test_query337_00016", + "negative_test_query337_00000", + "negative_test_query200_00014", + "negative_test_query337_00004", + "negative_test_query200_00007", + "negative_test_query200_00015", + "negative_test_query337_00007", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query159_00013", + "negative_test_query200_00000", + "negative_test_query38_00008", + "negative_test_query337_00001", + "negative_test_query200_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query338", + "ranked_ids": [ + "apositive_test_query338_00002", + "apositive_test_query338_00008", + "apositive_test_query338_00005", + "apositive_test_query338_00001", + "negative_test_query114_00004", + "apositive_test_query338_00000", + "apositive_test_query338_00006", + "apositive_test_query338_00004", + "apositive_test_query338_00003", + "negative_test_query338_00003", + "negative_test_query114_00007", + "negative_test_query81_00006", + "negative_test_query53_00008", + "apositive_test_query114_00005", + "apositive_test_query81_00000", + "negative_test_query95_00004", + "negative_test_query81_00004", + "negative_test_query76_00005", + "apositive_test_query114_00006", + "negative_test_query119_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query339_00001", + "negative_test_query339_00008", + "negative_test_query339_00009", + "negative_test_query339_00004", + "negative_test_query339_00012", + "apositive_test_query339_00000", + "negative_test_query192_00011", + "negative_test_query339_00016", + "negative_test_query96_00016", + "apositive_test_query96_00001", + "negative_test_query339_00017", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query339_00001", + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query339_00006", + "negative_test_query96_00004", + "negative_test_query96_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00000", + "apositive_test_query352_00002", + "negative_test_query34_00002", + "apositive_test_query34_00004", + "apositive_test_query352_00003", + "negative_test_query352_00004", + "negative_test_query34_00013", + "apositive_test_query352_00004", + "negative_test_query34_00008", + "negative_test_query352_00010", + "negative_test_query352_00000", + "negative_test_query34_00000", + "apositive_test_query352_00001", + "apositive_test_query34_00000", + "negative_test_query34_00014", + "apositive_test_query34_00001", + "negative_test_query352_00012", + "negative_test_query34_00009", + "negative_test_query352_00013", + "negative_test_query352_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00004", + "apositive_test_query340_00000", + "negative_test_query88_00006", + "negative_test_query340_00008", + "negative_test_query340_00014", + "apositive_test_query340_00001", + "negative_test_query88_00007", + "apositive_test_query340_00002", + "negative_test_query340_00007", + "negative_test_query340_00012", + "negative_test_query340_00001", + "negative_test_query340_00006", + "negative_test_query88_00000", + "apositive_test_query340_00003", + "negative_test_query340_00011", + "negative_test_query23_00012", + "apositive_test_query174_00012", + "negative_test_query88_00001", + "negative_test_query88_00013", + "apositive_test_query174_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.25 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query96_00005", + "negative_test_query214_00005", + "apositive_test_query341_00003", + "apositive_test_query341_00005", + "negative_test_query341_00003", + "apositive_test_query341_00001", + "negative_test_query341_00000", + "negative_test_query196_00009", + "apositive_test_query341_00002", + "negative_test_query341_00009", + "apositive_test_query32_00007", + "negative_test_query32_00003", + "negative_test_query196_00006", + "negative_test_query73_00014", + "negative_test_query341_00002", + "negative_test_query316_00008", + "negative_test_query96_00013", + "negative_test_query96_00007", + "negative_test_query341_00005", + "negative_test_query101_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query212_00009", + "negative_test_query342_00015", + "negative_test_query342_00009", + "apositive_test_query342_00000", + "negative_test_query68_00005", + "negative_test_query84_00007", + "negative_test_query200_00009", + "negative_test_query49_00002", + "negative_test_query134_00013", + "negative_test_query342_00016", + "negative_test_query342_00003", + "negative_test_query98_00017", + "negative_test_query77_00004", + "negative_test_query78_00015", + "negative_test_query136_00003", + "negative_test_query245_00001", + "negative_test_query324_00015", + "negative_test_query77_00002", + "negative_test_query325_00015", + "negative_test_query98_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query343", + "ranked_ids": [ + "negative_test_query103_00010", + "apositive_test_query343_00001", + "apositive_test_query185_00001", + "negative_test_query343_00008", + "apositive_test_query279_00003", + "apositive_test_query217_00002", + "apositive_test_query217_00004", + "apositive_test_query185_00008", + "apositive_test_query28_00006", + "negative_test_query225_00008", + "apositive_test_query343_00000", + "negative_test_query307_00006", + "apositive_test_query307_00008", + "apositive_test_query279_00002", + "negative_test_query103_00008", + "apositive_test_query136_00004", + "apositive_test_query133_00005", + "apositive_test_query287_00003", + "negative_test_query360_00017", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query282_00008", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query83_00008", + "negative_test_query282_00004", + "negative_test_query344_00005", + "negative_test_query345_00012", + "negative_test_query83_00005", + "negative_test_query223_00016", + "apositive_test_query221_00002", + "negative_test_query344_00004", + "apositive_test_query83_00006", + "negative_test_query344_00002", + "negative_test_query344_00012", + "negative_test_query231_00015", + "negative_test_query344_00000", + "negative_test_query282_00001", + "negative_test_query188_00000", + "negative_test_query282_00000", + "negative_test_query344_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query345_00003", + "negative_test_query345_00007", + "negative_test_query345_00011", + "negative_test_query90_00003", + "negative_test_query9_00002", + "negative_test_query130_00004", + "apositive_test_query345_00004", + "negative_test_query47_00003", + "apositive_test_query345_00001", + "negative_test_query47_00008", + "negative_test_query184_00009", + "negative_test_query259_00007", + "apositive_test_query203_00005", + "apositive_test_query345_00000", + "negative_test_query345_00010", + "apositive_test_query0_00008", + "apositive_test_query90_00001", + "apositive_test_query9_00000", + "negative_test_query221_00002", + "negative_test_query345_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query346_00009", + "negative_test_query346_00002", + "negative_test_query346_00001", + "negative_test_query157_00005", + "negative_test_query49_00007", + "negative_test_query346_00010", + "negative_test_query346_00008", + "apositive_test_query346_00001", + "negative_test_query346_00014", + "negative_test_query346_00012", + "negative_test_query346_00017", + "apositive_test_query346_00000", + "negative_test_query346_00006", + "negative_test_query346_00003", + "negative_test_query174_00001", + "negative_test_query19_00012", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "negative_test_query15_00000", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query347", + "ranked_ids": [ + "apositive_test_query347_00000", + "negative_test_query347_00000", + "apositive_test_query347_00001", + "negative_test_query298_00003", + "negative_test_query347_00014", + "negative_test_query347_00016", + "negative_test_query298_00006", + "negative_test_query347_00001", + "negative_test_query31_00003", + "negative_test_query347_00015", + "negative_test_query251_00007", + "negative_test_query347_00002", + "negative_test_query347_00007", + "negative_test_query347_00006", + "negative_test_query347_00012", + "negative_test_query347_00011", + "negative_test_query347_00010", + "negative_test_query347_00009", + "negative_test_query347_00008", + "negative_test_query347_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query348_00004", + "negative_test_query348_00008", + "negative_test_query348_00006", + "apositive_test_query348_00001", + "apositive_test_query115_00002", + "apositive_test_query348_00009", + "apositive_test_query348_00006", + "apositive_test_query115_00004", + "apositive_test_query227_00002", + "apositive_test_query348_00005", + "negative_test_query115_00002", + "negative_test_query115_00006", + "negative_test_query359_00014", + "apositive_test_query348_00000", + "negative_test_query115_00012", + "negative_test_query351_00010", + "negative_test_query341_00008", + "negative_test_query297_00017", + "apositive_test_query115_00001", + "negative_test_query31_00017" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "negative_test_query253_00007", + "negative_test_query74_00001", + "negative_test_query253_00000", + "negative_test_query349_00015", + "apositive_test_query137_00002", + "apositive_test_query253_00005", + "apositive_test_query253_00008", + "negative_test_query349_00003", + "negative_test_query194_00001", + "negative_test_query312_00016", + "apositive_test_query74_00002", + "negative_test_query124_00013", + "negative_test_query13_00002", + "apositive_test_query74_00003", + "apositive_test_query42_00004", + "negative_test_query253_00009", + "negative_test_query157_00009", + "apositive_test_query253_00009", + "negative_test_query349_00012", + "apositive_test_query355_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6096199500078984, + "recall@5": 0.75 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query139_00000", + "apositive_test_query35_00001", + "apositive_test_query35_00000", + "apositive_test_query35_00003", + "negative_test_query205_00005", + "negative_test_query35_00002", + "negative_test_query139_00016", + "apositive_test_query146_00007", + "negative_test_query35_00012", + "negative_test_query262_00002", + "negative_test_query198_00011", + "negative_test_query123_00006", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query101_00006", + "negative_test_query231_00012", + "negative_test_query124_00007", + "negative_test_query269_00006", + "negative_test_query8_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query350_00015", + "negative_test_query350_00014", + "negative_test_query350_00010", + "negative_test_query120_00007", + "negative_test_query14_00001", + "negative_test_query192_00004", + "negative_test_query350_00013", + "negative_test_query149_00005", + "negative_test_query179_00001", + "negative_test_query192_00005", + "negative_test_query192_00013", + "negative_test_query350_00011", + "negative_test_query316_00006", + "negative_test_query224_00010", + "negative_test_query63_00000", + "negative_test_query350_00017", + "negative_test_query331_00012", + "apositive_test_query33_00008", + "negative_test_query350_00001", + "negative_test_query350_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query351_00000", + "negative_test_query156_00006", + "apositive_test_query299_00000", + "negative_test_query351_00002", + "negative_test_query92_00005", + "negative_test_query351_00006", + "negative_test_query351_00014", + "apositive_test_query311_00000", + "apositive_test_query113_00001", + "apositive_test_query60_00003", + "apositive_test_query60_00000", + "apositive_test_query60_00010", + "negative_test_query234_00009", + "negative_test_query53_00011", + "apositive_test_query218_00004", + "negative_test_query8_00003", + "apositive_test_query335_00004", + "apositive_test_query193_00009", + "apositive_test_query60_00009", + "apositive_test_query60_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query34_00001", + "apositive_test_query352_00003", + "negative_test_query352_00010", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "negative_test_query34_00008", + "apositive_test_query34_00004", + "apositive_test_query352_00002", + "negative_test_query352_00004", + "negative_test_query34_00002", + "apositive_test_query352_00000", + "negative_test_query34_00000", + "apositive_test_query34_00000", + "negative_test_query352_00000", + "apositive_test_query34_00002", + "negative_test_query34_00012", + "negative_test_query352_00008", + "negative_test_query352_00013", + "apositive_test_query352_00001", + "negative_test_query34_00014" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.2 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00009", + "apositive_test_query353_00005", + "apositive_test_query353_00000", + "apositive_test_query204_00012", + "apositive_test_query353_00011", + "negative_test_query353_00002", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00002", + "negative_test_query353_00003", + "negative_test_query108_00002", + "apositive_test_query204_00007", + "apositive_test_query204_00015", + "apositive_test_query353_00007", + "apositive_test_query353_00009", + "negative_test_query175_00012", + "negative_test_query71_00008", + "apositive_test_query51_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.625 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "apositive_test_query354_00006", + "apositive_test_query354_00004", + "apositive_test_query354_00001", + "negative_test_query354_00004", + "negative_test_query309_00014", + "negative_test_query194_00002", + "negative_test_query309_00013", + "negative_test_query358_00003", + "apositive_test_query354_00005", + "negative_test_query309_00000", + "apositive_test_query320_00006", + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query331_00014", + "negative_test_query280_00007", + "negative_test_query148_00009", + "apositive_test_query147_00002", + "negative_test_query354_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query355_00001", + "negative_test_query355_00009", + "negative_test_query355_00007", + "apositive_test_query355_00002", + "apositive_test_query355_00005", + "negative_test_query355_00003", + "apositive_test_query355_00003", + "negative_test_query355_00005", + "negative_test_query355_00008", + "apositive_test_query355_00006", + "negative_test_query355_00002", + "negative_test_query264_00017", + "negative_test_query87_00004", + "apositive_test_query38_00006", + "negative_test_query159_00017", + "negative_test_query104_00001", + "negative_test_query68_00000", + "negative_test_query264_00003", + "negative_test_query215_00001", + "apositive_test_query100_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query356_00002", + "apositive_test_query356_00004", + "apositive_test_query356_00000", + "apositive_test_query66_00003", + "negative_test_query253_00001", + "negative_test_query47_00011", + "apositive_test_query30_00005", + "negative_test_query102_00014", + "negative_test_query15_00006", + "negative_test_query102_00010", + "negative_test_query334_00000", + "apositive_test_query20_00001", + "apositive_test_query33_00000", + "apositive_test_query356_00008", + "apositive_test_query127_00007", + "negative_test_query280_00004", + "apositive_test_query295_00000", + "apositive_test_query203_00006", + "apositive_test_query88_00000", + "apositive_test_query154_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00009", + "apositive_test_query117_00004", + "negative_test_query58_00008", + "apositive_test_query221_00004", + "apositive_test_query119_00002", + "apositive_test_query357_00003", + "apositive_test_query357_00004", + "apositive_test_query221_00000", + "negative_test_query300_00007", + "negative_test_query58_00009", + "apositive_test_query119_00005", + "apositive_test_query119_00012", + "apositive_test_query119_00010", + "apositive_test_query119_00000", + "apositive_test_query314_00000", + "negative_test_query46_00004", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query119_00001", + "apositive_test_query221_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query358", + "ranked_ids": [ + "negative_test_query309_00005", + "negative_test_query123_00007", + "apositive_test_query358_00006", + "apositive_test_query11_00007", + "apositive_test_query309_00000", + "apositive_test_query358_00004", + "apositive_test_query358_00007", + "negative_test_query309_00006", + "apositive_test_query358_00003", + "negative_test_query11_00000", + "apositive_test_query161_00000", + "apositive_test_query358_00000", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "negative_test_query279_00010", + "negative_test_query132_00001", + "negative_test_query354_00008", + "apositive_test_query358_00009", + "negative_test_query309_00015", + "negative_test_query42_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00004", + "negative_test_query359_00002", + "apositive_test_query0_00003", + "apositive_test_query0_00006", + "negative_test_query305_00000", + "negative_test_query359_00007", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "negative_test_query184_00016", + "negative_test_query268_00014", + "apositive_test_query259_00008", + "negative_test_query0_00001", + "apositive_test_query0_00002", + "apositive_test_query0_00010", + "negative_test_query0_00007", + "apositive_test_query0_00000", + "negative_test_query259_00006", + "apositive_test_query0_00007", + "negative_test_query130_00000", + "apositive_test_query259_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query36", + "ranked_ids": [ + "negative_test_query112_00004", + "apositive_test_query286_00006", + "apositive_test_query36_00003", + "negative_test_query112_00016", + "negative_test_query112_00015", + "negative_test_query36_00014", + "negative_test_query112_00010", + "apositive_test_query106_00003", + "apositive_test_query286_00005", + "negative_test_query216_00002", + "negative_test_query36_00001", + "negative_test_query286_00000", + "negative_test_query112_00013", + "apositive_test_query136_00013", + "negative_test_query260_00010", + "negative_test_query260_00009", + "negative_test_query6_00003", + "negative_test_query296_00013", + "negative_test_query80_00007", + "negative_test_query36_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00001", + "negative_test_query189_00014", + "apositive_test_query153_00011", + "apositive_test_query360_00000", + "negative_test_query189_00013", + "apositive_test_query142_00004", + "negative_test_query216_00011", + "negative_test_query360_00004", + "negative_test_query240_00010", + "negative_test_query246_00001", + "negative_test_query312_00004", + "negative_test_query281_00000", + "apositive_test_query69_00014", + "negative_test_query5_00014", + "negative_test_query360_00009", + "negative_test_query360_00005", + "apositive_test_query310_00006", + "apositive_test_query152_00002", + "negative_test_query148_00018", + "negative_test_query175_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.125 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query123_00001", + "apositive_test_query177_00000", + "negative_test_query231_00006", + "negative_test_query209_00001", + "apositive_test_query37_00001", + "negative_test_query155_00001", + "apositive_test_query158_00007", + "negative_test_query209_00000", + "apositive_test_query213_00006", + "negative_test_query131_00011", + "apositive_test_query158_00002", + "negative_test_query223_00010", + "apositive_test_query108_00000", + "negative_test_query57_00009", + "negative_test_query152_00008", + "negative_test_query20_00003", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query209_00002", + "negative_test_query123_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query38", + "ranked_ids": [ + "apositive_test_query38_00003", + "negative_test_query267_00016", + "negative_test_query38_00010", + "negative_test_query267_00015", + "negative_test_query159_00010", + "negative_test_query337_00017", + "negative_test_query267_00001", + "negative_test_query337_00000", + "negative_test_query47_00016", + "apositive_test_query38_00007", + "apositive_test_query38_00000", + "negative_test_query38_00006", + "apositive_test_query38_00002", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query175_00005", + "apositive_test_query247_00005", + "apositive_test_query337_00000", + "negative_test_query79_00002", + "negative_test_query94_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query39", + "ranked_ids": [ + "apositive_test_query39_00003", + "apositive_test_query39_00001", + "apositive_test_query39_00004", + "negative_test_query18_00004", + "negative_test_query39_00000", + "apositive_test_query39_00002", + "negative_test_query39_00004", + "negative_test_query39_00012", + "negative_test_query39_00006", + "negative_test_query39_00007", + "apositive_test_query39_00000", + "negative_test_query39_00009", + "negative_test_query29_00004", + "negative_test_query331_00010", + "negative_test_query66_00008", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query39_00003", + "negative_test_query232_00009", + "negative_test_query232_00017" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.35714285714285715 + }, + "query_id": "test_query4", + "ranked_ids": [ + "apositive_test_query4_00003", + "apositive_test_query4_00000", + "apositive_test_query4_00010", + "apositive_test_query4_00001", + "apositive_test_query4_00012", + "apositive_test_query4_00004", + "negative_test_query266_00009", + "negative_test_query4_00001", + "negative_test_query266_00004", + "apositive_test_query4_00005", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "negative_test_query149_00015", + "negative_test_query312_00011", + "negative_test_query19_00004", + "apositive_test_query19_00000", + "negative_test_query266_00012", + "negative_test_query4_00002", + "negative_test_query15_00000", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00004", + "negative_test_query40_00010", + "negative_test_query40_00013", + "negative_test_query40_00006", + "negative_test_query40_00000", + "negative_test_query40_00007", + "negative_test_query40_00001", + "negative_test_query40_00005", + "negative_test_query40_00003", + "negative_test_query40_00011", + "negative_test_query66_00005", + "negative_test_query40_00009", + "negative_test_query40_00012", + "apositive_test_query40_00002", + "negative_test_query347_00003", + "negative_test_query328_00006", + "negative_test_query106_00009", + "negative_test_query214_00015", + "apositive_test_query223_00001", + "negative_test_query324_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.75369761125927, + "recall@5": 0.75 + }, + "query_id": "test_query41", + "ranked_ids": [ + "apositive_test_query41_00002", + "negative_test_query41_00000", + "apositive_test_query41_00000", + "apositive_test_query41_00001", + "negative_test_query41_00004", + "negative_test_query41_00007", + "negative_test_query41_00010", + "negative_test_query41_00002", + "negative_test_query41_00012", + "negative_test_query41_00006", + "negative_test_query282_00003", + "apositive_test_query249_00012", + "negative_test_query144_00016", + "apositive_test_query41_00003", + "negative_test_query41_00011", + "negative_test_query226_00011", + "negative_test_query41_00014", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query98_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query42_00008", + "negative_test_query42_00002", + "apositive_test_query42_00003", + "negative_test_query42_00000", + "negative_test_query42_00003", + "apositive_test_query42_00001", + "apositive_test_query42_00000", + "negative_test_query42_00001", + "negative_test_query137_00005", + "apositive_test_query42_00004", + "apositive_test_query42_00005", + "negative_test_query42_00005", + "apositive_test_query42_00002", + "negative_test_query42_00010", + "negative_test_query42_00011", + "apositive_test_query276_00012", + "negative_test_query42_00007", + "negative_test_query42_00006", + "apositive_test_query162_00008", + "negative_test_query250_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query43_00000", + "negative_test_query43_00009", + "apositive_test_query43_00002", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query333_00001", + "negative_test_query43_00000", + "negative_test_query179_00006", + "apositive_test_query146_00003", + "apositive_test_query161_00002", + "apositive_test_query276_00001", + "apositive_test_query146_00004", + "apositive_test_query314_00009", + "apositive_test_query161_00005", + "negative_test_query146_00002", + "apositive_test_query153_00008", + "negative_test_query319_00002", + "negative_test_query82_00002", + "apositive_test_query153_00014", + "apositive_test_query218_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query44", + "ranked_ids": [ + "negative_test_query27_00011", + "apositive_test_query44_00001", + "negative_test_query27_00002", + "negative_test_query27_00008", + "negative_test_query330_00009", + "negative_test_query141_00007", + "negative_test_query293_00003", + "negative_test_query293_00015", + "negative_test_query10_00012", + "negative_test_query44_00002", + "apositive_test_query20_00003", + "apositive_test_query293_00000", + "apositive_test_query141_00005", + "negative_test_query180_00000", + "negative_test_query190_00011", + "negative_test_query330_00012", + "negative_test_query21_00009", + "apositive_test_query245_00000", + "negative_test_query293_00000", + "negative_test_query293_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query45_00013", + "apositive_test_query25_00000", + "negative_test_query45_00001", + "apositive_test_query45_00000", + "negative_test_query182_00016", + "negative_test_query45_00004", + "negative_test_query45_00007", + "negative_test_query45_00006", + "negative_test_query45_00005", + "apositive_test_query25_00005", + "negative_test_query143_00009", + "negative_test_query143_00011", + "negative_test_query328_00005", + "negative_test_query45_00003", + "negative_test_query342_00005", + "negative_test_query143_00014", + "negative_test_query143_00004", + "negative_test_query45_00016", + "negative_test_query45_00009", + "apositive_test_query165_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "negative_test_query39_00011", + "negative_test_query56_00001", + "apositive_test_query142_00001", + "apositive_test_query79_00005", + "apositive_test_query146_00013", + "negative_test_query79_00012", + "negative_test_query359_00005", + "negative_test_query46_00011", + "negative_test_query46_00003", + "negative_test_query105_00006", + "negative_test_query168_00003", + "negative_test_query46_00001", + "apositive_test_query186_00001", + "negative_test_query334_00004", + "negative_test_query213_00012", + "negative_test_query313_00010", + "negative_test_query46_00012", + "negative_test_query238_00001", + "negative_test_query108_00003", + "apositive_test_query276_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query47_00001", + "negative_test_query47_00013", + "apositive_test_query201_00002", + "negative_test_query47_00002", + "negative_test_query345_00002", + "apositive_test_query331_00002", + "apositive_test_query201_00004", + "negative_test_query47_00012", + "negative_test_query331_00013", + "apositive_test_query20_00005", + "negative_test_query47_00006", + "negative_test_query195_00005", + "apositive_test_query201_00006", + "negative_test_query331_00014", + "negative_test_query47_00011", + "negative_test_query331_00005", + "negative_test_query328_00007", + "negative_test_query325_00007", + "negative_test_query144_00018", + "negative_test_query41_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "negative_test_query307_00005", + "apositive_test_query28_00013", + "apositive_test_query185_00012", + "apositive_test_query48_00000", + "negative_test_query48_00016", + "negative_test_query48_00009", + "negative_test_query48_00007", + "apositive_test_query185_00004", + "apositive_test_query28_00010", + "apositive_test_query343_00001", + "negative_test_query103_00012", + "negative_test_query48_00018", + "negative_test_query48_00011", + "negative_test_query307_00001", + "negative_test_query48_00012", + "negative_test_query307_00003", + "negative_test_query48_00000", + "negative_test_query48_00004", + "negative_test_query49_00001", + "apositive_test_query307_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query49", + "ranked_ids": [ + "apositive_test_query49_00001", + "negative_test_query49_00002", + "negative_test_query342_00005", + "negative_test_query169_00000", + "negative_test_query176_00016", + "apositive_test_query29_00006", + "apositive_test_query54_00009", + "negative_test_query106_00004", + "negative_test_query49_00001", + "negative_test_query328_00009", + "negative_test_query52_00011", + "negative_test_query156_00007", + "negative_test_query72_00017", + "apositive_test_query19_00001", + "apositive_test_query204_00003", + "negative_test_query15_00008", + "apositive_test_query353_00004", + "negative_test_query49_00007", + "negative_test_query122_00016", + "negative_test_query156_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query5", + "ranked_ids": [ + "apositive_test_query5_00000", + "negative_test_query5_00005", + "negative_test_query246_00002", + "negative_test_query246_00013", + "negative_test_query67_00010", + "apositive_test_query5_00001", + "apositive_test_query5_00002", + "negative_test_query5_00000", + "negative_test_query5_00001", + "negative_test_query5_00014", + "negative_test_query5_00003", + "negative_test_query5_00007", + "negative_test_query5_00010", + "negative_test_query301_00003", + "negative_test_query5_00004", + "negative_test_query275_00001", + "negative_test_query5_00013", + "negative_test_query5_00006", + "negative_test_query67_00003", + "negative_test_query246_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00014", + "apositive_test_query186_00005", + "apositive_test_query50_00003", + "apositive_test_query50_00002", + "apositive_test_query50_00006", + "negative_test_query336_00011", + "apositive_test_query50_00009", + "negative_test_query336_00004", + "apositive_test_query50_00005", + "apositive_test_query50_00008", + "apositive_test_query50_00013", + "apositive_test_query50_00012", + "apositive_test_query50_00015", + "apositive_test_query50_00001", + "negative_test_query66_00002", + "negative_test_query50_00001", + "negative_test_query119_00002", + "apositive_test_query131_00000", + "negative_test_query223_00011", + "negative_test_query131_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.4 + }, + "query_id": "test_query51", + "ranked_ids": [ + "negative_test_query51_00006", + "apositive_test_query51_00000", + "apositive_test_query51_00001", + "apositive_test_query51_00008", + "apositive_test_query51_00005", + "apositive_test_query51_00007", + "negative_test_query51_00000", + "negative_test_query97_00001", + "apositive_test_query51_00006", + "negative_test_query97_00002", + "negative_test_query51_00004", + "negative_test_query97_00006", + "negative_test_query97_00011", + "negative_test_query97_00007", + "negative_test_query209_00008", + "negative_test_query204_00001", + "negative_test_query231_00012", + "apositive_test_query51_00009", + "apositive_test_query267_00001", + "negative_test_query267_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query257_00005", + "apositive_test_query286_00014", + "apositive_test_query286_00013", + "negative_test_query52_00007", + "apositive_test_query286_00006", + "negative_test_query54_00004", + "negative_test_query257_00009", + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00001", + "apositive_test_query257_00002", + "apositive_test_query257_00005", + "apositive_test_query257_00004", + "negative_test_query52_00005", + "apositive_test_query69_00007", + "negative_test_query257_00007", + "apositive_test_query257_00000", + "apositive_test_query52_00001", + "negative_test_query52_00004", + "apositive_test_query257_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00016", + "negative_test_query161_00007", + "negative_test_query169_00003", + "negative_test_query258_00000", + "negative_test_query258_00008", + "negative_test_query53_00000", + "negative_test_query258_00013", + "apositive_test_query217_00004", + "negative_test_query258_00018", + "negative_test_query230_00009", + "apositive_test_query28_00006", + "negative_test_query180_00012", + "negative_test_query150_00014", + "negative_test_query354_00011", + "negative_test_query150_00015", + "negative_test_query354_00001", + "negative_test_query132_00015", + "apositive_test_query248_00005", + "negative_test_query348_00001", + "negative_test_query3_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query54_00006", + "negative_test_query44_00014", + "negative_test_query140_00006", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query56_00011", + "negative_test_query51_00006", + "negative_test_query97_00002", + "apositive_test_query51_00007", + "apositive_test_query51_00008", + "apositive_test_query51_00006", + "apositive_test_query323_00006", + "negative_test_query200_00006", + "apositive_test_query204_00002", + "apositive_test_query335_00002", + "negative_test_query44_00007", + "negative_test_query136_00004", + "apositive_test_query86_00000", + "apositive_test_query156_00000", + "negative_test_query301_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query55", + "ranked_ids": [ + "apositive_test_query55_00000", + "negative_test_query55_00002", + "apositive_test_query55_00001", + "negative_test_query55_00016", + "negative_test_query55_00005", + "apositive_test_query55_00002", + "negative_test_query55_00003", + "negative_test_query55_00012", + "negative_test_query55_00009", + "negative_test_query55_00010", + "negative_test_query234_00011", + "negative_test_query61_00008", + "negative_test_query55_00011", + "negative_test_query55_00006", + "negative_test_query139_00015", + "negative_test_query55_00007", + "negative_test_query55_00013", + "negative_test_query55_00015", + "negative_test_query55_00008", + "negative_test_query55_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.125 + }, + "query_id": "test_query56", + "ranked_ids": [ + "negative_test_query56_00000", + "negative_test_query56_00006", + "negative_test_query56_00005", + "negative_test_query56_00003", + "apositive_test_query56_00005", + "negative_test_query111_00006", + "negative_test_query359_00005", + "apositive_test_query46_00001", + "negative_test_query105_00006", + "negative_test_query46_00011", + "negative_test_query119_00005", + "negative_test_query200_00014", + "negative_test_query56_00002", + "apositive_test_query56_00006", + "negative_test_query56_00004", + "negative_test_query46_00014", + "negative_test_query186_00007", + "negative_test_query102_00001", + "apositive_test_query238_00001", + "negative_test_query337_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query57", + "ranked_ids": [ + "negative_test_query295_00003", + "apositive_test_query57_00005", + "apositive_test_query57_00003", + "apositive_test_query57_00008", + "negative_test_query313_00012", + "apositive_test_query57_00000", + "apositive_test_query57_00006", + "negative_test_query46_00006", + "negative_test_query57_00001", + "negative_test_query111_00010", + "negative_test_query334_00013", + "negative_test_query334_00001", + "apositive_test_query74_00000", + "negative_test_query65_00010", + "apositive_test_query203_00000", + "apositive_test_query146_00000", + "apositive_test_query338_00002", + "negative_test_query57_00010", + "apositive_test_query259_00006", + "negative_test_query198_00016" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "negative_test_query58_00010", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query58_00015", + "negative_test_query308_00004", + "negative_test_query240_00016", + "negative_test_query240_00008", + "apositive_test_query311_00003", + "apositive_test_query311_00004", + "negative_test_query27_00017", + "negative_test_query240_00005", + "negative_test_query57_00004", + "negative_test_query240_00002", + "negative_test_query11_00004", + "negative_test_query311_00012", + "negative_test_query27_00016", + "negative_test_query240_00015", + "negative_test_query240_00004", + "negative_test_query346_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "apositive_test_query59_00000", + "negative_test_query59_00012", + "negative_test_query59_00010", + "negative_test_query59_00006", + "negative_test_query59_00011", + "negative_test_query59_00014", + "negative_test_query116_00000", + "negative_test_query59_00009", + "negative_test_query227_00008", + "negative_test_query59_00018", + "negative_test_query116_00003", + "negative_test_query116_00013", + "negative_test_query59_00015", + "negative_test_query59_00017", + "negative_test_query59_00005", + "negative_test_query81_00003", + "negative_test_query66_00008", + "apositive_test_query104_00000", + "negative_test_query280_00003", + "negative_test_query202_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "error": "ranking must contain exactly 20 items.", + "error_type": "RerankParseError", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query6", + "ranked_ids": [], + "valid": false + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.25 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00007", + "apositive_test_query60_00003", + "negative_test_query243_00004", + "apositive_test_query60_00013", + "apositive_test_query60_00001", + "apositive_test_query60_00000", + "apositive_test_query60_00006", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query218_00007", + "apositive_test_query60_00005", + "apositive_test_query128_00001", + "negative_test_query60_00003", + "negative_test_query304_00003", + "apositive_test_query285_00001", + "apositive_test_query285_00007", + "negative_test_query304_00002", + "negative_test_query60_00001", + "apositive_test_query60_00009", + "apositive_test_query218_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7095272044910244, + "recall@5": 0.75 + }, + "query_id": "test_query61", + "ranked_ids": [ + "apositive_test_query61_00003", + "negative_test_query61_00014", + "negative_test_query173_00003", + "apositive_test_query61_00002", + "apositive_test_query61_00001", + "negative_test_query61_00006", + "negative_test_query61_00010", + "negative_test_query61_00000", + "negative_test_query61_00003", + "negative_test_query61_00015", + "negative_test_query211_00005", + "negative_test_query61_00005", + "negative_test_query129_00005", + "negative_test_query201_00004", + "negative_test_query210_00012", + "apositive_test_query173_00004", + "negative_test_query61_00004", + "apositive_test_query101_00002", + "negative_test_query327_00016", + "negative_test_query61_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00004", + "negative_test_query62_00000", + "negative_test_query62_00002", + "negative_test_query62_00001", + "negative_test_query62_00013", + "negative_test_query62_00008", + "negative_test_query62_00010", + "negative_test_query15_00011", + "negative_test_query62_00003", + "negative_test_query62_00011", + "apositive_test_query62_00001", + "negative_test_query287_00007", + "negative_test_query16_00001", + "negative_test_query250_00003", + "negative_test_query352_00005", + "negative_test_query219_00006", + "negative_test_query88_00015", + "negative_test_query62_00006", + "negative_test_query38_00009", + "negative_test_query222_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00001", + "negative_test_query297_00006", + "negative_test_query63_00002", + "negative_test_query224_00007", + "negative_test_query63_00003", + "negative_test_query63_00001", + "negative_test_query63_00005", + "negative_test_query321_00002", + "apositive_test_query63_00004", + "apositive_test_query63_00006", + "negative_test_query63_00010", + "negative_test_query63_00009", + "apositive_test_query63_00008", + "apositive_test_query141_00001", + "negative_test_query78_00006", + "negative_test_query264_00015", + "apositive_test_query288_00004", + "negative_test_query331_00010", + "negative_test_query241_00011", + "negative_test_query63_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query64_00015", + "negative_test_query205_00009", + "negative_test_query64_00013", + "negative_test_query211_00009", + "negative_test_query64_00009", + "negative_test_query63_00009", + "negative_test_query64_00010", + "negative_test_query64_00016", + "negative_test_query149_00017", + "negative_test_query120_00003", + "negative_test_query120_00000", + "apositive_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query63_00000", + "negative_test_query214_00004", + "apositive_test_query34_00003", + "negative_test_query310_00005", + "negative_test_query352_00001", + "negative_test_query234_00010", + "negative_test_query149_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "negative_test_query65_00004", + "apositive_test_query65_00001", + "apositive_test_query65_00000", + "apositive_test_query134_00001", + "apositive_test_query356_00004", + "negative_test_query23_00006", + "negative_test_query30_00011", + "negative_test_query47_00005", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query268_00012", + "negative_test_query260_00013", + "apositive_test_query62_00000", + "negative_test_query6_00008", + "negative_test_query6_00003", + "negative_test_query261_00003", + "negative_test_query230_00001", + "apositive_test_query260_00000", + "negative_test_query65_00000", + "negative_test_query260_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.5 + }, + "query_id": "test_query66", + "ranked_ids": [ + "apositive_test_query66_00004", + "negative_test_query66_00011", + "apositive_test_query66_00005", + "apositive_test_query66_00003", + "apositive_test_query20_00005", + "apositive_test_query66_00002", + "apositive_test_query20_00001", + "negative_test_query220_00001", + "negative_test_query188_00014", + "negative_test_query220_00010", + "negative_test_query161_00008", + "negative_test_query344_00007", + "apositive_test_query220_00000", + "negative_test_query154_00009", + "negative_test_query161_00003", + "apositive_test_query188_00000", + "apositive_test_query220_00003", + "apositive_test_query188_00003", + "negative_test_query188_00006", + "negative_test_query9_00012" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query67", + "ranked_ids": [ + "negative_test_query67_00005", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "negative_test_query67_00011", + "negative_test_query67_00001", + "apositive_test_query67_00000", + "negative_test_query67_00010", + "apositive_test_query286_00004", + "apositive_test_query286_00010", + "negative_test_query246_00009", + "apositive_test_query286_00007", + "negative_test_query65_00008", + "negative_test_query55_00005", + "negative_test_query260_00013", + "negative_test_query6_00008", + "negative_test_query67_00008", + "negative_test_query246_00016", + "negative_test_query123_00008", + "negative_test_query67_00007", + "negative_test_query67_00015" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query68", + "ranked_ids": [ + "negative_test_query68_00001", + "negative_test_query126_00008", + "apositive_test_query68_00002", + "negative_test_query330_00009", + "negative_test_query68_00013", + "negative_test_query68_00014", + "apositive_test_query68_00000", + "negative_test_query151_00008", + "negative_test_query212_00000", + "apositive_test_query68_00001", + "apositive_test_query330_00001", + "apositive_test_query252_00001", + "negative_test_query288_00004", + "apositive_test_query5_00002", + "apositive_test_query89_00003", + "negative_test_query330_00011", + "negative_test_query311_00005", + "apositive_test_query326_00005", + "negative_test_query68_00016", + "negative_test_query68_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00002", + "apositive_test_query69_00011", + "apositive_test_query257_00003", + "apositive_test_query69_00000", + "apositive_test_query69_00013", + "negative_test_query169_00002", + "apositive_test_query355_00001", + "negative_test_query18_00010", + "apositive_test_query69_00008", + "apositive_test_query69_00010", + "apositive_test_query69_00005", + "apositive_test_query69_00009", + "apositive_test_query54_00003", + "apositive_test_query69_00012", + "apositive_test_query281_00016", + "apositive_test_query107_00007", + "apositive_test_query52_00001", + "apositive_test_query174_00016", + "negative_test_query277_00001", + "apositive_test_query225_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "negative_test_query7_00005", + "apositive_test_query119_00006", + "apositive_test_query212_00002", + "apositive_test_query212_00001", + "negative_test_query306_00008", + "apositive_test_query326_00007", + "negative_test_query159_00009", + "negative_test_query184_00006", + "negative_test_query7_00001", + "negative_test_query159_00004", + "apositive_test_query159_00001", + "negative_test_query159_00010", + "negative_test_query38_00010", + "negative_test_query264_00013", + "negative_test_query290_00006", + "negative_test_query231_00011", + "negative_test_query327_00011", + "apositive_test_query38_00003", + "negative_test_query79_00002", + "negative_test_query84_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00004", + "negative_test_query70_00011", + "apositive_test_query70_00002", + "negative_test_query227_00013", + "negative_test_query227_00006", + "apositive_test_query70_00001", + "negative_test_query70_00012", + "negative_test_query70_00000", + "negative_test_query227_00000", + "negative_test_query227_00012", + "negative_test_query227_00003", + "negative_test_query227_00014", + "negative_test_query70_00006", + "apositive_test_query227_00000", + "negative_test_query283_00014", + "negative_test_query348_00007", + "negative_test_query59_00004", + "apositive_test_query170_00001", + "negative_test_query227_00010", + "negative_test_query70_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query71", + "ranked_ids": [ + "negative_test_query259_00002", + "apositive_test_query71_00003", + "negative_test_query71_00009", + "negative_test_query71_00013", + "negative_test_query71_00001", + "negative_test_query186_00000", + "negative_test_query247_00002", + "negative_test_query71_00015", + "negative_test_query313_00001", + "apositive_test_query71_00001", + "negative_test_query71_00011", + "negative_test_query71_00000", + "apositive_test_query71_00002", + "negative_test_query65_00002", + "apositive_test_query71_00000", + "negative_test_query231_00014", + "negative_test_query71_00005", + "negative_test_query71_00002", + "apositive_test_query30_00005", + "apositive_test_query209_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "apositive_test_query72_00000", + "negative_test_query72_00007", + "negative_test_query72_00008", + "negative_test_query72_00001", + "negative_test_query72_00016", + "negative_test_query72_00003", + "negative_test_query72_00015", + "negative_test_query72_00013", + "negative_test_query55_00005", + "negative_test_query92_00001", + "negative_test_query17_00009", + "apositive_test_query175_00001", + "negative_test_query228_00000", + "negative_test_query72_00006", + "negative_test_query285_00009", + "negative_test_query265_00003", + "negative_test_query185_00000", + "negative_test_query185_00005", + "negative_test_query233_00009", + "negative_test_query72_00011" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query73", + "ranked_ids": [ + "apositive_test_query322_00003", + "negative_test_query73_00008", + "negative_test_query322_00009", + "negative_test_query21_00005", + "negative_test_query73_00010", + "apositive_test_query333_00005", + "apositive_test_query73_00000", + "negative_test_query128_00004", + "apositive_test_query333_00002", + "negative_test_query228_00013", + "negative_test_query333_00005", + "apositive_test_query254_00004", + "negative_test_query233_00015", + "apositive_test_query322_00008", + "negative_test_query246_00000", + "negative_test_query73_00001", + "negative_test_query73_00013", + "negative_test_query216_00013", + "negative_test_query73_00006", + "apositive_test_query333_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query74_00000", + "apositive_test_query74_00001", + "apositive_test_query74_00002", + "negative_test_query74_00001", + "negative_test_query74_00000", + "negative_test_query74_00013", + "negative_test_query74_00015", + "apositive_test_query74_00003", + "negative_test_query74_00014", + "negative_test_query95_00009", + "apositive_test_query114_00001", + "negative_test_query44_00006", + "negative_test_query85_00005", + "apositive_test_query113_00001", + "negative_test_query129_00008", + "apositive_test_query60_00006", + "apositive_test_query238_00003", + "negative_test_query74_00003", + "negative_test_query238_00013", + "negative_test_query203_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query75", + "ranked_ids": [ + "negative_test_query130_00012", + "apositive_test_query75_00000", + "apositive_test_query75_00005", + "apositive_test_query75_00001", + "negative_test_query89_00000", + "negative_test_query75_00006", + "apositive_test_query75_00006", + "negative_test_query112_00018", + "negative_test_query323_00011", + "negative_test_query75_00012", + "negative_test_query82_00004", + "negative_test_query99_00006", + "negative_test_query319_00004", + "negative_test_query67_00017", + "negative_test_query6_00008", + "negative_test_query75_00003", + "negative_test_query209_00004", + "negative_test_query75_00004", + "negative_test_query260_00013", + "apositive_test_query75_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query76", + "ranked_ids": [ + "apositive_test_query76_00004", + "apositive_test_query76_00006", + "negative_test_query76_00010", + "apositive_test_query76_00001", + "negative_test_query258_00008", + "negative_test_query53_00009", + "apositive_test_query76_00000", + "negative_test_query76_00011", + "negative_test_query147_00007", + "negative_test_query76_00009", + "negative_test_query181_00008", + "negative_test_query76_00007", + "negative_test_query53_00001", + "negative_test_query279_00009", + "negative_test_query91_00004", + "negative_test_query309_00011", + "negative_test_query76_00002", + "negative_test_query76_00008", + "apositive_test_query76_00002", + "negative_test_query147_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query98_00014", + "negative_test_query77_00013", + "negative_test_query263_00002", + "negative_test_query329_00015", + "negative_test_query315_00003", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query287_00012", + "negative_test_query14_00010", + "negative_test_query215_00004", + "negative_test_query216_00006", + "negative_test_query87_00016", + "negative_test_query148_00013", + "negative_test_query150_00011", + "negative_test_query216_00012", + "negative_test_query47_00010", + "negative_test_query144_00003", + "negative_test_query342_00007", + "negative_test_query301_00003", + "negative_test_query326_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6508205185601091, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query78", + "ranked_ids": [ + "apositive_test_query78_00000", + "negative_test_query290_00013", + "negative_test_query214_00010", + "negative_test_query290_00006", + "apositive_test_query78_00001", + "negative_test_query78_00016", + "apositive_test_query38_00003", + "negative_test_query78_00007", + "negative_test_query200_00009", + "apositive_test_query326_00000", + "negative_test_query274_00014", + "negative_test_query7_00005", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query272_00001", + "negative_test_query236_00018", + "negative_test_query192_00009", + "negative_test_query144_00000", + "negative_test_query329_00002", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "negative_test_query79_00002", + "apositive_test_query79_00004", + "negative_test_query56_00002", + "apositive_test_query221_00004", + "negative_test_query158_00000", + "negative_test_query79_00013", + "apositive_test_query79_00000", + "apositive_test_query306_00010", + "negative_test_query313_00013", + "apositive_test_query119_00005", + "negative_test_query79_00009", + "negative_test_query303_00016", + "negative_test_query79_00004", + "negative_test_query105_00011", + "apositive_test_query157_00008", + "negative_test_query157_00000", + "apositive_test_query295_00001", + "apositive_test_query113_00002", + "negative_test_query53_00015", + "negative_test_query38_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query8_00004", + "negative_test_query8_00014", + "negative_test_query8_00001", + "negative_test_query8_00011", + "negative_test_query205_00007", + "apositive_test_query8_00000", + "negative_test_query192_00006", + "negative_test_query8_00015", + "apositive_test_query192_00003", + "apositive_test_query243_00005", + "negative_test_query192_00003", + "negative_test_query8_00008", + "apositive_test_query192_00002", + "apositive_test_query243_00010", + "negative_test_query8_00012", + "negative_test_query329_00005", + "negative_test_query8_00000", + "negative_test_query96_00010" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query80_00004", + "negative_test_query179_00005", + "apositive_test_query80_00002", + "apositive_test_query179_00001", + "negative_test_query80_00003", + "apositive_test_query189_00004", + "negative_test_query80_00001", + "negative_test_query80_00012", + "negative_test_query289_00005", + "negative_test_query85_00007", + "negative_test_query80_00009", + "apositive_test_query36_00000", + "negative_test_query273_00005", + "apositive_test_query179_00003", + "apositive_test_query136_00013", + "negative_test_query287_00005", + "negative_test_query179_00006", + "negative_test_query36_00010", + "negative_test_query102_00009", + "negative_test_query270_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query81", + "ranked_ids": [ + "apositive_test_query81_00000", + "negative_test_query81_00005", + "apositive_test_query81_00004", + "apositive_test_query81_00005", + "negative_test_query81_00006", + "apositive_test_query81_00001", + "apositive_test_query81_00008", + "negative_test_query81_00001", + "negative_test_query81_00000", + "negative_test_query81_00008", + "negative_test_query81_00009", + "apositive_test_query81_00006", + "negative_test_query81_00007", + "negative_test_query81_00010", + "negative_test_query81_00004", + "negative_test_query207_00008", + "negative_test_query27_00006", + "apositive_test_query81_00003", + "negative_test_query246_00017", + "negative_test_query145_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00003", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "apositive_test_query319_00004", + "negative_test_query319_00009", + "apositive_test_query82_00004", + "apositive_test_query82_00003", + "negative_test_query82_00009", + "negative_test_query319_00008", + "negative_test_query319_00003", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query319_00006", + "negative_test_query340_00013", + "negative_test_query82_00003", + "apositive_test_query82_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query83_00002", + "apositive_test_query229_00001", + "apositive_test_query83_00004", + "negative_test_query168_00007", + "apositive_test_query168_00000", + "apositive_test_query95_00000", + "apositive_test_query168_00001", + "apositive_test_query146_00014", + "apositive_test_query164_00000", + "apositive_test_query83_00003", + "negative_test_query348_00006", + "negative_test_query0_00006", + "apositive_test_query83_00000", + "apositive_test_query217_00006", + "negative_test_query357_00002", + "negative_test_query279_00012", + "negative_test_query309_00016", + "negative_test_query287_00004", + "negative_test_query28_00005", + "negative_test_query48_00006" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "error": "ranking must be a permutation of 1..20.", + "error_type": "RerankParseError", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query84", + "ranked_ids": [], + "valid": false + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query85_00000", + "negative_test_query85_00012", + "negative_test_query85_00000", + "apositive_test_query85_00002", + "negative_test_query85_00003", + "negative_test_query112_00008", + "apositive_test_query109_00004", + "apositive_test_query85_00003", + "negative_test_query85_00009", + "apositive_test_query85_00001", + "negative_test_query106_00003", + "negative_test_query85_00011", + "negative_test_query85_00001", + "apositive_test_query286_00004", + "apositive_test_query286_00006", + "apositive_test_query81_00008", + "apositive_test_query284_00004", + "apositive_test_query133_00013", + "apositive_test_query287_00004", + "negative_test_query44_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query86", + "ranked_ids": [ + "apositive_test_query86_00003", + "apositive_test_query86_00001", + "negative_test_query86_00013", + "negative_test_query86_00001", + "negative_test_query86_00012", + "negative_test_query310_00007", + "apositive_test_query86_00000", + "apositive_test_query86_00002", + "negative_test_query86_00007", + "negative_test_query86_00002", + "negative_test_query86_00000", + "negative_test_query86_00008", + "negative_test_query299_00007", + "negative_test_query86_00014", + "negative_test_query86_00011", + "negative_test_query65_00013", + "negative_test_query245_00012", + "negative_test_query86_00003", + "negative_test_query210_00012", + "negative_test_query188_00013" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query87", + "ranked_ids": [ + "apositive_test_query87_00001", + "apositive_test_query87_00000", + "negative_test_query87_00004", + "negative_test_query87_00013", + "negative_test_query87_00000", + "negative_test_query87_00007", + "negative_test_query87_00008", + "negative_test_query87_00016", + "negative_test_query87_00014", + "negative_test_query87_00015", + "negative_test_query87_00011", + "negative_test_query87_00010", + "negative_test_query87_00006", + "negative_test_query275_00006", + "negative_test_query315_00006", + "negative_test_query87_00002", + "negative_test_query200_00011", + "negative_test_query182_00017", + "negative_test_query263_00002", + "apositive_test_query302_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "negative_test_query88_00005", + "apositive_test_query88_00001", + "negative_test_query88_00013", + "negative_test_query325_00006", + "negative_test_query88_00003", + "negative_test_query88_00016", + "negative_test_query340_00012", + "negative_test_query88_00002", + "negative_test_query184_00013", + "negative_test_query88_00010", + "negative_test_query88_00008", + "negative_test_query88_00017", + "apositive_test_query250_00002", + "negative_test_query105_00014", + "apositive_test_query147_00002", + "apositive_test_query223_00000", + "negative_test_query240_00009", + "negative_test_query88_00012", + "negative_test_query334_00006", + "negative_test_query203_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query110_00011", + "negative_test_query89_00004", + "negative_test_query326_00006", + "apositive_test_query89_00004", + "negative_test_query89_00008", + "negative_test_query89_00000", + "apositive_test_query75_00006", + "negative_test_query89_00005", + "apositive_test_query89_00000", + "negative_test_query130_00003", + "negative_test_query156_00013", + "negative_test_query233_00005", + "apositive_test_query255_00000", + "negative_test_query278_00009", + "negative_test_query278_00000", + "apositive_test_query68_00001", + "negative_test_query325_00012", + "negative_test_query68_00004", + "negative_test_query212_00000", + "negative_test_query89_00007" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00006", + "negative_test_query9_00008", + "negative_test_query221_00002", + "negative_test_query90_00004", + "apositive_test_query9_00000", + "negative_test_query9_00005", + "negative_test_query9_00001", + "negative_test_query9_00018", + "negative_test_query9_00011", + "negative_test_query90_00009", + "negative_test_query305_00009", + "negative_test_query221_00011", + "apositive_test_query90_00001", + "negative_test_query9_00004", + "negative_test_query9_00013", + "negative_test_query9_00015", + "negative_test_query90_00007", + "negative_test_query9_00016", + "negative_test_query221_00007", + "negative_test_query9_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query231_00002", + "negative_test_query90_00010", + "apositive_test_query90_00000", + "negative_test_query90_00011", + "negative_test_query90_00017", + "negative_test_query310_00007", + "negative_test_query90_00012", + "negative_test_query90_00002", + "negative_test_query127_00001", + "negative_test_query231_00014", + "negative_test_query227_00012", + "negative_test_query9_00008", + "negative_test_query90_00000", + "negative_test_query90_00006", + "negative_test_query90_00015", + "negative_test_query227_00006", + "apositive_test_query304_00000", + "negative_test_query227_00001", + "apositive_test_query14_00002", + "negative_test_query261_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query91", + "ranked_ids": [ + "apositive_test_query91_00002", + "apositive_test_query91_00000", + "negative_test_query279_00011", + "negative_test_query292_00009", + "apositive_test_query91_00006", + "negative_test_query91_00010", + "negative_test_query358_00004", + "negative_test_query319_00002", + "negative_test_query82_00002", + "apositive_test_query11_00006", + "negative_test_query350_00004", + "apositive_test_query157_00001", + "negative_test_query315_00013", + "negative_test_query130_00010", + "negative_test_query118_00015", + "negative_test_query102_00002", + "negative_test_query303_00011", + "apositive_test_query342_00001", + "negative_test_query59_00006", + "apositive_test_query157_00000" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.5 + }, + "query_id": "test_query92", + "ranked_ids": [ + "apositive_test_query92_00003", + "apositive_test_query92_00005", + "negative_test_query92_00003", + "negative_test_query283_00001", + "apositive_test_query92_00001", + "apositive_test_query92_00000", + "apositive_test_query92_00004", + "apositive_test_query92_00002", + "negative_test_query105_00016", + "apositive_test_query198_00001", + "negative_test_query92_00013", + "negative_test_query100_00014", + "apositive_test_query258_00000", + "apositive_test_query99_00003", + "negative_test_query100_00012", + "apositive_test_query100_00001", + "negative_test_query230_00006", + "negative_test_query1_00012", + "negative_test_query122_00018", + "negative_test_query283_00009" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query93_00004", + "apositive_test_query93_00000", + "apositive_test_query243_00006", + "negative_test_query93_00003", + "apositive_test_query93_00003", + "negative_test_query219_00000", + "negative_test_query195_00001", + "negative_test_query277_00003", + "negative_test_query280_00012", + "apositive_test_query335_00003", + "negative_test_query335_00003", + "negative_test_query92_00012", + "apositive_test_query323_00003", + "negative_test_query129_00003", + "negative_test_query5_00015", + "negative_test_query335_00011", + "negative_test_query195_00006", + "apositive_test_query160_00007", + "negative_test_query321_00001", + "negative_test_query108_00003" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query94_00001", + "apositive_test_query127_00006", + "apositive_test_query94_00002", + "apositive_test_query127_00001", + "negative_test_query94_00003", + "apositive_test_query94_00005", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "apositive_test_query127_00004", + "apositive_test_query94_00004", + "apositive_test_query94_00007", + "apositive_test_query94_00000", + "apositive_test_query127_00010", + "negative_test_query94_00005", + "apositive_test_query127_00008", + "negative_test_query94_00002", + "apositive_test_query94_00006", + "negative_test_query155_00008", + "negative_test_query94_00009", + "apositive_test_query127_00015" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "apositive_test_query95_00001", + "apositive_test_query95_00000", + "negative_test_query95_00012", + "negative_test_query95_00005", + "negative_test_query95_00015", + "negative_test_query95_00003", + "apositive_test_query253_00007", + "negative_test_query12_00015", + "negative_test_query95_00002", + "negative_test_query95_00010", + "negative_test_query12_00017", + "negative_test_query95_00013", + "negative_test_query244_00010", + "apositive_test_query177_00000", + "negative_test_query95_00014", + "negative_test_query95_00006", + "negative_test_query57_00007", + "negative_test_query95_00008", + "negative_test_query334_00012", + "apositive_test_query292_00001" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query339_00000", + "negative_test_query192_00011", + "negative_test_query339_00016", + "negative_test_query96_00016", + "negative_test_query339_00017", + "negative_test_query339_00008", + "negative_test_query339_00006", + "negative_test_query339_00004", + "apositive_test_query339_00001", + "negative_test_query96_00013", + "negative_test_query339_00009", + "negative_test_query96_00003", + "apositive_test_query96_00001", + "negative_test_query339_00014", + "negative_test_query339_00001", + "negative_test_query96_00015", + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.5 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "apositive_test_query97_00003", + "apositive_test_query97_00002", + "apositive_test_query97_00001", + "negative_test_query97_00009", + "negative_test_query97_00012", + "apositive_test_query97_00005", + "apositive_test_query289_00003", + "negative_test_query97_00006", + "apositive_test_query51_00000", + "negative_test_query113_00006", + "apositive_test_query51_00002", + "negative_test_query16_00014", + "negative_test_query113_00009", + "apositive_test_query267_00000", + "negative_test_query97_00011", + "negative_test_query84_00008", + "apositive_test_query201_00006", + "negative_test_query209_00008", + "negative_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00004", + "negative_test_query98_00016", + "negative_test_query98_00007", + "apositive_test_query342_00000", + "negative_test_query98_00017", + "negative_test_query231_00014", + "apositive_test_query14_00003", + "negative_test_query98_00011", + "negative_test_query98_00018", + "negative_test_query98_00005", + "negative_test_query129_00003", + "negative_test_query212_00006", + "apositive_test_query14_00002", + "apositive_test_query201_00004", + "negative_test_query197_00007", + "negative_test_query23_00013", + "negative_test_query39_00005", + "negative_test_query300_00000", + "negative_test_query316_00014", + "negative_test_query345_00002" + ], + "valid": true + }, + { + "algorithm": "acurank_k5_b1", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query99", + "ranked_ids": [ + "apositive_test_query193_00008", + "negative_test_query330_00008", + "apositive_test_query193_00012", + "negative_test_query296_00013", + "apositive_test_query193_00010", + "apositive_test_query99_00001", + "negative_test_query174_00002", + "negative_test_query193_00002", + "negative_test_query193_00001", + "negative_test_query193_00005", + "negative_test_query256_00002", + "apositive_test_query347_00002", + "apositive_test_query153_00012", + "apositive_test_query99_00007", + "apositive_test_query333_00003", + "negative_test_query27_00018", + "negative_test_query129_00001", + "apositive_test_query253_00003", + "negative_test_query27_00011", + "negative_test_query129_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.25 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00000", + "negative_test_query9_00017", + "apositive_test_query0_00010", + "apositive_test_query0_00002", + "apositive_test_query107_00004", + "negative_test_query130_00002", + "negative_test_query36_00011", + "apositive_test_query0_00006", + "apositive_test_query0_00003", + "apositive_test_query0_00004", + "negative_test_query359_00002", + "negative_test_query201_00000", + "apositive_test_query153_00004", + "negative_test_query0_00007", + "negative_test_query0_00000", + "negative_test_query0_00001", + "negative_test_query221_00006", + "negative_test_query259_00006", + "negative_test_query357_00000", + "apositive_test_query130_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query1", + "ranked_ids": [ + "negative_test_query1_00009", + "apositive_test_query348_00003", + "negative_test_query256_00005", + "apositive_test_query1_00000", + "negative_test_query61_00013", + "negative_test_query61_00014", + "negative_test_query157_00007", + "negative_test_query61_00002", + "negative_test_query61_00003", + "negative_test_query297_00011", + "negative_test_query184_00015", + "negative_test_query1_00005", + "negative_test_query1_00001", + "negative_test_query331_00002", + "apositive_test_query188_00002", + "negative_test_query202_00001", + "negative_test_query120_00006", + "negative_test_query110_00007", + "negative_test_query27_00017", + "negative_test_query245_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query10", + "ranked_ids": [ + "negative_test_query10_00010", + "apositive_test_query10_00002", + "negative_test_query10_00011", + "negative_test_query10_00000", + "negative_test_query169_00008", + "negative_test_query10_00007", + "apositive_test_query10_00001", + "apositive_test_query10_00000", + "negative_test_query104_00011", + "negative_test_query266_00009", + "apositive_test_query44_00001", + "negative_test_query10_00004", + "negative_test_query10_00009", + "negative_test_query10_00005", + "negative_test_query135_00014", + "negative_test_query83_00000", + "apositive_test_query69_00016", + "apositive_test_query69_00000", + "negative_test_query72_00002", + "apositive_test_query286_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.6 + }, + "query_id": "test_query100", + "ranked_ids": [ + "negative_test_query100_00013", + "apositive_test_query100_00004", + "apositive_test_query100_00003", + "apositive_test_query100_00000", + "negative_test_query100_00010", + "apositive_test_query100_00002", + "negative_test_query100_00014", + "negative_test_query100_00001", + "negative_test_query100_00009", + "negative_test_query290_00015", + "negative_test_query147_00000", + "negative_test_query15_00000", + "negative_test_query237_00010", + "apositive_test_query338_00002", + "negative_test_query258_00017", + "apositive_test_query181_00000", + "apositive_test_query338_00005", + "negative_test_query271_00006", + "negative_test_query252_00005", + "negative_test_query323_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query101", + "ranked_ids": [ + "negative_test_query61_00013", + "negative_test_query53_00000", + "apositive_test_query101_00002", + "negative_test_query35_00004", + "negative_test_query61_00002", + "negative_test_query61_00000", + "apositive_test_query101_00005", + "negative_test_query68_00009", + "negative_test_query103_00009", + "apositive_test_query291_00000", + "apositive_test_query336_00001", + "negative_test_query61_00014", + "negative_test_query191_00014", + "negative_test_query189_00014", + "apositive_test_query73_00002", + "negative_test_query173_00013", + "negative_test_query241_00005", + "negative_test_query258_00000", + "apositive_test_query196_00000", + "negative_test_query134_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query108_00009", + "apositive_test_query238_00000", + "negative_test_query131_00009", + "negative_test_query183_00009", + "negative_test_query313_00016", + "apositive_test_query30_00005", + "negative_test_query183_00012", + "apositive_test_query47_00001", + "negative_test_query203_00007", + "negative_test_query102_00014", + "negative_test_query205_00016", + "negative_test_query209_00006", + "negative_test_query219_00004", + "negative_test_query7_00001", + "negative_test_query232_00010", + "apositive_test_query20_00001", + "negative_test_query47_00011", + "apositive_test_query356_00008", + "negative_test_query90_00016", + "negative_test_query47_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "negative_test_query307_00008", + "negative_test_query103_00003", + "negative_test_query185_00003", + "negative_test_query217_00005", + "negative_test_query48_00017", + "negative_test_query48_00003", + "negative_test_query103_00006", + "apositive_test_query103_00005", + "negative_test_query103_00011", + "apositive_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query103_00000", + "apositive_test_query103_00000", + "apositive_test_query103_00001", + "negative_test_query185_00005", + "negative_test_query222_00009", + "apositive_test_query185_00002", + "negative_test_query48_00014", + "negative_test_query72_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "apositive_test_query104_00000", + "negative_test_query104_00018", + "negative_test_query104_00002", + "negative_test_query104_00011", + "negative_test_query104_00001", + "negative_test_query343_00013", + "negative_test_query104_00012", + "negative_test_query104_00014", + "negative_test_query104_00009", + "negative_test_query104_00016", + "negative_test_query104_00010", + "negative_test_query104_00005", + "negative_test_query104_00007", + "negative_test_query104_00013", + "negative_test_query104_00015", + "negative_test_query104_00003", + "negative_test_query104_00004", + "negative_test_query48_00010", + "apositive_test_query28_00011", + "apositive_test_query48_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query105", + "ranked_ids": [ + "apositive_test_query105_00000", + "negative_test_query105_00011", + "negative_test_query97_00007", + "negative_test_query105_00006", + "negative_test_query142_00010", + "negative_test_query79_00004", + "negative_test_query119_00005", + "negative_test_query56_00002", + "negative_test_query79_00002", + "negative_test_query131_00011", + "negative_test_query105_00005", + "negative_test_query168_00005", + "negative_test_query267_00011", + "apositive_test_query157_00008", + "apositive_test_query83_00004", + "negative_test_query295_00005", + "apositive_test_query158_00002", + "negative_test_query223_00010", + "negative_test_query313_00013", + "negative_test_query359_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query106", + "ranked_ids": [ + "apositive_test_query106_00003", + "apositive_test_query106_00005", + "negative_test_query286_00000", + "negative_test_query106_00012", + "apositive_test_query286_00005", + "negative_test_query326_00008", + "negative_test_query36_00001", + "apositive_test_query106_00000", + "negative_test_query106_00000", + "negative_test_query147_00001", + "apositive_test_query106_00001", + "negative_test_query212_00002", + "negative_test_query106_00007", + "apositive_test_query177_00008", + "negative_test_query162_00007", + "negative_test_query114_00003", + "apositive_test_query42_00006", + "negative_test_query106_00006", + "negative_test_query292_00004", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "negative_test_query46_00012", + "negative_test_query142_00004", + "apositive_test_query142_00004", + "apositive_test_query357_00003", + "negative_test_query146_00000", + "negative_test_query305_00015", + "negative_test_query142_00000", + "apositive_test_query36_00002", + "negative_test_query333_00005", + "apositive_test_query221_00001", + "negative_test_query105_00015", + "negative_test_query31_00013", + "apositive_test_query142_00002", + "negative_test_query105_00008", + "apositive_test_query211_00001", + "negative_test_query296_00001", + "apositive_test_query276_00003", + "negative_test_query160_00001", + "negative_test_query296_00008", + "negative_test_query124_00005" + ], + "retry_attempt": 1, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.25 + }, + "query_id": "test_query108", + "ranked_ids": [ + "apositive_test_query108_00000", + "apositive_test_query108_00007", + "negative_test_query108_00011", + "negative_test_query311_00006", + "negative_test_query108_00007", + "apositive_test_query108_00002", + "apositive_test_query108_00003", + "negative_test_query108_00008", + "negative_test_query313_00012", + "negative_test_query102_00012", + "apositive_test_query108_00005", + "negative_test_query168_00001", + "negative_test_query108_00006", + "negative_test_query57_00009", + "negative_test_query334_00013", + "negative_test_query108_00010", + "negative_test_query108_00001", + "negative_test_query250_00009", + "negative_test_query250_00011", + "negative_test_query102_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.4 + }, + "query_id": "test_query109", + "ranked_ids": [ + "negative_test_query190_00003", + "apositive_test_query109_00003", + "negative_test_query109_00003", + "negative_test_query6_00006", + "apositive_test_query109_00001", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query109_00000", + "negative_test_query109_00014", + "negative_test_query109_00013", + "negative_test_query38_00006", + "negative_test_query51_00004", + "negative_test_query109_00000", + "negative_test_query109_00012", + "negative_test_query268_00010", + "apositive_test_query248_00006", + "negative_test_query109_00009", + "negative_test_query109_00008", + "negative_test_query109_00011", + "negative_test_query109_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.375 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00000", + "apositive_test_query11_00006", + "apositive_test_query11_00001", + "negative_test_query226_00010", + "negative_test_query11_00004", + "apositive_test_query11_00004", + "negative_test_query11_00002", + "negative_test_query358_00005", + "negative_test_query132_00013", + "negative_test_query132_00003", + "negative_test_query309_00002", + "apositive_test_query358_00006", + "apositive_test_query11_00007", + "negative_test_query309_00001", + "negative_test_query309_00004", + "apositive_test_query358_00007", + "apositive_test_query301_00003", + "negative_test_query123_00007", + "negative_test_query358_00001", + "apositive_test_query11_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "negative_test_query89_00006", + "negative_test_query110_00014", + "negative_test_query110_00005", + "negative_test_query110_00013", + "negative_test_query110_00002", + "negative_test_query343_00001", + "apositive_test_query180_00001", + "negative_test_query260_00015", + "apositive_test_query330_00001", + "negative_test_query110_00004", + "negative_test_query17_00008", + "negative_test_query110_00007", + "negative_test_query10_00005", + "apositive_test_query344_00002", + "apositive_test_query11_00002", + "negative_test_query110_00006", + "negative_test_query311_00011", + "apositive_test_query138_00000", + "apositive_test_query242_00006", + "apositive_test_query291_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query111_00015", + "negative_test_query134_00006", + "negative_test_query213_00011", + "negative_test_query111_00018", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query65_00013", + "negative_test_query223_00000", + "apositive_test_query306_00002", + "negative_test_query58_00013", + "negative_test_query111_00008", + "negative_test_query137_00010", + "apositive_test_query162_00005", + "negative_test_query245_00001", + "negative_test_query65_00007", + "apositive_test_query174_00008", + "apositive_test_query308_00003", + "negative_test_query14_00015", + "negative_test_query243_00001", + "negative_test_query292_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "negative_test_query67_00003", + "negative_test_query80_00007", + "negative_test_query112_00004", + "negative_test_query85_00002", + "apositive_test_query286_00006", + "negative_test_query36_00014", + "apositive_test_query36_00003", + "apositive_test_query286_00005", + "negative_test_query112_00015", + "apositive_test_query67_00001", + "negative_test_query112_00013", + "apositive_test_query106_00003", + "negative_test_query216_00002", + "negative_test_query36_00010", + "negative_test_query112_00010", + "negative_test_query246_00016", + "negative_test_query112_00016", + "apositive_test_query246_00001", + "apositive_test_query246_00000", + "negative_test_query6_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query113", + "ranked_ids": [ + "negative_test_query38_00003", + "apositive_test_query113_00001", + "negative_test_query67_00014", + "negative_test_query6_00016", + "apositive_test_query113_00000", + "negative_test_query7_00016", + "apositive_test_query67_00000", + "apositive_test_query186_00005", + "negative_test_query216_00011", + "negative_test_query67_00002", + "apositive_test_query188_00001", + "negative_test_query113_00003", + "apositive_test_query113_00002", + "negative_test_query201_00008", + "negative_test_query105_00007", + "negative_test_query188_00005", + "negative_test_query223_00003", + "apositive_test_query79_00001", + "negative_test_query105_00000", + "negative_test_query47_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query114", + "ranked_ids": [ + "apositive_test_query114_00000", + "apositive_test_query114_00001", + "apositive_test_query114_00003", + "negative_test_query80_00008", + "negative_test_query114_00003", + "negative_test_query349_00003", + "apositive_test_query114_00004", + "negative_test_query262_00007", + "negative_test_query114_00002", + "apositive_test_query124_00004", + "apositive_test_query114_00006", + "apositive_test_query106_00003", + "negative_test_query106_00009", + "negative_test_query114_00008", + "negative_test_query114_00005", + "negative_test_query114_00004", + "negative_test_query114_00007", + "negative_test_query114_00010", + "negative_test_query114_00001", + "negative_test_query81_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query115", + "ranked_ids": [ + "apositive_test_query115_00004", + "apositive_test_query115_00001", + "apositive_test_query115_00000", + "apositive_test_query115_00002", + "negative_test_query115_00006", + "apositive_test_query115_00003", + "apositive_test_query348_00000", + "negative_test_query115_00004", + "negative_test_query115_00002", + "negative_test_query115_00012", + "apositive_test_query348_00009", + "negative_test_query276_00006", + "negative_test_query205_00004", + "apositive_test_query133_00010", + "negative_test_query121_00008", + "negative_test_query115_00003", + "negative_test_query115_00007", + "negative_test_query139_00000", + "apositive_test_query121_00001", + "negative_test_query115_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "apositive_test_query116_00000", + "apositive_test_query49_00000", + "negative_test_query59_00013", + "negative_test_query116_00003", + "negative_test_query116_00011", + "negative_test_query116_00012", + "negative_test_query116_00002", + "negative_test_query98_00005", + "negative_test_query116_00007", + "negative_test_query116_00009", + "negative_test_query116_00014", + "negative_test_query116_00013", + "negative_test_query116_00000", + "negative_test_query116_00015", + "negative_test_query116_00008", + "negative_test_query59_00007", + "negative_test_query116_00004", + "negative_test_query261_00013", + "negative_test_query116_00006", + "negative_test_query116_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.25 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00003", + "negative_test_query117_00010", + "negative_test_query117_00009", + "apositive_test_query354_00003", + "apositive_test_query117_00004", + "apositive_test_query354_00000", + "negative_test_query117_00007", + "apositive_test_query117_00000", + "negative_test_query63_00001", + "negative_test_query117_00004", + "negative_test_query117_00001", + "negative_test_query117_00008", + "negative_test_query117_00003", + "negative_test_query175_00013", + "negative_test_query194_00002", + "apositive_test_query117_00006", + "negative_test_query48_00000", + "apositive_test_query117_00007", + "apositive_test_query134_00004", + "negative_test_query202_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "apositive_test_query155_00005", + "negative_test_query23_00005", + "apositive_test_query174_00010", + "negative_test_query340_00007", + "negative_test_query296_00007", + "apositive_test_query174_00004", + "apositive_test_query174_00005", + "apositive_test_query174_00000", + "apositive_test_query174_00003", + "apositive_test_query310_00001", + "apositive_test_query340_00002", + "negative_test_query296_00010", + "negative_test_query88_00007", + "apositive_test_query217_00000", + "negative_test_query84_00012", + "negative_test_query171_00005", + "negative_test_query171_00016", + "apositive_test_query171_00001", + "apositive_test_query174_00001", + "apositive_test_query174_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query119", + "ranked_ids": [ + "apositive_test_query119_00005", + "negative_test_query119_00005", + "apositive_test_query221_00004", + "apositive_test_query113_00002", + "apositive_test_query119_00001", + "negative_test_query56_00002", + "negative_test_query158_00001", + "negative_test_query152_00004", + "apositive_test_query119_00012", + "apositive_test_query306_00010", + "negative_test_query79_00002", + "apositive_test_query79_00004", + "negative_test_query359_00005", + "negative_test_query157_00001", + "negative_test_query158_00000", + "negative_test_query157_00007", + "negative_test_query119_00002", + "negative_test_query223_00011", + "apositive_test_query26_00000", + "apositive_test_query105_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query12_00004", + "negative_test_query12_00011", + "negative_test_query12_00006", + "negative_test_query12_00014", + "negative_test_query12_00007", + "negative_test_query12_00001", + "negative_test_query12_00003", + "negative_test_query12_00017", + "apositive_test_query12_00000", + "negative_test_query12_00015", + "negative_test_query231_00013", + "apositive_test_query185_00000", + "apositive_test_query217_00005", + "negative_test_query12_00016", + "negative_test_query12_00013", + "negative_test_query12_00008", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query234_00014", + "negative_test_query175_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00008", + "negative_test_query120_00010", + "negative_test_query120_00012", + "apositive_test_query32_00007", + "negative_test_query120_00002", + "negative_test_query1_00009", + "negative_test_query121_00002", + "negative_test_query303_00018", + "negative_test_query148_00014", + "apositive_test_query120_00003", + "negative_test_query352_00013", + "apositive_test_query120_00002", + "negative_test_query120_00001", + "apositive_test_query120_00001", + "negative_test_query196_00006", + "apositive_test_query120_00000", + "apositive_test_query117_00007", + "negative_test_query121_00012", + "negative_test_query120_00006", + "apositive_test_query352_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00005", + "apositive_test_query121_00002", + "negative_test_query121_00008", + "apositive_test_query121_00004", + "apositive_test_query121_00000", + "negative_test_query121_00003", + "apositive_test_query121_00001", + "apositive_test_query121_00003", + "apositive_test_query307_00010", + "negative_test_query121_00000", + "negative_test_query256_00003", + "negative_test_query121_00013", + "negative_test_query121_00012", + "negative_test_query308_00009", + "negative_test_query44_00012", + "negative_test_query53_00003", + "negative_test_query121_00007", + "negative_test_query313_00006", + "negative_test_query115_00009", + "negative_test_query133_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query17_00012", + "apositive_test_query122_00000", + "negative_test_query195_00008", + "apositive_test_query175_00004", + "negative_test_query86_00006", + "negative_test_query210_00003", + "negative_test_query86_00001", + "negative_test_query64_00017", + "apositive_test_query173_00000", + "apositive_test_query134_00000", + "negative_test_query173_00003", + "negative_test_query297_00000", + "negative_test_query179_00014", + "negative_test_query358_00000", + "negative_test_query134_00007", + "negative_test_query61_00015", + "apositive_test_query254_00002", + "negative_test_query179_00002", + "negative_test_query5_00001", + "apositive_test_query173_00001" + ], + "retry_attempt": 1, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query123", + "ranked_ids": [ + "apositive_test_query123_00000", + "negative_test_query123_00013", + "apositive_test_query123_00002", + "negative_test_query123_00005", + "negative_test_query123_00010", + "negative_test_query155_00001", + "apositive_test_query123_00003", + "negative_test_query161_00009", + "negative_test_query155_00009", + "negative_test_query123_00008", + "negative_test_query309_00008", + "negative_test_query123_00007", + "negative_test_query269_00006", + "apositive_test_query123_00004", + "negative_test_query123_00001", + "negative_test_query123_00004", + "negative_test_query340_00012", + "negative_test_query123_00006", + "negative_test_query123_00000", + "negative_test_query123_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "apositive_test_query194_00000", + "apositive_test_query13_00000", + "negative_test_query347_00010", + "negative_test_query42_00005", + "apositive_test_query162_00002", + "negative_test_query262_00011", + "negative_test_query124_00003", + "apositive_test_query162_00001", + "negative_test_query349_00017", + "apositive_test_query13_00004", + "negative_test_query93_00010", + "negative_test_query187_00003", + "negative_test_query123_00009", + "negative_test_query42_00006", + "negative_test_query187_00006", + "negative_test_query309_00012", + "negative_test_query144_00010", + "negative_test_query76_00004", + "negative_test_query171_00002", + "negative_test_query339_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query125", + "ranked_ids": [ + "apositive_test_query125_00000", + "negative_test_query125_00013", + "apositive_test_query286_00009", + "apositive_test_query286_00006", + "apositive_test_query125_00003", + "negative_test_query125_00014", + "negative_test_query125_00006", + "negative_test_query12_00009", + "negative_test_query140_00017", + "negative_test_query360_00016", + "negative_test_query3_00000", + "apositive_test_query273_00003", + "negative_test_query6_00003", + "negative_test_query260_00012", + "negative_test_query125_00003", + "negative_test_query125_00004", + "negative_test_query125_00000", + "negative_test_query125_00009", + "negative_test_query301_00012", + "negative_test_query125_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query126", + "ranked_ids": [ + "apositive_test_query126_00000", + "negative_test_query126_00000", + "negative_test_query190_00017", + "negative_test_query126_00008", + "apositive_test_query151_00002", + "apositive_test_query151_00003", + "negative_test_query151_00006", + "negative_test_query126_00006", + "negative_test_query126_00017", + "negative_test_query126_00007", + "negative_test_query151_00008", + "negative_test_query190_00005", + "apositive_test_query330_00002", + "apositive_test_query5_00002", + "negative_test_query27_00013", + "negative_test_query131_00009", + "apositive_test_query330_00001", + "negative_test_query311_00005", + "negative_test_query220_00000", + "negative_test_query12_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00004", + "apositive_test_query127_00003", + "negative_test_query94_00007", + "apositive_test_query127_00008", + "apositive_test_query127_00000", + "negative_test_query94_00009", + "apositive_test_query127_00012", + "apositive_test_query127_00006", + "negative_test_query94_00003", + "apositive_test_query127_00014", + "apositive_test_query127_00011", + "apositive_test_query127_00009", + "apositive_test_query127_00013", + "negative_test_query198_00008", + "apositive_test_query127_00007", + "apositive_test_query127_00016", + "apositive_test_query94_00005", + "negative_test_query267_00009", + "apositive_test_query163_00000", + "apositive_test_query311_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query128", + "ranked_ids": [ + "apositive_test_query128_00001", + "negative_test_query256_00011", + "apositive_test_query128_00000", + "negative_test_query283_00006", + "negative_test_query310_00009", + "negative_test_query21_00017", + "apositive_test_query216_00002", + "apositive_test_query142_00000", + "apositive_test_query80_00005", + "negative_test_query128_00010", + "negative_test_query8_00017", + "negative_test_query75_00012", + "negative_test_query21_00000", + "apositive_test_query21_00000", + "negative_test_query128_00006", + "apositive_test_query196_00004", + "apositive_test_query32_00001", + "apositive_test_query8_00000", + "negative_test_query128_00011", + "negative_test_query192_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query129", + "ranked_ids": [ + "apositive_test_query129_00000", + "negative_test_query211_00012", + "apositive_test_query129_00001", + "negative_test_query117_00006", + "negative_test_query245_00014", + "apositive_test_query356_00006", + "apositive_test_query244_00001", + "negative_test_query63_00002", + "negative_test_query129_00010", + "negative_test_query165_00012", + "apositive_test_query150_00000", + "negative_test_query156_00017", + "negative_test_query63_00006", + "negative_test_query84_00008", + "apositive_test_query308_00004", + "negative_test_query16_00007", + "negative_test_query187_00012", + "negative_test_query342_00013", + "apositive_test_query155_00002", + "apositive_test_query249_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query13", + "ranked_ids": [ + "negative_test_query13_00001", + "apositive_test_query13_00002", + "apositive_test_query13_00000", + "negative_test_query13_00004", + "apositive_test_query13_00003", + "apositive_test_query13_00004", + "apositive_test_query13_00012", + "apositive_test_query13_00008", + "apositive_test_query13_00010", + "negative_test_query13_00006", + "apositive_test_query13_00006", + "negative_test_query13_00002", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query13_00003", + "negative_test_query13_00000", + "apositive_test_query13_00007", + "apositive_test_query13_00009", + "negative_test_query13_00005", + "negative_test_query245_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query130", + "ranked_ids": [ + "apositive_test_query130_00005", + "negative_test_query284_00001", + "apositive_test_query90_00001", + "apositive_test_query130_00002", + "negative_test_query130_00008", + "apositive_test_query284_00003", + "apositive_test_query323_00004", + "negative_test_query284_00013", + "negative_test_query130_00001", + "negative_test_query198_00011", + "negative_test_query198_00003", + "negative_test_query90_00014", + "negative_test_query284_00009", + "negative_test_query345_00002", + "apositive_test_query284_00001", + "negative_test_query130_00006", + "apositive_test_query233_00000", + "apositive_test_query275_00000", + "apositive_test_query30_00002", + "negative_test_query289_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.44149241373678083, + "recall@5": 0.5 + }, + "query_id": "test_query131", + "ranked_ids": [ + "negative_test_query313_00008", + "apositive_test_query131_00001", + "apositive_test_query131_00000", + "negative_test_query131_00001", + "apositive_test_query50_00008", + "negative_test_query345_00009", + "negative_test_query186_00012", + "negative_test_query247_00004", + "negative_test_query284_00002", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query168_00013", + "negative_test_query184_00012", + "negative_test_query131_00002", + "apositive_test_query106_00000", + "apositive_test_query344_00003", + "negative_test_query357_00001", + "negative_test_query284_00006", + "negative_test_query158_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "negative_test_query132_00006", + "negative_test_query123_00012", + "negative_test_query81_00005", + "negative_test_query81_00001", + "negative_test_query132_00010", + "negative_test_query132_00012", + "negative_test_query132_00016", + "negative_test_query132_00018", + "negative_test_query181_00008", + "negative_test_query132_00004", + "negative_test_query11_00006", + "negative_test_query240_00011", + "apositive_test_query161_00007", + "negative_test_query132_00000", + "negative_test_query304_00000", + "negative_test_query184_00015", + "negative_test_query148_00011", + "apositive_test_query187_00000", + "apositive_test_query3_00004", + "negative_test_query246_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.2 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query287_00001", + "apositive_test_query133_00000", + "apositive_test_query133_00001", + "apositive_test_query133_00014", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query133_00003", + "apositive_test_query133_00002", + "apositive_test_query174_00011", + "apositive_test_query129_00002", + "apositive_test_query129_00007", + "apositive_test_query129_00005", + "apositive_test_query129_00004", + "apositive_test_query91_00002", + "negative_test_query206_00017", + "negative_test_query310_00002", + "negative_test_query59_00008", + "negative_test_query12_00014", + "apositive_test_query129_00006", + "negative_test_query260_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query134", + "ranked_ids": [ + "negative_test_query134_00006", + "apositive_test_query276_00004", + "apositive_test_query134_00002", + "apositive_test_query213_00001", + "negative_test_query107_00003", + "apositive_test_query259_00011", + "negative_test_query134_00000", + "negative_test_query134_00010", + "negative_test_query359_00011", + "negative_test_query108_00003", + "negative_test_query238_00001", + "negative_test_query46_00004", + "negative_test_query134_00008", + "apositive_test_query357_00000", + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query142_00001", + "negative_test_query46_00011", + "negative_test_query218_00009", + "apositive_test_query238_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6509209298071326, + "recall@5": 1.0 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00003", + "apositive_test_query135_00000", + "negative_test_query135_00017", + "apositive_test_query135_00001", + "negative_test_query135_00011", + "negative_test_query135_00004", + "negative_test_query135_00007", + "negative_test_query135_00008", + "negative_test_query325_00012", + "negative_test_query135_00005", + "negative_test_query135_00015", + "negative_test_query297_00017", + "negative_test_query135_00010", + "apositive_test_query133_00011", + "negative_test_query135_00001", + "negative_test_query287_00009", + "negative_test_query135_00014", + "negative_test_query135_00013", + "negative_test_query135_00009", + "apositive_test_query28_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query136", + "ranked_ids": [ + "apositive_test_query136_00004", + "apositive_test_query136_00000", + "apositive_test_query136_00010", + "apositive_test_query136_00002", + "apositive_test_query136_00011", + "apositive_test_query136_00014", + "apositive_test_query136_00009", + "apositive_test_query136_00005", + "negative_test_query301_00001", + "apositive_test_query136_00001", + "negative_test_query136_00003", + "negative_test_query128_00011", + "apositive_test_query136_00007", + "apositive_test_query136_00013", + "negative_test_query129_00001", + "apositive_test_query60_00014", + "negative_test_query60_00002", + "negative_test_query8_00010", + "negative_test_query245_00003", + "negative_test_query149_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.34620964322385606, + "recall@5": 0.5 + }, + "query_id": "test_query137", + "ranked_ids": [ + "negative_test_query253_00007", + "negative_test_query74_00001", + "apositive_test_query137_00002", + "negative_test_query137_00014", + "apositive_test_query137_00000", + "negative_test_query137_00003", + "negative_test_query13_00002", + "apositive_test_query137_00003", + "negative_test_query262_00009", + "negative_test_query349_00006", + "negative_test_query137_00012", + "negative_test_query74_00003", + "negative_test_query74_00012", + "apositive_test_query253_00000", + "negative_test_query95_00004", + "negative_test_query207_00006", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "negative_test_query137_00000", + "negative_test_query149_00013" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query138", + "ranked_ids": [ + "apositive_test_query138_00000", + "negative_test_query138_00014", + "negative_test_query138_00003", + "negative_test_query138_00013", + "negative_test_query138_00004", + "negative_test_query138_00006", + "negative_test_query138_00015", + "negative_test_query138_00005", + "negative_test_query138_00008", + "negative_test_query138_00001", + "apositive_test_query356_00003", + "negative_test_query132_00009", + "negative_test_query213_00006", + "negative_test_query183_00004", + "apositive_test_query138_00001", + "apositive_test_query138_00002", + "apositive_test_query244_00005", + "apositive_test_query23_00003", + "negative_test_query240_00000", + "negative_test_query184_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query35_00001", + "apositive_test_query139_00000", + "apositive_test_query139_00001", + "apositive_test_query35_00000", + "negative_test_query101_00006", + "negative_test_query262_00002", + "negative_test_query231_00012", + "negative_test_query320_00007", + "apositive_test_query146_00007", + "negative_test_query124_00007", + "negative_test_query163_00000", + "negative_test_query35_00010", + "apositive_test_query354_00001", + "negative_test_query198_00011", + "negative_test_query8_00009", + "negative_test_query350_00005", + "negative_test_query298_00008", + "negative_test_query123_00006", + "negative_test_query269_00006", + "negative_test_query81_00002" + ], + "retry_attempt": 2, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query14", + "ranked_ids": [ + "apositive_test_query14_00003", + "apositive_test_query216_00002", + "apositive_test_query77_00000", + "negative_test_query260_00000", + "negative_test_query260_00014", + "negative_test_query14_00001", + "negative_test_query199_00009", + "negative_test_query155_00009", + "apositive_test_query296_00005", + "negative_test_query53_00003", + "negative_test_query155_00001", + "negative_test_query179_00001", + "apositive_test_query64_00000", + "apositive_test_query26_00006", + "negative_test_query14_00007", + "negative_test_query14_00005", + "negative_test_query14_00006", + "negative_test_query14_00004", + "negative_test_query102_00011", + "negative_test_query350_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00003", + "negative_test_query140_00016", + "negative_test_query140_00013", + "negative_test_query140_00005", + "apositive_test_query281_00013", + "negative_test_query140_00010", + "apositive_test_query18_00003", + "negative_test_query140_00001", + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query18_00001", + "negative_test_query140_00014", + "negative_test_query18_00013", + "apositive_test_query273_00005", + "negative_test_query140_00018", + "negative_test_query140_00002", + "negative_test_query257_00012", + "negative_test_query140_00009", + "negative_test_query18_00006", + "negative_test_query140_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query141", + "ranked_ids": [ + "apositive_test_query288_00007", + "negative_test_query105_00014", + "apositive_test_query288_00003", + "apositive_test_query141_00006", + "negative_test_query245_00008", + "apositive_test_query141_00005", + "negative_test_query220_00011", + "apositive_test_query141_00000", + "apositive_test_query288_00005", + "negative_test_query57_00002", + "negative_test_query325_00008", + "negative_test_query141_00010", + "apositive_test_query246_00001", + "negative_test_query245_00016", + "negative_test_query141_00006", + "negative_test_query176_00003", + "negative_test_query25_00006", + "negative_test_query308_00005", + "negative_test_query344_00013", + "negative_test_query245_00013" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query142", + "ranked_ids": [ + "apositive_test_query142_00002", + "negative_test_query105_00008", + "negative_test_query142_00003", + "negative_test_query46_00001", + "apositive_test_query142_00001", + "negative_test_query223_00005", + "negative_test_query238_00001", + "negative_test_query203_00003", + "negative_test_query46_00002", + "negative_test_query105_00006", + "apositive_test_query142_00003", + "negative_test_query359_00005", + "negative_test_query39_00008", + "negative_test_query142_00007", + "negative_test_query337_00016", + "negative_test_query102_00001", + "negative_test_query142_00009", + "negative_test_query46_00012", + "negative_test_query200_00014", + "negative_test_query142_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query143", + "ranked_ids": [ + "apositive_test_query143_00000", + "negative_test_query90_00014", + "negative_test_query143_00015", + "negative_test_query143_00005", + "negative_test_query143_00009", + "negative_test_query143_00006", + "apositive_test_query25_00006", + "apositive_test_query308_00007", + "negative_test_query143_00011", + "negative_test_query143_00013", + "negative_test_query143_00014", + "apositive_test_query25_00005", + "negative_test_query56_00005", + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query143_00000", + "negative_test_query143_00012", + "negative_test_query143_00010", + "negative_test_query143_00007", + "negative_test_query143_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query144_00012", + "negative_test_query41_00001", + "negative_test_query41_00009", + "negative_test_query144_00010", + "negative_test_query144_00018", + "negative_test_query144_00011", + "negative_test_query315_00011", + "negative_test_query144_00006", + "negative_test_query198_00016", + "negative_test_query144_00003", + "negative_test_query144_00017", + "negative_test_query166_00001", + "negative_test_query331_00015", + "negative_test_query144_00016", + "negative_test_query144_00001", + "apositive_test_query41_00000", + "negative_test_query144_00009", + "negative_test_query226_00011", + "negative_test_query41_00011", + "negative_test_query154_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query145", + "ranked_ids": [ + "apositive_test_query145_00002", + "negative_test_query305_00002", + "apositive_test_query213_00004", + "apositive_test_query225_00002", + "negative_test_query112_00007", + "apositive_test_query305_00002", + "apositive_test_query160_00010", + "negative_test_query230_00002", + "apositive_test_query28_00008", + "apositive_test_query107_00001", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "negative_test_query187_00014", + "negative_test_query211_00013", + "negative_test_query53_00012", + "negative_test_query118_00000", + "negative_test_query79_00005", + "negative_test_query171_00003", + "apositive_test_query207_00004", + "negative_test_query190_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.1875 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query146_00010", + "apositive_test_query146_00004", + "apositive_test_query161_00005", + "apositive_test_query146_00005", + "apositive_test_query314_00009", + "negative_test_query65_00006", + "apositive_test_query161_00008", + "apositive_test_query153_00008", + "apositive_test_query276_00001", + "negative_test_query146_00002", + "apositive_test_query314_00004", + "apositive_test_query146_00003", + "apositive_test_query314_00001", + "apositive_test_query314_00003", + "negative_test_query146_00001", + "negative_test_query99_00001", + "apositive_test_query3_00010", + "negative_test_query358_00008", + "negative_test_query144_00008", + "apositive_test_query161_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query147", + "ranked_ids": [ + "apositive_test_query147_00004", + "negative_test_query76_00000", + "apositive_test_query147_00002", + "negative_test_query309_00014", + "apositive_test_query161_00004", + "negative_test_query147_00013", + "negative_test_query258_00008", + "negative_test_query53_00009", + "apositive_test_query76_00001", + "negative_test_query354_00007", + "apositive_test_query147_00003", + "apositive_test_query76_00004", + "negative_test_query147_00008", + "negative_test_query11_00009", + "negative_test_query200_00011", + "negative_test_query309_00016", + "negative_test_query147_00005", + "negative_test_query181_00008", + "negative_test_query17_00009", + "negative_test_query180_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query148_00018", + "negative_test_query124_00010", + "negative_test_query210_00013", + "negative_test_query148_00005", + "negative_test_query148_00012", + "negative_test_query148_00017", + "negative_test_query148_00008", + "negative_test_query321_00013", + "negative_test_query80_00007", + "negative_test_query112_00013", + "negative_test_query196_00004", + "negative_test_query210_00006", + "apositive_test_query304_00001", + "negative_test_query148_00014", + "negative_test_query210_00005", + "negative_test_query89_00007", + "negative_test_query210_00008", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query315_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "apositive_test_query149_00001", + "apositive_test_query149_00000", + "negative_test_query149_00001", + "negative_test_query149_00010", + "negative_test_query149_00002", + "negative_test_query149_00017", + "negative_test_query63_00009", + "negative_test_query149_00012", + "negative_test_query149_00008", + "negative_test_query149_00013", + "negative_test_query332_00004", + "negative_test_query85_00009", + "negative_test_query149_00011", + "apositive_test_query225_00004", + "negative_test_query332_00008", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query149_00015", + "apositive_test_query113_00003", + "negative_test_query74_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00007", + "apositive_test_query15_00001", + "apositive_test_query325_00000", + "negative_test_query15_00005", + "negative_test_query15_00017", + "negative_test_query266_00013", + "negative_test_query15_00011", + "negative_test_query325_00011", + "negative_test_query15_00014", + "negative_test_query266_00006", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query266_00008", + "negative_test_query15_00010", + "negative_test_query15_00012", + "negative_test_query15_00008", + "negative_test_query15_00013", + "negative_test_query176_00011", + "negative_test_query15_00015", + "negative_test_query15_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query150", + "ranked_ids": [ + "apositive_test_query20_00007", + "apositive_test_query150_00003", + "apositive_test_query89_00000", + "apositive_test_query20_00010", + "negative_test_query150_00010", + "negative_test_query250_00013", + "negative_test_query306_00004", + "negative_test_query267_00006", + "apositive_test_query112_00000", + "negative_test_query213_00007", + "negative_test_query63_00004", + "negative_test_query254_00003", + "negative_test_query350_00018", + "apositive_test_query158_00001", + "negative_test_query139_00005", + "negative_test_query1_00012", + "negative_test_query62_00009", + "negative_test_query301_00007", + "negative_test_query346_00008", + "negative_test_query313_00017" + ], + "retry_attempt": 1, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query126_00017", + "negative_test_query126_00007", + "apositive_test_query151_00002", + "negative_test_query190_00017", + "negative_test_query190_00005", + "negative_test_query126_00000", + "negative_test_query126_00006", + "apositive_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query151_00006", + "negative_test_query293_00008", + "negative_test_query44_00002", + "negative_test_query151_00015", + "negative_test_query190_00015", + "negative_test_query151_00004", + "negative_test_query190_00003", + "apositive_test_query300_00001", + "negative_test_query190_00014", + "negative_test_query151_00003", + "negative_test_query201_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query237_00009", + "negative_test_query237_00011", + "apositive_test_query152_00009", + "negative_test_query342_00005", + "negative_test_query262_00010", + "negative_test_query312_00014", + "negative_test_query237_00004", + "apositive_test_query152_00005", + "negative_test_query95_00017", + "apositive_test_query194_00008", + "negative_test_query237_00000", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query152_00004", + "negative_test_query124_00010", + "negative_test_query349_00001", + "negative_test_query349_00005", + "negative_test_query253_00004", + "negative_test_query182_00004", + "negative_test_query250_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.1875 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00015", + "apositive_test_query153_00001", + "negative_test_query112_00007", + "negative_test_query250_00006", + "apositive_test_query153_00003", + "apositive_test_query153_00004", + "apositive_test_query153_00002", + "apositive_test_query153_00007", + "apositive_test_query153_00000", + "negative_test_query359_00003", + "negative_test_query112_00015", + "apositive_test_query213_00004", + "negative_test_query283_00015", + "negative_test_query278_00006", + "negative_test_query278_00007", + "apositive_test_query20_00006", + "negative_test_query228_00002", + "negative_test_query2_00001", + "negative_test_query322_00001", + "apositive_test_query176_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1 + }, + "query_id": "test_query154", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query20_00003", + "apositive_test_query20_00007", + "apositive_test_query154_00005", + "negative_test_query102_00010", + "negative_test_query331_00012", + "apositive_test_query66_00000", + "apositive_test_query23_00004", + "apositive_test_query154_00006", + "apositive_test_query20_00010", + "negative_test_query245_00006", + "negative_test_query356_00004", + "negative_test_query311_00005", + "negative_test_query68_00015", + "apositive_test_query356_00005", + "apositive_test_query33_00000", + "negative_test_query41_00007", + "apositive_test_query358_00008", + "negative_test_query133_00002", + "negative_test_query334_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00009", + "negative_test_query123_00014", + "apositive_test_query3_00008", + "negative_test_query155_00002", + "apositive_test_query155_00005", + "apositive_test_query155_00001", + "negative_test_query155_00004", + "negative_test_query155_00001", + "negative_test_query155_00012", + "negative_test_query155_00003", + "apositive_test_query355_00007", + "negative_test_query155_00006", + "negative_test_query155_00005", + "negative_test_query155_00010", + "negative_test_query119_00001", + "negative_test_query199_00016", + "negative_test_query111_00016", + "negative_test_query155_00000", + "apositive_test_query40_00004", + "negative_test_query99_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "apositive_test_query156_00000", + "apositive_test_query210_00001", + "negative_test_query330_00006", + "negative_test_query156_00001", + "negative_test_query140_00008", + "negative_test_query15_00010", + "apositive_test_query330_00002", + "negative_test_query246_00016", + "apositive_test_query154_00005", + "apositive_test_query25_00009", + "negative_test_query299_00014", + "negative_test_query32_00004", + "negative_test_query92_00010", + "negative_test_query1_00012", + "negative_test_query283_00001", + "negative_test_query92_00003", + "negative_test_query283_00009", + "negative_test_query49_00005", + "apositive_test_query39_00002", + "negative_test_query92_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "negative_test_query237_00004", + "negative_test_query237_00002", + "apositive_test_query152_00001", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query157_00010", + "negative_test_query152_00003", + "negative_test_query271_00003", + "apositive_test_query152_00000", + "negative_test_query237_00009", + "apositive_test_query157_00001", + "apositive_test_query182_00000", + "apositive_test_query237_00002", + "apositive_test_query237_00004", + "negative_test_query312_00017", + "negative_test_query157_00005", + "apositive_test_query237_00000", + "apositive_test_query152_00005", + "negative_test_query152_00008", + "apositive_test_query157_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query158", + "ranked_ids": [ + "negative_test_query102_00003", + "negative_test_query247_00010", + "negative_test_query26_00000", + "negative_test_query102_00015", + "apositive_test_query186_00003", + "negative_test_query158_00006", + "negative_test_query334_00016", + "negative_test_query105_00013", + "negative_test_query213_00002", + "negative_test_query334_00009", + "apositive_test_query158_00003", + "negative_test_query26_00003", + "negative_test_query223_00007", + "negative_test_query26_00004", + "apositive_test_query158_00000", + "negative_test_query223_00002", + "negative_test_query26_00005", + "negative_test_query164_00004", + "negative_test_query26_00010", + "apositive_test_query158_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query159", + "ranked_ids": [ + "negative_test_query159_00005", + "negative_test_query38_00010", + "apositive_test_query212_00002", + "apositive_test_query159_00001", + "apositive_test_query212_00001", + "apositive_test_query326_00007", + "apositive_test_query38_00003", + "negative_test_query264_00013", + "apositive_test_query212_00003", + "negative_test_query159_00016", + "negative_test_query264_00003", + "negative_test_query66_00003", + "negative_test_query7_00005", + "negative_test_query159_00011", + "negative_test_query264_00012", + "apositive_test_query50_00004", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query122_00002", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query16", + "ranked_ids": [ + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query16_00000", + "negative_test_query16_00011", + "negative_test_query308_00004", + "negative_test_query16_00008", + "negative_test_query16_00002", + "apositive_test_query286_00011", + "negative_test_query208_00012", + "apositive_test_query16_00002", + "negative_test_query16_00012", + "negative_test_query280_00007", + "negative_test_query237_00010", + "negative_test_query6_00015", + "apositive_test_query16_00004", + "negative_test_query16_00001", + "negative_test_query16_00010", + "negative_test_query16_00009", + "negative_test_query240_00003", + "negative_test_query16_00003" + ], + "retry_attempt": 1, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query160", + "ranked_ids": [ + "apositive_test_query160_00000", + "apositive_test_query160_00009", + "negative_test_query160_00003", + "negative_test_query214_00013", + "negative_test_query160_00004", + "negative_test_query149_00007", + "apositive_test_query160_00004", + "negative_test_query211_00009", + "negative_test_query64_00009", + "negative_test_query64_00015", + "apositive_test_query64_00000", + "apositive_test_query355_00007", + "apositive_test_query120_00002", + "negative_test_query215_00010", + "negative_test_query106_00004", + "negative_test_query176_00016", + "negative_test_query330_00005", + "negative_test_query271_00010", + "negative_test_query304_00007", + "apositive_test_query260_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query161", + "ranked_ids": [ + "negative_test_query155_00012", + "negative_test_query199_00004", + "apositive_test_query276_00010", + "apositive_test_query3_00006", + "apositive_test_query161_00005", + "apositive_test_query37_00004", + "negative_test_query260_00004", + "apositive_test_query161_00006", + "negative_test_query57_00001", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "negative_test_query301_00015", + "apositive_test_query314_00006", + "apositive_test_query305_00002", + "apositive_test_query161_00008", + "negative_test_query22_00015", + "negative_test_query71_00006", + "apositive_test_query261_00004", + "apositive_test_query314_00004", + "apositive_test_query301_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.25 + }, + "query_id": "test_query162", + "ranked_ids": [ + "negative_test_query177_00002", + "apositive_test_query162_00003", + "apositive_test_query162_00006", + "apositive_test_query114_00001", + "apositive_test_query162_00009", + "apositive_test_query194_00006", + "apositive_test_query194_00003", + "negative_test_query177_00005", + "negative_test_query162_00001", + "negative_test_query240_00008", + "negative_test_query95_00013", + "apositive_test_query194_00009", + "negative_test_query343_00012", + "apositive_test_query162_00010", + "negative_test_query162_00004", + "apositive_test_query162_00000", + "negative_test_query95_00006", + "apositive_test_query162_00005", + "apositive_test_query162_00004", + "negative_test_query230_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query197_00002", + "negative_test_query163_00003", + "negative_test_query163_00008", + "negative_test_query163_00013", + "negative_test_query197_00001", + "apositive_test_query152_00007", + "apositive_test_query357_00004", + "negative_test_query163_00011", + "negative_test_query265_00015", + "negative_test_query202_00002", + "negative_test_query16_00013", + "negative_test_query245_00001", + "negative_test_query350_00017", + "negative_test_query90_00011", + "negative_test_query58_00015", + "negative_test_query328_00006", + "negative_test_query47_00001", + "negative_test_query9_00005", + "apositive_test_query172_00006", + "negative_test_query224_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "negative_test_query158_00008", + "apositive_test_query164_00000", + "negative_test_query164_00015", + "negative_test_query131_00014", + "negative_test_query2_00000", + "negative_test_query1_00003", + "negative_test_query85_00010", + "apositive_test_query146_00014", + "negative_test_query344_00009", + "negative_test_query208_00014", + "negative_test_query321_00011", + "apositive_test_query297_00000", + "negative_test_query332_00001", + "apositive_test_query13_00000", + "negative_test_query165_00003", + "negative_test_query101_00005", + "negative_test_query360_00012", + "negative_test_query100_00013", + "negative_test_query245_00005", + "negative_test_query309_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query122_00003", + "apositive_test_query165_00001", + "negative_test_query165_00013", + "negative_test_query86_00010", + "negative_test_query165_00000", + "negative_test_query149_00012", + "negative_test_query299_00011", + "negative_test_query86_00006", + "negative_test_query165_00007", + "negative_test_query285_00007", + "negative_test_query327_00010", + "apositive_test_query219_00000", + "negative_test_query285_00004", + "negative_test_query348_00001", + "negative_test_query264_00016", + "negative_test_query215_00013", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query327_00016", + "negative_test_query38_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query166", + "ranked_ids": [ + "negative_test_query166_00012", + "apositive_test_query166_00005", + "apositive_test_query166_00002", + "apositive_test_query166_00003", + "apositive_test_query166_00000", + "negative_test_query166_00004", + "negative_test_query166_00013", + "negative_test_query166_00008", + "negative_test_query166_00006", + "negative_test_query249_00000", + "negative_test_query165_00015", + "negative_test_query255_00003", + "negative_test_query166_00000", + "negative_test_query166_00007", + "negative_test_query166_00005", + "apositive_test_query166_00001", + "negative_test_query166_00003", + "negative_test_query166_00001", + "negative_test_query224_00014", + "apositive_test_query224_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query167_00012", + "negative_test_query167_00014", + "negative_test_query278_00016", + "negative_test_query167_00015", + "negative_test_query167_00002", + "negative_test_query167_00003", + "negative_test_query167_00016", + "apositive_test_query288_00005", + "negative_test_query167_00004", + "negative_test_query167_00011", + "negative_test_query167_00007", + "negative_test_query167_00009", + "negative_test_query278_00010", + "negative_test_query141_00010", + "negative_test_query167_00005", + "negative_test_query167_00010", + "negative_test_query167_00001", + "negative_test_query167_00013", + "negative_test_query167_00000", + "negative_test_query167_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6096199500078984, + "recall@5": 0.75 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query83_00003", + "apositive_test_query168_00000", + "apositive_test_query168_00001", + "apositive_test_query168_00002", + "apositive_test_query190_00001", + "negative_test_query6_00001", + "negative_test_query168_00014", + "negative_test_query270_00003", + "negative_test_query6_00005", + "apositive_test_query340_00000", + "negative_test_query168_00007", + "negative_test_query168_00009", + "apositive_test_query286_00007", + "negative_test_query168_00000", + "negative_test_query6_00000", + "negative_test_query83_00003", + "negative_test_query344_00009", + "negative_test_query85_00010", + "apositive_test_query83_00004", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "apositive_test_query69_00000", + "apositive_test_query286_00008", + "apositive_test_query80_00003", + "negative_test_query104_00001", + "apositive_test_query291_00009", + "negative_test_query72_00013", + "apositive_test_query180_00001", + "negative_test_query10_00005", + "negative_test_query180_00004", + "negative_test_query89_00006", + "apositive_test_query242_00006", + "apositive_test_query330_00001", + "negative_test_query10_00013", + "negative_test_query49_00017", + "negative_test_query72_00001", + "negative_test_query72_00016", + "negative_test_query343_00001", + "negative_test_query260_00015", + "apositive_test_query266_00000", + "negative_test_query169_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00000", + "negative_test_query17_00014", + "negative_test_query17_00002", + "negative_test_query17_00016", + "negative_test_query17_00012", + "negative_test_query17_00005", + "negative_test_query130_00007", + "apositive_test_query153_00013", + "apositive_test_query355_00007", + "negative_test_query10_00005", + "negative_test_query148_00001", + "negative_test_query17_00003", + "negative_test_query312_00004", + "negative_test_query292_00010", + "negative_test_query34_00003", + "apositive_test_query225_00007", + "negative_test_query17_00015", + "negative_test_query305_00004", + "negative_test_query302_00011", + "negative_test_query164_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5555555555555556 + }, + "query_id": "test_query170", + "ranked_ids": [ + "apositive_test_query170_00005", + "apositive_test_query170_00006", + "apositive_test_query170_00000", + "apositive_test_query170_00004", + "apositive_test_query170_00002", + "negative_test_query170_00001", + "negative_test_query170_00000", + "negative_test_query170_00007", + "negative_test_query170_00008", + "negative_test_query348_00001", + "negative_test_query227_00002", + "apositive_test_query170_00001", + "negative_test_query170_00004", + "negative_test_query227_00010", + "negative_test_query170_00005", + "apositive_test_query297_00000", + "negative_test_query0_00006", + "negative_test_query297_00005", + "apositive_test_query170_00007", + "apositive_test_query70_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query171", + "ranked_ids": [ + "negative_test_query171_00005", + "apositive_test_query171_00001", + "negative_test_query219_00010", + "negative_test_query171_00004", + "negative_test_query171_00011", + "negative_test_query171_00003", + "negative_test_query219_00002", + "negative_test_query24_00007", + "negative_test_query346_00005", + "negative_test_query277_00007", + "negative_test_query191_00006", + "negative_test_query171_00013", + "negative_test_query171_00000", + "negative_test_query171_00015", + "negative_test_query261_00000", + "apositive_test_query224_00000", + "negative_test_query171_00001", + "negative_test_query126_00013", + "negative_test_query171_00007", + "negative_test_query171_00017" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.375 + }, + "query_id": "test_query172", + "ranked_ids": [ + "apositive_test_query172_00003", + "negative_test_query172_00007", + "apositive_test_query172_00002", + "apositive_test_query172_00004", + "apositive_test_query254_00003", + "negative_test_query192_00010", + "apositive_test_query124_00002", + "apositive_test_query13_00003", + "negative_test_query350_00009", + "negative_test_query192_00000", + "negative_test_query208_00007", + "negative_test_query349_00001", + "apositive_test_query136_00011", + "negative_test_query208_00015", + "negative_test_query173_00006", + "apositive_test_query136_00001", + "negative_test_query8_00000", + "negative_test_query64_00006", + "apositive_test_query136_00009", + "apositive_test_query172_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.5 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query173_00011", + "apositive_test_query173_00004", + "apositive_test_query101_00004", + "apositive_test_query173_00002", + "apositive_test_query173_00000", + "apositive_test_query173_00001", + "negative_test_query64_00017", + "apositive_test_query173_00003", + "negative_test_query64_00004", + "negative_test_query339_00006", + "negative_test_query303_00005", + "negative_test_query64_00013", + "negative_test_query173_00006", + "negative_test_query173_00003", + "negative_test_query64_00006", + "negative_test_query173_00013", + "negative_test_query350_00002", + "negative_test_query173_00000", + "apositive_test_query134_00000", + "negative_test_query222_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00000", + "apositive_test_query174_00012", + "apositive_test_query174_00005", + "apositive_test_query174_00004", + "apositive_test_query174_00003", + "apositive_test_query174_00008", + "apositive_test_query174_00010", + "apositive_test_query174_00011", + "apositive_test_query174_00001", + "apositive_test_query174_00002", + "apositive_test_query174_00009", + "apositive_test_query174_00007", + "apositive_test_query174_00014", + "negative_test_query19_00012", + "apositive_test_query4_00012", + "negative_test_query262_00005", + "apositive_test_query174_00006", + "apositive_test_query174_00015", + "apositive_test_query174_00013", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00004", + "apositive_test_query175_00003", + "negative_test_query204_00002", + "negative_test_query97_00007", + "negative_test_query175_00013", + "negative_test_query175_00010", + "negative_test_query38_00006", + "apositive_test_query204_00010", + "apositive_test_query175_00000", + "apositive_test_query50_00015", + "apositive_test_query353_00005", + "apositive_test_query204_00004", + "negative_test_query105_00016", + "negative_test_query175_00000", + "negative_test_query204_00001", + "apositive_test_query175_00005", + "negative_test_query105_00012", + "apositive_test_query353_00010", + "negative_test_query204_00000", + "negative_test_query7_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query284_00002", + "negative_test_query144_00018", + "apositive_test_query56_00006", + "negative_test_query9_00007", + "negative_test_query176_00015", + "negative_test_query9_00003", + "negative_test_query93_00000", + "negative_test_query284_00006", + "apositive_test_query128_00007", + "negative_test_query195_00012", + "negative_test_query221_00007", + "apositive_test_query344_00003", + "negative_test_query24_00014", + "negative_test_query35_00005", + "negative_test_query195_00006", + "negative_test_query224_00003", + "negative_test_query191_00008", + "negative_test_query200_00005", + "negative_test_query337_00005", + "apositive_test_query92_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.1 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query172_00008", + "negative_test_query177_00006", + "negative_test_query177_00008", + "negative_test_query152_00005", + "apositive_test_query177_00004", + "negative_test_query124_00012", + "negative_test_query124_00001", + "negative_test_query147_00011", + "negative_test_query177_00004", + "apositive_test_query177_00006", + "negative_test_query147_00002", + "negative_test_query147_00005", + "apositive_test_query230_00005", + "apositive_test_query229_00008", + "apositive_test_query162_00009", + "apositive_test_query162_00010", + "negative_test_query147_00009", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query178", + "ranked_ids": [ + "apositive_test_query178_00004", + "negative_test_query178_00007", + "negative_test_query178_00000", + "negative_test_query178_00010", + "apositive_test_query285_00006", + "negative_test_query178_00003", + "negative_test_query178_00001", + "apositive_test_query178_00005", + "negative_test_query178_00012", + "negative_test_query165_00001", + "apositive_test_query60_00011", + "negative_test_query321_00017", + "negative_test_query285_00008", + "negative_test_query285_00009", + "negative_test_query274_00001", + "negative_test_query214_00002", + "negative_test_query43_00001", + "negative_test_query31_00004", + "apositive_test_query352_00001", + "negative_test_query327_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "apositive_test_query32_00012", + "negative_test_query14_00001", + "negative_test_query191_00003", + "negative_test_query179_00001", + "negative_test_query179_00014", + "apositive_test_query173_00003", + "negative_test_query241_00006", + "negative_test_query316_00001", + "negative_test_query191_00011", + "negative_test_query196_00011", + "apositive_test_query316_00001", + "negative_test_query279_00010", + "negative_test_query173_00013", + "negative_test_query260_00012", + "negative_test_query64_00003", + "apositive_test_query173_00000", + "negative_test_query206_00004", + "negative_test_query303_00005", + "negative_test_query297_00000", + "negative_test_query206_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query18", + "ranked_ids": [ + "apositive_test_query18_00003", + "apositive_test_query18_00001", + "negative_test_query140_00003", + "negative_test_query18_00006", + "negative_test_query18_00011", + "apositive_test_query281_00013", + "apositive_test_query18_00000", + "negative_test_query18_00012", + "apositive_test_query18_00002", + "apositive_test_query18_00004", + "negative_test_query257_00003", + "negative_test_query18_00013", + "negative_test_query88_00002", + "negative_test_query215_00000", + "apositive_test_query288_00004", + "negative_test_query156_00018", + "negative_test_query257_00012", + "negative_test_query287_00004", + "negative_test_query88_00012", + "negative_test_query300_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9674679834891693, + "recall@5": 1.0 + }, + "query_id": "test_query180", + "ranked_ids": [ + "apositive_test_query180_00002", + "apositive_test_query180_00000", + "negative_test_query180_00007", + "apositive_test_query180_00001", + "negative_test_query180_00004", + "negative_test_query180_00016", + "negative_test_query112_00014", + "negative_test_query207_00007", + "negative_test_query176_00001", + "apositive_test_query262_00000", + "negative_test_query169_00015", + "apositive_test_query262_00002", + "negative_test_query180_00002", + "apositive_test_query262_00003", + "negative_test_query312_00018", + "negative_test_query180_00011", + "negative_test_query246_00003", + "negative_test_query180_00006", + "negative_test_query114_00003", + "negative_test_query95_00014" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00012", + "negative_test_query181_00006", + "negative_test_query181_00017", + "negative_test_query181_00005", + "negative_test_query181_00010", + "apositive_test_query256_00003", + "negative_test_query181_00008", + "negative_test_query181_00018", + "negative_test_query181_00007", + "negative_test_query181_00009", + "negative_test_query181_00002", + "negative_test_query96_00006", + "negative_test_query320_00007", + "negative_test_query183_00014", + "negative_test_query181_00014", + "negative_test_query181_00004", + "negative_test_query181_00001", + "negative_test_query181_00016", + "negative_test_query348_00007", + "apositive_test_query119_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "apositive_test_query182_00000", + "negative_test_query182_00017", + "negative_test_query182_00011", + "negative_test_query182_00004", + "apositive_test_query152_00009", + "negative_test_query182_00002", + "negative_test_query237_00004", + "negative_test_query237_00006", + "negative_test_query77_00002", + "negative_test_query342_00003", + "negative_test_query342_00009", + "negative_test_query182_00018", + "negative_test_query240_00001", + "negative_test_query337_00012", + "negative_test_query230_00008", + "negative_test_query296_00010", + "negative_test_query237_00001", + "negative_test_query237_00007", + "negative_test_query342_00001", + "negative_test_query342_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00014", + "negative_test_query183_00009", + "negative_test_query183_00012", + "negative_test_query183_00015", + "negative_test_query7_00007", + "negative_test_query183_00018", + "negative_test_query7_00008", + "apositive_test_query7_00001", + "negative_test_query183_00013", + "negative_test_query183_00007", + "negative_test_query183_00008", + "negative_test_query183_00000", + "negative_test_query112_00017", + "negative_test_query183_00006", + "negative_test_query238_00007", + "negative_test_query7_00009", + "negative_test_query7_00014", + "apositive_test_query178_00001", + "negative_test_query183_00010", + "negative_test_query7_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "negative_test_query216_00014", + "negative_test_query158_00008", + "apositive_test_query20_00010", + "negative_test_query216_00002", + "negative_test_query156_00001", + "apositive_test_query216_00001", + "negative_test_query41_00013", + "negative_test_query216_00008", + "negative_test_query344_00010", + "apositive_test_query216_00000", + "negative_test_query216_00007", + "apositive_test_query72_00000", + "apositive_test_query216_00004", + "apositive_test_query33_00006", + "negative_test_query101_00007", + "apositive_test_query316_00000", + "apositive_test_query33_00004", + "negative_test_query228_00011", + "negative_test_query322_00000", + "negative_test_query216_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query185", + "ranked_ids": [ + "negative_test_query185_00005", + "negative_test_query48_00017", + "negative_test_query103_00001", + "negative_test_query72_00017", + "apositive_test_query185_00002", + "negative_test_query72_00009", + "negative_test_query48_00009", + "negative_test_query185_00000", + "negative_test_query185_00002", + "negative_test_query350_00007", + "negative_test_query72_00011", + "negative_test_query139_00014", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query72_00012", + "apositive_test_query307_00008", + "apositive_test_query185_00003", + "negative_test_query103_00000", + "apositive_test_query43_00006", + "negative_test_query48_00014" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "negative_test_query247_00012", + "negative_test_query247_00007", + "negative_test_query186_00005", + "apositive_test_query50_00000", + "apositive_test_query50_00010", + "apositive_test_query306_00003", + "apositive_test_query79_00005", + "apositive_test_query50_00001", + "negative_test_query334_00017", + "negative_test_query313_00015", + "negative_test_query313_00014", + "negative_test_query46_00001", + "negative_test_query47_00009", + "negative_test_query336_00008", + "negative_test_query142_00009", + "negative_test_query29_00003", + "negative_test_query313_00005", + "negative_test_query105_00001", + "negative_test_query259_00003", + "negative_test_query57_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query176_00008", + "negative_test_query184_00015", + "negative_test_query279_00002", + "negative_test_query13_00005", + "apositive_test_query33_00008", + "apositive_test_query66_00000", + "negative_test_query245_00015", + "negative_test_query356_00004", + "negative_test_query245_00006", + "negative_test_query306_00004", + "negative_test_query230_00000", + "apositive_test_query358_00008", + "negative_test_query168_00002", + "apositive_test_query154_00005", + "negative_test_query331_00002", + "apositive_test_query20_00010", + "apositive_test_query311_00002", + "apositive_test_query176_00000", + "negative_test_query62_00009", + "apositive_test_query127_00014" + ], + "retry_attempt": 2, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "apositive_test_query188_00000", + "negative_test_query112_00017", + "negative_test_query220_00011", + "negative_test_query344_00013", + "negative_test_query344_00007", + "negative_test_query188_00009", + "negative_test_query20_00006", + "negative_test_query188_00013", + "apositive_test_query220_00003", + "negative_test_query188_00008", + "apositive_test_query146_00009", + "negative_test_query188_00010", + "negative_test_query161_00010", + "apositive_test_query20_00006", + "apositive_test_query20_00009", + "negative_test_query188_00003", + "negative_test_query188_00004", + "negative_test_query188_00011", + "negative_test_query188_00014", + "apositive_test_query220_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00001", + "apositive_test_query189_00001", + "negative_test_query189_00004", + "apositive_test_query189_00003", + "negative_test_query189_00007", + "negative_test_query189_00013", + "apositive_test_query85_00002", + "negative_test_query260_00007", + "negative_test_query140_00016", + "negative_test_query189_00002", + "negative_test_query18_00006", + "negative_test_query287_00000", + "apositive_test_query189_00002", + "negative_test_query189_00006", + "negative_test_query189_00000", + "negative_test_query85_00007", + "negative_test_query44_00005", + "negative_test_query55_00003", + "negative_test_query190_00001", + "apositive_test_query254_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query19", + "ranked_ids": [ + "apositive_test_query19_00000", + "negative_test_query19_00003", + "apositive_test_query4_00010", + "apositive_test_query4_00000", + "negative_test_query169_00014", + "apositive_test_query174_00007", + "negative_test_query292_00004", + "negative_test_query4_00001", + "negative_test_query229_00002", + "negative_test_query106_00004", + "negative_test_query106_00007", + "negative_test_query15_00000", + "negative_test_query19_00006", + "negative_test_query19_00008", + "negative_test_query266_00010", + "apositive_test_query174_00014", + "negative_test_query141_00000", + "negative_test_query174_00001", + "negative_test_query141_00001", + "negative_test_query167_00018" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "negative_test_query260_00007", + "negative_test_query85_00014", + "apositive_test_query85_00000", + "negative_test_query109_00006", + "negative_test_query85_00006", + "apositive_test_query260_00003", + "negative_test_query233_00008", + "negative_test_query271_00005", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query181_00003", + "negative_test_query246_00002", + "negative_test_query265_00004", + "negative_test_query205_00016", + "apositive_test_query311_00000", + "negative_test_query6_00007", + "negative_test_query345_00006", + "apositive_test_query234_00001", + "negative_test_query190_00013", + "negative_test_query264_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00001", + "negative_test_query181_00013", + "apositive_test_query191_00000", + "negative_test_query191_00010", + "negative_test_query256_00009", + "negative_test_query191_00003", + "apositive_test_query63_00003", + "negative_test_query297_00002", + "negative_test_query241_00007", + "negative_test_query191_00011", + "apositive_test_query191_00002", + "negative_test_query191_00012", + "negative_test_query64_00003", + "negative_test_query96_00006", + "negative_test_query60_00000", + "negative_test_query332_00012", + "apositive_test_query341_00003", + "apositive_test_query341_00005", + "negative_test_query274_00002", + "negative_test_query191_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00002", + "apositive_test_query192_00000", + "negative_test_query192_00010", + "negative_test_query322_00008", + "negative_test_query192_00013", + "negative_test_query172_00005", + "negative_test_query192_00009", + "apositive_test_query192_00004", + "negative_test_query192_00004", + "negative_test_query192_00012", + "negative_test_query120_00004", + "negative_test_query192_00001", + "negative_test_query192_00005", + "negative_test_query192_00002", + "negative_test_query192_00008", + "negative_test_query350_00010", + "negative_test_query120_00007", + "negative_test_query350_00013", + "apositive_test_query254_00007", + "negative_test_query192_00014" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query193", + "ranked_ids": [ + "apositive_test_query99_00003", + "apositive_test_query99_00002", + "apositive_test_query193_00011", + "apositive_test_query193_00008", + "apositive_test_query193_00001", + "apositive_test_query193_00000", + "apositive_test_query193_00002", + "negative_test_query99_00009", + "apositive_test_query193_00004", + "apositive_test_query153_00012", + "negative_test_query193_00002", + "negative_test_query193_00003", + "negative_test_query193_00006", + "apositive_test_query193_00009", + "apositive_test_query193_00005", + "negative_test_query146_00001", + "apositive_test_query146_00008", + "apositive_test_query276_00006", + "negative_test_query146_00003", + "negative_test_query193_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query194", + "ranked_ids": [ + "apositive_test_query194_00007", + "apositive_test_query124_00000", + "apositive_test_query162_00006", + "apositive_test_query194_00000", + "apositive_test_query42_00001", + "apositive_test_query194_00001", + "negative_test_query262_00007", + "apositive_test_query114_00001", + "negative_test_query194_00002", + "negative_test_query62_00014", + "apositive_test_query194_00006", + "negative_test_query137_00011", + "negative_test_query177_00007", + "negative_test_query162_00000", + "negative_test_query95_00016", + "apositive_test_query254_00006", + "negative_test_query148_00009", + "negative_test_query262_00011", + "negative_test_query194_00003", + "negative_test_query89_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query195", + "ranked_ids": [ + "apositive_test_query195_00002", + "negative_test_query222_00003", + "apositive_test_query195_00001", + "negative_test_query195_00001", + "negative_test_query195_00008", + "negative_test_query195_00013", + "negative_test_query149_00012", + "negative_test_query195_00002", + "apositive_test_query195_00005", + "negative_test_query321_00005", + "negative_test_query321_00008", + "negative_test_query109_00008", + "negative_test_query285_00000", + "negative_test_query195_00004", + "apositive_test_query249_00012", + "negative_test_query205_00006", + "apositive_test_query251_00000", + "negative_test_query195_00005", + "negative_test_query166_00009", + "negative_test_query199_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query196_00003", + "apositive_test_query32_00011", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "apositive_test_query32_00004", + "apositive_test_query32_00006", + "negative_test_query96_00013", + "negative_test_query235_00008", + "negative_test_query179_00001", + "negative_test_query139_00011", + "negative_test_query173_00012", + "apositive_test_query32_00002", + "negative_test_query196_00007", + "negative_test_query64_00005", + "negative_test_query32_00002", + "negative_test_query14_00001", + "negative_test_query196_00001", + "negative_test_query8_00013", + "negative_test_query341_00008", + "negative_test_query325_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query197", + "ranked_ids": [ + "negative_test_query163_00001", + "apositive_test_query197_00000", + "negative_test_query163_00002", + "negative_test_query197_00016", + "negative_test_query163_00012", + "apositive_test_query197_00002", + "negative_test_query197_00006", + "negative_test_query197_00004", + "negative_test_query197_00010", + "apositive_test_query189_00001", + "apositive_test_query34_00000", + "apositive_test_query163_00001", + "apositive_test_query163_00000", + "negative_test_query163_00009", + "negative_test_query197_00013", + "negative_test_query178_00008", + "negative_test_query321_00006", + "negative_test_query34_00012", + "negative_test_query28_00001", + "apositive_test_query197_00001" + ], + "retry_attempt": 1, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query239_00011", + "negative_test_query198_00007", + "negative_test_query239_00013", + "negative_test_query239_00010", + "negative_test_query239_00009", + "negative_test_query239_00001", + "negative_test_query198_00010", + "negative_test_query198_00002", + "negative_test_query239_00007", + "negative_test_query198_00008", + "negative_test_query352_00002", + "negative_test_query198_00004", + "negative_test_query198_00000", + "negative_test_query112_00005", + "negative_test_query294_00002", + "negative_test_query343_00002", + "negative_test_query62_00016", + "negative_test_query293_00010", + "negative_test_query293_00011", + "negative_test_query219_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query199_00014", + "negative_test_query199_00015", + "negative_test_query199_00002", + "negative_test_query199_00001", + "negative_test_query199_00013", + "negative_test_query139_00014", + "negative_test_query199_00006", + "negative_test_query350_00007", + "negative_test_query221_00008", + "negative_test_query54_00003", + "negative_test_query140_00014", + "negative_test_query213_00011", + "negative_test_query134_00006", + "negative_test_query305_00007", + "negative_test_query199_00005", + "negative_test_query250_00006", + "apositive_test_query43_00006", + "negative_test_query185_00002", + "apositive_test_query244_00002", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query2_00009", + "negative_test_query2_00012", + "negative_test_query2_00016", + "negative_test_query44_00008", + "negative_test_query2_00002", + "negative_test_query125_00012", + "negative_test_query190_00016", + "apositive_test_query217_00006", + "negative_test_query15_00000", + "negative_test_query2_00011", + "negative_test_query343_00004", + "negative_test_query85_00007", + "apositive_test_query57_00004", + "negative_test_query2_00013", + "negative_test_query352_00003", + "negative_test_query2_00010", + "negative_test_query197_00004", + "negative_test_query348_00005", + "negative_test_query126_00014", + "negative_test_query104_00007" + ], + "retry_attempt": 1, + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query154_00006", + "apositive_test_query20_00010", + "negative_test_query154_00002", + "apositive_test_query20_00009", + "negative_test_query33_00007", + "negative_test_query188_00000", + "apositive_test_query154_00005", + "apositive_test_query154_00002", + "apositive_test_query66_00000", + "negative_test_query356_00004", + "apositive_test_query20_00003", + "negative_test_query66_00002", + "apositive_test_query154_00004", + "apositive_test_query20_00007", + "apositive_test_query23_00004", + "apositive_test_query356_00005", + "apositive_test_query33_00008", + "negative_test_query220_00005", + "apositive_test_query33_00000", + "negative_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query200_00004", + "negative_test_query264_00006", + "negative_test_query337_00009", + "negative_test_query337_00007", + "negative_test_query200_00000", + "negative_test_query200_00013", + "negative_test_query38_00005", + "negative_test_query264_00007", + "negative_test_query200_00007", + "negative_test_query200_00005", + "negative_test_query200_00010", + "negative_test_query337_00005", + "negative_test_query303_00009", + "negative_test_query200_00009", + "negative_test_query200_00015", + "negative_test_query337_00004", + "negative_test_query200_00006", + "negative_test_query200_00012", + "negative_test_query8_00008", + "negative_test_query88_00013" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query201", + "ranked_ids": [ + "negative_test_query231_00012", + "negative_test_query47_00012", + "apositive_test_query201_00002", + "negative_test_query0_00004", + "negative_test_query71_00013", + "apositive_test_query0_00011", + "apositive_test_query201_00006", + "negative_test_query209_00008", + "negative_test_query47_00013", + "negative_test_query259_00002", + "negative_test_query71_00009", + "apositive_test_query30_00005", + "negative_test_query71_00000", + "negative_test_query71_00002", + "negative_test_query71_00005", + "negative_test_query65_00002", + "apositive_test_query71_00001", + "apositive_test_query20_00005", + "apositive_test_query30_00001", + "negative_test_query159_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query202_00016", + "negative_test_query202_00004", + "negative_test_query202_00009", + "negative_test_query321_00018", + "negative_test_query176_00011", + "negative_test_query202_00010", + "negative_test_query202_00000", + "negative_test_query202_00001", + "negative_test_query202_00008", + "apositive_test_query302_00006", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "apositive_test_query309_00000", + "negative_test_query235_00001", + "apositive_test_query206_00000", + "negative_test_query202_00006", + "negative_test_query312_00012", + "negative_test_query15_00010", + "negative_test_query188_00011", + "apositive_test_query61_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2 + }, + "query_id": "test_query203", + "ranked_ids": [ + "apositive_test_query203_00007", + "negative_test_query203_00006", + "negative_test_query203_00008", + "apositive_test_query110_00003", + "apositive_test_query203_00000", + "negative_test_query203_00005", + "apositive_test_query284_00003", + "apositive_test_query203_00009", + "negative_test_query209_00000", + "negative_test_query130_00001", + "apositive_test_query30_00005", + "apositive_test_query203_00004", + "negative_test_query284_00014", + "apositive_test_query130_00002", + "negative_test_query250_00005", + "apositive_test_query203_00003", + "negative_test_query250_00009", + "negative_test_query250_00011", + "negative_test_query35_00004", + "negative_test_query250_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.11764705882352941 + }, + "query_id": "test_query204", + "ranked_ids": [ + "apositive_test_query353_00002", + "apositive_test_query204_00002", + "apositive_test_query353_00005", + "apositive_test_query204_00000", + "negative_test_query353_00003", + "apositive_test_query204_00009", + "negative_test_query204_00000", + "negative_test_query204_00002", + "apositive_test_query175_00004", + "apositive_test_query353_00000", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "apositive_test_query353_00011", + "apositive_test_query204_00007", + "negative_test_query353_00004", + "negative_test_query97_00007", + "negative_test_query204_00001", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query97_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query205_00015", + "negative_test_query35_00007", + "negative_test_query139_00007", + "negative_test_query205_00003", + "apositive_test_query205_00000", + "negative_test_query31_00015", + "negative_test_query35_00008", + "negative_test_query205_00007", + "negative_test_query256_00000", + "negative_test_query228_00016", + "negative_test_query205_00008", + "negative_test_query115_00000", + "negative_test_query96_00011", + "negative_test_query256_00015", + "negative_test_query341_00003", + "negative_test_query256_00007", + "negative_test_query73_00000", + "negative_test_query73_00011", + "negative_test_query256_00006", + "apositive_test_query256_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query206_00002", + "negative_test_query206_00013", + "negative_test_query206_00001", + "negative_test_query206_00007", + "negative_test_query206_00014", + "negative_test_query299_00001", + "negative_test_query206_00000", + "negative_test_query206_00015", + "negative_test_query303_00015", + "apositive_test_query348_00001", + "negative_test_query70_00011", + "negative_test_query206_00003", + "negative_test_query2_00016", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00005", + "negative_test_query303_00003", + "negative_test_query206_00018", + "negative_test_query299_00002", + "negative_test_query64_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.5 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00005", + "apositive_test_query207_00000", + "apositive_test_query207_00004", + "negative_test_query207_00002", + "negative_test_query207_00011", + "negative_test_query207_00005", + "apositive_test_query207_00002", + "negative_test_query207_00012", + "negative_test_query207_00006", + "negative_test_query207_00001", + "negative_test_query207_00008", + "apositive_test_query13_00001", + "negative_test_query207_00009", + "apositive_test_query207_00001", + "negative_test_query207_00000", + "negative_test_query207_00003", + "negative_test_query207_00007", + "negative_test_query207_00013", + "negative_test_query37_00004", + "negative_test_query207_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "apositive_test_query208_00000", + "negative_test_query208_00000", + "negative_test_query208_00010", + "negative_test_query208_00017", + "negative_test_query208_00018", + "negative_test_query211_00014", + "negative_test_query208_00008", + "negative_test_query208_00003", + "negative_test_query208_00009", + "negative_test_query208_00016", + "negative_test_query256_00011", + "negative_test_query61_00012", + "negative_test_query208_00004", + "negative_test_query128_00003", + "apositive_test_query128_00002", + "negative_test_query73_00008", + "negative_test_query128_00001", + "apositive_test_query128_00007", + "negative_test_query208_00012", + "negative_test_query208_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query209", + "ranked_ids": [ + "apositive_test_query209_00001", + "negative_test_query209_00007", + "negative_test_query223_00003", + "negative_test_query26_00002", + "negative_test_query231_00006", + "apositive_test_query37_00001", + "apositive_test_query26_00007", + "apositive_test_query158_00004", + "apositive_test_query158_00009", + "negative_test_query347_00004", + "negative_test_query223_00010", + "negative_test_query102_00016", + "negative_test_query102_00010", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "apositive_test_query119_00008", + "negative_test_query209_00001", + "negative_test_query223_00007", + "negative_test_query26_00004", + "negative_test_query313_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "apositive_test_query21_00000", + "negative_test_query128_00006", + "negative_test_query21_00005", + "negative_test_query330_00000", + "apositive_test_query128_00003", + "negative_test_query21_00010", + "negative_test_query21_00009", + "negative_test_query128_00004", + "negative_test_query8_00006", + "negative_test_query21_00012", + "negative_test_query234_00006", + "negative_test_query21_00001", + "negative_test_query350_00002", + "negative_test_query224_00015", + "negative_test_query10_00008", + "apositive_test_query128_00002", + "negative_test_query234_00007", + "negative_test_query208_00000", + "apositive_test_query128_00004", + "negative_test_query208_00017" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query288_00006", + "negative_test_query210_00010", + "negative_test_query210_00006", + "negative_test_query148_00007", + "negative_test_query210_00013", + "negative_test_query321_00013", + "negative_test_query202_00014", + "negative_test_query148_00008", + "negative_test_query288_00007", + "negative_test_query199_00015", + "negative_test_query148_00014", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query210_00007", + "negative_test_query335_00004", + "negative_test_query210_00015", + "negative_test_query156_00015", + "negative_test_query17_00002", + "negative_test_query300_00000", + "negative_test_query210_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query211", + "ranked_ids": [ + "apositive_test_query211_00001", + "negative_test_query211_00015", + "negative_test_query211_00000", + "negative_test_query189_00008", + "negative_test_query211_00012", + "negative_test_query83_00010", + "negative_test_query137_00012", + "apositive_test_query101_00000", + "negative_test_query159_00006", + "negative_test_query32_00005", + "negative_test_query195_00007", + "apositive_test_query162_00010", + "apositive_test_query163_00004", + "negative_test_query239_00014", + "apositive_test_query176_00000", + "apositive_test_query198_00000", + "negative_test_query100_00012", + "negative_test_query167_00015", + "negative_test_query293_00013", + "negative_test_query202_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query212", + "ranked_ids": [ + "apositive_test_query326_00000", + "apositive_test_query212_00004", + "negative_test_query167_00017", + "apositive_test_query98_00000", + "apositive_test_query326_00004", + "negative_test_query77_00004", + "negative_test_query183_00009", + "negative_test_query183_00012", + "negative_test_query282_00009", + "negative_test_query126_00017", + "apositive_test_query66_00005", + "apositive_test_query89_00004", + "negative_test_query167_00016", + "negative_test_query346_00006", + "apositive_test_query244_00007", + "negative_test_query282_00010", + "negative_test_query344_00008", + "apositive_test_query71_00001", + "negative_test_query190_00017", + "apositive_test_query11_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00004", + "negative_test_query213_00011", + "apositive_test_query213_00000", + "apositive_test_query213_00002", + "apositive_test_query36_00002", + "negative_test_query313_00017", + "apositive_test_query213_00003", + "apositive_test_query305_00003", + "apositive_test_query158_00001", + "apositive_test_query259_00001", + "negative_test_query357_00007", + "negative_test_query213_00001", + "negative_test_query213_00005", + "negative_test_query137_00015", + "negative_test_query224_00015", + "negative_test_query138_00016", + "negative_test_query213_00004", + "apositive_test_query259_00000", + "negative_test_query138_00001", + "negative_test_query112_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "apositive_test_query214_00000", + "negative_test_query214_00004", + "negative_test_query214_00010", + "apositive_test_query323_00002", + "negative_test_query214_00016", + "negative_test_query214_00013", + "apositive_test_query323_00003", + "negative_test_query214_00015", + "negative_test_query103_00009", + "negative_test_query214_00001", + "apositive_test_query234_00001", + "negative_test_query324_00012", + "apositive_test_query160_00009", + "apositive_test_query75_00003", + "apositive_test_query17_00001", + "negative_test_query214_00003", + "negative_test_query17_00013", + "apositive_test_query89_00000", + "negative_test_query324_00002", + "negative_test_query9_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00000", + "negative_test_query215_00011", + "negative_test_query215_00004", + "apositive_test_query215_00000", + "negative_test_query257_00001", + "apositive_test_query80_00005", + "negative_test_query215_00012", + "negative_test_query215_00010", + "negative_test_query257_00003", + "negative_test_query215_00009", + "negative_test_query215_00015", + "negative_test_query215_00001", + "negative_test_query293_00013", + "negative_test_query215_00008", + "negative_test_query215_00006", + "negative_test_query215_00007", + "negative_test_query215_00018", + "negative_test_query263_00001", + "negative_test_query295_00004", + "negative_test_query215_00017" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.6 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query216_00000", + "apositive_test_query154_00005", + "apositive_test_query216_00004", + "apositive_test_query216_00002", + "negative_test_query216_00014", + "apositive_test_query330_00002", + "negative_test_query216_00002", + "negative_test_query216_00008", + "apositive_test_query216_00001", + "negative_test_query112_00002", + "apositive_test_query20_00001", + "negative_test_query44_00002", + "negative_test_query216_00007", + "negative_test_query246_00016", + "negative_test_query216_00006", + "negative_test_query216_00000", + "negative_test_query216_00005", + "negative_test_query216_00009", + "apositive_test_query33_00004", + "apositive_test_query182_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "negative_test_query217_00011", + "apositive_test_query307_00010", + "apositive_test_query307_00000", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00000", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query28_00002", + "negative_test_query217_00004", + "apositive_test_query213_00004", + "negative_test_query112_00007", + "negative_test_query230_00002", + "negative_test_query53_00012", + "negative_test_query76_00003", + "negative_test_query72_00005", + "apositive_test_query107_00001", + "apositive_test_query160_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query218", + "ranked_ids": [ + "negative_test_query134_00003", + "apositive_test_query218_00005", + "apositive_test_query218_00004", + "apositive_test_query218_00003", + "negative_test_query58_00010", + "negative_test_query218_00000", + "negative_test_query218_00012", + "negative_test_query218_00003", + "apositive_test_query134_00000", + "negative_test_query218_00009", + "apositive_test_query314_00015", + "negative_test_query134_00009", + "negative_test_query218_00002", + "negative_test_query274_00008", + "negative_test_query43_00006", + "negative_test_query304_00005", + "apositive_test_query60_00012", + "negative_test_query43_00010", + "apositive_test_query218_00000", + "negative_test_query165_00013" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "apositive_test_query219_00000", + "negative_test_query219_00014", + "negative_test_query219_00016", + "apositive_test_query219_00001", + "negative_test_query165_00010", + "negative_test_query219_00017", + "negative_test_query219_00008", + "negative_test_query219_00011", + "negative_test_query219_00015", + "negative_test_query227_00007", + "negative_test_query171_00005", + "negative_test_query24_00007", + "negative_test_query187_00015", + "negative_test_query308_00006", + "negative_test_query59_00000", + "negative_test_query219_00012", + "apositive_test_query177_00004", + "negative_test_query305_00005", + "negative_test_query183_00016", + "negative_test_query219_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "apositive_test_query22_00000", + "negative_test_query22_00008", + "negative_test_query22_00015", + "negative_test_query22_00002", + "negative_test_query22_00004", + "negative_test_query22_00012", + "negative_test_query22_00001", + "negative_test_query22_00003", + "negative_test_query22_00000", + "negative_test_query22_00007", + "negative_test_query22_00009", + "negative_test_query22_00017", + "negative_test_query22_00018", + "negative_test_query22_00005", + "negative_test_query22_00013", + "negative_test_query22_00016", + "negative_test_query66_00004", + "negative_test_query22_00006", + "apositive_test_query162_00007", + "apositive_test_query177_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query220", + "ranked_ids": [ + "negative_test_query344_00007", + "apositive_test_query220_00004", + "apositive_test_query220_00003", + "apositive_test_query220_00000", + "apositive_test_query220_00001", + "negative_test_query188_00008", + "negative_test_query188_00013", + "negative_test_query188_00004", + "negative_test_query161_00010", + "apositive_test_query188_00000", + "negative_test_query154_00009", + "negative_test_query188_00007", + "apositive_test_query146_00009", + "negative_test_query220_00011", + "negative_test_query344_00013", + "negative_test_query188_00014", + "negative_test_query188_00009", + "negative_test_query20_00002", + "negative_test_query220_00004", + "apositive_test_query20_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query142_00004", + "negative_test_query65_00013", + "negative_test_query277_00010", + "apositive_test_query306_00006", + "negative_test_query15_00005", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "negative_test_query278_00004", + "negative_test_query184_00001", + "apositive_test_query244_00008", + "apositive_test_query180_00002", + "negative_test_query12_00005", + "negative_test_query39_00007", + "negative_test_query65_00007", + "apositive_test_query174_00006", + "negative_test_query171_00008", + "apositive_test_query162_00005", + "negative_test_query180_00011", + "negative_test_query244_00006", + "apositive_test_query92_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query222", + "ranked_ids": [ + "apositive_test_query275_00000", + "apositive_test_query275_00008", + "apositive_test_query275_00001", + "apositive_test_query275_00004", + "negative_test_query271_00015", + "negative_test_query271_00004", + "apositive_test_query275_00005", + "negative_test_query275_00004", + "apositive_test_query222_00001", + "negative_test_query222_00014", + "negative_test_query222_00006", + "negative_test_query222_00011", + "apositive_test_query222_00000", + "apositive_test_query289_00002", + "negative_test_query289_00006", + "apositive_test_query275_00006", + "negative_test_query222_00017", + "negative_test_query290_00004", + "negative_test_query275_00002", + "apositive_test_query302_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query359_00008", + "negative_test_query51_00003", + "apositive_test_query201_00004", + "apositive_test_query259_00008", + "negative_test_query323_00005", + "negative_test_query47_00007", + "apositive_test_query259_00011", + "apositive_test_query259_00003", + "negative_test_query47_00002", + "negative_test_query107_00004", + "negative_test_query359_00014", + "negative_test_query345_00002", + "apositive_test_query107_00005", + "apositive_test_query260_00003", + "negative_test_query109_00006", + "negative_test_query47_00001", + "negative_test_query259_00006", + "negative_test_query164_00011", + "apositive_test_query201_00006", + "negative_test_query6_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query224", + "ranked_ids": [ + "negative_test_query350_00016", + "apositive_test_query224_00000", + "negative_test_query224_00010", + "apositive_test_query224_00003", + "apositive_test_query256_00001", + "negative_test_query350_00015", + "negative_test_query98_00018", + "negative_test_query149_00008", + "negative_test_query224_00011", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "negative_test_query224_00012", + "negative_test_query350_00017", + "negative_test_query224_00008", + "negative_test_query205_00009", + "apositive_test_query179_00002", + "negative_test_query224_00014", + "apositive_test_query83_00000", + "negative_test_query212_00013", + "negative_test_query304_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00001", + "apositive_test_query225_00000", + "negative_test_query145_00000", + "apositive_test_query225_00003", + "apositive_test_query225_00007", + "apositive_test_query145_00001", + "apositive_test_query225_00004", + "negative_test_query149_00011", + "apositive_test_query225_00005", + "apositive_test_query145_00005", + "negative_test_query225_00008", + "apositive_test_query169_00002", + "apositive_test_query225_00009", + "negative_test_query225_00006", + "negative_test_query183_00015", + "negative_test_query343_00009", + "negative_test_query161_00004", + "apositive_test_query225_00002", + "negative_test_query225_00000", + "negative_test_query54_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.8333333333333334 + }, + "query_id": "test_query226", + "ranked_ids": [ + "apositive_test_query226_00002", + "apositive_test_query226_00004", + "apositive_test_query226_00001", + "apositive_test_query226_00000", + "apositive_test_query226_00005", + "negative_test_query226_00001", + "negative_test_query226_00009", + "negative_test_query226_00005", + "negative_test_query226_00000", + "apositive_test_query226_00003", + "apositive_test_query153_00013", + "negative_test_query226_00007", + "negative_test_query226_00012", + "negative_test_query226_00003", + "negative_test_query226_00010", + "negative_test_query226_00013", + "negative_test_query226_00004", + "negative_test_query226_00006", + "negative_test_query226_00011", + "negative_test_query360_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00003", + "negative_test_query70_00011", + "negative_test_query227_00002", + "apositive_test_query227_00000", + "negative_test_query227_00006", + "negative_test_query227_00012", + "negative_test_query227_00004", + "apositive_test_query70_00001", + "negative_test_query227_00000", + "negative_test_query227_00014", + "negative_test_query235_00006", + "negative_test_query227_00013", + "negative_test_query59_00004", + "negative_test_query227_00010", + "negative_test_query227_00005", + "negative_test_query227_00011", + "negative_test_query321_00004", + "apositive_test_query227_00001", + "negative_test_query115_00007", + "apositive_test_query348_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query228", + "ranked_ids": [ + "negative_test_query228_00014", + "negative_test_query228_00012", + "negative_test_query73_00016", + "negative_test_query228_00011", + "negative_test_query228_00003", + "apositive_test_query228_00000", + "negative_test_query228_00002", + "apositive_test_query322_00007", + "apositive_test_query333_00001", + "negative_test_query322_00001", + "apositive_test_query318_00001", + "apositive_test_query322_00003", + "apositive_test_query73_00001", + "negative_test_query179_00010", + "negative_test_query165_00011", + "negative_test_query228_00000", + "negative_test_query228_00004", + "negative_test_query2_00001", + "negative_test_query216_00013", + "apositive_test_query254_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query229_00003", + "apositive_test_query229_00002", + "apositive_test_query229_00000", + "negative_test_query229_00005", + "apositive_test_query229_00007", + "negative_test_query229_00008", + "negative_test_query229_00007", + "apositive_test_query158_00006", + "apositive_test_query146_00004", + "apositive_test_query158_00002", + "negative_test_query355_00001", + "negative_test_query44_00005", + "negative_test_query229_00006", + "negative_test_query229_00000", + "negative_test_query229_00009", + "apositive_test_query18_00004", + "apositive_test_query69_00009", + "negative_test_query0_00001", + "apositive_test_query94_00007", + "negative_test_query94_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query23", + "ranked_ids": [ + "apositive_test_query23_00000", + "apositive_test_query23_00001", + "negative_test_query66_00010", + "apositive_test_query23_00002", + "negative_test_query356_00008", + "apositive_test_query52_00002", + "negative_test_query239_00012", + "negative_test_query23_00014", + "apositive_test_query54_00007", + "apositive_test_query54_00004", + "negative_test_query292_00013", + "apositive_test_query52_00000", + "apositive_test_query52_00001", + "apositive_test_query52_00004", + "negative_test_query259_00007", + "apositive_test_query39_00002", + "apositive_test_query146_00015", + "apositive_test_query225_00000", + "negative_test_query133_00003", + "negative_test_query307_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query230_00000", + "negative_test_query230_00007", + "apositive_test_query3_00001", + "negative_test_query99_00003", + "apositive_test_query3_00000", + "apositive_test_query3_00003", + "apositive_test_query3_00006", + "negative_test_query269_00010", + "apositive_test_query3_00009", + "negative_test_query3_00000", + "apositive_test_query230_00003", + "negative_test_query3_00004", + "negative_test_query3_00005", + "negative_test_query3_00006", + "negative_test_query230_00010", + "negative_test_query329_00000", + "negative_test_query23_00012", + "negative_test_query280_00006", + "apositive_test_query280_00002", + "negative_test_query280_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query231", + "ranked_ids": [ + "negative_test_query9_00017", + "apositive_test_query130_00000", + "negative_test_query221_00006", + "apositive_test_query0_00001", + "negative_test_query345_00011", + "negative_test_query75_00000", + "negative_test_query231_00001", + "negative_test_query359_00008", + "apositive_test_query130_00004", + "negative_test_query213_00004", + "negative_test_query130_00000", + "negative_test_query201_00011", + "apositive_test_query231_00003", + "apositive_test_query201_00006", + "apositive_test_query201_00002", + "negative_test_query47_00006", + "negative_test_query306_00006", + "negative_test_query130_00010", + "apositive_test_query119_00006", + "negative_test_query47_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "apositive_test_query232_00000", + "negative_test_query232_00015", + "negative_test_query232_00014", + "negative_test_query232_00018", + "negative_test_query232_00006", + "negative_test_query232_00012", + "negative_test_query232_00013", + "negative_test_query232_00004", + "negative_test_query232_00016", + "negative_test_query232_00011", + "negative_test_query232_00005", + "negative_test_query232_00017", + "negative_test_query232_00008", + "negative_test_query232_00002", + "negative_test_query232_00010", + "negative_test_query232_00000", + "negative_test_query232_00003", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query233", + "ranked_ids": [ + "apositive_test_query233_00002", + "negative_test_query233_00006", + "apositive_test_query233_00000", + "negative_test_query233_00004", + "negative_test_query2_00010", + "negative_test_query233_00001", + "negative_test_query190_00007", + "apositive_test_query2_00000", + "negative_test_query176_00017", + "negative_test_query233_00010", + "negative_test_query233_00015", + "negative_test_query219_00015", + "negative_test_query233_00005", + "negative_test_query233_00009", + "negative_test_query246_00001", + "negative_test_query263_00009", + "apositive_test_query275_00003", + "negative_test_query130_00008", + "negative_test_query2_00014", + "negative_test_query130_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00015", + "apositive_test_query304_00002", + "apositive_test_query304_00001", + "apositive_test_query304_00003", + "apositive_test_query234_00003", + "negative_test_query304_00001", + "negative_test_query214_00006", + "negative_test_query304_00012", + "negative_test_query234_00011", + "negative_test_query304_00005", + "negative_test_query234_00010", + "negative_test_query304_00008", + "negative_test_query10_00008", + "negative_test_query234_00005", + "negative_test_query214_00008", + "negative_test_query285_00011", + "negative_test_query304_00009", + "apositive_test_query43_00005", + "negative_test_query304_00013", + "apositive_test_query218_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00002", + "negative_test_query235_00014", + "apositive_test_query32_00006", + "negative_test_query235_00008", + "apositive_test_query235_00000", + "apositive_test_query32_00000", + "negative_test_query235_00000", + "apositive_test_query32_00004", + "apositive_test_query235_00001", + "negative_test_query235_00013", + "negative_test_query24_00017", + "negative_test_query235_00009", + "negative_test_query196_00000", + "negative_test_query173_00005", + "negative_test_query235_00015", + "apositive_test_query235_00003", + "negative_test_query219_00013", + "negative_test_query235_00004", + "negative_test_query61_00001", + "negative_test_query235_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "negative_test_query236_00006", + "negative_test_query236_00004", + "negative_test_query279_00005", + "apositive_test_query236_00000", + "negative_test_query236_00000", + "negative_test_query222_00000", + "negative_test_query236_00009", + "negative_test_query279_00012", + "apositive_test_query279_00002", + "negative_test_query236_00005", + "negative_test_query22_00016", + "negative_test_query236_00014", + "negative_test_query236_00012", + "negative_test_query236_00013", + "negative_test_query236_00011", + "negative_test_query279_00003", + "negative_test_query236_00007", + "negative_test_query236_00015", + "negative_test_query236_00018", + "negative_test_query236_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "apositive_test_query237_00001", + "negative_test_query237_00004", + "negative_test_query312_00001", + "negative_test_query237_00000", + "negative_test_query237_00002", + "negative_test_query237_00006", + "apositive_test_query237_00000", + "negative_test_query237_00005", + "negative_test_query290_00010", + "apositive_test_query237_00002", + "negative_test_query237_00007", + "negative_test_query237_00001", + "negative_test_query237_00009", + "negative_test_query152_00003", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "negative_test_query152_00008", + "negative_test_query261_00012", + "apositive_test_query152_00004", + "apositive_test_query157_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "negative_test_query108_00007", + "negative_test_query13_00003", + "negative_test_query102_00007", + "negative_test_query223_00005", + "apositive_test_query142_00003", + "apositive_test_query146_00013", + "negative_test_query46_00012", + "negative_test_query56_00000", + "negative_test_query56_00008", + "apositive_test_query306_00003", + "negative_test_query203_00003", + "negative_test_query123_00001", + "apositive_test_query276_00008", + "apositive_test_query29_00007", + "negative_test_query111_00013", + "negative_test_query238_00011", + "apositive_test_query29_00001", + "negative_test_query29_00006", + "negative_test_query111_00008", + "negative_test_query142_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "negative_test_query239_00015", + "negative_test_query239_00003", + "apositive_test_query239_00000", + "negative_test_query239_00000", + "negative_test_query239_00001", + "negative_test_query239_00013", + "negative_test_query239_00010", + "negative_test_query198_00007", + "negative_test_query239_00011", + "negative_test_query239_00009", + "negative_test_query352_00002", + "negative_test_query198_00002", + "negative_test_query239_00014", + "negative_test_query163_00004", + "negative_test_query239_00007", + "negative_test_query268_00005", + "negative_test_query198_00008", + "negative_test_query61_00004", + "negative_test_query183_00015", + "negative_test_query167_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "negative_test_query24_00012", + "negative_test_query24_00009", + "negative_test_query24_00002", + "apositive_test_query24_00000", + "negative_test_query24_00001", + "negative_test_query24_00004", + "negative_test_query24_00008", + "apositive_test_query106_00002", + "negative_test_query250_00008", + "negative_test_query128_00002", + "negative_test_query24_00000", + "negative_test_query52_00012", + "negative_test_query190_00016", + "apositive_test_query226_00004", + "negative_test_query15_00000", + "apositive_test_query57_00004", + "negative_test_query356_00003", + "negative_test_query323_00000", + "negative_test_query252_00005", + "negative_test_query198_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "apositive_test_query240_00000", + "negative_test_query240_00012", + "negative_test_query240_00006", + "negative_test_query240_00002", + "negative_test_query240_00010", + "negative_test_query240_00004", + "negative_test_query240_00015", + "negative_test_query240_00007", + "apositive_test_query311_00000", + "negative_test_query27_00010", + "negative_test_query6_00000", + "negative_test_query240_00016", + "negative_test_query240_00017", + "negative_test_query240_00018", + "negative_test_query240_00003", + "negative_test_query240_00011", + "negative_test_query246_00002", + "negative_test_query282_00013", + "negative_test_query293_00004", + "negative_test_query297_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "negative_test_query206_00004", + "negative_test_query206_00011", + "negative_test_query206_00009", + "negative_test_query241_00010", + "negative_test_query139_00007", + "negative_test_query191_00011", + "negative_test_query241_00005", + "negative_test_query241_00008", + "negative_test_query205_00003", + "negative_test_query298_00014", + "apositive_test_query241_00000", + "negative_test_query241_00006", + "negative_test_query256_00000", + "negative_test_query191_00014", + "negative_test_query241_00016", + "negative_test_query241_00017", + "negative_test_query241_00002", + "negative_test_query173_00013", + "negative_test_query258_00000", + "negative_test_query53_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.2 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query242_00002", + "apositive_test_query291_00000", + "apositive_test_query242_00001", + "apositive_test_query291_00001", + "apositive_test_query242_00005", + "apositive_test_query247_00004", + "apositive_test_query291_00008", + "apositive_test_query242_00013", + "apositive_test_query291_00005", + "apositive_test_query242_00008", + "apositive_test_query247_00005", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query242_00004", + "apositive_test_query242_00010", + "negative_test_query291_00004", + "negative_test_query291_00003", + "negative_test_query291_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.38461538461538464 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query243_00000", + "apositive_test_query243_00002", + "apositive_test_query243_00005", + "apositive_test_query243_00012", + "apositive_test_query243_00003", + "negative_test_query243_00002", + "negative_test_query192_00010", + "apositive_test_query243_00011", + "negative_test_query303_00005", + "negative_test_query192_00007", + "apositive_test_query173_00004", + "apositive_test_query160_00003", + "negative_test_query223_00004", + "negative_test_query274_00014", + "negative_test_query289_00000", + "negative_test_query243_00000", + "negative_test_query205_00007", + "negative_test_query250_00013", + "negative_test_query64_00003", + "negative_test_query184_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query244", + "ranked_ids": [ + "apositive_test_query244_00002", + "negative_test_query244_00005", + "negative_test_query244_00001", + "apositive_test_query244_00001", + "apositive_test_query244_00000", + "negative_test_query244_00003", + "negative_test_query244_00008", + "negative_test_query244_00010", + "negative_test_query244_00009", + "negative_test_query244_00000", + "negative_test_query244_00002", + "apositive_test_query244_00007", + "apositive_test_query244_00004", + "negative_test_query244_00004", + "apositive_test_query244_00005", + "apositive_test_query244_00006", + "apositive_test_query42_00002", + "negative_test_query126_00015", + "apositive_test_query23_00003", + "apositive_test_query356_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.38364913903790027, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query64_00013", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query245_00000", + "apositive_test_query245_00001", + "negative_test_query159_00003", + "negative_test_query135_00013", + "negative_test_query27_00002", + "negative_test_query121_00002", + "negative_test_query122_00002", + "apositive_test_query141_00002", + "negative_test_query245_00000", + "negative_test_query27_00011", + "negative_test_query293_00003", + "negative_test_query202_00004", + "apositive_test_query141_00006", + "negative_test_query245_00003", + "negative_test_query245_00010", + "negative_test_query202_00000", + "negative_test_query336_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query246_00002", + "negative_test_query246_00016", + "negative_test_query246_00012", + "negative_test_query252_00003", + "negative_test_query135_00017", + "negative_test_query6_00014", + "negative_test_query198_00012", + "negative_test_query311_00005", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query246_00001", + "negative_test_query6_00006", + "apositive_test_query68_00002", + "negative_test_query190_00003", + "negative_test_query44_00002", + "negative_test_query317_00014", + "apositive_test_query0_00001", + "negative_test_query293_00000", + "apositive_test_query216_00000", + "negative_test_query134_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query247", + "ranked_ids": [ + "negative_test_query291_00003", + "apositive_test_query242_00002", + "apositive_test_query242_00001", + "apositive_test_query291_00000", + "negative_test_query247_00005", + "apositive_test_query247_00004", + "apositive_test_query247_00005", + "negative_test_query242_00000", + "negative_test_query247_00011", + "negative_test_query291_00001", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "apositive_test_query242_00010", + "negative_test_query291_00005", + "apositive_test_query291_00001", + "negative_test_query291_00000", + "negative_test_query186_00010", + "negative_test_query105_00012", + "apositive_test_query242_00012", + "negative_test_query247_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.375 + }, + "query_id": "test_query248", + "ranked_ids": [ + "apositive_test_query248_00007", + "negative_test_query248_00008", + "apositive_test_query248_00002", + "apositive_test_query248_00005", + "negative_test_query248_00011", + "negative_test_query248_00002", + "negative_test_query248_00010", + "apositive_test_query29_00000", + "apositive_test_query29_00004", + "apositive_test_query248_00003", + "apositive_test_query29_00005", + "negative_test_query251_00017", + "negative_test_query248_00006", + "negative_test_query248_00004", + "negative_test_query29_00001", + "apositive_test_query248_00001", + "negative_test_query248_00009", + "negative_test_query205_00017", + "negative_test_query248_00000", + "negative_test_query248_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query249", + "ranked_ids": [ + "apositive_test_query249_00000", + "apositive_test_query249_00004", + "negative_test_query249_00001", + "negative_test_query122_00016", + "negative_test_query278_00016", + "apositive_test_query177_00004", + "negative_test_query156_00007", + "negative_test_query156_00002", + "negative_test_query193_00001", + "negative_test_query71_00006", + "negative_test_query325_00015", + "negative_test_query326_00005", + "negative_test_query301_00013", + "negative_test_query72_00017", + "apositive_test_query112_00000", + "apositive_test_query281_00001", + "negative_test_query210_00010", + "negative_test_query302_00007", + "apositive_test_query204_00003", + "apositive_test_query353_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query25", + "ranked_ids": [ + "apositive_test_query25_00000", + "apositive_test_query25_00003", + "negative_test_query25_00005", + "apositive_test_query25_00001", + "negative_test_query25_00007", + "apositive_test_query25_00007", + "apositive_test_query25_00005", + "apositive_test_query25_00010", + "apositive_test_query25_00004", + "apositive_test_query25_00008", + "negative_test_query25_00002", + "apositive_test_query25_00009", + "negative_test_query328_00000", + "negative_test_query45_00014", + "apositive_test_query25_00006", + "negative_test_query143_00000", + "negative_test_query25_00001", + "negative_test_query30_00003", + "negative_test_query346_00006", + "negative_test_query9_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query250", + "ranked_ids": [ + "negative_test_query250_00009", + "apositive_test_query203_00000", + "apositive_test_query250_00000", + "negative_test_query250_00013", + "apositive_test_query203_00006", + "negative_test_query79_00013", + "negative_test_query159_00014", + "negative_test_query186_00007", + "negative_test_query250_00001", + "negative_test_query192_00009", + "negative_test_query184_00015", + "negative_test_query64_00005", + "negative_test_query159_00004", + "negative_test_query250_00011", + "negative_test_query111_00015", + "negative_test_query57_00009", + "negative_test_query102_00010", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query250_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query251", + "ranked_ids": [ + "apositive_test_query251_00001", + "negative_test_query234_00006", + "negative_test_query326_00000", + "apositive_test_query347_00000", + "negative_test_query56_00005", + "apositive_test_query134_00003", + "negative_test_query251_00003", + "negative_test_query251_00004", + "negative_test_query14_00000", + "negative_test_query189_00011", + "negative_test_query322_00002", + "negative_test_query298_00003", + "negative_test_query160_00001", + "negative_test_query347_00014", + "apositive_test_query29_00002", + "negative_test_query143_00001", + "negative_test_query8_00009", + "negative_test_query268_00011", + "negative_test_query170_00008", + "negative_test_query269_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query252", + "ranked_ids": [ + "negative_test_query252_00007", + "apositive_test_query252_00000", + "apositive_test_query252_00003", + "negative_test_query252_00009", + "negative_test_query252_00011", + "negative_test_query44_00011", + "negative_test_query318_00006", + "negative_test_query321_00004", + "apositive_test_query110_00001", + "negative_test_query14_00006", + "negative_test_query212_00000", + "apositive_test_query252_00002", + "negative_test_query252_00010", + "negative_test_query252_00012", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query330_00006", + "apositive_test_query125_00001", + "negative_test_query252_00001", + "negative_test_query16_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3 + }, + "query_id": "test_query253", + "ranked_ids": [ + "apositive_test_query253_00003", + "apositive_test_query253_00004", + "apositive_test_query253_00007", + "apositive_test_query13_00012", + "negative_test_query172_00008", + "negative_test_query194_00001", + "apositive_test_query349_00000", + "negative_test_query124_00001", + "apositive_test_query194_00008", + "negative_test_query253_00005", + "apositive_test_query74_00002", + "negative_test_query226_00013", + "negative_test_query95_00017", + "apositive_test_query162_00008", + "negative_test_query177_00000", + "apositive_test_query186_00004", + "apositive_test_query74_00003", + "negative_test_query253_00003", + "negative_test_query253_00004", + "negative_test_query124_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query254", + "ranked_ids": [ + "apositive_test_query254_00001", + "apositive_test_query254_00002", + "apositive_test_query254_00000", + "negative_test_query254_00008", + "apositive_test_query254_00006", + "apositive_test_query254_00004", + "negative_test_query350_00000", + "apositive_test_query254_00010", + "apositive_test_query254_00003", + "apositive_test_query254_00005", + "negative_test_query316_00013", + "negative_test_query270_00010", + "negative_test_query128_00004", + "negative_test_query21_00005", + "negative_test_query136_00003", + "negative_test_query228_00016", + "negative_test_query21_00009", + "apositive_test_query333_00004", + "negative_test_query254_00004", + "negative_test_query35_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "negative_test_query89_00004", + "apositive_test_query255_00000", + "apositive_test_query25_00004", + "negative_test_query278_00015", + "negative_test_query255_00005", + "negative_test_query255_00004", + "negative_test_query255_00017", + "negative_test_query255_00006", + "apositive_test_query278_00000", + "negative_test_query278_00000", + "negative_test_query255_00018", + "negative_test_query208_00008", + "negative_test_query255_00013", + "apositive_test_query322_00001", + "negative_test_query255_00010", + "negative_test_query246_00003", + "negative_test_query255_00000", + "apositive_test_query313_00000", + "negative_test_query326_00006", + "negative_test_query110_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.75369761125927, + "recall@5": 0.75 + }, + "query_id": "test_query256", + "ranked_ids": [ + "apositive_test_query256_00003", + "negative_test_query256_00003", + "apositive_test_query256_00001", + "apositive_test_query256_00000", + "negative_test_query82_00001", + "negative_test_query183_00008", + "negative_test_query279_00000", + "apositive_test_query256_00002", + "apositive_test_query43_00001", + "apositive_test_query216_00002", + "negative_test_query256_00008", + "negative_test_query319_00001", + "negative_test_query5_00010", + "apositive_test_query142_00002", + "negative_test_query256_00011", + "apositive_test_query119_00000", + "negative_test_query347_00007", + "negative_test_query7_00009", + "negative_test_query135_00016", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query257", + "ranked_ids": [ + "apositive_test_query257_00004", + "apositive_test_query257_00001", + "apositive_test_query257_00000", + "apositive_test_query69_00007", + "apositive_test_query257_00005", + "apositive_test_query257_00002", + "apositive_test_query286_00014", + "negative_test_query52_00007", + "negative_test_query52_00005", + "negative_test_query257_00007", + "negative_test_query52_00004", + "apositive_test_query52_00001", + "negative_test_query257_00005", + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00003", + "apositive_test_query54_00003", + "negative_test_query52_00002", + "negative_test_query54_00005", + "apositive_test_query151_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query320_00002", + "negative_test_query82_00000", + "negative_test_query181_00002", + "negative_test_query162_00004", + "negative_test_query319_00000", + "negative_test_query259_00001", + "negative_test_query101_00001", + "negative_test_query38_00002", + "apositive_test_query301_00002", + "negative_test_query173_00010", + "negative_test_query228_00005", + "negative_test_query206_00016", + "negative_test_query311_00002", + "negative_test_query167_00007", + "negative_test_query144_00010", + "negative_test_query169_00016", + "negative_test_query171_00002", + "negative_test_query339_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query259", + "ranked_ids": [ + "apositive_test_query259_00004", + "apositive_test_query259_00008", + "negative_test_query259_00002", + "apositive_test_query359_00002", + "apositive_test_query231_00001", + "negative_test_query213_00004", + "apositive_test_query259_00006", + "apositive_test_query259_00003", + "apositive_test_query345_00000", + "negative_test_query111_00010", + "negative_test_query259_00001", + "apositive_test_query54_00011", + "negative_test_query250_00015", + "apositive_test_query259_00007", + "negative_test_query259_00006", + "negative_test_query47_00004", + "negative_test_query359_00012", + "negative_test_query30_00009", + "negative_test_query359_00003", + "apositive_test_query0_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "negative_test_query26_00008", + "apositive_test_query26_00005", + "negative_test_query247_00010", + "negative_test_query26_00003", + "apositive_test_query158_00003", + "negative_test_query26_00009", + "apositive_test_query186_00003", + "negative_test_query158_00006", + "negative_test_query102_00015", + "negative_test_query334_00016", + "negative_test_query223_00002", + "negative_test_query26_00005", + "apositive_test_query102_00000", + "apositive_test_query26_00004", + "negative_test_query105_00013", + "negative_test_query164_00004", + "negative_test_query119_00006", + "negative_test_query158_00009", + "negative_test_query26_00001", + "negative_test_query238_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "apositive_test_query6_00001", + "apositive_test_query6_00002", + "apositive_test_query6_00000", + "negative_test_query6_00002", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query6_00005", + "negative_test_query125_00001", + "apositive_test_query109_00001", + "negative_test_query260_00011", + "negative_test_query6_00004", + "negative_test_query6_00011", + "negative_test_query2_00004", + "negative_test_query6_00012", + "negative_test_query260_00009", + "negative_test_query252_00004", + "negative_test_query6_00009", + "apositive_test_query186_00001", + "negative_test_query260_00002", + "apositive_test_query109_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query261", + "ranked_ids": [ + "negative_test_query261_00005", + "negative_test_query116_00004", + "negative_test_query261_00007", + "negative_test_query261_00006", + "negative_test_query261_00002", + "negative_test_query261_00000", + "negative_test_query261_00011", + "negative_test_query261_00010", + "negative_test_query261_00003", + "negative_test_query261_00001", + "apositive_test_query261_00003", + "apositive_test_query261_00002", + "apositive_test_query261_00000", + "negative_test_query261_00013", + "negative_test_query261_00008", + "negative_test_query261_00004", + "negative_test_query118_00006", + "negative_test_query2_00015", + "negative_test_query261_00009", + "apositive_test_query29_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query262", + "ranked_ids": [ + "apositive_test_query262_00003", + "negative_test_query312_00018", + "apositive_test_query262_00000", + "negative_test_query106_00010", + "apositive_test_query42_00006", + "apositive_test_query194_00002", + "negative_test_query124_00012", + "negative_test_query266_00003", + "negative_test_query301_00005", + "negative_test_query262_00000", + "negative_test_query312_00003", + "negative_test_query152_00006", + "negative_test_query172_00000", + "negative_test_query106_00005", + "negative_test_query312_00000", + "apositive_test_query106_00003", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query326_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query263", + "ranked_ids": [ + "apositive_test_query263_00002", + "apositive_test_query292_00002", + "apositive_test_query291_00010", + "negative_test_query135_00004", + "negative_test_query169_00014", + "apositive_test_query242_00011", + "negative_test_query253_00009", + "apositive_test_query37_00002", + "negative_test_query263_00005", + "apositive_test_query231_00001", + "negative_test_query201_00010", + "negative_test_query88_00003", + "negative_test_query213_00005", + "negative_test_query357_00006", + "negative_test_query190_00009", + "apositive_test_query155_00003", + "negative_test_query125_00012", + "negative_test_query307_00005", + "negative_test_query71_00006", + "negative_test_query213_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "apositive_test_query264_00000", + "negative_test_query337_00002", + "apositive_test_query38_00003", + "negative_test_query264_00008", + "negative_test_query264_00011", + "negative_test_query264_00004", + "negative_test_query264_00014", + "negative_test_query264_00006", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query200_00002", + "negative_test_query200_00011", + "negative_test_query264_00007", + "negative_test_query337_00012", + "negative_test_query159_00017", + "negative_test_query200_00017", + "negative_test_query264_00002", + "negative_test_query264_00018", + "negative_test_query159_00013", + "negative_test_query264_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query265", + "ranked_ids": [ + "apositive_test_query265_00000", + "apositive_test_query265_00001", + "negative_test_query190_00013", + "negative_test_query265_00000", + "negative_test_query274_00008", + "negative_test_query265_00007", + "negative_test_query265_00002", + "negative_test_query265_00009", + "apositive_test_query234_00001", + "negative_test_query52_00000", + "negative_test_query86_00007", + "negative_test_query165_00007", + "negative_test_query139_00000", + "apositive_test_query156_00000", + "negative_test_query165_00016", + "negative_test_query86_00008", + "negative_test_query271_00005", + "negative_test_query92_00012", + "apositive_test_query78_00002", + "negative_test_query66_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "negative_test_query176_00018", + "negative_test_query266_00008", + "apositive_test_query266_00000", + "negative_test_query141_00000", + "negative_test_query266_00001", + "negative_test_query106_00004", + "negative_test_query266_00009", + "apositive_test_query54_00009", + "negative_test_query266_00005", + "negative_test_query266_00002", + "negative_test_query266_00014", + "negative_test_query169_00014", + "negative_test_query106_00001", + "negative_test_query174_00001", + "negative_test_query266_00007", + "negative_test_query266_00018", + "apositive_test_query19_00002", + "negative_test_query266_00000", + "negative_test_query266_00017", + "negative_test_query266_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "negative_test_query79_00002", + "negative_test_query359_00005", + "negative_test_query98_00013", + "apositive_test_query113_00002", + "negative_test_query79_00004", + "negative_test_query267_00002", + "negative_test_query186_00006", + "apositive_test_query295_00001", + "negative_test_query295_00002", + "negative_test_query5_00012", + "apositive_test_query250_00000", + "negative_test_query313_00016", + "negative_test_query30_00000", + "apositive_test_query50_00004", + "apositive_test_query157_00008", + "negative_test_query345_00013", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "negative_test_query204_00001", + "negative_test_query115_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query268", + "ranked_ids": [ + "apositive_test_query268_00000", + "negative_test_query47_00017", + "negative_test_query268_00000", + "negative_test_query268_00013", + "apositive_test_query51_00003", + "negative_test_query268_00007", + "apositive_test_query268_00004", + "negative_test_query268_00003", + "apositive_test_query268_00002", + "negative_test_query268_00004", + "negative_test_query323_00002", + "negative_test_query335_00002", + "negative_test_query268_00012", + "apositive_test_query201_00003", + "negative_test_query268_00008", + "negative_test_query195_00009", + "negative_test_query268_00005", + "negative_test_query210_00007", + "negative_test_query228_00007", + "negative_test_query335_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query269", + "ranked_ids": [ + "negative_test_query269_00001", + "negative_test_query269_00009", + "negative_test_query269_00003", + "apositive_test_query296_00005", + "negative_test_query269_00007", + "negative_test_query269_00005", + "apositive_test_query269_00000", + "negative_test_query269_00010", + "negative_test_query269_00000", + "negative_test_query269_00015", + "negative_test_query269_00004", + "negative_test_query269_00008", + "negative_test_query269_00002", + "negative_test_query296_00003", + "negative_test_query329_00014", + "negative_test_query269_00012", + "negative_test_query329_00004", + "apositive_test_query269_00003", + "negative_test_query269_00011", + "apositive_test_query296_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "negative_test_query27_00004", + "negative_test_query27_00009", + "negative_test_query147_00000", + "negative_test_query76_00009", + "apositive_test_query154_00000", + "negative_test_query11_00008", + "negative_test_query164_00007", + "negative_test_query259_00006", + "negative_test_query138_00006", + "negative_test_query358_00003", + "negative_test_query315_00001", + "apositive_test_query81_00007", + "apositive_test_query127_00010", + "negative_test_query33_00004", + "apositive_test_query59_00000", + "negative_test_query130_00006", + "negative_test_query30_00009", + "apositive_test_query29_00004", + "negative_test_query27_00014", + "apositive_test_query16_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query270", + "ranked_ids": [ + "apositive_test_query270_00001", + "negative_test_query270_00006", + "negative_test_query270_00012", + "apositive_test_query270_00000", + "negative_test_query270_00000", + "negative_test_query270_00015", + "negative_test_query270_00005", + "negative_test_query270_00004", + "negative_test_query270_00003", + "negative_test_query270_00016", + "negative_test_query270_00014", + "negative_test_query270_00009", + "negative_test_query270_00008", + "negative_test_query270_00002", + "negative_test_query270_00011", + "negative_test_query270_00010", + "apositive_test_query270_00002", + "apositive_test_query81_00000", + "negative_test_query76_00005", + "negative_test_query81_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query271", + "ranked_ids": [ + "apositive_test_query271_00000", + "apositive_test_query271_00001", + "negative_test_query271_00007", + "negative_test_query120_00003", + "negative_test_query78_00010", + "apositive_test_query10_00002", + "negative_test_query241_00018", + "negative_test_query330_00008", + "negative_test_query327_00010", + "apositive_test_query189_00000", + "apositive_test_query109_00001", + "negative_test_query227_00009", + "apositive_test_query67_00000", + "negative_test_query84_00014", + "apositive_test_query314_00007", + "negative_test_query169_00009", + "negative_test_query198_00015", + "negative_test_query221_00012", + "negative_test_query90_00001", + "negative_test_query112_00014" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query41_00008", + "negative_test_query272_00003", + "negative_test_query272_00010", + "negative_test_query272_00016", + "negative_test_query272_00015", + "negative_test_query144_00000", + "negative_test_query272_00008", + "negative_test_query272_00001", + "negative_test_query272_00017", + "negative_test_query272_00002", + "apositive_test_query272_00000", + "negative_test_query328_00000", + "negative_test_query272_00014", + "negative_test_query18_00014", + "negative_test_query272_00000", + "negative_test_query272_00009", + "negative_test_query272_00013", + "negative_test_query332_00013", + "negative_test_query272_00012", + "apositive_test_query251_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00008", + "negative_test_query18_00011", + "apositive_test_query281_00008", + "negative_test_query18_00006", + "apositive_test_query18_00000", + "negative_test_query273_00000", + "apositive_test_query18_00004", + "negative_test_query273_00004", + "apositive_test_query273_00005", + "apositive_test_query273_00000", + "negative_test_query273_00006", + "apositive_test_query18_00001", + "negative_test_query18_00012", + "apositive_test_query18_00003", + "negative_test_query273_00009", + "apositive_test_query273_00004", + "negative_test_query166_00009", + "negative_test_query300_00015", + "negative_test_query140_00013", + "apositive_test_query36_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query274", + "ranked_ids": [ + "apositive_test_query274_00000", + "negative_test_query274_00000", + "apositive_test_query274_00003", + "negative_test_query274_00014", + "negative_test_query274_00007", + "negative_test_query352_00012", + "negative_test_query274_00009", + "negative_test_query274_00011", + "negative_test_query274_00005", + "apositive_test_query274_00001", + "negative_test_query218_00011", + "negative_test_query200_00009", + "negative_test_query34_00007", + "negative_test_query274_00012", + "negative_test_query34_00014", + "negative_test_query256_00015", + "apositive_test_query327_00002", + "negative_test_query274_00004", + "negative_test_query78_00007", + "negative_test_query352_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query275_00003", + "apositive_test_query275_00000", + "apositive_test_query275_00007", + "apositive_test_query275_00006", + "negative_test_query202_00005", + "apositive_test_query275_00008", + "negative_test_query263_00009", + "apositive_test_query275_00004", + "negative_test_query275_00007", + "negative_test_query289_00013", + "negative_test_query159_00017", + "negative_test_query149_00016", + "apositive_test_query275_00005", + "negative_test_query275_00000", + "negative_test_query275_00004", + "negative_test_query264_00017", + "apositive_test_query110_00001", + "apositive_test_query93_00003", + "negative_test_query16_00001", + "negative_test_query87_00017" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query22_00004", + "negative_test_query199_00012", + "apositive_test_query314_00005", + "apositive_test_query276_00005", + "negative_test_query153_00001", + "apositive_test_query153_00006", + "negative_test_query146_00000", + "apositive_test_query314_00006", + "negative_test_query113_00011", + "apositive_test_query146_00015", + "apositive_test_query276_00004", + "negative_test_query280_00002", + "negative_test_query276_00001", + "apositive_test_query161_00002", + "negative_test_query146_00001", + "negative_test_query251_00012", + "apositive_test_query314_00007", + "apositive_test_query276_00006", + "apositive_test_query3_00010", + "negative_test_query146_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.787702056960637, + "recall@5": 0.75 + }, + "query_id": "test_query277", + "ranked_ids": [ + "apositive_test_query277_00001", + "apositive_test_query277_00000", + "negative_test_query277_00007", + "negative_test_query277_00015", + "apositive_test_query277_00002", + "apositive_test_query277_00003", + "negative_test_query277_00000", + "negative_test_query277_00010", + "negative_test_query277_00003", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "negative_test_query345_00000", + "negative_test_query277_00006", + "negative_test_query277_00013", + "negative_test_query277_00004", + "negative_test_query277_00014", + "negative_test_query313_00009", + "negative_test_query277_00008", + "apositive_test_query107_00007", + "negative_test_query277_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query278_00011", + "negative_test_query278_00002", + "negative_test_query140_00006", + "apositive_test_query263_00005", + "apositive_test_query4_00013", + "negative_test_query260_00009", + "negative_test_query260_00002", + "negative_test_query15_00007", + "apositive_test_query36_00001", + "negative_test_query177_00001", + "negative_test_query278_00009", + "negative_test_query255_00018", + "negative_test_query27_00005", + "negative_test_query324_00011", + "negative_test_query158_00003", + "negative_test_query260_00011", + "apositive_test_query186_00001", + "negative_test_query64_00007", + "negative_test_query255_00016", + "negative_test_query110_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query279_00003", + "apositive_test_query279_00000", + "apositive_test_query177_00004", + "apositive_test_query177_00006", + "apositive_test_query76_00002", + "negative_test_query279_00014", + "negative_test_query279_00009", + "apositive_test_query279_00004", + "negative_test_query309_00011", + "apositive_test_query358_00007", + "apositive_test_query279_00001", + "negative_test_query279_00013", + "apositive_test_query3_00000", + "apositive_test_query76_00005", + "negative_test_query279_00007", + "apositive_test_query348_00007", + "apositive_test_query3_00002", + "negative_test_query181_00008", + "apositive_test_query162_00009", + "negative_test_query132_00013" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query28", + "ranked_ids": [ + "apositive_test_query28_00008", + "negative_test_query48_00009", + "apositive_test_query217_00000", + "apositive_test_query28_00009", + "apositive_test_query217_00003", + "negative_test_query185_00003", + "negative_test_query72_00004", + "apositive_test_query28_00003", + "negative_test_query217_00011", + "negative_test_query307_00004", + "apositive_test_query28_00007", + "negative_test_query28_00002", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00001", + "negative_test_query72_00018", + "negative_test_query217_00004", + "negative_test_query307_00003", + "negative_test_query185_00001", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.5 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "apositive_test_query280_00000", + "negative_test_query280_00004", + "apositive_test_query280_00004", + "negative_test_query280_00002", + "negative_test_query280_00012", + "negative_test_query280_00003", + "negative_test_query280_00005", + "negative_test_query280_00008", + "negative_test_query106_00006", + "negative_test_query9_00012", + "negative_test_query280_00006", + "negative_test_query280_00013", + "apositive_test_query280_00003", + "negative_test_query106_00005", + "negative_test_query280_00010", + "apositive_test_query253_00009", + "negative_test_query129_00008", + "apositive_test_query74_00001", + "negative_test_query236_00018" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.10526315789473684 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query273_00000", + "apositive_test_query281_00004", + "apositive_test_query273_00002", + "apositive_test_query281_00012", + "negative_test_query273_00003", + "apositive_test_query281_00008", + "apositive_test_query281_00010", + "apositive_test_query281_00007", + "negative_test_query273_00004", + "negative_test_query257_00012", + "apositive_test_query273_00008", + "apositive_test_query281_00014", + "negative_test_query273_00007", + "negative_test_query125_00011", + "apositive_test_query281_00003", + "apositive_test_query281_00001", + "apositive_test_query281_00016", + "apositive_test_query281_00011", + "negative_test_query249_00000", + "apositive_test_query273_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6509209298071326, + "recall@5": 1.0 + }, + "query_id": "test_query282", + "ranked_ids": [ + "negative_test_query344_00001", + "apositive_test_query282_00000", + "negative_test_query282_00007", + "apositive_test_query282_00001", + "negative_test_query282_00004", + "negative_test_query282_00001", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query282_00006", + "negative_test_query282_00000", + "negative_test_query344_00000", + "negative_test_query282_00003", + "negative_test_query282_00016", + "negative_test_query282_00010", + "negative_test_query344_00008", + "negative_test_query282_00012", + "negative_test_query282_00009", + "apositive_test_query20_00004", + "apositive_test_query220_00002", + "negative_test_query344_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query283_00012", + "negative_test_query93_00004", + "negative_test_query283_00014", + "negative_test_query138_00005", + "negative_test_query283_00004", + "negative_test_query138_00009", + "negative_test_query283_00000", + "negative_test_query93_00010", + "apositive_test_query93_00002", + "negative_test_query283_00007", + "negative_test_query283_00003", + "negative_test_query93_00006", + "negative_test_query270_00006", + "negative_test_query93_00002", + "negative_test_query283_00016", + "negative_test_query31_00010", + "negative_test_query93_00009", + "negative_test_query265_00013", + "negative_test_query92_00010", + "negative_test_query299_00017" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "apositive_test_query284_00003", + "negative_test_query284_00009", + "negative_test_query284_00001", + "negative_test_query141_00002", + "apositive_test_query259_00009", + "apositive_test_query284_00001", + "negative_test_query114_00006", + "apositive_test_query141_00002", + "apositive_test_query163_00002", + "negative_test_query109_00009", + "negative_test_query284_00012", + "apositive_test_query79_00004", + "negative_test_query284_00004", + "apositive_test_query198_00002", + "negative_test_query14_00014", + "apositive_test_query162_00005", + "apositive_test_query163_00003", + "apositive_test_query141_00006", + "negative_test_query288_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5 + }, + "query_id": "test_query285", + "ranked_ids": [ + "apositive_test_query285_00004", + "apositive_test_query285_00007", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "apositive_test_query285_00002", + "apositive_test_query285_00003", + "apositive_test_query285_00006", + "apositive_test_query285_00005", + "negative_test_query285_00002", + "negative_test_query285_00004", + "negative_test_query304_00009", + "negative_test_query285_00007", + "negative_test_query285_00008", + "negative_test_query285_00005", + "negative_test_query285_00000", + "negative_test_query285_00011", + "negative_test_query285_00010", + "negative_test_query86_00001", + "negative_test_query285_00001", + "negative_test_query285_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.2 + }, + "query_id": "test_query286", + "ranked_ids": [ + "apositive_test_query286_00010", + "negative_test_query257_00005", + "apositive_test_query286_00013", + "apositive_test_query286_00012", + "negative_test_query257_00001", + "negative_test_query286_00000", + "apositive_test_query286_00005", + "negative_test_query36_00001", + "apositive_test_query286_00006", + "negative_test_query54_00004", + "negative_test_query125_00012", + "apositive_test_query257_00001", + "apositive_test_query257_00000", + "negative_test_query52_00001", + "apositive_test_query80_00000", + "negative_test_query286_00003", + "negative_test_query36_00013", + "negative_test_query52_00005", + "negative_test_query139_00008", + "apositive_test_query286_00014" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query287_00001", + "apositive_test_query133_00001", + "apositive_test_query133_00008", + "apositive_test_query133_00002", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00014", + "apositive_test_query287_00004", + "apositive_test_query133_00004", + "apositive_test_query133_00006", + "apositive_test_query133_00012", + "negative_test_query301_00010", + "apositive_test_query133_00013", + "negative_test_query153_00003", + "negative_test_query359_00011", + "negative_test_query133_00002", + "apositive_test_query153_00003", + "apositive_test_query4_00009", + "negative_test_query134_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.125 + }, + "query_id": "test_query288", + "ranked_ids": [ + "apositive_test_query141_00006", + "negative_test_query288_00007", + "apositive_test_query141_00002", + "negative_test_query288_00002", + "apositive_test_query288_00002", + "negative_test_query288_00001", + "negative_test_query17_00002", + "negative_test_query199_00010", + "negative_test_query93_00000", + "negative_test_query288_00006", + "apositive_test_query210_00001", + "negative_test_query213_00007", + "negative_test_query148_00004", + "negative_test_query199_00015", + "negative_test_query210_00013", + "apositive_test_query210_00002", + "apositive_test_query304_00001", + "negative_test_query148_00014", + "negative_test_query210_00005", + "negative_test_query288_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00006", + "negative_test_query7_00006", + "negative_test_query334_00014", + "apositive_test_query289_00002", + "negative_test_query47_00009", + "negative_test_query289_00004", + "apositive_test_query289_00004", + "negative_test_query113_00009", + "negative_test_query289_00001", + "negative_test_query289_00013", + "apositive_test_query275_00005", + "negative_test_query289_00000", + "negative_test_query247_00001", + "apositive_test_query289_00000", + "negative_test_query305_00011", + "negative_test_query267_00005", + "apositive_test_query157_00008", + "negative_test_query250_00016", + "negative_test_query107_00004", + "negative_test_query289_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.375 + }, + "query_id": "test_query29", + "ranked_ids": [ + "apositive_test_query29_00007", + "apositive_test_query29_00004", + "negative_test_query248_00009", + "apositive_test_query29_00000", + "negative_test_query29_00005", + "apositive_test_query29_00005", + "negative_test_query39_00011", + "negative_test_query213_00012", + "negative_test_query292_00015", + "negative_test_query29_00009", + "negative_test_query29_00006", + "negative_test_query29_00010", + "apositive_test_query29_00002", + "negative_test_query29_00008", + "apositive_test_query29_00001", + "apositive_test_query29_00003", + "negative_test_query248_00007", + "negative_test_query29_00007", + "apositive_test_query29_00006", + "negative_test_query29_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query290", + "ranked_ids": [ + "apositive_test_query290_00003", + "apositive_test_query290_00001", + "negative_test_query98_00003", + "negative_test_query237_00010", + "negative_test_query312_00001", + "negative_test_query290_00006", + "negative_test_query237_00013", + "negative_test_query290_00005", + "apositive_test_query290_00002", + "apositive_test_query237_00002", + "negative_test_query290_00008", + "negative_test_query59_00002", + "negative_test_query116_00002", + "negative_test_query290_00002", + "negative_test_query342_00005", + "negative_test_query290_00009", + "negative_test_query290_00014", + "negative_test_query290_00003", + "negative_test_query290_00000", + "negative_test_query157_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query291", + "ranked_ids": [ + "negative_test_query186_00000", + "apositive_test_query242_00001", + "apositive_test_query242_00002", + "apositive_test_query291_00008", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "apositive_test_query291_00000", + "apositive_test_query242_00005", + "apositive_test_query242_00010", + "negative_test_query291_00004", + "negative_test_query291_00003", + "apositive_test_query242_00013", + "apositive_test_query291_00005", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query291_00005", + "apositive_test_query247_00005", + "negative_test_query291_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query292", + "ranked_ids": [ + "apositive_test_query292_00001", + "negative_test_query292_00010", + "apositive_test_query292_00000", + "negative_test_query106_00007", + "negative_test_query292_00012", + "negative_test_query292_00004", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query99_00004", + "negative_test_query292_00000", + "negative_test_query292_00014", + "negative_test_query298_00000", + "negative_test_query292_00005", + "negative_test_query292_00007", + "negative_test_query292_00008", + "negative_test_query139_00001", + "negative_test_query205_00000", + "negative_test_query61_00011", + "negative_test_query292_00006", + "negative_test_query151_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query293", + "ranked_ids": [ + "apositive_test_query293_00002", + "negative_test_query293_00015", + "apositive_test_query178_00003", + "negative_test_query89_00002", + "negative_test_query293_00011", + "negative_test_query293_00006", + "apositive_test_query293_00000", + "negative_test_query293_00000", + "negative_test_query220_00007", + "apositive_test_query293_00001", + "negative_test_query27_00008", + "negative_test_query27_00011", + "negative_test_query293_00009", + "negative_test_query27_00002", + "negative_test_query293_00003", + "apositive_test_query44_00001", + "apositive_test_query52_00005", + "negative_test_query293_00016", + "negative_test_query167_00013", + "negative_test_query293_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00011", + "apositive_test_query294_00001", + "negative_test_query294_00001", + "negative_test_query294_00004", + "negative_test_query294_00016", + "negative_test_query294_00008", + "negative_test_query294_00002", + "negative_test_query294_00012", + "negative_test_query111_00015", + "negative_test_query294_00003", + "negative_test_query294_00005", + "negative_test_query294_00007", + "negative_test_query277_00005", + "negative_test_query58_00003", + "negative_test_query156_00003", + "negative_test_query294_00013", + "negative_test_query294_00006", + "negative_test_query314_00001", + "apositive_test_query170_00005", + "apositive_test_query177_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query295", + "ranked_ids": [ + "apositive_test_query57_00000", + "apositive_test_query295_00001", + "apositive_test_query295_00000", + "negative_test_query26_00002", + "negative_test_query295_00008", + "apositive_test_query295_00002", + "negative_test_query168_00005", + "negative_test_query295_00002", + "apositive_test_query223_00001", + "negative_test_query57_00009", + "negative_test_query295_00013", + "negative_test_query295_00011", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query295_00016", + "negative_test_query115_00005", + "negative_test_query295_00001", + "negative_test_query295_00004", + "negative_test_query295_00012", + "negative_test_query295_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query296", + "ranked_ids": [ + "apositive_test_query296_00002", + "apositive_test_query296_00001", + "apositive_test_query296_00004", + "apositive_test_query296_00005", + "negative_test_query340_00009", + "apositive_test_query296_00003", + "negative_test_query329_00011", + "negative_test_query296_00012", + "negative_test_query112_00012", + "negative_test_query340_00014", + "negative_test_query296_00003", + "negative_test_query296_00013", + "negative_test_query296_00002", + "negative_test_query296_00005", + "negative_test_query296_00001", + "apositive_test_query296_00000", + "negative_test_query296_00009", + "negative_test_query296_00008", + "negative_test_query296_00000", + "negative_test_query0_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query297_00012", + "apositive_test_query297_00000", + "negative_test_query297_00014", + "negative_test_query64_00003", + "negative_test_query297_00010", + "apositive_test_query139_00001", + "negative_test_query297_00003", + "negative_test_query297_00007", + "negative_test_query117_00006", + "negative_test_query191_00011", + "negative_test_query297_00008", + "negative_test_query70_00005", + "negative_test_query191_00010", + "negative_test_query329_00007", + "negative_test_query297_00013", + "negative_test_query297_00004", + "negative_test_query297_00001", + "negative_test_query191_00003", + "negative_test_query297_00002", + "negative_test_query64_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "apositive_test_query298_00002", + "apositive_test_query298_00000", + "negative_test_query216_00010", + "negative_test_query298_00006", + "negative_test_query298_00003", + "negative_test_query182_00011", + "apositive_test_query347_00001", + "negative_test_query298_00004", + "apositive_test_query347_00002", + "negative_test_query240_00001", + "apositive_test_query56_00002", + "negative_test_query175_00009", + "negative_test_query193_00006", + "negative_test_query202_00011", + "negative_test_query347_00014", + "apositive_test_query298_00001", + "negative_test_query182_00010", + "negative_test_query298_00001", + "negative_test_query347_00000", + "negative_test_query347_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query299_00005", + "negative_test_query299_00006", + "apositive_test_query141_00002", + "apositive_test_query346_00001", + "apositive_test_query265_00002", + "apositive_test_query325_00002", + "negative_test_query288_00007", + "negative_test_query300_00000", + "negative_test_query351_00009", + "negative_test_query3_00003", + "negative_test_query89_00008", + "negative_test_query125_00005", + "apositive_test_query101_00003", + "negative_test_query134_00000", + "negative_test_query305_00007", + "negative_test_query92_00010", + "negative_test_query210_00008", + "negative_test_query288_00000", + "negative_test_query58_00005", + "apositive_test_query227_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query3", + "ranked_ids": [ + "apositive_test_query3_00000", + "apositive_test_query3_00001", + "apositive_test_query3_00003", + "negative_test_query3_00000", + "negative_test_query83_00007", + "negative_test_query147_00011", + "negative_test_query3_00007", + "apositive_test_query3_00008", + "negative_test_query279_00000", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "apositive_test_query3_00007", + "negative_test_query230_00007", + "negative_test_query230_00000", + "negative_test_query329_00000", + "negative_test_query23_00012", + "negative_test_query230_00010", + "negative_test_query3_00005", + "negative_test_query147_00014", + "apositive_test_query230_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query30", + "ranked_ids": [ + "apositive_test_query71_00000", + "apositive_test_query71_00002", + "apositive_test_query30_00004", + "negative_test_query71_00010", + "apositive_test_query30_00003", + "apositive_test_query306_00002", + "negative_test_query259_00002", + "negative_test_query231_00002", + "apositive_test_query30_00001", + "apositive_test_query209_00005", + "apositive_test_query30_00005", + "negative_test_query201_00005", + "negative_test_query231_00008", + "negative_test_query71_00005", + "negative_test_query182_00014", + "negative_test_query357_00002", + "apositive_test_query30_00000", + "negative_test_query231_00000", + "negative_test_query356_00010", + "negative_test_query97_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query15_00000", + "negative_test_query252_00005", + "negative_test_query300_00009", + "negative_test_query141_00001", + "apositive_test_query300_00001", + "negative_test_query300_00013", + "negative_test_query300_00008", + "apositive_test_query300_00002", + "negative_test_query300_00010", + "negative_test_query300_00006", + "negative_test_query300_00011", + "negative_test_query300_00005", + "negative_test_query266_00010", + "negative_test_query312_00011", + "negative_test_query190_00004", + "negative_test_query185_00001", + "negative_test_query167_00018", + "negative_test_query141_00000", + "negative_test_query190_00016", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query301", + "ranked_ids": [ + "apositive_test_query301_00000", + "negative_test_query301_00011", + "apositive_test_query301_00002", + "negative_test_query301_00015", + "negative_test_query301_00013", + "apositive_test_query301_00001", + "negative_test_query301_00002", + "negative_test_query301_00006", + "negative_test_query301_00014", + "negative_test_query301_00007", + "negative_test_query148_00009", + "negative_test_query21_00018", + "negative_test_query2_00008", + "negative_test_query301_00001", + "negative_test_query301_00000", + "negative_test_query316_00017", + "apositive_test_query171_00000", + "apositive_test_query86_00002", + "apositive_test_query86_00000", + "negative_test_query297_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query302", + "ranked_ids": [ + "apositive_test_query308_00009", + "apositive_test_query308_00001", + "negative_test_query308_00009", + "negative_test_query308_00003", + "apositive_test_query324_00001", + "apositive_test_query302_00002", + "apositive_test_query302_00001", + "apositive_test_query324_00002", + "apositive_test_query302_00000", + "apositive_test_query302_00003", + "negative_test_query302_00010", + "apositive_test_query302_00005", + "negative_test_query308_00008", + "apositive_test_query308_00008", + "negative_test_query302_00001", + "negative_test_query302_00012", + "apositive_test_query302_00004", + "negative_test_query302_00000", + "negative_test_query308_00002", + "negative_test_query308_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query243_00000", + "negative_test_query303_00017", + "apositive_test_query303_00000", + "negative_test_query61_00011", + "negative_test_query35_00003", + "negative_test_query303_00002", + "negative_test_query274_00014", + "negative_test_query206_00009", + "negative_test_query303_00005", + "negative_test_query173_00001", + "negative_test_query206_00011", + "negative_test_query307_00006", + "negative_test_query196_00007", + "negative_test_query8_00013", + "apositive_test_query86_00001", + "negative_test_query316_00009", + "negative_test_query206_00006", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query303_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.592512031964586, + "recall@5": 0.75 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query234_00003", + "apositive_test_query304_00001", + "apositive_test_query304_00002", + "negative_test_query43_00001", + "apositive_test_query304_00003", + "negative_test_query304_00012", + "negative_test_query304_00005", + "negative_test_query214_00002", + "negative_test_query304_00004", + "negative_test_query43_00006", + "apositive_test_query285_00005", + "negative_test_query304_00013", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query234_00010", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "negative_test_query304_00008", + "negative_test_query304_00007", + "negative_test_query304_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query305", + "ranked_ids": [ + "apositive_test_query305_00001", + "apositive_test_query357_00000", + "negative_test_query323_00003", + "negative_test_query134_00006", + "apositive_test_query127_00005", + "apositive_test_query71_00003", + "negative_test_query305_00005", + "negative_test_query56_00004", + "negative_test_query65_00013", + "negative_test_query305_00014", + "apositive_test_query259_00011", + "negative_test_query164_00011", + "negative_test_query51_00003", + "negative_test_query0_00007", + "negative_test_query203_00006", + "negative_test_query0_00004", + "negative_test_query0_00003", + "negative_test_query310_00000", + "negative_test_query224_00015", + "negative_test_query47_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query306", + "ranked_ids": [ + "negative_test_query71_00013", + "apositive_test_query30_00002", + "apositive_test_query71_00002", + "negative_test_query259_00002", + "apositive_test_query30_00001", + "negative_test_query209_00008", + "apositive_test_query119_00006", + "negative_test_query182_00014", + "apositive_test_query209_00005", + "negative_test_query159_00009", + "apositive_test_query306_00002", + "apositive_test_query71_00003", + "apositive_test_query127_00007", + "negative_test_query142_00002", + "negative_test_query305_00014", + "negative_test_query71_00012", + "apositive_test_query71_00001", + "negative_test_query204_00001", + "negative_test_query71_00011", + "apositive_test_query289_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "negative_test_query307_00006", + "apositive_test_query170_00005", + "negative_test_query217_00003", + "negative_test_query48_00018", + "negative_test_query360_00003", + "negative_test_query132_00008", + "apositive_test_query177_00004", + "apositive_test_query279_00002", + "negative_test_query144_00004", + "negative_test_query103_00012", + "negative_test_query163_00010", + "apositive_test_query3_00003", + "negative_test_query70_00010", + "negative_test_query346_00017", + "negative_test_query251_00001", + "negative_test_query187_00011", + "apositive_test_query13_00005", + "negative_test_query207_00004", + "apositive_test_query279_00003", + "negative_test_query18_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.3 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query308_00001", + "apositive_test_query324_00002", + "apositive_test_query308_00002", + "apositive_test_query308_00003", + "apositive_test_query324_00001", + "apositive_test_query308_00000", + "negative_test_query324_00014", + "negative_test_query308_00003", + "negative_test_query308_00007", + "apositive_test_query7_00000", + "apositive_test_query308_00008", + "negative_test_query308_00008", + "apositive_test_query341_00005", + "apositive_test_query324_00000", + "negative_test_query159_00013", + "apositive_test_query308_00009", + "negative_test_query324_00012", + "negative_test_query308_00009", + "negative_test_query94_00011", + "apositive_test_query310_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "negative_test_query123_00007", + "negative_test_query309_00003", + "apositive_test_query309_00000", + "negative_test_query309_00006", + "apositive_test_query358_00004", + "apositive_test_query358_00007", + "negative_test_query113_00013", + "apositive_test_query358_00006", + "apositive_test_query358_00003", + "negative_test_query309_00005", + "negative_test_query113_00011", + "negative_test_query354_00002", + "negative_test_query11_00000", + "negative_test_query309_00015", + "negative_test_query309_00012", + "negative_test_query354_00008", + "apositive_test_query161_00008", + "negative_test_query81_00002", + "negative_test_query42_00009", + "apositive_test_query76_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query31_00013", + "negative_test_query129_00000", + "negative_test_query31_00011", + "negative_test_query31_00010", + "negative_test_query31_00001", + "negative_test_query31_00000", + "negative_test_query31_00006", + "negative_test_query31_00002", + "negative_test_query31_00007", + "apositive_test_query31_00000", + "negative_test_query31_00008", + "negative_test_query31_00003", + "negative_test_query313_00007", + "negative_test_query31_00015", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "apositive_test_query213_00005", + "apositive_test_query249_00009", + "negative_test_query326_00005", + "negative_test_query38_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4 + }, + "query_id": "test_query310", + "ranked_ids": [ + "apositive_test_query310_00008", + "apositive_test_query310_00002", + "apositive_test_query310_00001", + "negative_test_query36_00015", + "apositive_test_query310_00009", + "apositive_test_query310_00000", + "apositive_test_query310_00003", + "apositive_test_query310_00006", + "negative_test_query148_00010", + "negative_test_query241_00001", + "negative_test_query59_00005", + "apositive_test_query218_00005", + "negative_test_query329_00003", + "negative_test_query179_00013", + "negative_test_query74_00002", + "negative_test_query191_00010", + "negative_test_query330_00007", + "negative_test_query148_00003", + "negative_test_query292_00007", + "negative_test_query310_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query311_00005", + "negative_test_query311_00008", + "negative_test_query240_00014", + "negative_test_query240_00008", + "apositive_test_query311_00000", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query134_00001", + "negative_test_query311_00012", + "apositive_test_query330_00001", + "negative_test_query246_00006", + "apositive_test_query330_00002", + "negative_test_query202_00017", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query12_00008", + "apositive_test_query5_00002", + "negative_test_query288_00004", + "negative_test_query330_00009", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "negative_test_query237_00000", + "apositive_test_query312_00000", + "negative_test_query191_00000", + "apositive_test_query237_00001", + "negative_test_query312_00006", + "negative_test_query206_00010", + "apositive_test_query220_00005", + "apositive_test_query212_00004", + "negative_test_query242_00001", + "apositive_test_query152_00004", + "negative_test_query157_00002", + "apositive_test_query98_00000", + "negative_test_query246_00015", + "negative_test_query343_00016", + "negative_test_query283_00016", + "apositive_test_query326_00004", + "negative_test_query182_00002", + "negative_test_query212_00003", + "negative_test_query294_00015", + "negative_test_query305_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query56_00001", + "negative_test_query46_00003", + "negative_test_query46_00011", + "negative_test_query111_00013", + "negative_test_query119_00005", + "negative_test_query46_00005", + "negative_test_query57_00001", + "negative_test_query359_00005", + "apositive_test_query142_00005", + "negative_test_query336_00001", + "negative_test_query334_00013", + "negative_test_query334_00001", + "negative_test_query313_00010", + "apositive_test_query108_00001", + "negative_test_query313_00012", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "negative_test_query313_00002", + "negative_test_query158_00000", + "apositive_test_query155_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query314_00010", + "apositive_test_query314_00005", + "apositive_test_query314_00002", + "apositive_test_query276_00007", + "apositive_test_query314_00006", + "negative_test_query37_00006", + "apositive_test_query276_00001", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query314_00011", + "apositive_test_query314_00012", + "apositive_test_query314_00009", + "apositive_test_query314_00004", + "apositive_test_query314_00013", + "apositive_test_query3_00009", + "apositive_test_query314_00016", + "negative_test_query314_00001", + "apositive_test_query314_00003", + "apositive_test_query314_00007", + "negative_test_query314_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9060254355346823, + "recall@5": 1.0 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query315_00000", + "negative_test_query269_00004", + "apositive_test_query315_00002", + "apositive_test_query315_00001", + "negative_test_query333_00007", + "apositive_test_query319_00005", + "apositive_test_query82_00005", + "apositive_test_query85_00003", + "negative_test_query85_00013", + "negative_test_query85_00005", + "apositive_test_query319_00001", + "negative_test_query319_00001", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query345_00006", + "negative_test_query85_00014", + "negative_test_query315_00009", + "apositive_test_query82_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query316", + "ranked_ids": [ + "negative_test_query316_00002", + "apositive_test_query316_00001", + "apositive_test_query316_00000", + "negative_test_query316_00004", + "negative_test_query316_00000", + "negative_test_query316_00005", + "apositive_test_query63_00007", + "negative_test_query316_00011", + "negative_test_query301_00005", + "negative_test_query316_00016", + "negative_test_query101_00007", + "negative_test_query322_00000", + "negative_test_query158_00008", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query316_00008", + "negative_test_query316_00001", + "apositive_test_query216_00000", + "apositive_test_query72_00000", + "negative_test_query228_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00004", + "negative_test_query317_00014", + "negative_test_query317_00008", + "negative_test_query317_00005", + "negative_test_query317_00013", + "apositive_test_query317_00000", + "negative_test_query317_00010", + "apositive_test_query317_00001", + "negative_test_query317_00000", + "negative_test_query317_00015", + "negative_test_query317_00006", + "negative_test_query317_00012", + "negative_test_query116_00014", + "negative_test_query55_00014", + "negative_test_query116_00012", + "negative_test_query271_00002", + "negative_test_query283_00013", + "negative_test_query331_00009", + "negative_test_query317_00017", + "negative_test_query317_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00000", + "apositive_test_query318_00002", + "negative_test_query318_00015", + "negative_test_query318_00004", + "negative_test_query318_00009", + "apositive_test_query318_00001", + "apositive_test_query318_00003", + "negative_test_query318_00005", + "negative_test_query73_00004", + "negative_test_query318_00008", + "negative_test_query73_00005", + "negative_test_query73_00000", + "negative_test_query318_00012", + "negative_test_query73_00012", + "negative_test_query318_00010", + "negative_test_query318_00003", + "negative_test_query318_00000", + "negative_test_query318_00002", + "negative_test_query318_00014", + "negative_test_query322_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query319", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "apositive_test_query82_00006", + "negative_test_query82_00009", + "negative_test_query319_00010", + "apositive_test_query82_00001", + "negative_test_query319_00008", + "negative_test_query82_00010", + "apositive_test_query319_00001", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query319_00006", + "negative_test_query340_00013", + "negative_test_query319_00009", + "apositive_test_query319_00003", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "negative_test_query82_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query136_00003", + "apositive_test_query32_00013", + "negative_test_query211_00016", + "negative_test_query196_00001", + "apositive_test_query192_00000", + "apositive_test_query196_00000", + "negative_test_query341_00010", + "negative_test_query211_00005", + "negative_test_query339_00006", + "negative_test_query189_00014", + "apositive_test_query32_00002", + "negative_test_query55_00011", + "negative_test_query55_00014", + "apositive_test_query336_00001", + "apositive_test_query357_00000", + "apositive_test_query17_00000", + "negative_test_query78_00007", + "negative_test_query321_00001", + "negative_test_query68_00009", + "negative_test_query83_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query320", + "ranked_ids": [ + "apositive_test_query320_00003", + "negative_test_query320_00005", + "negative_test_query320_00004", + "negative_test_query320_00008", + "negative_test_query91_00007", + "negative_test_query258_00012", + "negative_test_query212_00006", + "negative_test_query53_00014", + "negative_test_query318_00011", + "apositive_test_query238_00000", + "negative_test_query131_00009", + "negative_test_query230_00001", + "negative_test_query344_00010", + "negative_test_query100_00001", + "negative_test_query193_00006", + "negative_test_query43_00000", + "negative_test_query216_00014", + "apositive_test_query267_00001", + "negative_test_query304_00005", + "negative_test_query290_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "negative_test_query321_00000", + "negative_test_query34_00012", + "negative_test_query321_00010", + "apositive_test_query321_00000", + "negative_test_query321_00002", + "negative_test_query126_00005", + "negative_test_query352_00005", + "negative_test_query141_00009", + "negative_test_query197_00004", + "apositive_test_query217_00006", + "negative_test_query103_00006", + "apositive_test_query185_00009", + "negative_test_query103_00011", + "negative_test_query136_00004", + "negative_test_query86_00014", + "negative_test_query15_00013", + "negative_test_query226_00002", + "negative_test_query126_00011", + "negative_test_query322_00002", + "negative_test_query210_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query322", + "ranked_ids": [ + "apositive_test_query322_00000", + "apositive_test_query322_00005", + "apositive_test_query322_00002", + "negative_test_query322_00003", + "apositive_test_query322_00004", + "apositive_test_query322_00001", + "apositive_test_query322_00006", + "negative_test_query73_00010", + "apositive_test_query322_00003", + "negative_test_query318_00002", + "apositive_test_query228_00001", + "negative_test_query322_00009", + "negative_test_query73_00008", + "negative_test_query255_00010", + "negative_test_query322_00006", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query322_00002", + "negative_test_query322_00005", + "apositive_test_query254_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query323", + "ranked_ids": [ + "apositive_test_query130_00001", + "apositive_test_query323_00002", + "apositive_test_query323_00001", + "negative_test_query323_00012", + "apositive_test_query323_00004", + "negative_test_query323_00000", + "apositive_test_query323_00000", + "apositive_test_query323_00006", + "negative_test_query75_00002", + "negative_test_query323_00003", + "negative_test_query345_00011", + "negative_test_query323_00006", + "negative_test_query323_00007", + "negative_test_query323_00001", + "negative_test_query209_00004", + "negative_test_query255_00002", + "negative_test_query201_00009", + "negative_test_query75_00007", + "negative_test_query15_00006", + "negative_test_query75_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query324", + "ranked_ids": [ + "negative_test_query308_00003", + "apositive_test_query324_00000", + "negative_test_query324_00014", + "apositive_test_query324_00001", + "apositive_test_query308_00002", + "apositive_test_query324_00002", + "apositive_test_query308_00003", + "apositive_test_query308_00000", + "negative_test_query324_00005", + "negative_test_query324_00004", + "negative_test_query324_00006", + "negative_test_query308_00007", + "apositive_test_query308_00001", + "negative_test_query308_00009", + "negative_test_query324_00007", + "negative_test_query324_00003", + "negative_test_query324_00015", + "negative_test_query324_00016", + "negative_test_query218_00003", + "apositive_test_query308_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query176_00003", + "negative_test_query325_00007", + "negative_test_query325_00013", + "apositive_test_query293_00002", + "negative_test_query293_00015", + "negative_test_query325_00002", + "negative_test_query15_00015", + "negative_test_query293_00000", + "negative_test_query15_00005", + "negative_test_query266_00014", + "negative_test_query325_00008", + "negative_test_query169_00014", + "negative_test_query15_00017", + "negative_test_query15_00016", + "apositive_test_query19_00002", + "apositive_test_query178_00003", + "apositive_test_query246_00001", + "negative_test_query19_00008", + "negative_test_query191_00011", + "negative_test_query297_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query326", + "ranked_ids": [ + "apositive_test_query326_00000", + "apositive_test_query212_00004", + "apositive_test_query326_00004", + "negative_test_query156_00013", + "negative_test_query178_00012", + "apositive_test_query326_00003", + "apositive_test_query54_00002", + "apositive_test_query326_00001", + "apositive_test_query355_00002", + "apositive_test_query23_00000", + "negative_test_query17_00011", + "negative_test_query81_00003", + "negative_test_query107_00011", + "apositive_test_query52_00002", + "negative_test_query265_00007", + "negative_test_query183_00001", + "apositive_test_query294_00001", + "negative_test_query181_00007", + "negative_test_query265_00005", + "negative_test_query45_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query327", + "ranked_ids": [ + "negative_test_query321_00014", + "apositive_test_query327_00000", + "apositive_test_query327_00001", + "negative_test_query327_00014", + "negative_test_query61_00003", + "apositive_test_query327_00002", + "negative_test_query327_00012", + "negative_test_query327_00000", + "negative_test_query327_00003", + "negative_test_query327_00013", + "negative_test_query303_00010", + "negative_test_query327_00010", + "negative_test_query271_00007", + "negative_test_query335_00004", + "negative_test_query195_00007", + "negative_test_query61_00006", + "negative_test_query227_00009", + "apositive_test_query271_00000", + "negative_test_query299_00010", + "apositive_test_query271_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query328", + "ranked_ids": [ + "negative_test_query328_00003", + "negative_test_query328_00002", + "apositive_test_query328_00001", + "negative_test_query217_00005", + "apositive_test_query328_00004", + "negative_test_query59_00017", + "negative_test_query328_00006", + "negative_test_query328_00000", + "apositive_test_query328_00002", + "apositive_test_query328_00007", + "apositive_test_query40_00003", + "negative_test_query138_00013", + "negative_test_query129_00002", + "negative_test_query24_00006", + "negative_test_query92_00009", + "negative_test_query328_00007", + "negative_test_query144_00005", + "negative_test_query222_00016", + "negative_test_query261_00009", + "apositive_test_query227_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query329", + "ranked_ids": [ + "apositive_test_query329_00001", + "negative_test_query269_00000", + "negative_test_query62_00017", + "negative_test_query269_00012", + "negative_test_query329_00011", + "negative_test_query329_00006", + "negative_test_query329_00004", + "negative_test_query329_00013", + "negative_test_query298_00015", + "negative_test_query329_00009", + "negative_test_query329_00001", + "negative_test_query329_00002", + "negative_test_query329_00010", + "apositive_test_query329_00002", + "negative_test_query329_00003", + "apositive_test_query7_00001", + "negative_test_query183_00006", + "negative_test_query35_00000", + "negative_test_query202_00016", + "negative_test_query38_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.3 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00004", + "apositive_test_query33_00009", + "apositive_test_query33_00006", + "apositive_test_query20_00010", + "apositive_test_query33_00003", + "apositive_test_query154_00007", + "apositive_test_query33_00004", + "negative_test_query216_00004", + "negative_test_query161_00008", + "negative_test_query220_00010", + "negative_test_query154_00006", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "apositive_test_query284_00004", + "negative_test_query33_00006", + "negative_test_query344_00010", + "negative_test_query356_00006", + "apositive_test_query220_00003", + "negative_test_query216_00014", + "apositive_test_query220_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query330", + "ranked_ids": [ + "negative_test_query252_00007", + "negative_test_query27_00018", + "negative_test_query180_00000", + "negative_test_query325_00016", + "apositive_test_query281_00012", + "apositive_test_query330_00002", + "negative_test_query330_00009", + "negative_test_query330_00008", + "apositive_test_query60_00014", + "negative_test_query330_00012", + "negative_test_query311_00009", + "negative_test_query27_00011", + "negative_test_query27_00013", + "negative_test_query288_00006", + "negative_test_query330_00000", + "negative_test_query330_00010", + "negative_test_query330_00001", + "apositive_test_query284_00003", + "apositive_test_query288_00002", + "negative_test_query129_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6797310500037655, + "recall@5": 1.0 + }, + "query_id": "test_query331", + "ranked_ids": [ + "negative_test_query331_00013", + "apositive_test_query331_00001", + "negative_test_query331_00000", + "apositive_test_query331_00000", + "apositive_test_query331_00002", + "negative_test_query331_00012", + "negative_test_query331_00008", + "negative_test_query334_00014", + "negative_test_query250_00016", + "apositive_test_query223_00001", + "negative_test_query110_00007", + "negative_test_query251_00009", + "negative_test_query202_00001", + "negative_test_query303_00001", + "negative_test_query113_00000", + "negative_test_query312_00012", + "negative_test_query154_00000", + "negative_test_query214_00015", + "negative_test_query3_00008", + "negative_test_query331_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.6 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00001", + "negative_test_query202_00010", + "apositive_test_query332_00003", + "apositive_test_query332_00002", + "negative_test_query115_00008", + "negative_test_query202_00002", + "apositive_test_query348_00008", + "negative_test_query30_00010", + "negative_test_query245_00001", + "apositive_test_query162_00005", + "negative_test_query332_00009", + "negative_test_query244_00006", + "negative_test_query332_00002", + "apositive_test_query142_00004", + "negative_test_query12_00005", + "negative_test_query171_00008", + "apositive_test_query203_00002", + "negative_test_query65_00007", + "negative_test_query180_00011", + "negative_test_query145_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00002", + "negative_test_query318_00010", + "apositive_test_query322_00003", + "negative_test_query322_00009", + "negative_test_query333_00005", + "negative_test_query67_00000", + "apositive_test_query333_00000", + "negative_test_query333_00008", + "negative_test_query73_00008", + "negative_test_query246_00000", + "negative_test_query318_00005", + "negative_test_query73_00005", + "negative_test_query128_00004", + "negative_test_query21_00005", + "negative_test_query136_00003", + "negative_test_query73_00014", + "apositive_test_query228_00000", + "negative_test_query216_00013", + "apositive_test_query322_00000", + "apositive_test_query254_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query155_00010", + "negative_test_query107_00001", + "negative_test_query111_00017", + "apositive_test_query57_00006", + "negative_test_query111_00005", + "negative_test_query111_00010", + "negative_test_query155_00013", + "negative_test_query111_00006", + "negative_test_query37_00000", + "negative_test_query334_00012", + "negative_test_query213_00009", + "negative_test_query170_00004", + "negative_test_query209_00003", + "negative_test_query102_00015", + "negative_test_query334_00010", + "apositive_test_query276_00009", + "apositive_test_query26_00001", + "negative_test_query164_00005", + "negative_test_query334_00016", + "negative_test_query223_00016" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.6 + }, + "query_id": "test_query335", + "ranked_ids": [ + "apositive_test_query335_00000", + "apositive_test_query335_00003", + "apositive_test_query335_00002", + "negative_test_query335_00006", + "negative_test_query335_00000", + "negative_test_query335_00004", + "negative_test_query335_00007", + "apositive_test_query335_00004", + "negative_test_query45_00010", + "negative_test_query210_00005", + "negative_test_query335_00003", + "negative_test_query335_00001", + "negative_test_query335_00009", + "negative_test_query321_00013", + "negative_test_query335_00013", + "apositive_test_query335_00001", + "negative_test_query335_00010", + "negative_test_query210_00006", + "negative_test_query92_00002", + "negative_test_query335_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "negative_test_query336_00009", + "negative_test_query336_00014", + "apositive_test_query50_00005", + "apositive_test_query50_00015", + "negative_test_query336_00007", + "apositive_test_query38_00005", + "negative_test_query336_00004", + "apositive_test_query50_00012", + "apositive_test_query186_00002", + "negative_test_query247_00006", + "apositive_test_query242_00012", + "negative_test_query247_00003", + "negative_test_query105_00011", + "apositive_test_query186_00000", + "negative_test_query108_00005", + "negative_test_query247_00000", + "negative_test_query247_00008", + "negative_test_query336_00015", + "negative_test_query186_00012", + "apositive_test_query113_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00010", + "negative_test_query337_00007", + "negative_test_query159_00013", + "apositive_test_query337_00000", + "negative_test_query200_00007", + "negative_test_query200_00000", + "negative_test_query337_00000", + "negative_test_query200_00001", + "negative_test_query200_00009", + "negative_test_query200_00015", + "apositive_test_query38_00003", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query38_00008", + "negative_test_query337_00001", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query337_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query338", + "ranked_ids": [ + "apositive_test_query338_00002", + "apositive_test_query338_00008", + "apositive_test_query338_00006", + "apositive_test_query338_00005", + "negative_test_query81_00006", + "apositive_test_query338_00001", + "apositive_test_query338_00003", + "apositive_test_query338_00000", + "negative_test_query338_00003", + "apositive_test_query81_00000", + "apositive_test_query338_00004", + "apositive_test_query114_00006", + "negative_test_query114_00004", + "negative_test_query114_00007", + "negative_test_query81_00004", + "negative_test_query95_00004", + "negative_test_query119_00003", + "negative_test_query76_00005", + "negative_test_query53_00008", + "apositive_test_query114_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query339_00001", + "apositive_test_query339_00000", + "negative_test_query339_00009", + "negative_test_query339_00008", + "negative_test_query339_00004", + "negative_test_query96_00016", + "negative_test_query339_00017", + "negative_test_query339_00006", + "negative_test_query192_00011", + "negative_test_query339_00016", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query339_00012", + "apositive_test_query96_00001", + "apositive_test_query96_00002", + "negative_test_query339_00001", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004", + "negative_test_query96_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00000", + "apositive_test_query352_00002", + "negative_test_query34_00008", + "apositive_test_query34_00004", + "apositive_test_query352_00003", + "apositive_test_query34_00000", + "negative_test_query34_00002", + "apositive_test_query34_00001", + "negative_test_query352_00004", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "negative_test_query352_00000", + "negative_test_query352_00010", + "apositive_test_query352_00001", + "negative_test_query34_00000", + "negative_test_query352_00008", + "negative_test_query352_00013", + "negative_test_query34_00009", + "negative_test_query34_00014", + "negative_test_query352_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00001", + "apositive_test_query340_00004", + "negative_test_query340_00008", + "apositive_test_query340_00000", + "negative_test_query88_00006", + "apositive_test_query340_00002", + "negative_test_query340_00011", + "negative_test_query340_00006", + "negative_test_query340_00014", + "negative_test_query340_00012", + "negative_test_query88_00007", + "negative_test_query88_00000", + "negative_test_query340_00001", + "negative_test_query340_00007", + "negative_test_query88_00013", + "apositive_test_query174_00002", + "apositive_test_query340_00003", + "apositive_test_query174_00012", + "negative_test_query23_00012", + "negative_test_query88_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.25 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query196_00009", + "apositive_test_query341_00005", + "apositive_test_query32_00007", + "apositive_test_query341_00001", + "negative_test_query341_00000", + "negative_test_query101_00003", + "negative_test_query32_00003", + "apositive_test_query341_00002", + "negative_test_query96_00005", + "apositive_test_query341_00003", + "negative_test_query341_00003", + "negative_test_query73_00014", + "negative_test_query196_00006", + "negative_test_query214_00005", + "negative_test_query316_00008", + "negative_test_query341_00009", + "negative_test_query341_00002", + "negative_test_query96_00013", + "negative_test_query96_00007", + "negative_test_query341_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query342_00009", + "negative_test_query200_00009", + "apositive_test_query342_00000", + "negative_test_query342_00015", + "negative_test_query77_00002", + "negative_test_query134_00013", + "negative_test_query84_00007", + "negative_test_query98_00017", + "negative_test_query98_00011", + "negative_test_query77_00004", + "negative_test_query342_00016", + "negative_test_query212_00009", + "negative_test_query68_00005", + "negative_test_query49_00002", + "negative_test_query136_00003", + "negative_test_query78_00015", + "negative_test_query245_00001", + "negative_test_query324_00015", + "negative_test_query325_00015", + "negative_test_query342_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query343", + "ranked_ids": [ + "negative_test_query103_00010", + "apositive_test_query343_00001", + "apositive_test_query185_00001", + "negative_test_query343_00008", + "negative_test_query103_00008", + "apositive_test_query185_00008", + "apositive_test_query307_00008", + "apositive_test_query217_00004", + "apositive_test_query28_00006", + "negative_test_query225_00008", + "apositive_test_query343_00000", + "apositive_test_query217_00002", + "apositive_test_query279_00003", + "negative_test_query307_00006", + "apositive_test_query136_00004", + "apositive_test_query133_00005", + "apositive_test_query287_00003", + "negative_test_query360_00017", + "apositive_test_query279_00002", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query282_00008", + "apositive_test_query83_00006", + "negative_test_query85_00010", + "negative_test_query344_00004", + "negative_test_query344_00005", + "negative_test_query344_00009", + "apositive_test_query221_00002", + "negative_test_query223_00016", + "negative_test_query345_00012", + "negative_test_query83_00008", + "negative_test_query282_00001", + "negative_test_query344_00002", + "negative_test_query83_00005", + "negative_test_query344_00012", + "negative_test_query344_00000", + "negative_test_query282_00004", + "negative_test_query344_00006", + "negative_test_query188_00000", + "negative_test_query231_00015", + "negative_test_query282_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query345_00007", + "negative_test_query345_00003", + "negative_test_query130_00004", + "negative_test_query345_00011", + "negative_test_query47_00003", + "negative_test_query184_00009", + "negative_test_query259_00007", + "apositive_test_query345_00000", + "apositive_test_query345_00001", + "negative_test_query90_00003", + "apositive_test_query0_00008", + "negative_test_query345_00010", + "negative_test_query221_00002", + "negative_test_query9_00002", + "apositive_test_query345_00004", + "negative_test_query47_00008", + "negative_test_query345_00001", + "apositive_test_query9_00000", + "apositive_test_query203_00005", + "apositive_test_query90_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query346_00001", + "negative_test_query346_00002", + "negative_test_query346_00009", + "apositive_test_query346_00001", + "negative_test_query346_00008", + "negative_test_query346_00012", + "negative_test_query346_00014", + "apositive_test_query346_00000", + "negative_test_query346_00017", + "negative_test_query346_00003", + "negative_test_query346_00006", + "negative_test_query157_00005", + "negative_test_query49_00007", + "negative_test_query346_00010", + "negative_test_query174_00001", + "negative_test_query19_00012", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "negative_test_query15_00000", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query347", + "ranked_ids": [ + "apositive_test_query347_00001", + "negative_test_query298_00003", + "negative_test_query347_00014", + "apositive_test_query347_00000", + "negative_test_query347_00000", + "negative_test_query298_00006", + "negative_test_query347_00015", + "negative_test_query31_00003", + "negative_test_query347_00016", + "negative_test_query347_00001", + "negative_test_query347_00012", + "negative_test_query251_00007", + "negative_test_query347_00007", + "negative_test_query347_00002", + "negative_test_query347_00005", + "negative_test_query347_00008", + "negative_test_query347_00006", + "negative_test_query347_00009", + "negative_test_query347_00010", + "negative_test_query347_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query348_00004", + "apositive_test_query348_00001", + "negative_test_query115_00006", + "negative_test_query115_00012", + "apositive_test_query115_00004", + "apositive_test_query115_00001", + "apositive_test_query348_00009", + "apositive_test_query115_00002", + "negative_test_query359_00014", + "apositive_test_query348_00000", + "apositive_test_query348_00006", + "negative_test_query348_00008", + "negative_test_query348_00006", + "apositive_test_query227_00002", + "negative_test_query341_00008", + "apositive_test_query348_00005", + "negative_test_query115_00002", + "negative_test_query297_00017", + "negative_test_query351_00010", + "negative_test_query31_00017" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "negative_test_query74_00001", + "negative_test_query253_00007", + "apositive_test_query253_00005", + "negative_test_query349_00015", + "apositive_test_query74_00002", + "apositive_test_query137_00002", + "negative_test_query253_00000", + "apositive_test_query253_00008", + "negative_test_query13_00002", + "negative_test_query349_00003", + "apositive_test_query74_00003", + "apositive_test_query42_00004", + "negative_test_query312_00016", + "apositive_test_query253_00009", + "negative_test_query124_00013", + "apositive_test_query355_00007", + "negative_test_query253_00009", + "negative_test_query157_00009", + "negative_test_query194_00001", + "negative_test_query349_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query35_00000", + "apositive_test_query139_00000", + "apositive_test_query35_00001", + "negative_test_query205_00005", + "negative_test_query101_00006", + "apositive_test_query35_00003", + "negative_test_query139_00016", + "negative_test_query35_00002", + "negative_test_query35_00012", + "apositive_test_query146_00007", + "negative_test_query123_00006", + "negative_test_query262_00002", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query231_00012", + "negative_test_query269_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query350_00014", + "negative_test_query350_00015", + "negative_test_query179_00001", + "negative_test_query350_00010", + "negative_test_query192_00004", + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query14_00001", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query350_00001", + "negative_test_query350_00006", + "negative_test_query192_00005", + "negative_test_query350_00011", + "negative_test_query192_00013", + "negative_test_query149_00005", + "negative_test_query63_00000", + "negative_test_query331_00012", + "negative_test_query316_00006", + "apositive_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query351_00000", + "negative_test_query234_00009", + "negative_test_query92_00005", + "apositive_test_query60_00000", + "apositive_test_query60_00003", + "apositive_test_query335_00004", + "negative_test_query53_00011", + "negative_test_query156_00006", + "apositive_test_query299_00000", + "negative_test_query351_00006", + "apositive_test_query113_00001", + "apositive_test_query60_00010", + "apositive_test_query218_00004", + "apositive_test_query60_00012", + "apositive_test_query60_00009", + "negative_test_query351_00002", + "apositive_test_query311_00000", + "negative_test_query351_00014", + "negative_test_query8_00003", + "apositive_test_query193_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.6 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query34_00001", + "apositive_test_query34_00004", + "apositive_test_query352_00002", + "apositive_test_query352_00003", + "apositive_test_query352_00004", + "apositive_test_query352_00000", + "negative_test_query34_00012", + "apositive_test_query34_00000", + "negative_test_query34_00002", + "negative_test_query352_00004", + "negative_test_query352_00000", + "negative_test_query34_00013", + "apositive_test_query352_00001", + "negative_test_query352_00013", + "negative_test_query34_00008", + "negative_test_query352_00010", + "negative_test_query34_00000", + "apositive_test_query34_00002", + "negative_test_query352_00008", + "negative_test_query34_00014" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00009", + "apositive_test_query204_00002", + "apositive_test_query353_00005", + "apositive_test_query353_00011", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "negative_test_query353_00003", + "apositive_test_query353_00000", + "negative_test_query108_00002", + "apositive_test_query204_00012", + "negative_test_query353_00002", + "apositive_test_query353_00009", + "apositive_test_query51_00002", + "apositive_test_query204_00015", + "apositive_test_query353_00007", + "negative_test_query175_00012", + "negative_test_query71_00008", + "apositive_test_query204_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.625 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "apositive_test_query354_00004", + "apositive_test_query354_00006", + "apositive_test_query354_00001", + "apositive_test_query16_00003", + "apositive_test_query354_00005", + "apositive_test_query147_00002", + "negative_test_query354_00004", + "negative_test_query309_00014", + "apositive_test_query16_00001", + "apositive_test_query320_00006", + "negative_test_query309_00013", + "negative_test_query194_00002", + "negative_test_query358_00003", + "negative_test_query280_00007", + "negative_test_query331_00014", + "negative_test_query148_00009", + "negative_test_query309_00000", + "negative_test_query354_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query355_00001", + "negative_test_query355_00007", + "negative_test_query355_00009", + "apositive_test_query355_00002", + "negative_test_query355_00005", + "apositive_test_query355_00005", + "apositive_test_query355_00003", + "negative_test_query355_00003", + "negative_test_query264_00003", + "apositive_test_query355_00006", + "apositive_test_query100_00001", + "negative_test_query355_00002", + "negative_test_query355_00008", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query87_00004", + "negative_test_query215_00001", + "negative_test_query68_00000", + "negative_test_query104_00001", + "apositive_test_query38_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query356_00000", + "apositive_test_query66_00003", + "apositive_test_query356_00004", + "apositive_test_query356_00002", + "negative_test_query102_00014", + "negative_test_query47_00011", + "apositive_test_query203_00006", + "apositive_test_query154_00009", + "apositive_test_query30_00005", + "negative_test_query102_00010", + "apositive_test_query33_00000", + "apositive_test_query20_00001", + "apositive_test_query127_00007", + "negative_test_query253_00001", + "negative_test_query15_00006", + "negative_test_query280_00004", + "apositive_test_query356_00008", + "negative_test_query334_00000", + "apositive_test_query295_00000", + "apositive_test_query88_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00001", + "apositive_test_query221_00000", + "apositive_test_query119_00005", + "apositive_test_query119_00009", + "apositive_test_query221_00002", + "apositive_test_query357_00004", + "apositive_test_query119_00000", + "negative_test_query58_00008", + "apositive_test_query117_00004", + "apositive_test_query119_00010", + "apositive_test_query221_00004", + "apositive_test_query32_00002", + "negative_test_query46_00004", + "apositive_test_query119_00012", + "apositive_test_query357_00003", + "apositive_test_query119_00002", + "apositive_test_query314_00000", + "negative_test_query300_00007", + "negative_test_query58_00009", + "negative_test_query196_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.3 + }, + "query_id": "test_query358", + "ranked_ids": [ + "negative_test_query123_00007", + "negative_test_query309_00006", + "apositive_test_query358_00004", + "apositive_test_query358_00007", + "apositive_test_query358_00006", + "negative_test_query11_00000", + "apositive_test_query11_00007", + "apositive_test_query358_00003", + "apositive_test_query358_00000", + "negative_test_query279_00010", + "apositive_test_query358_00009", + "apositive_test_query161_00000", + "negative_test_query132_00001", + "negative_test_query309_00005", + "apositive_test_query309_00000", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "negative_test_query354_00008", + "negative_test_query309_00015", + "negative_test_query42_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00003", + "apositive_test_query0_00006", + "negative_test_query359_00002", + "negative_test_query305_00000", + "apositive_test_query0_00007", + "apositive_test_query259_00004", + "apositive_test_query259_00008", + "negative_test_query359_00007", + "negative_test_query130_00000", + "apositive_test_query0_00000", + "negative_test_query268_00014", + "negative_test_query259_00006", + "apositive_test_query359_00002", + "apositive_test_query0_00004", + "negative_test_query0_00001", + "apositive_test_query0_00010", + "negative_test_query0_00007", + "negative_test_query184_00016", + "apositive_test_query0_00002", + "apositive_test_query259_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query36", + "ranked_ids": [ + "negative_test_query112_00015", + "apositive_test_query286_00006", + "negative_test_query112_00004", + "apositive_test_query36_00003", + "negative_test_query36_00014", + "negative_test_query112_00016", + "apositive_test_query286_00005", + "negative_test_query286_00000", + "negative_test_query112_00010", + "negative_test_query36_00010", + "negative_test_query260_00009", + "negative_test_query36_00001", + "apositive_test_query136_00013", + "negative_test_query216_00002", + "apositive_test_query106_00003", + "negative_test_query296_00013", + "negative_test_query6_00003", + "negative_test_query112_00013", + "negative_test_query260_00010", + "negative_test_query80_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00001", + "negative_test_query189_00014", + "negative_test_query360_00005", + "apositive_test_query69_00014", + "negative_test_query5_00014", + "apositive_test_query153_00011", + "negative_test_query281_00000", + "apositive_test_query360_00000", + "negative_test_query360_00004", + "apositive_test_query310_00006", + "negative_test_query189_00013", + "negative_test_query175_00009", + "negative_test_query312_00004", + "apositive_test_query152_00002", + "apositive_test_query142_00004", + "negative_test_query240_00010", + "negative_test_query148_00018", + "negative_test_query216_00011", + "negative_test_query246_00001", + "negative_test_query360_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query231_00006", + "apositive_test_query158_00007", + "apositive_test_query37_00001", + "negative_test_query223_00010", + "negative_test_query209_00000", + "apositive_test_query177_00000", + "apositive_test_query108_00000", + "negative_test_query209_00001", + "negative_test_query57_00009", + "apositive_test_query213_00006", + "negative_test_query123_00001", + "negative_test_query152_00008", + "negative_test_query155_00001", + "negative_test_query123_00011", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query20_00003", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query209_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query38", + "ranked_ids": [ + "negative_test_query267_00016", + "negative_test_query38_00010", + "negative_test_query267_00001", + "negative_test_query38_00006", + "negative_test_query159_00011", + "apositive_test_query38_00003", + "negative_test_query337_00000", + "apositive_test_query247_00005", + "negative_test_query337_00017", + "negative_test_query94_00011", + "negative_test_query47_00016", + "apositive_test_query38_00000", + "apositive_test_query337_00000", + "apositive_test_query38_00002", + "negative_test_query264_00012", + "negative_test_query79_00002", + "negative_test_query159_00010", + "negative_test_query267_00015", + "apositive_test_query38_00007", + "negative_test_query175_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.7142857142857143 + }, + "query_id": "test_query39", + "ranked_ids": [ + "apositive_test_query39_00004", + "apositive_test_query39_00001", + "apositive_test_query39_00003", + "apositive_test_query39_00002", + "apositive_test_query39_00000", + "negative_test_query39_00006", + "negative_test_query39_00009", + "negative_test_query39_00007", + "negative_test_query39_00004", + "negative_test_query66_00008", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00017", + "negative_test_query18_00004", + "negative_test_query39_00000", + "negative_test_query39_00012", + "negative_test_query331_00010", + "negative_test_query39_00003", + "negative_test_query232_00009", + "negative_test_query29_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.35714285714285715 + }, + "query_id": "test_query4", + "ranked_ids": [ + "apositive_test_query4_00000", + "apositive_test_query4_00004", + "apositive_test_query4_00010", + "apositive_test_query4_00001", + "apositive_test_query4_00003", + "negative_test_query4_00001", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "negative_test_query266_00004", + "negative_test_query312_00011", + "apositive_test_query4_00005", + "apositive_test_query4_00012", + "negative_test_query266_00009", + "negative_test_query174_00001", + "negative_test_query4_00002", + "apositive_test_query19_00000", + "negative_test_query149_00015", + "negative_test_query19_00004", + "negative_test_query266_00012", + "negative_test_query15_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00005", + "apositive_test_query40_00002", + "negative_test_query40_00006", + "negative_test_query40_00011", + "negative_test_query40_00004", + "negative_test_query40_00007", + "negative_test_query40_00001", + "negative_test_query40_00010", + "negative_test_query40_00013", + "negative_test_query40_00000", + "negative_test_query40_00003", + "negative_test_query40_00012", + "negative_test_query66_00005", + "negative_test_query214_00015", + "negative_test_query347_00003", + "negative_test_query328_00006", + "negative_test_query106_00009", + "negative_test_query40_00009", + "apositive_test_query223_00001", + "negative_test_query324_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7365896932159578, + "recall@5": 0.75 + }, + "query_id": "test_query41", + "ranked_ids": [ + "apositive_test_query41_00002", + "negative_test_query41_00000", + "apositive_test_query41_00000", + "negative_test_query41_00007", + "apositive_test_query41_00001", + "negative_test_query41_00004", + "negative_test_query41_00002", + "negative_test_query41_00010", + "negative_test_query282_00003", + "apositive_test_query249_00012", + "negative_test_query144_00016", + "negative_test_query41_00014", + "negative_test_query41_00012", + "negative_test_query41_00006", + "apositive_test_query41_00003", + "negative_test_query226_00011", + "negative_test_query41_00011", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query98_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.25 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query42_00002", + "negative_test_query42_00000", + "apositive_test_query42_00003", + "negative_test_query42_00003", + "apositive_test_query42_00002", + "negative_test_query42_00005", + "apositive_test_query42_00005", + "apositive_test_query42_00000", + "negative_test_query42_00008", + "apositive_test_query42_00001", + "negative_test_query42_00001", + "negative_test_query42_00010", + "negative_test_query42_00006", + "apositive_test_query42_00004", + "negative_test_query137_00005", + "negative_test_query42_00007", + "apositive_test_query276_00012", + "negative_test_query42_00011", + "negative_test_query250_00010", + "apositive_test_query162_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.25 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query43_00000", + "negative_test_query43_00009", + "negative_test_query333_00001", + "apositive_test_query43_00002", + "apositive_test_query153_00008", + "negative_test_query179_00006", + "apositive_test_query276_00001", + "negative_test_query146_00002", + "apositive_test_query161_00005", + "apositive_test_query161_00002", + "negative_test_query43_00000", + "apositive_test_query314_00009", + "apositive_test_query153_00014", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "apositive_test_query146_00004", + "apositive_test_query146_00003", + "apositive_test_query218_00002", + "negative_test_query319_00002", + "negative_test_query82_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query44", + "ranked_ids": [ + "apositive_test_query44_00001", + "negative_test_query27_00002", + "negative_test_query330_00009", + "negative_test_query141_00007", + "negative_test_query27_00011", + "negative_test_query44_00002", + "negative_test_query293_00003", + "negative_test_query10_00012", + "negative_test_query27_00008", + "apositive_test_query293_00000", + "apositive_test_query20_00003", + "negative_test_query180_00000", + "negative_test_query293_00015", + "negative_test_query293_00009", + "negative_test_query293_00000", + "apositive_test_query141_00005", + "apositive_test_query245_00000", + "negative_test_query21_00009", + "negative_test_query330_00012", + "negative_test_query190_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query45_00001", + "negative_test_query45_00013", + "negative_test_query143_00014", + "apositive_test_query45_00000", + "negative_test_query45_00003", + "apositive_test_query165_00000", + "negative_test_query45_00016", + "negative_test_query143_00004", + "negative_test_query45_00004", + "negative_test_query45_00007", + "negative_test_query45_00006", + "negative_test_query143_00011", + "negative_test_query45_00009", + "apositive_test_query25_00000", + "negative_test_query182_00016", + "negative_test_query328_00005", + "negative_test_query45_00005", + "negative_test_query143_00009", + "negative_test_query342_00005", + "apositive_test_query25_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "negative_test_query46_00011", + "negative_test_query56_00001", + "negative_test_query359_00005", + "negative_test_query46_00001", + "apositive_test_query79_00005", + "negative_test_query108_00003", + "negative_test_query105_00006", + "apositive_test_query146_00013", + "apositive_test_query142_00001", + "negative_test_query213_00012", + "negative_test_query238_00001", + "negative_test_query313_00010", + "negative_test_query168_00003", + "negative_test_query79_00012", + "apositive_test_query276_00008", + "negative_test_query39_00011", + "negative_test_query46_00003", + "negative_test_query334_00004", + "apositive_test_query186_00001", + "negative_test_query46_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query331_00002", + "negative_test_query345_00002", + "negative_test_query331_00013", + "negative_test_query47_00002", + "apositive_test_query201_00002", + "apositive_test_query47_00001", + "negative_test_query47_00013", + "negative_test_query331_00014", + "negative_test_query47_00006", + "negative_test_query47_00012", + "apositive_test_query20_00005", + "negative_test_query47_00011", + "apositive_test_query201_00004", + "negative_test_query144_00018", + "negative_test_query325_00007", + "negative_test_query41_00009", + "negative_test_query195_00005", + "negative_test_query331_00005", + "apositive_test_query201_00006", + "negative_test_query328_00007" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "apositive_test_query185_00012", + "negative_test_query48_00016", + "apositive_test_query48_00000", + "negative_test_query307_00005", + "negative_test_query48_00007", + "apositive_test_query28_00013", + "negative_test_query48_00009", + "apositive_test_query28_00010", + "negative_test_query103_00012", + "negative_test_query307_00003", + "negative_test_query48_00004", + "negative_test_query49_00001", + "apositive_test_query343_00001", + "negative_test_query48_00012", + "negative_test_query48_00011", + "apositive_test_query185_00004", + "negative_test_query48_00018", + "negative_test_query307_00001", + "negative_test_query48_00000", + "apositive_test_query307_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query49", + "ranked_ids": [ + "negative_test_query169_00000", + "apositive_test_query49_00001", + "negative_test_query106_00004", + "apositive_test_query54_00009", + "negative_test_query176_00016", + "negative_test_query49_00002", + "negative_test_query49_00007", + "negative_test_query49_00001", + "apositive_test_query19_00001", + "negative_test_query52_00011", + "negative_test_query156_00007", + "negative_test_query122_00016", + "negative_test_query342_00005", + "apositive_test_query29_00006", + "negative_test_query328_00009", + "negative_test_query15_00008", + "negative_test_query72_00017", + "negative_test_query156_00010", + "apositive_test_query204_00003", + "apositive_test_query353_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6797310500037655, + "recall@5": 1.0 + }, + "query_id": "test_query5", + "ranked_ids": [ + "negative_test_query246_00013", + "apositive_test_query5_00000", + "negative_test_query5_00005", + "apositive_test_query5_00002", + "apositive_test_query5_00001", + "negative_test_query5_00003", + "negative_test_query5_00006", + "negative_test_query5_00010", + "negative_test_query5_00007", + "negative_test_query5_00000", + "negative_test_query5_00001", + "negative_test_query5_00013", + "negative_test_query67_00003", + "negative_test_query67_00010", + "negative_test_query246_00002", + "negative_test_query275_00001", + "negative_test_query5_00004", + "negative_test_query5_00014", + "negative_test_query301_00003", + "negative_test_query246_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query186_00005", + "apositive_test_query50_00014", + "apositive_test_query50_00003", + "apositive_test_query50_00005", + "apositive_test_query50_00002", + "negative_test_query336_00004", + "apositive_test_query50_00006", + "negative_test_query50_00001", + "apositive_test_query50_00009", + "negative_test_query336_00011", + "apositive_test_query50_00008", + "negative_test_query66_00002", + "negative_test_query223_00011", + "apositive_test_query50_00013", + "apositive_test_query50_00015", + "apositive_test_query50_00001", + "apositive_test_query131_00000", + "apositive_test_query50_00012", + "negative_test_query131_00001", + "negative_test_query119_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2 + }, + "query_id": "test_query51", + "ranked_ids": [ + "apositive_test_query51_00001", + "apositive_test_query51_00005", + "negative_test_query51_00006", + "negative_test_query51_00000", + "negative_test_query97_00001", + "apositive_test_query51_00007", + "apositive_test_query51_00000", + "apositive_test_query51_00008", + "negative_test_query267_00007", + "negative_test_query97_00011", + "negative_test_query97_00002", + "negative_test_query51_00004", + "negative_test_query204_00001", + "apositive_test_query51_00006", + "negative_test_query97_00007", + "negative_test_query97_00006", + "apositive_test_query51_00009", + "negative_test_query209_00008", + "negative_test_query231_00012", + "apositive_test_query267_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query52_00007", + "negative_test_query257_00005", + "apositive_test_query257_00002", + "apositive_test_query69_00007", + "apositive_test_query257_00004", + "negative_test_query54_00004", + "apositive_test_query286_00006", + "apositive_test_query257_00001", + "apositive_test_query257_00003", + "apositive_test_query257_00000", + "apositive_test_query257_00005", + "apositive_test_query286_00014", + "negative_test_query257_00009", + "apositive_test_query286_00013", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query52_00005", + "negative_test_query257_00007", + "negative_test_query52_00004", + "apositive_test_query52_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00016", + "negative_test_query258_00008", + "apositive_test_query28_00006", + "negative_test_query161_00007", + "negative_test_query150_00015", + "negative_test_query354_00011", + "negative_test_query258_00013", + "negative_test_query258_00000", + "negative_test_query3_00008", + "negative_test_query169_00003", + "negative_test_query354_00001", + "negative_test_query53_00000", + "apositive_test_query217_00004", + "negative_test_query150_00014", + "negative_test_query230_00009", + "negative_test_query180_00012", + "negative_test_query258_00018", + "negative_test_query348_00001", + "apositive_test_query248_00005", + "negative_test_query132_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query54_00006", + "negative_test_query44_00014", + "negative_test_query140_00006", + "apositive_test_query51_00006", + "negative_test_query126_00008", + "negative_test_query97_00002", + "apositive_test_query51_00007", + "apositive_test_query51_00008", + "negative_test_query151_00008", + "apositive_test_query86_00000", + "negative_test_query51_00006", + "apositive_test_query323_00006", + "negative_test_query56_00011", + "negative_test_query44_00007", + "negative_test_query200_00006", + "negative_test_query301_00006", + "negative_test_query136_00004", + "apositive_test_query156_00000", + "apositive_test_query204_00002", + "apositive_test_query335_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4776237035032179, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query55", + "ranked_ids": [ + "negative_test_query55_00002", + "apositive_test_query55_00001", + "negative_test_query55_00005", + "negative_test_query55_00003", + "apositive_test_query55_00000", + "negative_test_query55_00016", + "negative_test_query55_00007", + "apositive_test_query55_00002", + "negative_test_query61_00008", + "negative_test_query55_00011", + "negative_test_query139_00015", + "negative_test_query234_00011", + "negative_test_query55_00009", + "negative_test_query55_00013", + "negative_test_query55_00008", + "negative_test_query55_00010", + "negative_test_query55_00012", + "negative_test_query55_00015", + "negative_test_query55_00000", + "negative_test_query55_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query56", + "ranked_ids": [ + "apositive_test_query56_00005", + "negative_test_query56_00003", + "negative_test_query56_00000", + "negative_test_query46_00011", + "negative_test_query186_00007", + "negative_test_query56_00006", + "negative_test_query105_00006", + "negative_test_query111_00006", + "negative_test_query56_00004", + "negative_test_query359_00005", + "negative_test_query119_00005", + "negative_test_query56_00002", + "negative_test_query337_00016", + "negative_test_query56_00005", + "apositive_test_query56_00006", + "negative_test_query46_00014", + "apositive_test_query238_00001", + "negative_test_query200_00014", + "apositive_test_query46_00001", + "negative_test_query102_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query57", + "ranked_ids": [ + "apositive_test_query57_00003", + "negative_test_query295_00003", + "apositive_test_query57_00005", + "apositive_test_query57_00006", + "apositive_test_query57_00008", + "negative_test_query46_00006", + "apositive_test_query57_00000", + "negative_test_query111_00010", + "negative_test_query57_00001", + "apositive_test_query338_00002", + "negative_test_query334_00001", + "negative_test_query57_00010", + "apositive_test_query74_00000", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query198_00016", + "negative_test_query65_00010", + "apositive_test_query259_00006", + "apositive_test_query146_00000", + "apositive_test_query203_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "negative_test_query58_00010", + "negative_test_query58_00015", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query240_00016", + "negative_test_query58_00001", + "negative_test_query57_00004", + "apositive_test_query311_00003", + "apositive_test_query311_00004", + "negative_test_query308_00004", + "negative_test_query11_00004", + "negative_test_query240_00005", + "negative_test_query346_00000", + "negative_test_query240_00002", + "negative_test_query240_00008", + "negative_test_query27_00017", + "negative_test_query27_00016", + "negative_test_query311_00012", + "negative_test_query240_00004", + "negative_test_query240_00015" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "apositive_test_query59_00000", + "negative_test_query59_00012", + "negative_test_query59_00010", + "negative_test_query59_00017", + "negative_test_query116_00013", + "negative_test_query59_00006", + "negative_test_query59_00014", + "negative_test_query59_00018", + "negative_test_query116_00003", + "negative_test_query59_00011", + "negative_test_query59_00015", + "negative_test_query202_00007", + "negative_test_query116_00000", + "negative_test_query59_00009", + "negative_test_query227_00008", + "apositive_test_query104_00000", + "negative_test_query66_00008", + "negative_test_query81_00003", + "negative_test_query59_00005", + "negative_test_query280_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query6", + "ranked_ids": [ + "apositive_test_query6_00000", + "apositive_test_query6_00002", + "apositive_test_query6_00001", + "negative_test_query6_00002", + "apositive_test_query305_00003", + "apositive_test_query109_00001", + "negative_test_query252_00004", + "negative_test_query58_00007", + "apositive_test_query242_00004", + "negative_test_query6_00001", + "negative_test_query112_00010", + "negative_test_query112_00004", + "negative_test_query296_00006", + "negative_test_query6_00012", + "negative_test_query179_00007", + "negative_test_query6_00004", + "negative_test_query296_00013", + "apositive_test_query189_00000", + "negative_test_query6_00016", + "negative_test_query6_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3125 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00003", + "apositive_test_query60_00007", + "apositive_test_query60_00001", + "apositive_test_query60_00013", + "apositive_test_query60_00006", + "negative_test_query243_00004", + "negative_test_query304_00002", + "apositive_test_query128_00001", + "apositive_test_query60_00005", + "negative_test_query31_00004", + "negative_test_query60_00001", + "negative_test_query60_00003", + "negative_test_query218_00007", + "apositive_test_query60_00000", + "apositive_test_query218_00004", + "apositive_test_query60_00009", + "apositive_test_query285_00001", + "apositive_test_query285_00007", + "negative_test_query304_00003", + "apositive_test_query60_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query61", + "ranked_ids": [ + "apositive_test_query61_00003", + "apositive_test_query61_00002", + "apositive_test_query61_00001", + "negative_test_query61_00014", + "negative_test_query61_00006", + "negative_test_query61_00010", + "negative_test_query61_00003", + "negative_test_query211_00005", + "negative_test_query173_00003", + "negative_test_query61_00015", + "negative_test_query210_00012", + "negative_test_query327_00016", + "negative_test_query61_00002", + "negative_test_query129_00005", + "negative_test_query61_00005", + "negative_test_query201_00004", + "negative_test_query61_00000", + "apositive_test_query101_00002", + "apositive_test_query173_00004", + "negative_test_query61_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00000", + "negative_test_query250_00003", + "negative_test_query62_00002", + "negative_test_query62_00011", + "apositive_test_query62_00001", + "negative_test_query62_00010", + "negative_test_query287_00007", + "negative_test_query219_00006", + "negative_test_query88_00015", + "negative_test_query222_00007", + "negative_test_query62_00006", + "negative_test_query15_00011", + "negative_test_query62_00004", + "negative_test_query62_00003", + "negative_test_query62_00001", + "negative_test_query62_00013", + "negative_test_query62_00008", + "negative_test_query352_00005", + "negative_test_query16_00001", + "negative_test_query38_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query63", + "ranked_ids": [ + "negative_test_query63_00005", + "apositive_test_query63_00006", + "negative_test_query224_00007", + "negative_test_query63_00002", + "apositive_test_query63_00001", + "negative_test_query63_00001", + "apositive_test_query63_00008", + "negative_test_query297_00006", + "negative_test_query63_00003", + "negative_test_query63_00010", + "negative_test_query331_00010", + "negative_test_query63_00009", + "apositive_test_query141_00001", + "negative_test_query321_00002", + "apositive_test_query63_00004", + "negative_test_query78_00006", + "negative_test_query264_00015", + "negative_test_query63_00004", + "apositive_test_query288_00004", + "negative_test_query241_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query64_00015", + "negative_test_query64_00009", + "negative_test_query64_00013", + "negative_test_query211_00009", + "negative_test_query63_00009", + "negative_test_query64_00016", + "negative_test_query205_00009", + "negative_test_query64_00010", + "apositive_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query310_00005", + "negative_test_query149_00001", + "negative_test_query63_00000", + "negative_test_query214_00004", + "negative_test_query149_00017", + "negative_test_query120_00003", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query120_00000", + "negative_test_query234_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "apositive_test_query65_00001", + "negative_test_query65_00004", + "apositive_test_query134_00001", + "apositive_test_query65_00000", + "negative_test_query260_00013", + "negative_test_query30_00011", + "negative_test_query47_00005", + "negative_test_query23_00006", + "negative_test_query6_00003", + "negative_test_query6_00008", + "apositive_test_query356_00004", + "negative_test_query335_00002", + "negative_test_query260_00012", + "apositive_test_query260_00000", + "apositive_test_query62_00000", + "negative_test_query230_00001", + "negative_test_query195_00009", + "negative_test_query261_00003", + "negative_test_query268_00012", + "negative_test_query65_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.5 + }, + "query_id": "test_query66", + "ranked_ids": [ + "apositive_test_query66_00004", + "apositive_test_query66_00003", + "negative_test_query220_00010", + "apositive_test_query188_00000", + "apositive_test_query66_00005", + "negative_test_query188_00014", + "apositive_test_query188_00003", + "apositive_test_query20_00005", + "apositive_test_query20_00001", + "negative_test_query161_00008", + "negative_test_query154_00009", + "negative_test_query161_00003", + "negative_test_query9_00012", + "negative_test_query344_00007", + "apositive_test_query220_00000", + "negative_test_query188_00006", + "negative_test_query66_00011", + "apositive_test_query66_00002", + "negative_test_query220_00001", + "apositive_test_query220_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query67", + "ranked_ids": [ + "negative_test_query67_00005", + "negative_test_query67_00011", + "apositive_test_query286_00010", + "negative_test_query67_00010", + "apositive_test_query67_00001", + "apositive_test_query67_00000", + "negative_test_query246_00016", + "apositive_test_query246_00000", + "apositive_test_query286_00007", + "negative_test_query67_00001", + "negative_test_query246_00009", + "negative_test_query260_00013", + "negative_test_query6_00008", + "apositive_test_query286_00004", + "negative_test_query67_00008", + "negative_test_query67_00007", + "negative_test_query67_00015", + "negative_test_query123_00008", + "negative_test_query65_00008", + "negative_test_query55_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query68", + "ranked_ids": [ + "apositive_test_query68_00002", + "negative_test_query212_00000", + "negative_test_query68_00001", + "apositive_test_query68_00000", + "negative_test_query68_00014", + "negative_test_query151_00008", + "apositive_test_query68_00001", + "apositive_test_query5_00002", + "negative_test_query330_00009", + "apositive_test_query252_00001", + "negative_test_query68_00007", + "apositive_test_query330_00001", + "apositive_test_query89_00003", + "negative_test_query288_00004", + "negative_test_query68_00013", + "negative_test_query126_00008", + "negative_test_query68_00016", + "negative_test_query311_00005", + "apositive_test_query326_00005", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00002", + "apositive_test_query69_00000", + "apositive_test_query257_00003", + "apositive_test_query69_00010", + "apositive_test_query69_00011", + "negative_test_query18_00010", + "apositive_test_query69_00005", + "apositive_test_query174_00016", + "apositive_test_query281_00016", + "apositive_test_query355_00001", + "apositive_test_query69_00008", + "apositive_test_query54_00003", + "apositive_test_query225_00001", + "apositive_test_query69_00013", + "negative_test_query169_00002", + "apositive_test_query69_00009", + "apositive_test_query69_00012", + "apositive_test_query107_00007", + "negative_test_query277_00001", + "apositive_test_query52_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "negative_test_query306_00008", + "apositive_test_query119_00006", + "apositive_test_query212_00001", + "apositive_test_query326_00007", + "apositive_test_query38_00003", + "apositive_test_query212_00002", + "negative_test_query7_00001", + "negative_test_query7_00005", + "negative_test_query159_00004", + "negative_test_query159_00009", + "negative_test_query184_00006", + "negative_test_query231_00011", + "negative_test_query84_00007", + "apositive_test_query159_00001", + "negative_test_query264_00013", + "negative_test_query38_00010", + "negative_test_query79_00002", + "negative_test_query327_00011", + "negative_test_query290_00006", + "negative_test_query159_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00004", + "negative_test_query70_00011", + "negative_test_query227_00013", + "apositive_test_query70_00002", + "negative_test_query227_00003", + "negative_test_query227_00012", + "negative_test_query70_00012", + "negative_test_query227_00006", + "negative_test_query227_00014", + "negative_test_query70_00000", + "apositive_test_query227_00000", + "apositive_test_query70_00001", + "negative_test_query70_00006", + "negative_test_query70_00003", + "negative_test_query227_00000", + "negative_test_query348_00007", + "negative_test_query227_00010", + "apositive_test_query170_00001", + "negative_test_query283_00014", + "negative_test_query59_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8048099750039491, + "recall@5": 0.75 + }, + "query_id": "test_query71", + "ranked_ids": [ + "apositive_test_query71_00003", + "apositive_test_query71_00000", + "negative_test_query71_00009", + "apositive_test_query71_00002", + "negative_test_query231_00014", + "negative_test_query186_00000", + "negative_test_query247_00002", + "negative_test_query71_00015", + "negative_test_query259_00002", + "negative_test_query71_00013", + "negative_test_query71_00005", + "negative_test_query71_00011", + "apositive_test_query30_00005", + "negative_test_query71_00001", + "negative_test_query65_00002", + "apositive_test_query71_00001", + "negative_test_query71_00000", + "apositive_test_query209_00005", + "negative_test_query71_00002", + "negative_test_query313_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "apositive_test_query72_00000", + "negative_test_query72_00001", + "negative_test_query72_00007", + "negative_test_query72_00015", + "negative_test_query72_00011", + "negative_test_query185_00005", + "negative_test_query72_00008", + "negative_test_query72_00003", + "negative_test_query72_00006", + "negative_test_query72_00016", + "negative_test_query72_00013", + "negative_test_query185_00000", + "negative_test_query17_00009", + "negative_test_query55_00005", + "negative_test_query92_00001", + "apositive_test_query175_00001", + "negative_test_query228_00000", + "negative_test_query285_00009", + "negative_test_query265_00003", + "negative_test_query233_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query73", + "ranked_ids": [ + "apositive_test_query322_00003", + "negative_test_query73_00013", + "apositive_test_query333_00005", + "negative_test_query216_00013", + "apositive_test_query73_00000", + "negative_test_query73_00010", + "negative_test_query128_00004", + "negative_test_query322_00009", + "negative_test_query21_00005", + "apositive_test_query254_00004", + "negative_test_query73_00001", + "apositive_test_query333_00000", + "negative_test_query228_00013", + "apositive_test_query333_00002", + "negative_test_query333_00005", + "negative_test_query73_00006", + "negative_test_query73_00008", + "apositive_test_query322_00008", + "negative_test_query233_00015", + "negative_test_query246_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7095272044910244, + "recall@5": 0.75 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query74_00000", + "negative_test_query74_00000", + "negative_test_query74_00015", + "apositive_test_query74_00001", + "apositive_test_query74_00002", + "negative_test_query95_00009", + "negative_test_query74_00013", + "negative_test_query74_00014", + "negative_test_query74_00001", + "apositive_test_query74_00003", + "negative_test_query44_00006", + "apositive_test_query114_00001", + "negative_test_query85_00005", + "apositive_test_query113_00001", + "apositive_test_query60_00006", + "negative_test_query129_00008", + "apositive_test_query238_00003", + "negative_test_query238_00013", + "negative_test_query74_00003", + "negative_test_query203_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query75", + "ranked_ids": [ + "apositive_test_query75_00005", + "apositive_test_query75_00001", + "negative_test_query130_00012", + "apositive_test_query75_00006", + "negative_test_query6_00008", + "apositive_test_query75_00000", + "negative_test_query82_00004", + "negative_test_query75_00012", + "negative_test_query99_00006", + "negative_test_query209_00004", + "negative_test_query319_00004", + "negative_test_query323_00011", + "negative_test_query260_00013", + "negative_test_query89_00000", + "negative_test_query75_00006", + "apositive_test_query75_00003", + "negative_test_query75_00003", + "negative_test_query75_00004", + "negative_test_query112_00018", + "negative_test_query67_00017" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query76", + "ranked_ids": [ + "apositive_test_query76_00006", + "negative_test_query76_00010", + "apositive_test_query76_00001", + "negative_test_query76_00011", + "apositive_test_query76_00000", + "apositive_test_query76_00004", + "negative_test_query258_00008", + "negative_test_query53_00009", + "negative_test_query76_00007", + "negative_test_query147_00007", + "negative_test_query181_00008", + "negative_test_query53_00001", + "negative_test_query279_00009", + "negative_test_query91_00004", + "apositive_test_query76_00002", + "negative_test_query76_00009", + "negative_test_query76_00002", + "negative_test_query76_00008", + "negative_test_query309_00011", + "negative_test_query147_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query98_00014", + "negative_test_query216_00006", + "negative_test_query77_00013", + "negative_test_query87_00016", + "negative_test_query263_00002", + "negative_test_query315_00003", + "negative_test_query150_00011", + "negative_test_query215_00004", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query148_00013", + "negative_test_query14_00010", + "negative_test_query216_00012", + "negative_test_query329_00015", + "negative_test_query342_00007", + "negative_test_query144_00003", + "negative_test_query287_00012", + "negative_test_query301_00003", + "negative_test_query326_00009", + "negative_test_query47_00010" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query78", + "ranked_ids": [ + "negative_test_query290_00013", + "apositive_test_query78_00001", + "negative_test_query290_00006", + "apositive_test_query78_00000", + "negative_test_query78_00007", + "negative_test_query214_00010", + "apositive_test_query38_00003", + "negative_test_query78_00016", + "negative_test_query274_00014", + "negative_test_query7_00005", + "negative_test_query200_00009", + "negative_test_query144_00000", + "negative_test_query272_00001", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query72_00005", + "apositive_test_query326_00000", + "negative_test_query236_00018", + "negative_test_query329_00002", + "negative_test_query192_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "apositive_test_query79_00004", + "apositive_test_query119_00005", + "negative_test_query56_00002", + "apositive_test_query221_00004", + "negative_test_query303_00016", + "negative_test_query79_00009", + "negative_test_query79_00013", + "negative_test_query157_00000", + "negative_test_query79_00002", + "apositive_test_query306_00010", + "apositive_test_query295_00001", + "negative_test_query79_00004", + "negative_test_query38_00007", + "negative_test_query53_00015", + "negative_test_query313_00013", + "negative_test_query158_00000", + "apositive_test_query79_00000", + "apositive_test_query113_00002", + "apositive_test_query157_00008", + "negative_test_query105_00011" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "negative_test_query196_00007", + "apositive_test_query192_00002", + "negative_test_query8_00014", + "negative_test_query8_00001", + "negative_test_query8_00015", + "apositive_test_query243_00005", + "negative_test_query8_00013", + "negative_test_query8_00004", + "negative_test_query8_00011", + "negative_test_query205_00007", + "negative_test_query192_00003", + "negative_test_query8_00008", + "apositive_test_query243_00010", + "negative_test_query96_00010", + "apositive_test_query8_00000", + "negative_test_query192_00006", + "apositive_test_query192_00003", + "negative_test_query8_00012", + "negative_test_query329_00005", + "negative_test_query8_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query80_00004", + "negative_test_query179_00005", + "negative_test_query80_00003", + "apositive_test_query80_00002", + "apositive_test_query189_00004", + "negative_test_query289_00005", + "negative_test_query80_00012", + "negative_test_query80_00001", + "apositive_test_query179_00001", + "negative_test_query80_00009", + "negative_test_query273_00005", + "negative_test_query287_00005", + "negative_test_query85_00007", + "apositive_test_query36_00000", + "negative_test_query102_00009", + "negative_test_query36_00010", + "apositive_test_query179_00003", + "apositive_test_query136_00013", + "negative_test_query179_00006", + "negative_test_query270_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query81", + "ranked_ids": [ + "apositive_test_query81_00000", + "apositive_test_query81_00004", + "apositive_test_query81_00005", + "negative_test_query81_00005", + "apositive_test_query81_00008", + "negative_test_query81_00010", + "negative_test_query81_00009", + "apositive_test_query81_00003", + "negative_test_query81_00006", + "negative_test_query81_00000", + "apositive_test_query81_00006", + "negative_test_query81_00001", + "negative_test_query207_00008", + "apositive_test_query81_00001", + "negative_test_query81_00008", + "negative_test_query81_00004", + "negative_test_query81_00007", + "negative_test_query27_00006", + "negative_test_query246_00017", + "negative_test_query145_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query82", + "ranked_ids": [ + "apositive_test_query82_00000", + "apositive_test_query319_00000", + "apositive_test_query82_00001", + "apositive_test_query82_00004", + "apositive_test_query82_00003", + "apositive_test_query319_00001", + "apositive_test_query319_00004", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query319_00010", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "negative_test_query319_00009", + "negative_test_query82_00009", + "negative_test_query319_00008", + "negative_test_query82_00010", + "apositive_test_query319_00003", + "negative_test_query319_00003", + "negative_test_query82_00003", + "negative_test_query340_00013" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query83_00004", + "apositive_test_query229_00001", + "apositive_test_query168_00001", + "apositive_test_query168_00000", + "apositive_test_query83_00002", + "apositive_test_query146_00014", + "apositive_test_query83_00003", + "negative_test_query168_00007", + "apositive_test_query164_00000", + "negative_test_query309_00016", + "negative_test_query287_00004", + "apositive_test_query95_00000", + "negative_test_query348_00006", + "negative_test_query28_00005", + "negative_test_query48_00006", + "apositive_test_query83_00000", + "apositive_test_query217_00006", + "negative_test_query279_00012", + "negative_test_query0_00006", + "negative_test_query357_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query84", + "ranked_ids": [ + "apositive_test_query84_00001", + "apositive_test_query84_00000", + "negative_test_query84_00006", + "negative_test_query84_00012", + "apositive_test_query155_00002", + "negative_test_query84_00009", + "negative_test_query84_00017", + "negative_test_query84_00000", + "negative_test_query84_00002", + "negative_test_query84_00001", + "negative_test_query84_00016", + "negative_test_query84_00008", + "apositive_test_query249_00013", + "negative_test_query84_00007", + "negative_test_query84_00005", + "negative_test_query43_00002", + "negative_test_query84_00013", + "negative_test_query224_00010", + "negative_test_query350_00017", + "negative_test_query268_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query85_00000", + "negative_test_query85_00012", + "apositive_test_query85_00002", + "negative_test_query85_00001", + "negative_test_query112_00008", + "negative_test_query85_00000", + "apositive_test_query85_00001", + "negative_test_query85_00009", + "apositive_test_query287_00004", + "apositive_test_query109_00004", + "negative_test_query85_00011", + "negative_test_query106_00003", + "apositive_test_query286_00006", + "apositive_test_query133_00013", + "negative_test_query85_00003", + "apositive_test_query85_00003", + "apositive_test_query284_00004", + "apositive_test_query81_00008", + "negative_test_query44_00008", + "apositive_test_query286_00004" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9828920819566879, + "recall@5": 1.0 + }, + "query_id": "test_query86", + "ranked_ids": [ + "apositive_test_query86_00003", + "apositive_test_query86_00002", + "apositive_test_query86_00000", + "negative_test_query86_00013", + "apositive_test_query86_00001", + "negative_test_query86_00000", + "negative_test_query86_00007", + "negative_test_query86_00002", + "negative_test_query86_00001", + "negative_test_query299_00007", + "negative_test_query86_00014", + "negative_test_query86_00008", + "negative_test_query86_00003", + "negative_test_query310_00007", + "negative_test_query86_00012", + "negative_test_query65_00013", + "negative_test_query210_00012", + "negative_test_query245_00012", + "negative_test_query86_00011", + "negative_test_query188_00013" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query87", + "ranked_ids": [ + "apositive_test_query87_00001", + "apositive_test_query87_00000", + "negative_test_query87_00006", + "negative_test_query87_00016", + "apositive_test_query302_00001", + "negative_test_query87_00013", + "negative_test_query87_00004", + "negative_test_query87_00011", + "negative_test_query87_00008", + "negative_test_query87_00000", + "negative_test_query87_00014", + "negative_test_query87_00015", + "negative_test_query87_00010", + "negative_test_query275_00006", + "negative_test_query87_00007", + "negative_test_query315_00006", + "negative_test_query87_00002", + "negative_test_query200_00011", + "negative_test_query182_00017", + "negative_test_query263_00002" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "apositive_test_query88_00001", + "negative_test_query88_00012", + "negative_test_query88_00017", + "negative_test_query88_00002", + "negative_test_query88_00005", + "negative_test_query88_00013", + "negative_test_query88_00003", + "negative_test_query340_00012", + "negative_test_query88_00008", + "negative_test_query88_00010", + "negative_test_query88_00016", + "negative_test_query325_00006", + "apositive_test_query250_00002", + "apositive_test_query147_00002", + "negative_test_query105_00014", + "negative_test_query184_00013", + "apositive_test_query223_00000", + "negative_test_query240_00009", + "negative_test_query334_00006", + "negative_test_query203_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query110_00011", + "negative_test_query156_00013", + "negative_test_query326_00006", + "negative_test_query89_00004", + "apositive_test_query89_00004", + "negative_test_query278_00000", + "negative_test_query89_00008", + "apositive_test_query75_00006", + "apositive_test_query255_00000", + "negative_test_query89_00007", + "negative_test_query278_00009", + "negative_test_query325_00012", + "negative_test_query89_00000", + "negative_test_query68_00004", + "negative_test_query212_00000", + "negative_test_query89_00005", + "negative_test_query130_00003", + "apositive_test_query68_00001", + "apositive_test_query89_00000", + "negative_test_query233_00005" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00008", + "negative_test_query9_00006", + "negative_test_query9_00001", + "apositive_test_query90_00001", + "negative_test_query90_00004", + "negative_test_query9_00011", + "negative_test_query90_00009", + "negative_test_query9_00018", + "apositive_test_query9_00000", + "negative_test_query221_00002", + "negative_test_query9_00005", + "negative_test_query9_00015", + "negative_test_query221_00007", + "negative_test_query9_00003", + "negative_test_query9_00004", + "negative_test_query9_00016", + "negative_test_query9_00013", + "negative_test_query221_00011", + "negative_test_query90_00007", + "negative_test_query305_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query231_00002", + "negative_test_query90_00012", + "negative_test_query90_00015", + "apositive_test_query90_00000", + "negative_test_query90_00010", + "negative_test_query310_00007", + "negative_test_query90_00011", + "negative_test_query90_00017", + "negative_test_query127_00001", + "negative_test_query90_00000", + "apositive_test_query14_00002", + "negative_test_query227_00001", + "apositive_test_query304_00000", + "negative_test_query90_00006", + "negative_test_query231_00014", + "negative_test_query90_00002", + "negative_test_query227_00012", + "negative_test_query261_00002", + "negative_test_query227_00006", + "negative_test_query9_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query91", + "ranked_ids": [ + "negative_test_query292_00009", + "apositive_test_query91_00000", + "apositive_test_query91_00006", + "negative_test_query303_00011", + "apositive_test_query91_00002", + "negative_test_query279_00011", + "negative_test_query319_00002", + "negative_test_query82_00002", + "apositive_test_query11_00006", + "negative_test_query91_00010", + "negative_test_query358_00004", + "apositive_test_query342_00001", + "negative_test_query315_00013", + "negative_test_query350_00004", + "negative_test_query130_00010", + "apositive_test_query157_00001", + "negative_test_query102_00002", + "apositive_test_query157_00000", + "negative_test_query118_00015", + "negative_test_query59_00006" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.8333333333333334 + }, + "query_id": "test_query92", + "ranked_ids": [ + "apositive_test_query92_00005", + "apositive_test_query92_00000", + "apositive_test_query92_00003", + "apositive_test_query92_00004", + "apositive_test_query92_00001", + "negative_test_query92_00013", + "negative_test_query105_00016", + "apositive_test_query92_00002", + "apositive_test_query100_00001", + "negative_test_query122_00018", + "negative_test_query100_00014", + "negative_test_query92_00003", + "negative_test_query283_00001", + "apositive_test_query198_00001", + "negative_test_query1_00012", + "negative_test_query283_00009", + "apositive_test_query99_00003", + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query100_00012" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query93_00004", + "apositive_test_query93_00003", + "apositive_test_query93_00000", + "negative_test_query93_00003", + "negative_test_query335_00003", + "negative_test_query129_00003", + "negative_test_query195_00006", + "apositive_test_query335_00003", + "negative_test_query280_00012", + "negative_test_query195_00001", + "negative_test_query321_00001", + "apositive_test_query160_00007", + "apositive_test_query243_00006", + "negative_test_query219_00000", + "negative_test_query92_00012", + "negative_test_query108_00003", + "negative_test_query277_00003", + "negative_test_query335_00011", + "negative_test_query5_00015", + "apositive_test_query323_00003" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00006", + "negative_test_query94_00003", + "apositive_test_query94_00002", + "apositive_test_query127_00004", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "negative_test_query94_00005", + "apositive_test_query127_00001", + "apositive_test_query94_00005", + "negative_test_query94_00009", + "apositive_test_query94_00004", + "apositive_test_query94_00000", + "apositive_test_query127_00015", + "apositive_test_query94_00001", + "negative_test_query94_00002", + "apositive_test_query127_00010", + "apositive_test_query94_00006", + "apositive_test_query94_00007", + "apositive_test_query127_00008", + "negative_test_query155_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "apositive_test_query95_00001", + "apositive_test_query95_00000", + "negative_test_query95_00013", + "negative_test_query95_00005", + "negative_test_query95_00010", + "negative_test_query12_00015", + "negative_test_query95_00015", + "negative_test_query95_00012", + "negative_test_query95_00002", + "apositive_test_query253_00007", + "negative_test_query95_00003", + "negative_test_query12_00017", + "negative_test_query95_00008", + "negative_test_query95_00014", + "negative_test_query244_00010", + "negative_test_query334_00012", + "apositive_test_query177_00000", + "negative_test_query57_00007", + "negative_test_query95_00006", + "apositive_test_query292_00001" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query339_00000", + "apositive_test_query96_00001", + "negative_test_query96_00016", + "negative_test_query339_00017", + "negative_test_query339_00006", + "apositive_test_query339_00001", + "negative_test_query339_00008", + "negative_test_query339_00004", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query96_00013", + "negative_test_query192_00011", + "negative_test_query339_00007", + "negative_test_query339_00016", + "apositive_test_query96_00002", + "negative_test_query339_00001", + "negative_test_query339_00005", + "negative_test_query96_00004", + "negative_test_query96_00003", + "negative_test_query339_00009" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.5 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "negative_test_query97_00012", + "apositive_test_query97_00002", + "apositive_test_query97_00003", + "apositive_test_query97_00001", + "negative_test_query113_00006", + "apositive_test_query97_00005", + "apositive_test_query289_00003", + "negative_test_query97_00009", + "negative_test_query97_00006", + "apositive_test_query51_00000", + "negative_test_query97_00011", + "apositive_test_query51_00002", + "negative_test_query84_00008", + "negative_test_query209_00008", + "negative_test_query113_00009", + "negative_test_query16_00014", + "apositive_test_query267_00000", + "apositive_test_query201_00006", + "negative_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00017", + "negative_test_query98_00007", + "apositive_test_query342_00000", + "negative_test_query231_00014", + "negative_test_query98_00004", + "negative_test_query98_00011", + "negative_test_query197_00007", + "negative_test_query129_00003", + "negative_test_query212_00006", + "negative_test_query39_00005", + "negative_test_query316_00014", + "negative_test_query98_00005", + "apositive_test_query201_00004", + "negative_test_query345_00002", + "negative_test_query98_00016", + "negative_test_query98_00018", + "apositive_test_query14_00003", + "apositive_test_query14_00002", + "negative_test_query23_00013", + "negative_test_query300_00000" + ], + "valid": true + }, + { + "algorithm": "tourrank_r2", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query99", + "ranked_ids": [ + "apositive_test_query193_00008", + "apositive_test_query99_00001", + "negative_test_query193_00002", + "apositive_test_query193_00012", + "negative_test_query193_00001", + "negative_test_query330_00008", + "apositive_test_query193_00010", + "negative_test_query296_00013", + "negative_test_query193_00005", + "apositive_test_query153_00012", + "apositive_test_query347_00002", + "apositive_test_query99_00007", + "negative_test_query256_00002", + "negative_test_query129_00003", + "negative_test_query174_00002", + "negative_test_query129_00001", + "negative_test_query27_00018", + "apositive_test_query253_00003", + "apositive_test_query333_00003", + "negative_test_query27_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.4166666666666667 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00003", + "apositive_test_query0_00006", + "apositive_test_query0_00004", + "negative_test_query359_00002", + "negative_test_query9_00017", + "negative_test_query0_00001", + "apositive_test_query0_00002", + "apositive_test_query107_00004", + "negative_test_query221_00006", + "negative_test_query259_00006", + "apositive_test_query130_00000", + "negative_test_query130_00002", + "negative_test_query201_00000", + "negative_test_query357_00000", + "apositive_test_query153_00004", + "negative_test_query0_00007", + "negative_test_query0_00000", + "negative_test_query36_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query1", + "ranked_ids": [ + "apositive_test_query1_00000", + "negative_test_query1_00009", + "negative_test_query1_00005", + "apositive_test_query348_00003", + "negative_test_query61_00013", + "negative_test_query61_00014", + "negative_test_query1_00001", + "negative_test_query256_00005", + "negative_test_query297_00011", + "negative_test_query184_00015", + "negative_test_query331_00002", + "apositive_test_query188_00002", + "negative_test_query202_00001", + "negative_test_query157_00007", + "negative_test_query61_00002", + "negative_test_query120_00006", + "negative_test_query110_00007", + "negative_test_query27_00017", + "negative_test_query245_00006", + "negative_test_query61_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.7328286204777911, + "recall@5": 1.0 + }, + "query_id": "test_query10", + "ranked_ids": [ + "negative_test_query10_00000", + "apositive_test_query10_00001", + "apositive_test_query10_00000", + "apositive_test_query10_00002", + "negative_test_query10_00011", + "negative_test_query10_00005", + "negative_test_query10_00010", + "negative_test_query169_00008", + "negative_test_query104_00011", + "negative_test_query135_00014", + "negative_test_query83_00000", + "negative_test_query266_00009", + "apositive_test_query69_00000", + "apositive_test_query69_00016", + "apositive_test_query44_00001", + "negative_test_query10_00007", + "negative_test_query72_00002", + "negative_test_query10_00004", + "negative_test_query10_00009", + "apositive_test_query286_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query100", + "ranked_ids": [ + "apositive_test_query100_00003", + "negative_test_query100_00001", + "apositive_test_query100_00000", + "negative_test_query15_00000", + "negative_test_query100_00013", + "negative_test_query100_00009", + "negative_test_query290_00015", + "negative_test_query237_00010", + "negative_test_query100_00010", + "apositive_test_query338_00002", + "negative_test_query258_00017", + "apositive_test_query100_00004", + "apositive_test_query181_00000", + "apositive_test_query338_00005", + "apositive_test_query100_00002", + "negative_test_query100_00014", + "negative_test_query271_00006", + "negative_test_query252_00005", + "negative_test_query323_00000", + "negative_test_query147_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query101_00005", + "apositive_test_query101_00002", + "apositive_test_query336_00001", + "apositive_test_query73_00002", + "negative_test_query173_00013", + "negative_test_query61_00002", + "negative_test_query61_00013", + "negative_test_query241_00005", + "negative_test_query68_00009", + "negative_test_query258_00000", + "negative_test_query53_00000", + "negative_test_query35_00004", + "apositive_test_query196_00000", + "negative_test_query103_00009", + "negative_test_query61_00014", + "negative_test_query61_00000", + "negative_test_query191_00014", + "negative_test_query134_00008", + "apositive_test_query291_00000", + "negative_test_query189_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query108_00009", + "negative_test_query102_00014", + "apositive_test_query20_00001", + "negative_test_query131_00009", + "negative_test_query313_00016", + "negative_test_query205_00016", + "negative_test_query47_00011", + "apositive_test_query30_00005", + "apositive_test_query47_00001", + "negative_test_query183_00012", + "apositive_test_query356_00008", + "negative_test_query183_00009", + "apositive_test_query238_00000", + "negative_test_query203_00007", + "negative_test_query209_00006", + "negative_test_query7_00001", + "negative_test_query219_00004", + "negative_test_query90_00016", + "negative_test_query47_00010", + "negative_test_query232_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "apositive_test_query103_00001", + "negative_test_query103_00003", + "negative_test_query103_00011", + "apositive_test_query185_00003", + "negative_test_query185_00005", + "negative_test_query217_00005", + "negative_test_query222_00009", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query307_00008", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query103_00000", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query48_00014", + "negative_test_query72_00012", + "negative_test_query103_00006", + "apositive_test_query103_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "apositive_test_query104_00000", + "negative_test_query104_00002", + "negative_test_query104_00018", + "negative_test_query104_00011", + "negative_test_query104_00001", + "negative_test_query104_00013", + "negative_test_query104_00015", + "negative_test_query104_00010", + "negative_test_query104_00003", + "negative_test_query343_00013", + "negative_test_query104_00005", + "negative_test_query104_00012", + "negative_test_query104_00014", + "negative_test_query104_00009", + "negative_test_query104_00016", + "negative_test_query104_00004", + "negative_test_query104_00007", + "negative_test_query48_00010", + "apositive_test_query28_00011", + "apositive_test_query48_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query105", + "ranked_ids": [ + "apositive_test_query105_00000", + "negative_test_query79_00002", + "negative_test_query105_00011", + "apositive_test_query158_00002", + "apositive_test_query83_00004", + "negative_test_query295_00005", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query119_00005", + "negative_test_query105_00005", + "negative_test_query105_00006", + "negative_test_query168_00005", + "negative_test_query142_00010", + "negative_test_query267_00011", + "negative_test_query97_00007", + "negative_test_query313_00013", + "apositive_test_query157_00008", + "negative_test_query359_00008", + "negative_test_query79_00004", + "negative_test_query56_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query106", + "ranked_ids": [ + "apositive_test_query106_00003", + "apositive_test_query106_00005", + "negative_test_query326_00008", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "apositive_test_query106_00000", + "negative_test_query286_00000", + "negative_test_query36_00001", + "apositive_test_query106_00001", + "negative_test_query114_00003", + "apositive_test_query42_00006", + "negative_test_query106_00012", + "negative_test_query212_00002", + "negative_test_query106_00000", + "apositive_test_query286_00005", + "negative_test_query106_00006", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "negative_test_query46_00012", + "apositive_test_query357_00003", + "apositive_test_query36_00002", + "negative_test_query146_00000", + "apositive_test_query276_00003", + "negative_test_query142_00000", + "negative_test_query160_00001", + "negative_test_query296_00008", + "negative_test_query333_00005", + "apositive_test_query221_00001", + "negative_test_query105_00015", + "negative_test_query31_00013", + "negative_test_query305_00015", + "negative_test_query124_00005", + "apositive_test_query142_00002", + "negative_test_query105_00008", + "apositive_test_query142_00004", + "apositive_test_query211_00001", + "negative_test_query296_00001", + "negative_test_query142_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.375 + }, + "query_id": "test_query108", + "ranked_ids": [ + "apositive_test_query108_00000", + "apositive_test_query108_00002", + "negative_test_query108_00011", + "apositive_test_query108_00003", + "negative_test_query57_00009", + "negative_test_query102_00012", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query311_00006", + "negative_test_query108_00010", + "apositive_test_query108_00007", + "negative_test_query108_00008", + "apositive_test_query108_00005", + "negative_test_query108_00001", + "negative_test_query168_00001", + "negative_test_query108_00006", + "negative_test_query250_00009", + "negative_test_query250_00011", + "negative_test_query102_00007", + "negative_test_query108_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query109", + "ranked_ids": [ + "negative_test_query190_00003", + "apositive_test_query109_00000", + "negative_test_query109_00003", + "negative_test_query109_00000", + "negative_test_query109_00012", + "negative_test_query268_00010", + "apositive_test_query109_00001", + "negative_test_query109_00013", + "negative_test_query109_00014", + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query248_00006", + "negative_test_query109_00009", + "negative_test_query109_00008", + "negative_test_query109_00011", + "negative_test_query109_00004", + "negative_test_query6_00006", + "negative_test_query38_00006", + "apositive_test_query109_00003", + "negative_test_query51_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.375 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00006", + "apositive_test_query11_00001", + "negative_test_query11_00004", + "apositive_test_query11_00007", + "negative_test_query132_00003", + "negative_test_query309_00001", + "negative_test_query226_00010", + "negative_test_query309_00002", + "negative_test_query309_00004", + "negative_test_query358_00005", + "apositive_test_query11_00004", + "apositive_test_query358_00006", + "apositive_test_query11_00000", + "apositive_test_query358_00007", + "negative_test_query132_00013", + "apositive_test_query301_00003", + "negative_test_query123_00007", + "apositive_test_query11_00003", + "negative_test_query358_00001", + "negative_test_query11_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "negative_test_query89_00006", + "negative_test_query110_00005", + "negative_test_query110_00013", + "negative_test_query110_00014", + "negative_test_query110_00002", + "negative_test_query10_00005", + "apositive_test_query344_00002", + "apositive_test_query11_00002", + "negative_test_query110_00006", + "negative_test_query311_00011", + "negative_test_query110_00004", + "apositive_test_query138_00000", + "negative_test_query17_00008", + "negative_test_query110_00007", + "apositive_test_query180_00001", + "negative_test_query343_00001", + "negative_test_query260_00015", + "apositive_test_query242_00006", + "apositive_test_query291_00009", + "apositive_test_query330_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query111_00015", + "negative_test_query137_00010", + "negative_test_query134_00006", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query213_00011", + "negative_test_query111_00018", + "negative_test_query223_00000", + "negative_test_query65_00013", + "apositive_test_query162_00005", + "negative_test_query111_00008", + "negative_test_query245_00001", + "negative_test_query65_00007", + "apositive_test_query306_00002", + "apositive_test_query174_00008", + "apositive_test_query308_00003", + "negative_test_query14_00015", + "negative_test_query58_00013", + "negative_test_query243_00001", + "negative_test_query292_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "negative_test_query246_00016", + "negative_test_query36_00014", + "negative_test_query112_00004", + "negative_test_query112_00010", + "apositive_test_query286_00005", + "apositive_test_query36_00003", + "apositive_test_query286_00006", + "apositive_test_query106_00003", + "negative_test_query112_00016", + "apositive_test_query246_00001", + "negative_test_query216_00002", + "negative_test_query112_00015", + "negative_test_query67_00003", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query85_00002", + "negative_test_query6_00003", + "negative_test_query36_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query113", + "ranked_ids": [ + "apositive_test_query113_00001", + "apositive_test_query113_00000", + "negative_test_query67_00014", + "negative_test_query7_00016", + "negative_test_query201_00008", + "negative_test_query113_00003", + "apositive_test_query113_00002", + "negative_test_query67_00002", + "negative_test_query38_00003", + "negative_test_query105_00007", + "apositive_test_query67_00000", + "negative_test_query188_00005", + "negative_test_query223_00003", + "apositive_test_query186_00005", + "negative_test_query216_00011", + "apositive_test_query79_00001", + "negative_test_query6_00016", + "negative_test_query105_00000", + "negative_test_query47_00005", + "apositive_test_query188_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query114", + "ranked_ids": [ + "apositive_test_query114_00001", + "apositive_test_query114_00000", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "apositive_test_query114_00003", + "negative_test_query114_00008", + "negative_test_query114_00002", + "negative_test_query349_00003", + "apositive_test_query124_00004", + "negative_test_query114_00005", + "negative_test_query114_00004", + "negative_test_query114_00007", + "negative_test_query114_00010", + "negative_test_query114_00001", + "negative_test_query114_00003", + "apositive_test_query114_00006", + "apositive_test_query106_00003", + "negative_test_query106_00009", + "negative_test_query262_00007", + "negative_test_query81_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query115", + "ranked_ids": [ + "apositive_test_query115_00002", + "apositive_test_query115_00001", + "apositive_test_query115_00004", + "apositive_test_query115_00000", + "negative_test_query115_00003", + "negative_test_query115_00002", + "negative_test_query115_00006", + "apositive_test_query115_00003", + "apositive_test_query348_00000", + "negative_test_query115_00007", + "negative_test_query115_00012", + "negative_test_query276_00006", + "apositive_test_query348_00009", + "negative_test_query115_00004", + "negative_test_query115_00010", + "negative_test_query139_00000", + "negative_test_query205_00004", + "apositive_test_query121_00001", + "apositive_test_query133_00010", + "negative_test_query121_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "apositive_test_query116_00000", + "apositive_test_query49_00000", + "negative_test_query116_00015", + "negative_test_query59_00013", + "negative_test_query116_00011", + "negative_test_query116_00013", + "negative_test_query116_00008", + "negative_test_query116_00003", + "negative_test_query59_00007", + "negative_test_query116_00002", + "negative_test_query116_00000", + "negative_test_query116_00004", + "negative_test_query261_00013", + "negative_test_query98_00005", + "negative_test_query116_00006", + "negative_test_query116_00010", + "negative_test_query116_00007", + "negative_test_query116_00009", + "negative_test_query116_00014", + "negative_test_query116_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.25 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00003", + "apositive_test_query117_00006", + "negative_test_query117_00010", + "negative_test_query117_00009", + "apositive_test_query354_00003", + "negative_test_query194_00002", + "negative_test_query63_00001", + "apositive_test_query354_00000", + "apositive_test_query117_00004", + "negative_test_query48_00000", + "apositive_test_query117_00007", + "negative_test_query117_00004", + "negative_test_query117_00003", + "negative_test_query117_00007", + "negative_test_query117_00001", + "negative_test_query117_00008", + "apositive_test_query117_00000", + "apositive_test_query134_00004", + "negative_test_query202_00006", + "negative_test_query175_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "negative_test_query296_00007", + "apositive_test_query155_00005", + "negative_test_query340_00007", + "negative_test_query23_00005", + "negative_test_query84_00012", + "apositive_test_query174_00010", + "apositive_test_query310_00001", + "negative_test_query171_00005", + "negative_test_query171_00016", + "apositive_test_query171_00001", + "apositive_test_query174_00003", + "apositive_test_query174_00004", + "apositive_test_query174_00005", + "apositive_test_query340_00002", + "negative_test_query296_00010", + "negative_test_query88_00007", + "apositive_test_query217_00000", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "apositive_test_query174_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query119", + "ranked_ids": [ + "apositive_test_query119_00005", + "apositive_test_query221_00004", + "apositive_test_query119_00012", + "negative_test_query119_00005", + "apositive_test_query306_00010", + "negative_test_query56_00002", + "negative_test_query158_00001", + "negative_test_query158_00000", + "negative_test_query79_00002", + "apositive_test_query113_00002", + "apositive_test_query79_00004", + "apositive_test_query119_00001", + "negative_test_query152_00004", + "negative_test_query359_00005", + "negative_test_query157_00001", + "negative_test_query157_00007", + "negative_test_query119_00002", + "negative_test_query223_00011", + "apositive_test_query26_00000", + "apositive_test_query105_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query12_00016", + "negative_test_query12_00014", + "negative_test_query12_00003", + "negative_test_query12_00004", + "negative_test_query12_00013", + "negative_test_query12_00008", + "negative_test_query12_00017", + "negative_test_query12_00007", + "apositive_test_query12_00000", + "negative_test_query12_00015", + "negative_test_query12_00011", + "negative_test_query12_00001", + "negative_test_query12_00006", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query231_00013", + "negative_test_query234_00014", + "apositive_test_query185_00000", + "apositive_test_query217_00005", + "negative_test_query175_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00008", + "negative_test_query120_00002", + "negative_test_query120_00010", + "negative_test_query1_00009", + "apositive_test_query120_00003", + "negative_test_query352_00013", + "apositive_test_query120_00002", + "negative_test_query120_00001", + "negative_test_query121_00002", + "apositive_test_query120_00000", + "apositive_test_query120_00001", + "apositive_test_query117_00007", + "negative_test_query303_00018", + "negative_test_query120_00012", + "negative_test_query121_00012", + "negative_test_query148_00014", + "negative_test_query120_00006", + "apositive_test_query32_00007", + "negative_test_query196_00006", + "apositive_test_query352_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00002", + "apositive_test_query121_00001", + "apositive_test_query121_00004", + "negative_test_query121_00008", + "apositive_test_query121_00005", + "apositive_test_query121_00000", + "apositive_test_query121_00003", + "negative_test_query121_00013", + "negative_test_query121_00000", + "negative_test_query121_00012", + "negative_test_query121_00003", + "negative_test_query308_00009", + "negative_test_query44_00012", + "apositive_test_query307_00010", + "negative_test_query256_00003", + "negative_test_query53_00003", + "negative_test_query121_00007", + "negative_test_query313_00006", + "negative_test_query115_00009", + "negative_test_query133_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query86_00006", + "negative_test_query61_00015", + "negative_test_query17_00012", + "apositive_test_query122_00000", + "apositive_test_query254_00002", + "negative_test_query210_00003", + "apositive_test_query173_00000", + "negative_test_query195_00008", + "apositive_test_query134_00000", + "negative_test_query86_00001", + "negative_test_query173_00003", + "negative_test_query64_00017", + "negative_test_query179_00002", + "negative_test_query297_00000", + "negative_test_query5_00001", + "apositive_test_query175_00004", + "negative_test_query179_00014", + "negative_test_query358_00000", + "negative_test_query134_00007", + "apositive_test_query173_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.4 + }, + "query_id": "test_query123", + "ranked_ids": [ + "negative_test_query123_00013", + "apositive_test_query123_00002", + "negative_test_query123_00008", + "negative_test_query155_00001", + "apositive_test_query123_00003", + "negative_test_query123_00005", + "apositive_test_query123_00004", + "negative_test_query123_00001", + "negative_test_query123_00004", + "negative_test_query123_00010", + "negative_test_query161_00009", + "negative_test_query123_00006", + "negative_test_query340_00012", + "negative_test_query309_00008", + "negative_test_query123_00007", + "negative_test_query269_00006", + "apositive_test_query123_00000", + "negative_test_query123_00000", + "negative_test_query123_00002", + "negative_test_query155_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "apositive_test_query13_00004", + "negative_test_query187_00003", + "apositive_test_query194_00000", + "negative_test_query347_00010", + "negative_test_query124_00003", + "apositive_test_query13_00000", + "negative_test_query42_00005", + "negative_test_query262_00011", + "negative_test_query123_00009", + "negative_test_query42_00006", + "apositive_test_query162_00002", + "negative_test_query187_00006", + "negative_test_query349_00017", + "negative_test_query309_00012", + "negative_test_query144_00010", + "negative_test_query76_00004", + "apositive_test_query162_00001", + "negative_test_query171_00002", + "negative_test_query339_00008", + "negative_test_query93_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query125_00009", + "negative_test_query125_00014", + "negative_test_query140_00017", + "negative_test_query360_00016", + "apositive_test_query125_00000", + "apositive_test_query125_00003", + "negative_test_query125_00003", + "negative_test_query125_00013", + "negative_test_query125_00004", + "negative_test_query125_00000", + "negative_test_query3_00000", + "apositive_test_query273_00003", + "negative_test_query301_00012", + "negative_test_query125_00011", + "negative_test_query6_00003", + "negative_test_query125_00006", + "negative_test_query12_00009", + "apositive_test_query286_00009", + "negative_test_query260_00012", + "apositive_test_query286_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query126", + "ranked_ids": [ + "apositive_test_query126_00000", + "negative_test_query126_00017", + "negative_test_query27_00013", + "negative_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "negative_test_query190_00017", + "negative_test_query126_00008", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query151_00008", + "apositive_test_query330_00001", + "negative_test_query131_00009", + "negative_test_query151_00006", + "negative_test_query190_00005", + "negative_test_query311_00005", + "apositive_test_query330_00002", + "negative_test_query220_00000", + "negative_test_query12_00008", + "apositive_test_query5_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00004", + "apositive_test_query127_00003", + "apositive_test_query127_00008", + "apositive_test_query127_00000", + "apositive_test_query127_00007", + "apositive_test_query127_00012", + "negative_test_query94_00009", + "apositive_test_query127_00016", + "negative_test_query267_00009", + "apositive_test_query94_00005", + "apositive_test_query127_00009", + "apositive_test_query127_00006", + "negative_test_query94_00003", + "apositive_test_query127_00011", + "apositive_test_query127_00014", + "negative_test_query94_00007", + "apositive_test_query127_00013", + "apositive_test_query163_00000", + "apositive_test_query311_00003", + "negative_test_query198_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query128", + "ranked_ids": [ + "negative_test_query256_00011", + "apositive_test_query128_00001", + "apositive_test_query128_00000", + "negative_test_query21_00017", + "negative_test_query283_00006", + "negative_test_query21_00000", + "apositive_test_query21_00000", + "negative_test_query8_00017", + "negative_test_query128_00006", + "apositive_test_query216_00002", + "apositive_test_query142_00000", + "apositive_test_query196_00004", + "apositive_test_query32_00001", + "apositive_test_query80_00005", + "negative_test_query75_00012", + "negative_test_query310_00009", + "apositive_test_query8_00000", + "negative_test_query128_00010", + "negative_test_query128_00011", + "negative_test_query192_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query129", + "ranked_ids": [ + "apositive_test_query129_00000", + "negative_test_query211_00012", + "apositive_test_query129_00001", + "negative_test_query156_00017", + "apositive_test_query244_00001", + "negative_test_query63_00002", + "negative_test_query63_00006", + "negative_test_query129_00010", + "negative_test_query165_00012", + "negative_test_query245_00014", + "apositive_test_query356_00006", + "negative_test_query117_00006", + "negative_test_query84_00008", + "apositive_test_query150_00000", + "apositive_test_query308_00004", + "negative_test_query16_00007", + "negative_test_query187_00012", + "negative_test_query342_00013", + "apositive_test_query155_00002", + "apositive_test_query249_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query13", + "ranked_ids": [ + "negative_test_query13_00001", + "apositive_test_query13_00006", + "apositive_test_query13_00000", + "negative_test_query13_00002", + "apositive_test_query13_00001", + "negative_test_query207_00013", + "apositive_test_query13_00002", + "apositive_test_query13_00003", + "negative_test_query13_00003", + "negative_test_query13_00000", + "apositive_test_query13_00007", + "apositive_test_query13_00009", + "apositive_test_query13_00008", + "apositive_test_query13_00010", + "negative_test_query13_00004", + "apositive_test_query13_00004", + "apositive_test_query13_00012", + "negative_test_query13_00006", + "negative_test_query13_00005", + "negative_test_query245_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query130", + "ranked_ids": [ + "apositive_test_query130_00005", + "negative_test_query90_00014", + "negative_test_query130_00001", + "negative_test_query198_00011", + "negative_test_query130_00008", + "negative_test_query198_00003", + "negative_test_query345_00002", + "apositive_test_query323_00004", + "apositive_test_query284_00001", + "negative_test_query130_00006", + "negative_test_query284_00001", + "apositive_test_query284_00003", + "apositive_test_query90_00001", + "apositive_test_query130_00002", + "negative_test_query284_00013", + "negative_test_query284_00009", + "apositive_test_query30_00002", + "apositive_test_query233_00000", + "apositive_test_query275_00000", + "negative_test_query289_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query131", + "ranked_ids": [ + "apositive_test_query131_00000", + "apositive_test_query131_00001", + "negative_test_query131_00001", + "negative_test_query313_00008", + "negative_test_query186_00012", + "negative_test_query131_00002", + "negative_test_query247_00004", + "apositive_test_query106_00000", + "negative_test_query284_00002", + "apositive_test_query344_00003", + "apositive_test_query50_00008", + "negative_test_query357_00001", + "negative_test_query284_00006", + "apositive_test_query158_00002", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query184_00012", + "negative_test_query158_00001", + "negative_test_query168_00013", + "negative_test_query345_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "negative_test_query132_00006", + "negative_test_query132_00000", + "negative_test_query132_00018", + "negative_test_query81_00005", + "negative_test_query123_00012", + "negative_test_query81_00001", + "negative_test_query132_00010", + "negative_test_query132_00012", + "negative_test_query304_00000", + "negative_test_query181_00008", + "negative_test_query132_00004", + "negative_test_query11_00006", + "negative_test_query184_00015", + "negative_test_query240_00011", + "negative_test_query132_00016", + "apositive_test_query161_00007", + "negative_test_query148_00011", + "apositive_test_query187_00000", + "apositive_test_query3_00004", + "negative_test_query246_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.2 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query287_00001", + "apositive_test_query133_00014", + "apositive_test_query133_00001", + "apositive_test_query133_00000", + "apositive_test_query129_00004", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query91_00002", + "apositive_test_query133_00002", + "negative_test_query206_00017", + "negative_test_query310_00002", + "negative_test_query59_00008", + "negative_test_query12_00014", + "apositive_test_query129_00006", + "apositive_test_query129_00002", + "apositive_test_query174_00011", + "apositive_test_query129_00007", + "negative_test_query260_00006", + "apositive_test_query129_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query134", + "ranked_ids": [ + "negative_test_query134_00006", + "negative_test_query134_00010", + "negative_test_query134_00008", + "negative_test_query107_00003", + "apositive_test_query134_00002", + "apositive_test_query213_00001", + "negative_test_query359_00011", + "negative_test_query108_00003", + "apositive_test_query357_00000", + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query142_00001", + "negative_test_query238_00001", + "negative_test_query46_00011", + "apositive_test_query259_00011", + "negative_test_query46_00004", + "negative_test_query134_00000", + "negative_test_query218_00009", + "apositive_test_query276_00004", + "apositive_test_query238_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00003", + "negative_test_query135_00017", + "negative_test_query135_00011", + "apositive_test_query135_00001", + "negative_test_query135_00007", + "apositive_test_query133_00011", + "negative_test_query135_00001", + "negative_test_query135_00005", + "negative_test_query287_00009", + "negative_test_query135_00014", + "negative_test_query135_00015", + "negative_test_query135_00013", + "negative_test_query135_00004", + "negative_test_query135_00009", + "negative_test_query297_00017", + "apositive_test_query135_00000", + "negative_test_query135_00008", + "negative_test_query325_00012", + "apositive_test_query28_00008", + "negative_test_query135_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query136", + "ranked_ids": [ + "apositive_test_query136_00004", + "negative_test_query128_00011", + "apositive_test_query136_00002", + "apositive_test_query136_00000", + "apositive_test_query136_00011", + "apositive_test_query136_00014", + "apositive_test_query136_00001", + "apositive_test_query136_00007", + "apositive_test_query136_00005", + "apositive_test_query136_00010", + "apositive_test_query136_00009", + "apositive_test_query136_00013", + "negative_test_query136_00003", + "negative_test_query301_00001", + "negative_test_query129_00001", + "apositive_test_query60_00014", + "negative_test_query60_00002", + "negative_test_query8_00010", + "negative_test_query149_00007", + "negative_test_query245_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query137", + "ranked_ids": [ + "negative_test_query137_00014", + "apositive_test_query137_00000", + "apositive_test_query253_00000", + "negative_test_query95_00004", + "negative_test_query137_00003", + "negative_test_query349_00006", + "negative_test_query207_00013", + "negative_test_query207_00006", + "apositive_test_query13_00001", + "negative_test_query253_00007", + "negative_test_query137_00012", + "negative_test_query13_00002", + "apositive_test_query137_00002", + "negative_test_query74_00001", + "negative_test_query137_00000", + "apositive_test_query137_00003", + "negative_test_query262_00009", + "negative_test_query149_00013", + "negative_test_query74_00003", + "negative_test_query74_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query138", + "ranked_ids": [ + "apositive_test_query138_00000", + "apositive_test_query138_00001", + "negative_test_query138_00014", + "negative_test_query138_00003", + "negative_test_query138_00013", + "negative_test_query138_00006", + "negative_test_query138_00015", + "negative_test_query138_00008", + "negative_test_query138_00004", + "apositive_test_query138_00002", + "negative_test_query138_00001", + "negative_test_query138_00005", + "apositive_test_query244_00005", + "apositive_test_query356_00003", + "apositive_test_query23_00003", + "negative_test_query132_00009", + "negative_test_query213_00006", + "negative_test_query183_00004", + "negative_test_query240_00000", + "negative_test_query184_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.49818925746641285, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query35_00001", + "apositive_test_query139_00000", + "apositive_test_query35_00000", + "apositive_test_query139_00001", + "negative_test_query101_00006", + "apositive_test_query146_00007", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query123_00006", + "negative_test_query231_00012", + "negative_test_query269_00006", + "negative_test_query81_00002", + "negative_test_query262_00002", + "negative_test_query320_00007", + "apositive_test_query354_00001", + "negative_test_query35_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query14", + "ranked_ids": [ + "apositive_test_query14_00003", + "negative_test_query155_00009", + "negative_test_query14_00007", + "negative_test_query14_00005", + "negative_test_query260_00014", + "apositive_test_query296_00005", + "apositive_test_query216_00002", + "negative_test_query53_00003", + "negative_test_query155_00001", + "negative_test_query14_00006", + "apositive_test_query77_00000", + "negative_test_query14_00001", + "negative_test_query179_00001", + "negative_test_query260_00000", + "apositive_test_query64_00000", + "negative_test_query14_00004", + "negative_test_query102_00011", + "negative_test_query199_00009", + "apositive_test_query26_00006", + "negative_test_query350_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00005", + "negative_test_query140_00003", + "apositive_test_query281_00013", + "negative_test_query140_00001", + "negative_test_query140_00016", + "apositive_test_query273_00005", + "negative_test_query140_00002", + "negative_test_query140_00010", + "negative_test_query140_00018", + "negative_test_query140_00014", + "negative_test_query140_00013", + "negative_test_query257_00012", + "apositive_test_query18_00001", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00009", + "negative_test_query18_00006", + "negative_test_query140_00007", + "apositive_test_query18_00003", + "negative_test_query18_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query141", + "ranked_ids": [ + "apositive_test_query288_00003", + "apositive_test_query141_00006", + "negative_test_query245_00008", + "apositive_test_query141_00005", + "negative_test_query245_00016", + "apositive_test_query141_00000", + "negative_test_query57_00002", + "negative_test_query141_00006", + "negative_test_query325_00008", + "negative_test_query176_00003", + "negative_test_query25_00006", + "negative_test_query220_00011", + "negative_test_query308_00005", + "negative_test_query344_00013", + "apositive_test_query288_00005", + "negative_test_query141_00010", + "apositive_test_query288_00007", + "negative_test_query105_00014", + "negative_test_query245_00013", + "apositive_test_query246_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query142", + "ranked_ids": [ + "apositive_test_query142_00001", + "negative_test_query238_00001", + "negative_test_query102_00001", + "negative_test_query46_00001", + "negative_test_query142_00003", + "negative_test_query46_00002", + "negative_test_query142_00009", + "apositive_test_query142_00003", + "negative_test_query46_00012", + "negative_test_query105_00006", + "negative_test_query203_00003", + "negative_test_query142_00007", + "negative_test_query223_00005", + "negative_test_query359_00005", + "negative_test_query39_00008", + "apositive_test_query142_00002", + "negative_test_query105_00008", + "negative_test_query142_00004", + "negative_test_query200_00014", + "negative_test_query337_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query143", + "ranked_ids": [ + "apositive_test_query143_00000", + "negative_test_query143_00012", + "negative_test_query143_00006", + "negative_test_query143_00011", + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query143_00005", + "negative_test_query143_00009", + "apositive_test_query25_00006", + "negative_test_query143_00000", + "negative_test_query90_00014", + "negative_test_query143_00014", + "negative_test_query143_00010", + "negative_test_query143_00013", + "negative_test_query143_00015", + "negative_test_query143_00007", + "apositive_test_query25_00005", + "negative_test_query143_00016", + "apositive_test_query308_00007", + "negative_test_query56_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query144_00012", + "negative_test_query41_00009", + "negative_test_query144_00010", + "negative_test_query144_00011", + "negative_test_query331_00015", + "negative_test_query144_00018", + "negative_test_query41_00001", + "negative_test_query315_00011", + "negative_test_query144_00003", + "negative_test_query144_00016", + "negative_test_query144_00001", + "negative_test_query144_00006", + "negative_test_query144_00017", + "negative_test_query166_00001", + "apositive_test_query41_00000", + "negative_test_query144_00009", + "negative_test_query154_00009", + "negative_test_query226_00011", + "negative_test_query41_00011", + "negative_test_query198_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query145", + "ranked_ids": [ + "apositive_test_query145_00002", + "negative_test_query230_00002", + "negative_test_query53_00012", + "apositive_test_query213_00004", + "apositive_test_query160_00010", + "negative_test_query118_00000", + "negative_test_query79_00005", + "apositive_test_query28_00008", + "apositive_test_query225_00002", + "negative_test_query112_00007", + "negative_test_query171_00003", + "apositive_test_query217_00007", + "apositive_test_query107_00001", + "apositive_test_query307_00000", + "apositive_test_query305_00002", + "apositive_test_query207_00004", + "negative_test_query187_00014", + "negative_test_query305_00002", + "negative_test_query190_00012", + "negative_test_query211_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.0625 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query146_00004", + "apositive_test_query3_00010", + "negative_test_query358_00008", + "apositive_test_query153_00008", + "apositive_test_query314_00009", + "apositive_test_query276_00001", + "negative_test_query146_00002", + "apositive_test_query146_00010", + "negative_test_query144_00008", + "apositive_test_query314_00004", + "apositive_test_query146_00003", + "apositive_test_query161_00007", + "apositive_test_query314_00001", + "negative_test_query65_00006", + "apositive_test_query146_00005", + "apositive_test_query314_00003", + "apositive_test_query161_00008", + "negative_test_query99_00001", + "negative_test_query146_00001", + "apositive_test_query161_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query147", + "ranked_ids": [ + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query147_00013", + "negative_test_query147_00008", + "negative_test_query309_00014", + "apositive_test_query161_00004", + "negative_test_query11_00009", + "apositive_test_query147_00002", + "negative_test_query258_00008", + "negative_test_query53_00009", + "negative_test_query200_00011", + "apositive_test_query147_00003", + "apositive_test_query76_00001", + "negative_test_query309_00016", + "negative_test_query354_00007", + "apositive_test_query76_00004", + "negative_test_query147_00005", + "negative_test_query180_00012", + "negative_test_query181_00008", + "negative_test_query17_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query124_00010", + "negative_test_query148_00005", + "negative_test_query148_00018", + "negative_test_query148_00012", + "negative_test_query148_00017", + "negative_test_query210_00013", + "negative_test_query89_00007", + "negative_test_query210_00008", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query112_00013", + "negative_test_query148_00008", + "negative_test_query210_00006", + "negative_test_query196_00004", + "negative_test_query148_00014", + "negative_test_query315_00002", + "negative_test_query321_00013", + "negative_test_query80_00007", + "apositive_test_query304_00001", + "negative_test_query210_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "negative_test_query149_00001", + "negative_test_query149_00011", + "negative_test_query149_00012", + "negative_test_query149_00010", + "apositive_test_query225_00004", + "negative_test_query149_00002", + "negative_test_query149_00008", + "negative_test_query332_00008", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query149_00017", + "apositive_test_query149_00001", + "negative_test_query149_00015", + "negative_test_query63_00009", + "negative_test_query149_00013", + "apositive_test_query113_00003", + "apositive_test_query149_00000", + "negative_test_query332_00004", + "negative_test_query85_00009", + "negative_test_query74_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query266_00008", + "negative_test_query15_00010", + "negative_test_query15_00012", + "negative_test_query15_00008", + "negative_test_query15_00007", + "negative_test_query15_00013", + "negative_test_query15_00014", + "apositive_test_query325_00000", + "negative_test_query176_00011", + "negative_test_query15_00015", + "apositive_test_query15_00001", + "negative_test_query15_00004", + "negative_test_query15_00011", + "negative_test_query325_00011", + "negative_test_query15_00005", + "negative_test_query15_00017", + "negative_test_query266_00013", + "negative_test_query266_00006", + "apositive_test_query242_00000", + "apositive_test_query291_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query150", + "ranked_ids": [ + "apositive_test_query150_00003", + "negative_test_query150_00010", + "negative_test_query306_00004", + "negative_test_query250_00013", + "negative_test_query139_00005", + "negative_test_query1_00012", + "apositive_test_query89_00000", + "apositive_test_query20_00010", + "negative_test_query62_00009", + "negative_test_query301_00007", + "negative_test_query213_00007", + "negative_test_query346_00008", + "negative_test_query63_00004", + "negative_test_query267_00006", + "apositive_test_query112_00000", + "negative_test_query254_00003", + "negative_test_query350_00018", + "apositive_test_query158_00001", + "negative_test_query313_00017", + "apositive_test_query20_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query126_00017", + "negative_test_query190_00017", + "apositive_test_query151_00002", + "negative_test_query126_00007", + "negative_test_query126_00000", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "apositive_test_query126_00000", + "negative_test_query190_00003", + "negative_test_query151_00006", + "apositive_test_query300_00001", + "negative_test_query190_00014", + "negative_test_query190_00005", + "negative_test_query293_00008", + "negative_test_query151_00003", + "negative_test_query44_00002", + "negative_test_query151_00015", + "negative_test_query190_00015", + "negative_test_query151_00004", + "negative_test_query201_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.3 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query237_00011", + "apositive_test_query152_00001", + "apositive_test_query152_00004", + "apositive_test_query152_00009", + "negative_test_query237_00000", + "negative_test_query312_00014", + "negative_test_query237_00004", + "negative_test_query124_00010", + "negative_test_query157_00010", + "apositive_test_query152_00005", + "negative_test_query95_00017", + "negative_test_query349_00001", + "negative_test_query342_00005", + "negative_test_query250_00002", + "negative_test_query349_00005", + "negative_test_query253_00004", + "negative_test_query182_00004", + "negative_test_query237_00009", + "apositive_test_query194_00008", + "negative_test_query262_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.25 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00003", + "apositive_test_query153_00015", + "apositive_test_query153_00002", + "apositive_test_query153_00004", + "negative_test_query283_00015", + "negative_test_query112_00007", + "negative_test_query278_00006", + "apositive_test_query153_00000", + "negative_test_query359_00003", + "apositive_test_query153_00007", + "negative_test_query278_00007", + "apositive_test_query153_00001", + "negative_test_query250_00006", + "apositive_test_query20_00006", + "negative_test_query112_00015", + "negative_test_query228_00002", + "negative_test_query2_00001", + "negative_test_query322_00001", + "apositive_test_query213_00004", + "apositive_test_query176_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query154", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query154_00006", + "apositive_test_query20_00003", + "apositive_test_query20_00007", + "apositive_test_query23_00004", + "apositive_test_query356_00005", + "negative_test_query331_00012", + "apositive_test_query33_00000", + "negative_test_query41_00007", + "apositive_test_query358_00008", + "negative_test_query133_00002", + "apositive_test_query20_00010", + "apositive_test_query154_00005", + "negative_test_query356_00004", + "negative_test_query245_00006", + "apositive_test_query66_00000", + "negative_test_query311_00005", + "negative_test_query334_00000", + "negative_test_query68_00015", + "negative_test_query102_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00006", + "negative_test_query155_00005", + "negative_test_query155_00003", + "negative_test_query155_00009", + "apositive_test_query3_00008", + "negative_test_query155_00004", + "apositive_test_query155_00005", + "negative_test_query155_00010", + "negative_test_query155_00001", + "negative_test_query155_00012", + "apositive_test_query355_00007", + "negative_test_query119_00001", + "negative_test_query199_00016", + "negative_test_query111_00016", + "apositive_test_query155_00001", + "negative_test_query155_00000", + "apositive_test_query40_00004", + "negative_test_query99_00001", + "negative_test_query123_00014", + "negative_test_query155_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "negative_test_query330_00006", + "apositive_test_query156_00000", + "negative_test_query92_00010", + "negative_test_query1_00012", + "negative_test_query283_00001", + "negative_test_query92_00003", + "negative_test_query283_00009", + "apositive_test_query330_00002", + "negative_test_query49_00005", + "apositive_test_query39_00002", + "apositive_test_query154_00005", + "negative_test_query92_00005", + "apositive_test_query210_00001", + "negative_test_query156_00001", + "apositive_test_query25_00009", + "negative_test_query246_00016", + "negative_test_query140_00008", + "negative_test_query299_00014", + "negative_test_query15_00010", + "negative_test_query32_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "apositive_test_query152_00005", + "negative_test_query237_00004", + "negative_test_query237_00009", + "negative_test_query157_00005", + "apositive_test_query237_00000", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "negative_test_query152_00008", + "apositive_test_query157_00001", + "negative_test_query152_00003", + "negative_test_query237_00002", + "apositive_test_query157_00002", + "apositive_test_query182_00000", + "apositive_test_query152_00004", + "apositive_test_query152_00000", + "negative_test_query237_00000", + "negative_test_query271_00003", + "apositive_test_query237_00002", + "negative_test_query312_00017", + "apositive_test_query237_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query158", + "ranked_ids": [ + "negative_test_query102_00015", + "negative_test_query334_00016", + "negative_test_query102_00003", + "negative_test_query223_00007", + "negative_test_query26_00004", + "apositive_test_query158_00000", + "negative_test_query223_00002", + "negative_test_query26_00005", + "apositive_test_query158_00003", + "apositive_test_query186_00003", + "negative_test_query105_00013", + "negative_test_query158_00006", + "negative_test_query164_00004", + "negative_test_query247_00010", + "negative_test_query26_00003", + "negative_test_query213_00002", + "negative_test_query26_00000", + "negative_test_query334_00009", + "negative_test_query26_00010", + "apositive_test_query158_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query159", + "ranked_ids": [ + "apositive_test_query159_00001", + "negative_test_query264_00013", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query38_00010", + "negative_test_query159_00016", + "negative_test_query159_00005", + "apositive_test_query50_00004", + "apositive_test_query326_00007", + "apositive_test_query212_00001", + "apositive_test_query212_00002", + "apositive_test_query38_00003", + "negative_test_query7_00005", + "negative_test_query264_00003", + "negative_test_query66_00003", + "apositive_test_query212_00003", + "negative_test_query122_00002", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.6 + }, + "query_id": "test_query16", + "ranked_ids": [ + "apositive_test_query16_00001", + "apositive_test_query16_00003", + "negative_test_query16_00002", + "negative_test_query308_00004", + "apositive_test_query16_00004", + "negative_test_query16_00008", + "negative_test_query16_00001", + "negative_test_query16_00000", + "negative_test_query16_00010", + "negative_test_query16_00009", + "apositive_test_query16_00002", + "negative_test_query16_00012", + "negative_test_query16_00011", + "negative_test_query280_00007", + "negative_test_query16_00003", + "negative_test_query240_00003", + "negative_test_query237_00010", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query208_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query160", + "ranked_ids": [ + "apositive_test_query160_00009", + "negative_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query330_00005", + "negative_test_query160_00004", + "apositive_test_query120_00002", + "negative_test_query211_00009", + "negative_test_query64_00009", + "negative_test_query64_00015", + "negative_test_query149_00007", + "apositive_test_query64_00000", + "apositive_test_query355_00007", + "negative_test_query271_00010", + "negative_test_query304_00007", + "negative_test_query214_00013", + "apositive_test_query160_00004", + "negative_test_query215_00010", + "negative_test_query106_00004", + "apositive_test_query260_00000", + "negative_test_query176_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query161", + "ranked_ids": [ + "negative_test_query199_00004", + "apositive_test_query161_00006", + "negative_test_query155_00012", + "apositive_test_query3_00006", + "apositive_test_query276_00010", + "apositive_test_query305_00002", + "apositive_test_query161_00008", + "apositive_test_query161_00005", + "negative_test_query22_00015", + "apositive_test_query37_00004", + "negative_test_query301_00015", + "negative_test_query71_00006", + "negative_test_query260_00004", + "negative_test_query57_00001", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "apositive_test_query261_00004", + "apositive_test_query314_00004", + "apositive_test_query314_00006", + "apositive_test_query301_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query162", + "ranked_ids": [ + "negative_test_query177_00002", + "apositive_test_query162_00009", + "apositive_test_query162_00010", + "negative_test_query162_00004", + "negative_test_query95_00006", + "apositive_test_query162_00003", + "apositive_test_query162_00000", + "negative_test_query162_00001", + "apositive_test_query162_00006", + "apositive_test_query162_00005", + "negative_test_query240_00008", + "apositive_test_query194_00006", + "negative_test_query95_00013", + "apositive_test_query162_00004", + "negative_test_query177_00005", + "apositive_test_query194_00009", + "negative_test_query343_00012", + "apositive_test_query194_00003", + "apositive_test_query114_00001", + "negative_test_query230_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query197_00002", + "negative_test_query163_00011", + "negative_test_query163_00008", + "negative_test_query90_00011", + "negative_test_query58_00015", + "negative_test_query328_00006", + "negative_test_query47_00001", + "negative_test_query265_00015", + "negative_test_query9_00005", + "negative_test_query202_00002", + "apositive_test_query172_00006", + "apositive_test_query357_00004", + "negative_test_query16_00013", + "negative_test_query163_00003", + "negative_test_query245_00001", + "negative_test_query197_00001", + "negative_test_query163_00013", + "negative_test_query224_00010", + "negative_test_query350_00017", + "apositive_test_query152_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query164_00000", + "negative_test_query131_00014", + "negative_test_query158_00008", + "negative_test_query2_00000", + "apositive_test_query146_00014", + "negative_test_query332_00001", + "apositive_test_query13_00000", + "negative_test_query164_00015", + "negative_test_query165_00003", + "negative_test_query360_00012", + "negative_test_query101_00005", + "negative_test_query208_00014", + "negative_test_query100_00013", + "negative_test_query1_00003", + "negative_test_query245_00005", + "negative_test_query344_00009", + "negative_test_query321_00011", + "negative_test_query85_00010", + "negative_test_query309_00012", + "apositive_test_query297_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query86_00006", + "negative_test_query165_00013", + "negative_test_query122_00003", + "negative_test_query86_00010", + "negative_test_query165_00007", + "apositive_test_query165_00001", + "negative_test_query165_00000", + "negative_test_query285_00007", + "negative_test_query285_00004", + "negative_test_query348_00001", + "negative_test_query264_00016", + "negative_test_query215_00013", + "negative_test_query149_00012", + "negative_test_query299_00011", + "negative_test_query327_00010", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query38_00010", + "negative_test_query327_00016", + "apositive_test_query219_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query166", + "ranked_ids": [ + "negative_test_query166_00012", + "apositive_test_query166_00005", + "negative_test_query166_00000", + "negative_test_query166_00013", + "apositive_test_query166_00003", + "negative_test_query166_00006", + "apositive_test_query166_00002", + "apositive_test_query166_00000", + "negative_test_query166_00007", + "negative_test_query166_00005", + "negative_test_query166_00004", + "apositive_test_query166_00001", + "negative_test_query166_00003", + "negative_test_query166_00001", + "negative_test_query249_00000", + "negative_test_query224_00014", + "apositive_test_query224_00000", + "negative_test_query165_00015", + "negative_test_query166_00008", + "negative_test_query255_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query167_00012", + "negative_test_query167_00002", + "negative_test_query167_00016", + "negative_test_query167_00004", + "negative_test_query167_00011", + "negative_test_query167_00005", + "negative_test_query167_00010", + "negative_test_query167_00014", + "negative_test_query167_00003", + "negative_test_query278_00016", + "negative_test_query167_00001", + "negative_test_query167_00013", + "negative_test_query167_00000", + "negative_test_query167_00008", + "negative_test_query167_00015", + "negative_test_query167_00007", + "negative_test_query167_00009", + "negative_test_query278_00010", + "apositive_test_query288_00005", + "negative_test_query141_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6096199500078984, + "recall@5": 0.75 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query83_00003", + "apositive_test_query168_00000", + "apositive_test_query168_00001", + "apositive_test_query168_00002", + "apositive_test_query190_00001", + "apositive_test_query286_00007", + "negative_test_query168_00014", + "negative_test_query6_00001", + "negative_test_query168_00000", + "negative_test_query6_00000", + "apositive_test_query340_00000", + "negative_test_query83_00003", + "negative_test_query270_00003", + "negative_test_query6_00005", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query168_00007", + "apositive_test_query83_00004", + "negative_test_query168_00009", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "apositive_test_query69_00000", + "negative_test_query49_00017", + "negative_test_query72_00013", + "negative_test_query104_00001", + "apositive_test_query286_00008", + "negative_test_query72_00001", + "negative_test_query72_00016", + "apositive_test_query180_00001", + "negative_test_query89_00006", + "negative_test_query343_00001", + "negative_test_query260_00015", + "apositive_test_query242_00006", + "apositive_test_query330_00001", + "apositive_test_query291_00009", + "apositive_test_query80_00003", + "negative_test_query10_00005", + "negative_test_query180_00004", + "apositive_test_query266_00000", + "negative_test_query10_00013", + "negative_test_query169_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00002", + "negative_test_query17_00005", + "negative_test_query17_00014", + "negative_test_query17_00000", + "negative_test_query17_00012", + "negative_test_query10_00005", + "negative_test_query34_00003", + "apositive_test_query225_00007", + "negative_test_query17_00015", + "negative_test_query17_00016", + "negative_test_query148_00001", + "negative_test_query17_00003", + "negative_test_query312_00004", + "negative_test_query305_00004", + "negative_test_query302_00011", + "negative_test_query130_00007", + "negative_test_query164_00006", + "apositive_test_query153_00013", + "negative_test_query292_00010", + "apositive_test_query355_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5555555555555556 + }, + "query_id": "test_query170", + "ranked_ids": [ + "apositive_test_query170_00005", + "apositive_test_query170_00006", + "apositive_test_query170_00000", + "apositive_test_query170_00002", + "apositive_test_query170_00004", + "negative_test_query170_00001", + "negative_test_query170_00005", + "apositive_test_query297_00000", + "negative_test_query0_00006", + "negative_test_query170_00008", + "negative_test_query170_00007", + "negative_test_query348_00001", + "apositive_test_query170_00001", + "negative_test_query297_00005", + "negative_test_query227_00002", + "negative_test_query170_00000", + "negative_test_query170_00004", + "apositive_test_query170_00007", + "apositive_test_query70_00003", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query171", + "ranked_ids": [ + "negative_test_query171_00013", + "negative_test_query171_00005", + "negative_test_query171_00000", + "negative_test_query171_00011", + "apositive_test_query171_00001", + "negative_test_query171_00004", + "negative_test_query219_00010", + "negative_test_query277_00007", + "negative_test_query171_00015", + "negative_test_query171_00017", + "negative_test_query261_00000", + "apositive_test_query224_00000", + "negative_test_query171_00001", + "negative_test_query126_00013", + "negative_test_query171_00007", + "negative_test_query171_00003", + "negative_test_query191_00006", + "negative_test_query24_00007", + "negative_test_query346_00005", + "negative_test_query219_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query172", + "ranked_ids": [ + "apositive_test_query172_00003", + "negative_test_query172_00007", + "apositive_test_query172_00004", + "negative_test_query350_00009", + "apositive_test_query254_00003", + "negative_test_query192_00010", + "apositive_test_query136_00011", + "negative_test_query208_00015", + "negative_test_query173_00006", + "apositive_test_query136_00001", + "negative_test_query8_00000", + "negative_test_query64_00006", + "apositive_test_query136_00009", + "negative_test_query192_00000", + "negative_test_query208_00007", + "negative_test_query349_00001", + "apositive_test_query172_00002", + "apositive_test_query124_00002", + "apositive_test_query172_00006", + "apositive_test_query13_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query173", + "ranked_ids": [ + "apositive_test_query173_00004", + "negative_test_query64_00017", + "negative_test_query173_00006", + "negative_test_query173_00003", + "negative_test_query64_00006", + "apositive_test_query173_00002", + "apositive_test_query173_00003", + "negative_test_query173_00011", + "negative_test_query173_00013", + "apositive_test_query173_00000", + "negative_test_query350_00002", + "negative_test_query173_00000", + "apositive_test_query173_00001", + "negative_test_query339_00006", + "negative_test_query64_00013", + "apositive_test_query134_00000", + "apositive_test_query101_00004", + "negative_test_query303_00005", + "negative_test_query64_00004", + "negative_test_query222_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00000", + "negative_test_query19_00012", + "apositive_test_query174_00012", + "apositive_test_query174_00001", + "apositive_test_query4_00012", + "negative_test_query262_00005", + "apositive_test_query174_00004", + "apositive_test_query174_00002", + "apositive_test_query174_00003", + "apositive_test_query174_00005", + "apositive_test_query174_00008", + "apositive_test_query174_00006", + "apositive_test_query174_00009", + "apositive_test_query174_00010", + "apositive_test_query174_00015", + "apositive_test_query174_00007", + "apositive_test_query174_00011", + "apositive_test_query174_00014", + "apositive_test_query174_00013", + "negative_test_query174_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00004", + "apositive_test_query175_00000", + "apositive_test_query175_00003", + "negative_test_query175_00013", + "apositive_test_query175_00005", + "negative_test_query175_00010", + "apositive_test_query50_00015", + "negative_test_query204_00002", + "negative_test_query38_00006", + "negative_test_query105_00012", + "apositive_test_query353_00005", + "apositive_test_query204_00010", + "negative_test_query204_00001", + "negative_test_query97_00007", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "negative_test_query105_00016", + "negative_test_query204_00000", + "negative_test_query7_00000", + "negative_test_query175_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query176_00015", + "apositive_test_query344_00003", + "negative_test_query195_00006", + "apositive_test_query128_00007", + "negative_test_query195_00012", + "negative_test_query224_00003", + "negative_test_query221_00007", + "negative_test_query9_00003", + "negative_test_query191_00008", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query284_00002", + "negative_test_query284_00006", + "negative_test_query24_00014", + "apositive_test_query92_00002", + "negative_test_query144_00018", + "negative_test_query9_00007", + "negative_test_query35_00005", + "negative_test_query93_00000", + "apositive_test_query56_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1 + }, + "query_id": "test_query177", + "ranked_ids": [ + "apositive_test_query177_00004", + "negative_test_query172_00008", + "negative_test_query124_00012", + "negative_test_query177_00004", + "negative_test_query177_00008", + "negative_test_query177_00006", + "apositive_test_query177_00006", + "apositive_test_query162_00009", + "apositive_test_query162_00010", + "negative_test_query147_00002", + "negative_test_query147_00005", + "apositive_test_query230_00005", + "negative_test_query152_00005", + "apositive_test_query229_00008", + "negative_test_query147_00009", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query147_00011", + "negative_test_query124_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query178", + "ranked_ids": [ + "apositive_test_query178_00004", + "negative_test_query178_00007", + "negative_test_query178_00001", + "apositive_test_query178_00005", + "apositive_test_query285_00006", + "negative_test_query165_00001", + "negative_test_query285_00009", + "negative_test_query274_00001", + "negative_test_query178_00000", + "negative_test_query214_00002", + "negative_test_query43_00001", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query178_00003", + "apositive_test_query352_00001", + "negative_test_query327_00004", + "negative_test_query321_00017", + "negative_test_query178_00010", + "negative_test_query178_00012", + "negative_test_query285_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "apositive_test_query173_00003", + "negative_test_query179_00014", + "apositive_test_query173_00000", + "negative_test_query14_00001", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query303_00005", + "negative_test_query297_00000", + "apositive_test_query316_00001", + "negative_test_query316_00001", + "negative_test_query279_00010", + "negative_test_query206_00009", + "negative_test_query179_00001", + "negative_test_query173_00013", + "negative_test_query260_00012", + "negative_test_query191_00003", + "negative_test_query64_00003", + "negative_test_query191_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.8 + }, + "query_id": "test_query18", + "ranked_ids": [ + "apositive_test_query18_00001", + "apositive_test_query18_00000", + "apositive_test_query18_00003", + "negative_test_query18_00006", + "apositive_test_query18_00002", + "apositive_test_query18_00004", + "apositive_test_query281_00013", + "negative_test_query18_00011", + "negative_test_query18_00012", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00003", + "negative_test_query18_00013", + "negative_test_query88_00002", + "negative_test_query156_00018", + "apositive_test_query288_00004", + "negative_test_query257_00012", + "negative_test_query287_00004", + "negative_test_query88_00012", + "negative_test_query300_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query180", + "ranked_ids": [ + "apositive_test_query180_00000", + "negative_test_query180_00002", + "apositive_test_query180_00001", + "negative_test_query180_00004", + "apositive_test_query262_00003", + "apositive_test_query262_00000", + "negative_test_query312_00018", + "negative_test_query180_00011", + "apositive_test_query180_00002", + "negative_test_query180_00007", + "negative_test_query180_00016", + "negative_test_query246_00003", + "negative_test_query180_00006", + "negative_test_query112_00014", + "negative_test_query169_00015", + "negative_test_query114_00003", + "negative_test_query207_00007", + "apositive_test_query262_00002", + "negative_test_query95_00014", + "negative_test_query176_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00006", + "negative_test_query181_00017", + "negative_test_query181_00012", + "negative_test_query181_00002", + "negative_test_query181_00005", + "negative_test_query181_00014", + "negative_test_query181_00007", + "negative_test_query181_00010", + "negative_test_query181_00004", + "negative_test_query181_00001", + "negative_test_query181_00018", + "negative_test_query181_00008", + "negative_test_query181_00016", + "apositive_test_query256_00003", + "negative_test_query96_00006", + "negative_test_query181_00009", + "negative_test_query348_00007", + "negative_test_query320_00007", + "apositive_test_query119_00011", + "negative_test_query183_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "apositive_test_query182_00000", + "negative_test_query182_00011", + "negative_test_query182_00017", + "negative_test_query182_00004", + "apositive_test_query152_00009", + "negative_test_query182_00018", + "negative_test_query240_00001", + "negative_test_query237_00004", + "negative_test_query337_00012", + "negative_test_query230_00008", + "negative_test_query182_00002", + "negative_test_query237_00006", + "negative_test_query296_00010", + "negative_test_query237_00001", + "negative_test_query237_00007", + "negative_test_query342_00003", + "negative_test_query342_00001", + "negative_test_query77_00002", + "negative_test_query342_00004", + "negative_test_query342_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00009", + "negative_test_query183_00006", + "negative_test_query183_00014", + "negative_test_query7_00007", + "negative_test_query183_00012", + "apositive_test_query7_00001", + "negative_test_query183_00015", + "negative_test_query183_00013", + "negative_test_query238_00007", + "negative_test_query183_00007", + "negative_test_query183_00008", + "negative_test_query7_00009", + "negative_test_query183_00018", + "negative_test_query7_00008", + "negative_test_query7_00014", + "negative_test_query112_00017", + "negative_test_query183_00000", + "apositive_test_query178_00001", + "negative_test_query183_00010", + "negative_test_query7_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "negative_test_query216_00014", + "negative_test_query158_00008", + "negative_test_query216_00007", + "apositive_test_query72_00000", + "negative_test_query216_00008", + "apositive_test_query216_00004", + "apositive_test_query33_00006", + "apositive_test_query316_00000", + "negative_test_query101_00007", + "negative_test_query156_00001", + "apositive_test_query33_00004", + "negative_test_query344_00010", + "apositive_test_query216_00000", + "apositive_test_query216_00001", + "negative_test_query228_00011", + "negative_test_query322_00000", + "negative_test_query216_00006", + "negative_test_query41_00013", + "apositive_test_query20_00010", + "negative_test_query216_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query185", + "ranked_ids": [ + "apositive_test_query185_00003", + "apositive_test_query307_00008", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "negative_test_query72_00009", + "negative_test_query185_00002", + "negative_test_query103_00001", + "negative_test_query350_00007", + "negative_test_query103_00000", + "negative_test_query185_00005", + "negative_test_query185_00000", + "negative_test_query72_00011", + "negative_test_query72_00017", + "negative_test_query48_00009", + "negative_test_query139_00014", + "apositive_test_query43_00006", + "apositive_test_query103_00000", + "negative_test_query48_00003", + "negative_test_query72_00012", + "negative_test_query48_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "negative_test_query247_00012", + "negative_test_query186_00005", + "negative_test_query46_00001", + "apositive_test_query306_00003", + "negative_test_query247_00007", + "negative_test_query313_00015", + "negative_test_query47_00009", + "apositive_test_query79_00005", + "negative_test_query336_00008", + "negative_test_query142_00009", + "apositive_test_query50_00000", + "negative_test_query29_00003", + "negative_test_query313_00005", + "negative_test_query105_00001", + "negative_test_query259_00003", + "apositive_test_query50_00001", + "apositive_test_query50_00010", + "negative_test_query57_00007", + "negative_test_query313_00014", + "negative_test_query334_00017" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query176_00008", + "negative_test_query245_00006", + "negative_test_query279_00002", + "negative_test_query306_00004", + "negative_test_query230_00000", + "apositive_test_query311_00002", + "apositive_test_query176_00000", + "apositive_test_query358_00008", + "negative_test_query62_00009", + "negative_test_query168_00002", + "apositive_test_query127_00014", + "apositive_test_query154_00005", + "negative_test_query184_00015", + "negative_test_query13_00005", + "negative_test_query331_00002", + "negative_test_query245_00015", + "apositive_test_query66_00000", + "apositive_test_query33_00008", + "negative_test_query356_00004", + "apositive_test_query20_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "apositive_test_query188_00000", + "negative_test_query188_00008", + "negative_test_query20_00006", + "negative_test_query188_00010", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "apositive_test_query20_00006", + "negative_test_query344_00007", + "apositive_test_query20_00009", + "negative_test_query188_00009", + "negative_test_query112_00017", + "negative_test_query188_00003", + "negative_test_query188_00013", + "negative_test_query220_00011", + "negative_test_query344_00013", + "negative_test_query188_00004", + "negative_test_query188_00011", + "apositive_test_query220_00004", + "negative_test_query188_00014", + "apositive_test_query220_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00007", + "negative_test_query189_00004", + "negative_test_query189_00001", + "apositive_test_query189_00002", + "negative_test_query189_00013", + "negative_test_query189_00006", + "negative_test_query189_00000", + "negative_test_query85_00007", + "apositive_test_query85_00002", + "apositive_test_query189_00003", + "apositive_test_query189_00001", + "negative_test_query140_00016", + "negative_test_query44_00005", + "negative_test_query55_00003", + "negative_test_query260_00007", + "negative_test_query18_00006", + "negative_test_query189_00002", + "negative_test_query287_00000", + "negative_test_query190_00001", + "apositive_test_query254_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query19", + "ranked_ids": [ + "negative_test_query19_00003", + "apositive_test_query19_00000", + "apositive_test_query4_00000", + "negative_test_query15_00000", + "negative_test_query19_00006", + "negative_test_query19_00008", + "negative_test_query266_00010", + "negative_test_query4_00001", + "apositive_test_query174_00014", + "negative_test_query229_00002", + "apositive_test_query4_00010", + "negative_test_query169_00014", + "apositive_test_query174_00007", + "negative_test_query106_00004", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query174_00001", + "negative_test_query141_00000", + "negative_test_query141_00001", + "negative_test_query167_00018" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query319_00008", + "apositive_test_query82_00008", + "apositive_test_query260_00003", + "negative_test_query271_00005", + "apositive_test_query311_00000", + "negative_test_query260_00007", + "negative_test_query85_00006", + "negative_test_query109_00006", + "negative_test_query6_00007", + "apositive_test_query85_00000", + "negative_test_query345_00006", + "apositive_test_query234_00001", + "negative_test_query190_00013", + "negative_test_query181_00003", + "negative_test_query246_00002", + "negative_test_query264_00008", + "negative_test_query233_00008", + "negative_test_query265_00004", + "negative_test_query85_00014", + "negative_test_query205_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "apositive_test_query191_00001", + "negative_test_query191_00010", + "apositive_test_query63_00003", + "negative_test_query96_00006", + "apositive_test_query191_00002", + "negative_test_query60_00000", + "negative_test_query241_00007", + "negative_test_query332_00012", + "apositive_test_query341_00003", + "negative_test_query191_00012", + "negative_test_query297_00002", + "negative_test_query181_00013", + "negative_test_query256_00009", + "negative_test_query191_00003", + "apositive_test_query341_00005", + "negative_test_query274_00002", + "negative_test_query191_00002", + "negative_test_query191_00011", + "negative_test_query64_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.4 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00002", + "negative_test_query120_00004", + "negative_test_query192_00001", + "apositive_test_query192_00000", + "negative_test_query192_00002", + "negative_test_query192_00013", + "negative_test_query192_00009", + "negative_test_query172_00005", + "negative_test_query192_00008", + "negative_test_query192_00010", + "negative_test_query120_00007", + "negative_test_query350_00010", + "negative_test_query350_00013", + "negative_test_query322_00008", + "apositive_test_query192_00004", + "apositive_test_query254_00007", + "negative_test_query192_00004", + "negative_test_query192_00012", + "negative_test_query192_00014", + "negative_test_query192_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.3076923076923077 + }, + "query_id": "test_query193", + "ranked_ids": [ + "apositive_test_query99_00003", + "apositive_test_query193_00000", + "apositive_test_query193_00009", + "apositive_test_query193_00001", + "apositive_test_query193_00004", + "apositive_test_query193_00011", + "apositive_test_query193_00005", + "negative_test_query99_00009", + "apositive_test_query153_00012", + "apositive_test_query99_00002", + "apositive_test_query193_00008", + "negative_test_query146_00001", + "negative_test_query193_00002", + "apositive_test_query146_00008", + "apositive_test_query276_00006", + "negative_test_query146_00003", + "negative_test_query193_00003", + "negative_test_query193_00006", + "apositive_test_query193_00002", + "negative_test_query193_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query194_00002", + "negative_test_query137_00011", + "apositive_test_query194_00007", + "negative_test_query177_00007", + "negative_test_query162_00000", + "negative_test_query95_00016", + "apositive_test_query254_00006", + "negative_test_query148_00009", + "apositive_test_query124_00000", + "negative_test_query62_00014", + "apositive_test_query162_00006", + "apositive_test_query194_00006", + "negative_test_query262_00011", + "apositive_test_query194_00000", + "negative_test_query194_00003", + "negative_test_query89_00007", + "apositive_test_query194_00001", + "apositive_test_query114_00001", + "apositive_test_query42_00001", + "negative_test_query262_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query195", + "ranked_ids": [ + "apositive_test_query195_00002", + "negative_test_query222_00003", + "apositive_test_query195_00001", + "negative_test_query195_00002", + "negative_test_query195_00013", + "apositive_test_query195_00005", + "negative_test_query285_00000", + "negative_test_query109_00008", + "negative_test_query195_00001", + "negative_test_query195_00008", + "negative_test_query321_00005", + "negative_test_query149_00012", + "negative_test_query195_00004", + "negative_test_query321_00008", + "apositive_test_query249_00012", + "negative_test_query205_00006", + "apositive_test_query251_00000", + "negative_test_query195_00005", + "negative_test_query166_00009", + "negative_test_query199_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query196_00003", + "apositive_test_query32_00011", + "apositive_test_query32_00006", + "apositive_test_query32_00012", + "negative_test_query196_00011", + "negative_test_query64_00005", + "negative_test_query235_00008", + "apositive_test_query32_00004", + "negative_test_query32_00002", + "apositive_test_query32_00002", + "negative_test_query14_00001", + "negative_test_query179_00001", + "negative_test_query139_00011", + "negative_test_query173_00012", + "negative_test_query196_00001", + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query96_00013", + "negative_test_query341_00008", + "negative_test_query325_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4776237035032179, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query197", + "ranked_ids": [ + "negative_test_query163_00001", + "apositive_test_query197_00000", + "negative_test_query197_00010", + "negative_test_query197_00016", + "apositive_test_query197_00002", + "negative_test_query197_00006", + "negative_test_query197_00013", + "apositive_test_query189_00001", + "negative_test_query197_00004", + "negative_test_query178_00008", + "apositive_test_query34_00000", + "negative_test_query163_00002", + "negative_test_query321_00006", + "negative_test_query34_00012", + "negative_test_query28_00001", + "apositive_test_query163_00001", + "apositive_test_query197_00001", + "apositive_test_query163_00000", + "negative_test_query163_00012", + "negative_test_query163_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query239_00011", + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query198_00000", + "negative_test_query112_00005", + "negative_test_query239_00013", + "negative_test_query198_00010", + "negative_test_query294_00002", + "negative_test_query239_00007", + "negative_test_query239_00001", + "negative_test_query239_00010", + "negative_test_query343_00002", + "negative_test_query62_00016", + "negative_test_query293_00010", + "negative_test_query293_00011", + "negative_test_query198_00002", + "negative_test_query198_00008", + "negative_test_query198_00004", + "negative_test_query352_00002", + "negative_test_query219_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query140_00014", + "negative_test_query199_00015", + "negative_test_query54_00003", + "negative_test_query305_00007", + "negative_test_query199_00014", + "negative_test_query139_00014", + "negative_test_query199_00005", + "negative_test_query199_00013", + "negative_test_query250_00006", + "negative_test_query350_00007", + "apositive_test_query43_00006", + "negative_test_query199_00006", + "negative_test_query185_00002", + "apositive_test_query244_00002", + "negative_test_query72_00005", + "negative_test_query199_00002", + "negative_test_query221_00008", + "negative_test_query213_00011", + "negative_test_query134_00006", + "negative_test_query199_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query44_00008", + "negative_test_query190_00016", + "negative_test_query2_00013", + "negative_test_query2_00012", + "negative_test_query2_00009", + "negative_test_query2_00002", + "negative_test_query2_00011", + "negative_test_query352_00003", + "negative_test_query2_00010", + "negative_test_query197_00004", + "apositive_test_query217_00006", + "negative_test_query85_00007", + "negative_test_query343_00004", + "negative_test_query2_00016", + "negative_test_query15_00000", + "negative_test_query125_00012", + "negative_test_query348_00005", + "negative_test_query126_00014", + "negative_test_query104_00007", + "apositive_test_query57_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query154_00006", + "apositive_test_query20_00010", + "negative_test_query154_00002", + "apositive_test_query20_00007", + "apositive_test_query23_00004", + "apositive_test_query356_00005", + "apositive_test_query154_00005", + "apositive_test_query20_00003", + "apositive_test_query20_00009", + "negative_test_query66_00002", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "apositive_test_query66_00000", + "apositive_test_query33_00008", + "negative_test_query188_00000", + "negative_test_query356_00004", + "apositive_test_query33_00000", + "negative_test_query220_00005", + "apositive_test_query154_00004", + "negative_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query264_00006", + "negative_test_query200_00009", + "negative_test_query200_00000", + "negative_test_query200_00015", + "negative_test_query337_00009", + "negative_test_query337_00004", + "negative_test_query200_00004", + "negative_test_query200_00006", + "negative_test_query200_00012", + "negative_test_query200_00010", + "negative_test_query200_00013", + "negative_test_query38_00005", + "negative_test_query200_00005", + "negative_test_query200_00007", + "negative_test_query337_00005", + "negative_test_query8_00008", + "negative_test_query264_00007", + "negative_test_query88_00013", + "negative_test_query337_00007", + "negative_test_query303_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query201", + "ranked_ids": [ + "negative_test_query231_00012", + "negative_test_query259_00002", + "negative_test_query71_00013", + "negative_test_query47_00012", + "apositive_test_query201_00006", + "negative_test_query0_00004", + "apositive_test_query30_00005", + "negative_test_query71_00000", + "negative_test_query71_00002", + "negative_test_query71_00005", + "negative_test_query65_00002", + "apositive_test_query71_00001", + "negative_test_query209_00008", + "negative_test_query71_00009", + "apositive_test_query20_00005", + "apositive_test_query30_00001", + "apositive_test_query0_00011", + "negative_test_query47_00013", + "negative_test_query159_00009", + "apositive_test_query201_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query235_00001", + "negative_test_query202_00000", + "negative_test_query202_00004", + "negative_test_query202_00009", + "negative_test_query202_00001", + "apositive_test_query206_00000", + "negative_test_query202_00008", + "negative_test_query202_00006", + "negative_test_query312_00012", + "apositive_test_query302_00006", + "negative_test_query202_00016", + "negative_test_query176_00011", + "negative_test_query321_00018", + "negative_test_query202_00010", + "negative_test_query352_00001", + "apositive_test_query34_00003", + "negative_test_query15_00010", + "apositive_test_query309_00000", + "negative_test_query188_00011", + "apositive_test_query61_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query203", + "ranked_ids": [ + "apositive_test_query110_00003", + "negative_test_query203_00006", + "negative_test_query203_00008", + "negative_test_query203_00005", + "negative_test_query250_00005", + "apositive_test_query203_00004", + "apositive_test_query30_00005", + "apositive_test_query284_00003", + "negative_test_query284_00014", + "apositive_test_query130_00002", + "apositive_test_query203_00009", + "negative_test_query209_00000", + "apositive_test_query203_00003", + "negative_test_query130_00001", + "negative_test_query250_00009", + "negative_test_query250_00011", + "apositive_test_query203_00000", + "negative_test_query35_00004", + "apositive_test_query203_00007", + "negative_test_query250_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.11764705882352941 + }, + "query_id": "test_query204", + "ranked_ids": [ + "apositive_test_query353_00005", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00002", + "negative_test_query204_00002", + "negative_test_query97_00007", + "negative_test_query353_00003", + "apositive_test_query353_00000", + "negative_test_query204_00001", + "apositive_test_query175_00004", + "apositive_test_query204_00009", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query353_00011", + "apositive_test_query204_00004", + "apositive_test_query353_00010", + "negative_test_query204_00000", + "apositive_test_query97_00001", + "apositive_test_query204_00007", + "negative_test_query353_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query205_00015", + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query96_00011", + "negative_test_query228_00016", + "negative_test_query205_00007", + "negative_test_query256_00015", + "negative_test_query341_00003", + "negative_test_query256_00000", + "apositive_test_query205_00000", + "negative_test_query256_00007", + "negative_test_query73_00000", + "negative_test_query73_00011", + "negative_test_query256_00006", + "negative_test_query35_00007", + "negative_test_query205_00008", + "negative_test_query31_00015", + "negative_test_query35_00008", + "apositive_test_query256_00002", + "negative_test_query115_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query206_00002", + "negative_test_query206_00014", + "negative_test_query206_00007", + "negative_test_query206_00015", + "negative_test_query206_00013", + "negative_test_query303_00015", + "negative_test_query206_00000", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query206_00001", + "negative_test_query206_00005", + "negative_test_query303_00003", + "apositive_test_query348_00001", + "negative_test_query70_00011", + "negative_test_query206_00003", + "negative_test_query299_00001", + "negative_test_query2_00016", + "negative_test_query206_00018", + "negative_test_query299_00002", + "negative_test_query64_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.5 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00000", + "negative_test_query207_00006", + "negative_test_query207_00002", + "apositive_test_query207_00004", + "apositive_test_query207_00005", + "negative_test_query207_00011", + "negative_test_query207_00009", + "negative_test_query207_00005", + "apositive_test_query207_00002", + "negative_test_query207_00001", + "apositive_test_query207_00001", + "negative_test_query207_00008", + "negative_test_query207_00012", + "apositive_test_query13_00001", + "negative_test_query207_00000", + "negative_test_query207_00003", + "negative_test_query207_00007", + "negative_test_query207_00013", + "negative_test_query37_00004", + "negative_test_query207_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "apositive_test_query208_00000", + "negative_test_query208_00000", + "negative_test_query128_00003", + "negative_test_query208_00016", + "negative_test_query208_00010", + "negative_test_query208_00017", + "negative_test_query208_00018", + "apositive_test_query128_00007", + "apositive_test_query128_00002", + "negative_test_query208_00008", + "negative_test_query73_00008", + "negative_test_query128_00001", + "negative_test_query211_00014", + "negative_test_query256_00011", + "negative_test_query208_00003", + "negative_test_query208_00012", + "negative_test_query61_00012", + "negative_test_query208_00004", + "negative_test_query208_00006", + "negative_test_query208_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query209", + "ranked_ids": [ + "apositive_test_query209_00001", + "negative_test_query209_00007", + "apositive_test_query37_00001", + "negative_test_query231_00006", + "negative_test_query102_00010", + "apositive_test_query158_00002", + "apositive_test_query26_00007", + "negative_test_query347_00004", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query223_00003", + "negative_test_query26_00002", + "negative_test_query102_00016", + "apositive_test_query119_00008", + "apositive_test_query158_00004", + "apositive_test_query158_00009", + "negative_test_query223_00007", + "negative_test_query209_00001", + "negative_test_query26_00004", + "negative_test_query313_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "apositive_test_query21_00000", + "negative_test_query224_00015", + "negative_test_query10_00008", + "apositive_test_query128_00003", + "apositive_test_query128_00002", + "negative_test_query234_00007", + "negative_test_query21_00010", + "negative_test_query8_00006", + "negative_test_query128_00004", + "negative_test_query208_00000", + "negative_test_query21_00005", + "negative_test_query21_00009", + "negative_test_query330_00000", + "negative_test_query128_00006", + "apositive_test_query128_00004", + "negative_test_query21_00012", + "negative_test_query208_00017", + "negative_test_query21_00001", + "negative_test_query234_00006", + "negative_test_query350_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query210_00013", + "negative_test_query148_00008", + "negative_test_query210_00015", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query148_00014", + "negative_test_query202_00014", + "negative_test_query17_00002", + "negative_test_query156_00015", + "negative_test_query288_00007", + "negative_test_query300_00000", + "negative_test_query288_00006", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query210_00007", + "negative_test_query210_00008", + "negative_test_query148_00007", + "negative_test_query210_00010", + "negative_test_query199_00015", + "negative_test_query335_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query211", + "ranked_ids": [ + "apositive_test_query211_00001", + "negative_test_query159_00006", + "negative_test_query239_00014", + "negative_test_query137_00012", + "apositive_test_query198_00000", + "apositive_test_query176_00000", + "negative_test_query100_00012", + "negative_test_query189_00008", + "negative_test_query32_00005", + "negative_test_query167_00015", + "negative_test_query195_00007", + "apositive_test_query162_00010", + "negative_test_query211_00012", + "negative_test_query293_00013", + "apositive_test_query101_00000", + "negative_test_query202_00006", + "negative_test_query83_00010", + "negative_test_query211_00000", + "apositive_test_query163_00004", + "negative_test_query211_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query212", + "ranked_ids": [ + "apositive_test_query326_00000", + "apositive_test_query212_00004", + "apositive_test_query326_00004", + "negative_test_query167_00017", + "negative_test_query167_00016", + "apositive_test_query98_00000", + "negative_test_query183_00009", + "negative_test_query183_00012", + "negative_test_query346_00006", + "apositive_test_query244_00007", + "apositive_test_query66_00005", + "negative_test_query282_00010", + "negative_test_query344_00008", + "negative_test_query282_00009", + "apositive_test_query71_00001", + "negative_test_query126_00017", + "negative_test_query190_00017", + "negative_test_query77_00004", + "apositive_test_query11_00007", + "apositive_test_query89_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00000", + "apositive_test_query213_00004", + "apositive_test_query213_00002", + "negative_test_query137_00015", + "negative_test_query213_00001", + "negative_test_query213_00005", + "negative_test_query213_00011", + "negative_test_query138_00016", + "negative_test_query224_00015", + "apositive_test_query213_00003", + "negative_test_query213_00004", + "apositive_test_query259_00001", + "apositive_test_query305_00003", + "apositive_test_query36_00002", + "apositive_test_query259_00000", + "negative_test_query357_00007", + "negative_test_query138_00001", + "negative_test_query112_00003", + "apositive_test_query158_00001", + "negative_test_query313_00017" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "apositive_test_query214_00000", + "negative_test_query214_00003", + "negative_test_query214_00013", + "negative_test_query214_00001", + "apositive_test_query234_00001", + "apositive_test_query323_00003", + "negative_test_query214_00004", + "negative_test_query214_00015", + "negative_test_query214_00010", + "negative_test_query17_00013", + "apositive_test_query89_00000", + "negative_test_query324_00002", + "apositive_test_query160_00009", + "apositive_test_query323_00002", + "negative_test_query214_00016", + "negative_test_query9_00008", + "negative_test_query324_00012", + "negative_test_query103_00009", + "apositive_test_query75_00003", + "apositive_test_query17_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00012", + "apositive_test_query215_00000", + "negative_test_query215_00007", + "negative_test_query215_00008", + "negative_test_query215_00006", + "negative_test_query215_00009", + "negative_test_query215_00018", + "negative_test_query215_00000", + "negative_test_query215_00010", + "negative_test_query257_00003", + "negative_test_query215_00011", + "negative_test_query215_00015", + "negative_test_query215_00001", + "negative_test_query263_00001", + "negative_test_query295_00004", + "negative_test_query257_00001", + "negative_test_query293_00013", + "negative_test_query215_00017", + "negative_test_query215_00004", + "apositive_test_query80_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query216_00000", + "apositive_test_query216_00004", + "negative_test_query216_00007", + "negative_test_query216_00002", + "negative_test_query216_00008", + "negative_test_query246_00016", + "apositive_test_query154_00005", + "negative_test_query216_00014", + "negative_test_query216_00006", + "apositive_test_query216_00002", + "apositive_test_query216_00001", + "apositive_test_query20_00001", + "negative_test_query216_00000", + "negative_test_query112_00002", + "negative_test_query44_00002", + "negative_test_query216_00009", + "negative_test_query216_00005", + "apositive_test_query330_00002", + "apositive_test_query33_00004", + "apositive_test_query182_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "apositive_test_query28_00008", + "apositive_test_query217_00007", + "apositive_test_query307_00000", + "negative_test_query217_00011", + "negative_test_query217_00004", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00000", + "negative_test_query28_00002", + "apositive_test_query213_00004", + "negative_test_query185_00003", + "apositive_test_query307_00010", + "negative_test_query112_00007", + "negative_test_query230_00002", + "negative_test_query53_00012", + "negative_test_query76_00003", + "negative_test_query307_00004", + "negative_test_query72_00005", + "apositive_test_query107_00001", + "apositive_test_query160_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query218", + "ranked_ids": [ + "negative_test_query134_00003", + "apositive_test_query218_00004", + "negative_test_query218_00009", + "negative_test_query218_00002", + "negative_test_query218_00000", + "negative_test_query218_00012", + "apositive_test_query218_00003", + "negative_test_query218_00003", + "apositive_test_query218_00005", + "negative_test_query274_00008", + "negative_test_query43_00006", + "negative_test_query58_00010", + "negative_test_query304_00005", + "apositive_test_query314_00015", + "apositive_test_query134_00000", + "negative_test_query43_00010", + "negative_test_query134_00009", + "apositive_test_query60_00012", + "apositive_test_query218_00000", + "negative_test_query165_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6509209298071326, + "recall@5": 1.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "negative_test_query219_00016", + "apositive_test_query219_00000", + "negative_test_query219_00014", + "apositive_test_query219_00001", + "negative_test_query308_00006", + "negative_test_query187_00015", + "negative_test_query165_00010", + "negative_test_query219_00008", + "negative_test_query59_00000", + "negative_test_query219_00011", + "negative_test_query219_00012", + "negative_test_query219_00015", + "apositive_test_query177_00004", + "negative_test_query227_00007", + "negative_test_query219_00017", + "negative_test_query305_00005", + "negative_test_query171_00005", + "negative_test_query183_00016", + "negative_test_query219_00005", + "negative_test_query24_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "negative_test_query22_00015", + "negative_test_query22_00003", + "apositive_test_query22_00000", + "negative_test_query22_00008", + "negative_test_query22_00004", + "negative_test_query22_00000", + "negative_test_query22_00005", + "negative_test_query22_00007", + "negative_test_query22_00009", + "negative_test_query22_00017", + "negative_test_query22_00018", + "negative_test_query22_00002", + "negative_test_query22_00012", + "negative_test_query66_00004", + "negative_test_query22_00013", + "negative_test_query22_00016", + "negative_test_query22_00006", + "negative_test_query22_00001", + "apositive_test_query162_00007", + "apositive_test_query177_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query220", + "ranked_ids": [ + "negative_test_query344_00007", + "apositive_test_query220_00003", + "apositive_test_query220_00000", + "apositive_test_query220_00001", + "apositive_test_query220_00004", + "negative_test_query220_00011", + "negative_test_query344_00013", + "apositive_test_query188_00000", + "negative_test_query188_00014", + "negative_test_query188_00013", + "negative_test_query188_00009", + "negative_test_query20_00002", + "negative_test_query220_00004", + "negative_test_query188_00004", + "negative_test_query154_00009", + "negative_test_query188_00008", + "negative_test_query188_00007", + "apositive_test_query146_00009", + "apositive_test_query20_00009", + "negative_test_query161_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query142_00004", + "negative_test_query65_00007", + "negative_test_query277_00010", + "apositive_test_query174_00006", + "apositive_test_query180_00002", + "negative_test_query278_00004", + "apositive_test_query306_00006", + "negative_test_query15_00005", + "negative_test_query184_00001", + "negative_test_query12_00005", + "negative_test_query171_00008", + "apositive_test_query244_00008", + "apositive_test_query162_00005", + "apositive_test_query203_00002", + "negative_test_query65_00013", + "negative_test_query30_00010", + "negative_test_query180_00011", + "negative_test_query244_00006", + "apositive_test_query92_00003", + "negative_test_query39_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8503449055347546, + "recall@5": 1.0 + }, + "query_id": "test_query222", + "ranked_ids": [ + "apositive_test_query222_00001", + "negative_test_query222_00014", + "negative_test_query222_00011", + "negative_test_query222_00006", + "apositive_test_query222_00000", + "apositive_test_query275_00001", + "negative_test_query222_00017", + "apositive_test_query289_00002", + "apositive_test_query275_00006", + "negative_test_query289_00006", + "negative_test_query290_00004", + "negative_test_query271_00015", + "apositive_test_query275_00000", + "apositive_test_query275_00005", + "negative_test_query271_00004", + "negative_test_query275_00002", + "apositive_test_query302_00000", + "apositive_test_query275_00008", + "negative_test_query275_00004", + "apositive_test_query275_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query359_00008", + "apositive_test_query259_00011", + "apositive_test_query259_00003", + "negative_test_query47_00001", + "apositive_test_query259_00008", + "negative_test_query359_00014", + "negative_test_query259_00006", + "negative_test_query323_00005", + "apositive_test_query201_00004", + "negative_test_query345_00002", + "negative_test_query164_00011", + "apositive_test_query201_00006", + "negative_test_query47_00002", + "negative_test_query51_00003", + "apositive_test_query107_00005", + "negative_test_query107_00004", + "negative_test_query47_00007", + "apositive_test_query260_00003", + "negative_test_query6_00007", + "negative_test_query109_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query224", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query350_00016", + "negative_test_query224_00012", + "negative_test_query224_00010", + "negative_test_query350_00017", + "apositive_test_query224_00003", + "negative_test_query205_00009", + "apositive_test_query256_00001", + "apositive_test_query179_00002", + "apositive_test_query32_00002", + "negative_test_query224_00014", + "apositive_test_query83_00000", + "negative_test_query224_00008", + "negative_test_query350_00015", + "negative_test_query98_00018", + "negative_test_query212_00013", + "negative_test_query149_00008", + "negative_test_query224_00011", + "negative_test_query304_00001", + "negative_test_query196_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.36363636363636365 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00001", + "apositive_test_query225_00000", + "negative_test_query145_00000", + "apositive_test_query225_00003", + "apositive_test_query225_00007", + "apositive_test_query225_00005", + "apositive_test_query225_00009", + "apositive_test_query225_00004", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query183_00015", + "negative_test_query343_00009", + "negative_test_query149_00011", + "negative_test_query225_00008", + "negative_test_query161_00004", + "apositive_test_query145_00005", + "apositive_test_query225_00002", + "negative_test_query225_00000", + "negative_test_query54_00002", + "apositive_test_query169_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query226", + "ranked_ids": [ + "apositive_test_query226_00003", + "apositive_test_query226_00005", + "negative_test_query226_00001", + "negative_test_query226_00007", + "negative_test_query226_00009", + "negative_test_query226_00012", + "negative_test_query226_00003", + "negative_test_query226_00010", + "negative_test_query226_00013", + "negative_test_query226_00004", + "negative_test_query226_00005", + "negative_test_query226_00006", + "negative_test_query226_00011", + "apositive_test_query226_00001", + "negative_test_query226_00000", + "apositive_test_query226_00000", + "apositive_test_query226_00004", + "apositive_test_query226_00002", + "negative_test_query360_00000", + "apositive_test_query153_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00003", + "negative_test_query227_00006", + "apositive_test_query227_00000", + "negative_test_query227_00012", + "negative_test_query227_00004", + "negative_test_query227_00002", + "apositive_test_query70_00001", + "negative_test_query227_00000", + "negative_test_query227_00010", + "negative_test_query227_00005", + "negative_test_query227_00014", + "negative_test_query227_00011", + "negative_test_query70_00011", + "apositive_test_query227_00001", + "negative_test_query321_00004", + "negative_test_query227_00013", + "negative_test_query235_00006", + "negative_test_query59_00004", + "negative_test_query115_00007", + "apositive_test_query348_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query228", + "ranked_ids": [ + "negative_test_query228_00014", + "negative_test_query228_00012", + "negative_test_query165_00011", + "apositive_test_query228_00000", + "negative_test_query228_00000", + "negative_test_query73_00016", + "negative_test_query228_00004", + "negative_test_query228_00002", + "negative_test_query228_00011", + "negative_test_query2_00001", + "negative_test_query322_00001", + "negative_test_query216_00013", + "negative_test_query228_00003", + "apositive_test_query322_00003", + "apositive_test_query254_00000", + "apositive_test_query318_00001", + "negative_test_query179_00010", + "apositive_test_query322_00007", + "apositive_test_query333_00001", + "apositive_test_query73_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query229_00002", + "apositive_test_query229_00003", + "apositive_test_query229_00000", + "negative_test_query229_00005", + "negative_test_query229_00006", + "negative_test_query229_00008", + "apositive_test_query229_00007", + "negative_test_query229_00000", + "negative_test_query229_00009", + "negative_test_query229_00007", + "apositive_test_query18_00004", + "apositive_test_query69_00009", + "negative_test_query355_00001", + "apositive_test_query158_00006", + "negative_test_query44_00005", + "apositive_test_query94_00007", + "negative_test_query0_00001", + "apositive_test_query146_00004", + "negative_test_query94_00010", + "apositive_test_query158_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query23", + "ranked_ids": [ + "apositive_test_query23_00000", + "apositive_test_query23_00002", + "negative_test_query66_00010", + "apositive_test_query23_00001", + "negative_test_query259_00007", + "negative_test_query356_00008", + "apositive_test_query39_00002", + "negative_test_query239_00012", + "apositive_test_query146_00015", + "negative_test_query23_00014", + "apositive_test_query54_00007", + "apositive_test_query225_00000", + "apositive_test_query54_00004", + "negative_test_query133_00003", + "negative_test_query292_00013", + "negative_test_query307_00002", + "apositive_test_query52_00000", + "apositive_test_query52_00001", + "apositive_test_query52_00002", + "apositive_test_query52_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query230_00007", + "negative_test_query230_00000", + "negative_test_query230_00010", + "apositive_test_query230_00003", + "apositive_test_query3_00001", + "negative_test_query329_00000", + "negative_test_query269_00010", + "apositive_test_query3_00000", + "negative_test_query23_00012", + "apositive_test_query3_00003", + "apositive_test_query3_00006", + "negative_test_query99_00003", + "negative_test_query3_00005", + "apositive_test_query3_00009", + "negative_test_query3_00004", + "negative_test_query3_00000", + "negative_test_query280_00006", + "apositive_test_query280_00002", + "negative_test_query280_00004", + "negative_test_query3_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query231", + "ranked_ids": [ + "negative_test_query345_00011", + "apositive_test_query130_00000", + "apositive_test_query0_00001", + "apositive_test_query231_00003", + "negative_test_query75_00000", + "apositive_test_query201_00006", + "apositive_test_query201_00002", + "negative_test_query231_00001", + "negative_test_query47_00006", + "negative_test_query306_00006", + "apositive_test_query130_00004", + "negative_test_query130_00010", + "negative_test_query9_00017", + "negative_test_query213_00004", + "apositive_test_query119_00006", + "negative_test_query130_00000", + "negative_test_query359_00008", + "negative_test_query201_00011", + "negative_test_query47_00005", + "negative_test_query221_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "apositive_test_query232_00000", + "negative_test_query232_00013", + "negative_test_query232_00014", + "negative_test_query232_00000", + "negative_test_query232_00011", + "negative_test_query232_00003", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query232_00008", + "negative_test_query232_00015", + "negative_test_query232_00018", + "negative_test_query232_00004", + "negative_test_query232_00002", + "negative_test_query232_00006", + "negative_test_query232_00010", + "negative_test_query232_00001", + "negative_test_query232_00016", + "negative_test_query232_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query233", + "ranked_ids": [ + "apositive_test_query233_00002", + "apositive_test_query233_00000", + "negative_test_query233_00001", + "negative_test_query2_00010", + "negative_test_query233_00010", + "negative_test_query233_00006", + "negative_test_query233_00005", + "negative_test_query233_00015", + "negative_test_query190_00007", + "negative_test_query233_00004", + "negative_test_query233_00009", + "negative_test_query246_00001", + "apositive_test_query2_00000", + "negative_test_query263_00009", + "negative_test_query219_00015", + "apositive_test_query275_00003", + "negative_test_query2_00014", + "negative_test_query130_00008", + "negative_test_query176_00017", + "negative_test_query130_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00015", + "apositive_test_query304_00003", + "negative_test_query234_00005", + "negative_test_query234_00011", + "negative_test_query304_00001", + "negative_test_query304_00005", + "negative_test_query214_00008", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query214_00006", + "negative_test_query234_00010", + "negative_test_query304_00008", + "apositive_test_query43_00005", + "apositive_test_query304_00001", + "negative_test_query304_00012", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query304_00013", + "apositive_test_query218_00002", + "negative_test_query10_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00002", + "negative_test_query235_00014", + "apositive_test_query32_00006", + "negative_test_query235_00008", + "apositive_test_query32_00000", + "negative_test_query235_00004", + "negative_test_query61_00001", + "negative_test_query235_00009", + "negative_test_query196_00000", + "negative_test_query235_00000", + "apositive_test_query235_00000", + "negative_test_query235_00011", + "apositive_test_query32_00004", + "negative_test_query173_00005", + "negative_test_query235_00015", + "apositive_test_query235_00003", + "apositive_test_query235_00001", + "negative_test_query235_00013", + "negative_test_query24_00017", + "negative_test_query219_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "apositive_test_query279_00002", + "negative_test_query236_00000", + "negative_test_query236_00014", + "negative_test_query236_00009", + "negative_test_query236_00012", + "negative_test_query279_00012", + "negative_test_query236_00006", + "negative_test_query236_00004", + "negative_test_query236_00013", + "negative_test_query279_00005", + "negative_test_query236_00011", + "negative_test_query279_00003", + "negative_test_query236_00018", + "negative_test_query22_00016", + "negative_test_query236_00005", + "negative_test_query236_00007", + "negative_test_query236_00015", + "negative_test_query222_00000", + "negative_test_query236_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "negative_test_query237_00006", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "apositive_test_query237_00002", + "negative_test_query152_00008", + "negative_test_query237_00007", + "apositive_test_query237_00001", + "negative_test_query237_00001", + "negative_test_query261_00012", + "negative_test_query237_00009", + "apositive_test_query237_00000", + "negative_test_query237_00004", + "negative_test_query237_00005", + "apositive_test_query152_00004", + "negative_test_query290_00010", + "negative_test_query152_00003", + "negative_test_query237_00000", + "negative_test_query312_00001", + "apositive_test_query157_00001", + "negative_test_query237_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "apositive_test_query146_00013", + "negative_test_query123_00001", + "negative_test_query46_00012", + "apositive_test_query276_00008", + "negative_test_query111_00013", + "apositive_test_query29_00007", + "negative_test_query56_00008", + "negative_test_query56_00000", + "negative_test_query238_00011", + "apositive_test_query29_00001", + "apositive_test_query142_00003", + "negative_test_query29_00006", + "negative_test_query102_00007", + "apositive_test_query306_00003", + "negative_test_query108_00007", + "negative_test_query111_00008", + "negative_test_query13_00003", + "negative_test_query142_00011", + "negative_test_query203_00003", + "negative_test_query223_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "negative_test_query239_00015", + "apositive_test_query239_00000", + "negative_test_query239_00003", + "negative_test_query239_00001", + "negative_test_query239_00014", + "negative_test_query198_00002", + "negative_test_query239_00000", + "negative_test_query239_00007", + "negative_test_query163_00004", + "negative_test_query239_00010", + "negative_test_query268_00005", + "negative_test_query198_00008", + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query239_00011", + "negative_test_query239_00013", + "negative_test_query352_00002", + "negative_test_query183_00015", + "negative_test_query61_00004", + "negative_test_query167_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "negative_test_query24_00009", + "negative_test_query24_00000", + "apositive_test_query226_00004", + "negative_test_query15_00000", + "negative_test_query24_00004", + "apositive_test_query57_00004", + "apositive_test_query24_00000", + "negative_test_query24_00008", + "apositive_test_query106_00002", + "negative_test_query24_00002", + "negative_test_query250_00008", + "negative_test_query24_00012", + "negative_test_query128_00002", + "negative_test_query356_00003", + "negative_test_query323_00000", + "negative_test_query252_00005", + "negative_test_query198_00006", + "negative_test_query52_00012", + "negative_test_query190_00016", + "negative_test_query24_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "apositive_test_query240_00000", + "negative_test_query240_00012", + "negative_test_query240_00006", + "negative_test_query240_00010", + "negative_test_query240_00004", + "negative_test_query240_00002", + "negative_test_query240_00015", + "negative_test_query240_00017", + "negative_test_query240_00007", + "negative_test_query240_00018", + "negative_test_query240_00003", + "negative_test_query240_00011", + "negative_test_query246_00002", + "apositive_test_query311_00000", + "negative_test_query27_00010", + "negative_test_query240_00016", + "negative_test_query282_00013", + "negative_test_query6_00000", + "negative_test_query293_00004", + "negative_test_query297_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "apositive_test_query241_00000", + "negative_test_query241_00010", + "negative_test_query241_00005", + "negative_test_query241_00016", + "negative_test_query241_00017", + "negative_test_query241_00002", + "negative_test_query206_00004", + "negative_test_query241_00008", + "negative_test_query173_00013", + "negative_test_query241_00006", + "negative_test_query206_00009", + "negative_test_query206_00011", + "negative_test_query256_00000", + "negative_test_query258_00000", + "negative_test_query53_00000", + "negative_test_query191_00011", + "negative_test_query139_00007", + "negative_test_query205_00003", + "negative_test_query298_00014", + "negative_test_query191_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query291_00000", + "apositive_test_query242_00002", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query242_00001", + "apositive_test_query291_00001", + "apositive_test_query247_00004", + "apositive_test_query242_00005", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query242_00004", + "apositive_test_query242_00010", + "negative_test_query291_00004", + "negative_test_query291_00003", + "negative_test_query291_00005", + "apositive_test_query291_00008", + "apositive_test_query242_00008", + "apositive_test_query247_00005", + "apositive_test_query242_00013", + "apositive_test_query291_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query243_00000", + "apositive_test_query243_00002", + "apositive_test_query243_00011", + "negative_test_query274_00014", + "negative_test_query289_00000", + "negative_test_query243_00000", + "apositive_test_query243_00005", + "apositive_test_query243_00012", + "negative_test_query205_00007", + "negative_test_query250_00013", + "apositive_test_query243_00003", + "negative_test_query303_00005", + "negative_test_query243_00002", + "negative_test_query192_00010", + "negative_test_query192_00007", + "apositive_test_query173_00004", + "apositive_test_query160_00003", + "negative_test_query64_00003", + "negative_test_query184_00004", + "negative_test_query223_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query244", + "ranked_ids": [ + "negative_test_query244_00001", + "apositive_test_query244_00000", + "negative_test_query244_00005", + "negative_test_query244_00008", + "apositive_test_query244_00004", + "negative_test_query244_00004", + "negative_test_query244_00009", + "apositive_test_query244_00002", + "apositive_test_query244_00001", + "negative_test_query244_00003", + "negative_test_query244_00000", + "negative_test_query244_00010", + "negative_test_query244_00002", + "apositive_test_query244_00005", + "apositive_test_query244_00007", + "apositive_test_query356_00003", + "apositive_test_query244_00006", + "apositive_test_query42_00002", + "negative_test_query126_00015", + "apositive_test_query23_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query245_00003", + "negative_test_query245_00010", + "negative_test_query64_00013", + "apositive_test_query245_00000", + "negative_test_query135_00013", + "negative_test_query245_00000", + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query245_00001", + "negative_test_query13_00005", + "negative_test_query245_00015", + "negative_test_query122_00002", + "negative_test_query202_00000", + "negative_test_query121_00002", + "negative_test_query293_00003", + "negative_test_query159_00003", + "apositive_test_query141_00006", + "negative_test_query336_00002", + "negative_test_query202_00004", + "apositive_test_query141_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query246_00001", + "negative_test_query252_00003", + "negative_test_query246_00012", + "negative_test_query246_00002", + "negative_test_query246_00016", + "negative_test_query190_00003", + "negative_test_query135_00017", + "negative_test_query6_00014", + "negative_test_query198_00012", + "negative_test_query44_00002", + "negative_test_query317_00014", + "negative_test_query6_00006", + "apositive_test_query0_00001", + "negative_test_query311_00005", + "negative_test_query293_00000", + "apositive_test_query216_00000", + "apositive_test_query68_00002", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "apositive_test_query242_00002", + "negative_test_query291_00005", + "apositive_test_query247_00004", + "apositive_test_query291_00001", + "negative_test_query291_00003", + "apositive_test_query247_00005", + "apositive_test_query242_00001", + "negative_test_query247_00005", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query291_00000", + "negative_test_query186_00010", + "negative_test_query247_00011", + "negative_test_query105_00012", + "apositive_test_query242_00010", + "negative_test_query291_00001", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "apositive_test_query242_00012", + "negative_test_query247_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.25 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00008", + "apositive_test_query248_00002", + "negative_test_query248_00011", + "apositive_test_query248_00005", + "negative_test_query248_00010", + "negative_test_query251_00017", + "apositive_test_query248_00007", + "negative_test_query248_00002", + "apositive_test_query29_00004", + "apositive_test_query248_00003", + "negative_test_query248_00006", + "negative_test_query248_00004", + "negative_test_query29_00001", + "apositive_test_query29_00000", + "apositive_test_query29_00005", + "apositive_test_query248_00001", + "negative_test_query248_00000", + "negative_test_query248_00009", + "negative_test_query205_00017", + "negative_test_query248_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query249", + "ranked_ids": [ + "apositive_test_query249_00000", + "negative_test_query156_00002", + "negative_test_query249_00001", + "negative_test_query193_00001", + "negative_test_query71_00006", + "apositive_test_query112_00000", + "negative_test_query325_00015", + "apositive_test_query177_00004", + "apositive_test_query249_00004", + "apositive_test_query281_00001", + "negative_test_query210_00010", + "negative_test_query326_00005", + "negative_test_query302_00007", + "negative_test_query278_00016", + "negative_test_query301_00013", + "negative_test_query122_00016", + "negative_test_query156_00007", + "negative_test_query72_00017", + "apositive_test_query204_00003", + "apositive_test_query353_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query25", + "ranked_ids": [ + "apositive_test_query25_00000", + "negative_test_query25_00007", + "negative_test_query45_00014", + "apositive_test_query25_00008", + "apositive_test_query25_00006", + "apositive_test_query25_00005", + "negative_test_query143_00000", + "negative_test_query25_00002", + "negative_test_query25_00001", + "apositive_test_query25_00007", + "apositive_test_query25_00003", + "negative_test_query25_00005", + "apositive_test_query25_00009", + "apositive_test_query25_00001", + "apositive_test_query25_00010", + "apositive_test_query25_00004", + "negative_test_query30_00003", + "negative_test_query346_00006", + "negative_test_query9_00001", + "negative_test_query328_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query250", + "ranked_ids": [ + "apositive_test_query250_00000", + "negative_test_query186_00007", + "negative_test_query250_00009", + "negative_test_query250_00011", + "negative_test_query250_00013", + "negative_test_query111_00015", + "apositive_test_query203_00000", + "negative_test_query79_00013", + "negative_test_query57_00009", + "negative_test_query102_00010", + "apositive_test_query203_00006", + "negative_test_query250_00001", + "negative_test_query159_00014", + "negative_test_query192_00009", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query159_00004", + "negative_test_query184_00015", + "negative_test_query64_00005", + "negative_test_query250_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query251", + "ranked_ids": [ + "apositive_test_query251_00001", + "negative_test_query189_00011", + "apositive_test_query134_00003", + "negative_test_query322_00002", + "negative_test_query56_00005", + "negative_test_query234_00006", + "negative_test_query251_00003", + "negative_test_query251_00004", + "negative_test_query298_00003", + "apositive_test_query29_00002", + "negative_test_query143_00001", + "negative_test_query160_00001", + "negative_test_query8_00009", + "negative_test_query326_00000", + "negative_test_query268_00011", + "negative_test_query14_00000", + "apositive_test_query347_00000", + "negative_test_query269_00007", + "negative_test_query170_00008", + "negative_test_query347_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query252", + "ranked_ids": [ + "apositive_test_query252_00000", + "apositive_test_query252_00003", + "negative_test_query252_00009", + "negative_test_query252_00012", + "negative_test_query126_00001", + "negative_test_query151_00000", + "negative_test_query252_00007", + "negative_test_query212_00000", + "negative_test_query330_00006", + "apositive_test_query125_00001", + "negative_test_query252_00011", + "negative_test_query318_00006", + "negative_test_query252_00001", + "negative_test_query321_00004", + "apositive_test_query252_00002", + "apositive_test_query110_00001", + "negative_test_query44_00011", + "negative_test_query16_00006", + "negative_test_query252_00010", + "negative_test_query14_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2 + }, + "query_id": "test_query253", + "ranked_ids": [ + "apositive_test_query253_00003", + "apositive_test_query194_00008", + "apositive_test_query253_00004", + "apositive_test_query13_00012", + "negative_test_query253_00005", + "apositive_test_query162_00008", + "negative_test_query177_00000", + "apositive_test_query253_00007", + "apositive_test_query186_00004", + "apositive_test_query74_00002", + "apositive_test_query74_00003", + "negative_test_query194_00001", + "apositive_test_query349_00000", + "negative_test_query124_00001", + "negative_test_query226_00013", + "negative_test_query253_00003", + "negative_test_query95_00017", + "negative_test_query253_00004", + "negative_test_query172_00008", + "negative_test_query124_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query254", + "ranked_ids": [ + "apositive_test_query254_00004", + "apositive_test_query254_00000", + "negative_test_query350_00000", + "negative_test_query254_00008", + "negative_test_query270_00010", + "apositive_test_query254_00005", + "apositive_test_query254_00002", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00010", + "apositive_test_query254_00003", + "negative_test_query136_00003", + "apositive_test_query254_00001", + "apositive_test_query254_00006", + "negative_test_query228_00016", + "negative_test_query21_00009", + "apositive_test_query333_00004", + "negative_test_query254_00004", + "negative_test_query35_00015", + "negative_test_query316_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "negative_test_query255_00000", + "apositive_test_query255_00000", + "negative_test_query255_00017", + "apositive_test_query322_00001", + "negative_test_query255_00010", + "negative_test_query246_00003", + "negative_test_query255_00006", + "negative_test_query326_00006", + "negative_test_query255_00018", + "apositive_test_query25_00004", + "negative_test_query208_00008", + "negative_test_query278_00015", + "apositive_test_query313_00000", + "negative_test_query255_00013", + "negative_test_query89_00004", + "negative_test_query255_00004", + "negative_test_query255_00005", + "negative_test_query110_00001", + "apositive_test_query278_00000", + "negative_test_query278_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.75369761125927, + "recall@5": 0.75 + }, + "query_id": "test_query256", + "ranked_ids": [ + "apositive_test_query256_00003", + "negative_test_query256_00003", + "apositive_test_query256_00002", + "apositive_test_query256_00001", + "negative_test_query256_00011", + "apositive_test_query256_00000", + "apositive_test_query119_00000", + "negative_test_query183_00008", + "negative_test_query347_00007", + "negative_test_query7_00009", + "apositive_test_query43_00001", + "apositive_test_query216_00002", + "negative_test_query279_00000", + "negative_test_query5_00010", + "negative_test_query256_00008", + "negative_test_query319_00001", + "negative_test_query82_00001", + "negative_test_query135_00016", + "apositive_test_query142_00002", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query257", + "ranked_ids": [ + "apositive_test_query257_00004", + "negative_test_query257_00005", + "apositive_test_query257_00001", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query52_00005", + "apositive_test_query257_00000", + "apositive_test_query286_00014", + "apositive_test_query257_00003", + "apositive_test_query257_00002", + "apositive_test_query69_00007", + "apositive_test_query257_00005", + "apositive_test_query52_00001", + "negative_test_query257_00007", + "negative_test_query52_00004", + "negative_test_query52_00007", + "apositive_test_query54_00003", + "negative_test_query54_00005", + "negative_test_query52_00002", + "apositive_test_query151_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query320_00002", + "negative_test_query228_00005", + "negative_test_query206_00016", + "negative_test_query311_00002", + "negative_test_query162_00004", + "negative_test_query167_00007", + "negative_test_query319_00000", + "negative_test_query82_00000", + "negative_test_query259_00001", + "negative_test_query181_00002", + "negative_test_query144_00010", + "negative_test_query38_00002", + "apositive_test_query301_00002", + "negative_test_query169_00016", + "negative_test_query101_00001", + "negative_test_query171_00002", + "negative_test_query339_00008", + "negative_test_query173_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.25 + }, + "query_id": "test_query259", + "ranked_ids": [ + "negative_test_query259_00002", + "apositive_test_query259_00006", + "apositive_test_query259_00007", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "apositive_test_query259_00008", + "negative_test_query259_00006", + "apositive_test_query259_00003", + "apositive_test_query231_00001", + "apositive_test_query345_00000", + "apositive_test_query54_00011", + "negative_test_query111_00010", + "negative_test_query47_00004", + "negative_test_query250_00015", + "negative_test_query359_00012", + "negative_test_query213_00004", + "negative_test_query259_00001", + "negative_test_query30_00009", + "negative_test_query359_00003", + "apositive_test_query0_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "apositive_test_query26_00005", + "negative_test_query223_00002", + "negative_test_query26_00008", + "negative_test_query102_00015", + "negative_test_query26_00005", + "apositive_test_query102_00000", + "apositive_test_query158_00003", + "apositive_test_query186_00003", + "apositive_test_query26_00004", + "negative_test_query105_00013", + "negative_test_query158_00006", + "negative_test_query164_00004", + "negative_test_query247_00010", + "negative_test_query26_00003", + "negative_test_query119_00006", + "negative_test_query158_00009", + "negative_test_query26_00009", + "negative_test_query26_00001", + "negative_test_query334_00016", + "negative_test_query238_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "apositive_test_query6_00002", + "negative_test_query260_00011", + "negative_test_query260_00009", + "apositive_test_query6_00001", + "apositive_test_query286_00011", + "negative_test_query6_00015", + "negative_test_query252_00004", + "negative_test_query6_00005", + "negative_test_query6_00009", + "negative_test_query6_00004", + "negative_test_query6_00011", + "apositive_test_query6_00000", + "negative_test_query125_00001", + "negative_test_query6_00002", + "negative_test_query2_00004", + "apositive_test_query109_00000", + "apositive_test_query186_00001", + "negative_test_query260_00002", + "apositive_test_query109_00001", + "negative_test_query6_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query261", + "ranked_ids": [ + "negative_test_query261_00005", + "negative_test_query261_00006", + "negative_test_query116_00004", + "negative_test_query261_00010", + "negative_test_query261_00013", + "negative_test_query261_00008", + "negative_test_query261_00007", + "negative_test_query261_00004", + "negative_test_query261_00002", + "negative_test_query261_00001", + "negative_test_query261_00003", + "negative_test_query261_00000", + "negative_test_query261_00011", + "apositive_test_query261_00002", + "apositive_test_query261_00003", + "negative_test_query118_00006", + "negative_test_query2_00015", + "apositive_test_query261_00000", + "negative_test_query261_00009", + "apositive_test_query29_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query262", + "ranked_ids": [ + "apositive_test_query262_00000", + "negative_test_query266_00003", + "negative_test_query106_00005", + "negative_test_query301_00005", + "negative_test_query262_00000", + "negative_test_query312_00000", + "negative_test_query312_00003", + "apositive_test_query106_00003", + "apositive_test_query262_00003", + "negative_test_query312_00018", + "negative_test_query106_00010", + "apositive_test_query194_00002", + "apositive_test_query177_00008", + "apositive_test_query42_00006", + "negative_test_query172_00000", + "negative_test_query147_00001", + "negative_test_query152_00006", + "negative_test_query162_00007", + "negative_test_query326_00008", + "negative_test_query124_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query263", + "ranked_ids": [ + "negative_test_query135_00004", + "negative_test_query169_00014", + "apositive_test_query263_00002", + "negative_test_query190_00009", + "apositive_test_query155_00003", + "apositive_test_query242_00011", + "apositive_test_query291_00010", + "negative_test_query253_00009", + "negative_test_query125_00012", + "negative_test_query307_00005", + "apositive_test_query37_00002", + "negative_test_query71_00006", + "negative_test_query88_00003", + "negative_test_query263_00005", + "apositive_test_query231_00001", + "apositive_test_query292_00002", + "negative_test_query201_00010", + "negative_test_query213_00001", + "negative_test_query213_00005", + "negative_test_query357_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "apositive_test_query264_00000", + "negative_test_query264_00011", + "negative_test_query264_00004", + "negative_test_query264_00014", + "negative_test_query200_00002", + "negative_test_query200_00017", + "negative_test_query264_00002", + "negative_test_query264_00018", + "negative_test_query159_00013", + "negative_test_query264_00006", + "negative_test_query200_00011", + "apositive_test_query38_00003", + "negative_test_query337_00002", + "negative_test_query159_00011", + "negative_test_query264_00008", + "negative_test_query264_00012", + "negative_test_query264_00007", + "negative_test_query337_00012", + "negative_test_query264_00016", + "negative_test_query159_00017" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query265", + "ranked_ids": [ + "apositive_test_query265_00000", + "apositive_test_query265_00001", + "negative_test_query265_00002", + "negative_test_query265_00000", + "negative_test_query86_00007", + "negative_test_query274_00008", + "apositive_test_query156_00000", + "negative_test_query165_00016", + "negative_test_query265_00009", + "negative_test_query86_00008", + "negative_test_query165_00007", + "negative_test_query271_00005", + "negative_test_query92_00012", + "negative_test_query265_00007", + "apositive_test_query234_00001", + "negative_test_query52_00000", + "apositive_test_query78_00002", + "negative_test_query66_00010", + "negative_test_query190_00013", + "negative_test_query139_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "negative_test_query266_00008", + "negative_test_query266_00018", + "apositive_test_query266_00000", + "apositive_test_query19_00002", + "negative_test_query141_00000", + "negative_test_query106_00004", + "negative_test_query266_00009", + "negative_test_query266_00001", + "negative_test_query266_00000", + "apositive_test_query54_00009", + "negative_test_query266_00017", + "negative_test_query266_00005", + "negative_test_query266_00002", + "negative_test_query176_00018", + "negative_test_query266_00014", + "negative_test_query266_00010", + "negative_test_query169_00014", + "negative_test_query106_00001", + "negative_test_query174_00001", + "negative_test_query266_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "negative_test_query79_00004", + "negative_test_query267_00002", + "apositive_test_query295_00001", + "apositive_test_query157_00008", + "negative_test_query98_00013", + "negative_test_query295_00002", + "negative_test_query359_00005", + "negative_test_query345_00013", + "negative_test_query79_00002", + "negative_test_query5_00012", + "negative_test_query186_00006", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query250_00000", + "negative_test_query313_00016", + "apositive_test_query50_00004", + "negative_test_query30_00000", + "negative_test_query204_00001", + "apositive_test_query113_00002", + "negative_test_query115_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query268", + "ranked_ids": [ + "negative_test_query268_00000", + "apositive_test_query268_00004", + "negative_test_query268_00012", + "apositive_test_query268_00000", + "apositive_test_query201_00003", + "negative_test_query268_00013", + "negative_test_query47_00017", + "negative_test_query268_00004", + "negative_test_query268_00003", + "apositive_test_query51_00003", + "apositive_test_query268_00002", + "negative_test_query268_00008", + "negative_test_query323_00002", + "negative_test_query268_00007", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query210_00007", + "negative_test_query268_00005", + "negative_test_query228_00007", + "negative_test_query335_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query269", + "ranked_ids": [ + "apositive_test_query269_00000", + "negative_test_query329_00014", + "negative_test_query269_00001", + "negative_test_query269_00009", + "negative_test_query269_00000", + "negative_test_query269_00015", + "negative_test_query269_00012", + "negative_test_query269_00004", + "negative_test_query269_00002", + "negative_test_query329_00004", + "negative_test_query269_00008", + "negative_test_query269_00010", + "negative_test_query269_00007", + "negative_test_query296_00003", + "apositive_test_query269_00003", + "apositive_test_query296_00005", + "negative_test_query269_00003", + "negative_test_query269_00005", + "apositive_test_query296_00003", + "negative_test_query269_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "negative_test_query27_00004", + "negative_test_query27_00009", + "negative_test_query164_00007", + "apositive_test_query59_00000", + "negative_test_query147_00000", + "negative_test_query138_00006", + "negative_test_query259_00006", + "negative_test_query76_00009", + "negative_test_query358_00003", + "negative_test_query315_00001", + "negative_test_query30_00009", + "negative_test_query130_00006", + "apositive_test_query29_00004", + "apositive_test_query81_00007", + "negative_test_query27_00014", + "apositive_test_query127_00010", + "apositive_test_query154_00000", + "negative_test_query33_00004", + "apositive_test_query16_00002", + "negative_test_query11_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query270", + "ranked_ids": [ + "negative_test_query270_00012", + "negative_test_query270_00000", + "negative_test_query270_00009", + "negative_test_query270_00004", + "negative_test_query270_00002", + "negative_test_query270_00008", + "negative_test_query270_00014", + "negative_test_query270_00006", + "negative_test_query270_00011", + "apositive_test_query270_00001", + "negative_test_query270_00003", + "negative_test_query270_00010", + "negative_test_query270_00015", + "apositive_test_query270_00000", + "apositive_test_query270_00002", + "negative_test_query270_00005", + "negative_test_query270_00016", + "apositive_test_query81_00000", + "negative_test_query81_00006", + "negative_test_query76_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query271", + "ranked_ids": [ + "apositive_test_query271_00000", + "apositive_test_query271_00001", + "negative_test_query330_00008", + "negative_test_query120_00003", + "apositive_test_query10_00002", + "negative_test_query78_00010", + "negative_test_query327_00010", + "negative_test_query271_00007", + "negative_test_query84_00014", + "apositive_test_query314_00007", + "negative_test_query169_00009", + "negative_test_query221_00012", + "negative_test_query198_00015", + "negative_test_query241_00018", + "apositive_test_query189_00000", + "apositive_test_query109_00001", + "negative_test_query90_00001", + "negative_test_query227_00009", + "negative_test_query112_00014", + "apositive_test_query67_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00016", + "negative_test_query272_00003", + "negative_test_query272_00015", + "negative_test_query144_00000", + "negative_test_query272_00001", + "negative_test_query272_00010", + "negative_test_query272_00002", + "apositive_test_query272_00000", + "negative_test_query272_00008", + "negative_test_query328_00000", + "negative_test_query272_00014", + "negative_test_query41_00008", + "negative_test_query272_00017", + "negative_test_query18_00014", + "negative_test_query272_00000", + "negative_test_query272_00009", + "negative_test_query272_00013", + "negative_test_query332_00013", + "negative_test_query272_00012", + "apositive_test_query251_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00008", + "apositive_test_query273_00005", + "apositive_test_query281_00008", + "negative_test_query18_00011", + "negative_test_query273_00000", + "apositive_test_query273_00000", + "apositive_test_query18_00000", + "negative_test_query273_00006", + "apositive_test_query18_00003", + "negative_test_query273_00009", + "apositive_test_query273_00004", + "apositive_test_query18_00001", + "apositive_test_query18_00004", + "negative_test_query18_00006", + "negative_test_query18_00012", + "negative_test_query166_00009", + "negative_test_query273_00004", + "negative_test_query140_00013", + "negative_test_query300_00015", + "apositive_test_query36_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query274", + "ranked_ids": [ + "apositive_test_query274_00000", + "negative_test_query274_00000", + "negative_test_query274_00014", + "apositive_test_query274_00003", + "negative_test_query274_00011", + "negative_test_query274_00012", + "negative_test_query34_00014", + "negative_test_query352_00012", + "negative_test_query274_00005", + "apositive_test_query274_00001", + "negative_test_query256_00015", + "negative_test_query218_00011", + "negative_test_query78_00007", + "negative_test_query274_00007", + "negative_test_query200_00009", + "negative_test_query274_00009", + "apositive_test_query327_00002", + "negative_test_query274_00004", + "negative_test_query352_00009", + "negative_test_query34_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.4166666666666667 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query275_00000", + "apositive_test_query275_00004", + "apositive_test_query275_00005", + "apositive_test_query275_00003", + "apositive_test_query275_00007", + "negative_test_query275_00000", + "apositive_test_query275_00006", + "negative_test_query202_00005", + "apositive_test_query275_00008", + "negative_test_query275_00004", + "negative_test_query289_00013", + "negative_test_query263_00009", + "negative_test_query159_00017", + "negative_test_query264_00017", + "negative_test_query275_00007", + "negative_test_query149_00016", + "apositive_test_query110_00001", + "negative_test_query16_00001", + "apositive_test_query93_00003", + "negative_test_query87_00017" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query22_00004", + "negative_test_query199_00012", + "apositive_test_query276_00005", + "apositive_test_query276_00004", + "negative_test_query276_00001", + "negative_test_query280_00002", + "negative_test_query153_00001", + "apositive_test_query153_00006", + "apositive_test_query161_00002", + "negative_test_query146_00000", + "negative_test_query146_00001", + "negative_test_query251_00012", + "apositive_test_query314_00007", + "apositive_test_query314_00005", + "apositive_test_query314_00006", + "negative_test_query113_00011", + "apositive_test_query276_00006", + "apositive_test_query3_00010", + "apositive_test_query146_00015", + "negative_test_query146_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query277", + "ranked_ids": [ + "apositive_test_query277_00001", + "apositive_test_query277_00003", + "apositive_test_query277_00000", + "negative_test_query277_00015", + "negative_test_query277_00007", + "negative_test_query277_00004", + "negative_test_query277_00014", + "negative_test_query277_00008", + "negative_test_query313_00009", + "negative_test_query277_00003", + "apositive_test_query107_00007", + "apositive_test_query277_00002", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "negative_test_query345_00000", + "negative_test_query277_00000", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query277_00009", + "negative_test_query277_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query278_00002", + "negative_test_query140_00006", + "apositive_test_query263_00005", + "negative_test_query278_00011", + "apositive_test_query4_00013", + "negative_test_query260_00011", + "apositive_test_query36_00001", + "negative_test_query260_00002", + "negative_test_query324_00011", + "negative_test_query158_00003", + "negative_test_query260_00009", + "apositive_test_query186_00001", + "negative_test_query64_00007", + "negative_test_query15_00007", + "negative_test_query177_00001", + "negative_test_query278_00009", + "negative_test_query255_00016", + "negative_test_query255_00018", + "negative_test_query110_00011", + "negative_test_query27_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.6 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query279_00000", + "apositive_test_query279_00003", + "apositive_test_query279_00001", + "negative_test_query279_00014", + "apositive_test_query177_00004", + "negative_test_query279_00009", + "apositive_test_query348_00007", + "apositive_test_query279_00004", + "apositive_test_query3_00000", + "apositive_test_query177_00006", + "apositive_test_query3_00002", + "negative_test_query279_00013", + "negative_test_query181_00008", + "negative_test_query309_00011", + "apositive_test_query76_00005", + "apositive_test_query358_00007", + "negative_test_query279_00007", + "apositive_test_query162_00009", + "negative_test_query132_00013", + "apositive_test_query76_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query28", + "ranked_ids": [ + "apositive_test_query217_00000", + "negative_test_query72_00004", + "apositive_test_query185_00006", + "negative_test_query222_00005", + "apositive_test_query217_00001", + "apositive_test_query28_00003", + "apositive_test_query28_00009", + "negative_test_query72_00018", + "apositive_test_query28_00008", + "negative_test_query217_00004", + "negative_test_query307_00003", + "apositive_test_query28_00007", + "apositive_test_query217_00003", + "negative_test_query28_00002", + "negative_test_query48_00009", + "negative_test_query185_00003", + "negative_test_query185_00001", + "negative_test_query217_00011", + "negative_test_query307_00004", + "negative_test_query72_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.5 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "apositive_test_query280_00000", + "negative_test_query280_00004", + "negative_test_query280_00005", + "apositive_test_query280_00004", + "negative_test_query280_00008", + "negative_test_query106_00006", + "negative_test_query280_00010", + "negative_test_query9_00012", + "apositive_test_query280_00003", + "negative_test_query280_00006", + "apositive_test_query253_00009", + "negative_test_query280_00002", + "negative_test_query280_00013", + "negative_test_query129_00008", + "apositive_test_query74_00001", + "negative_test_query236_00018", + "negative_test_query280_00012", + "negative_test_query280_00003", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query273_00002", + "apositive_test_query281_00004", + "apositive_test_query281_00010", + "apositive_test_query281_00012", + "apositive_test_query281_00007", + "apositive_test_query281_00016", + "apositive_test_query281_00003", + "apositive_test_query281_00001", + "apositive_test_query281_00008", + "negative_test_query273_00003", + "negative_test_query257_00012", + "apositive_test_query281_00011", + "apositive_test_query273_00008", + "negative_test_query273_00004", + "apositive_test_query281_00014", + "apositive_test_query273_00000", + "negative_test_query249_00000", + "apositive_test_query273_00003", + "negative_test_query273_00007", + "negative_test_query125_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5706417189553201, + "recall@5": 1.0 + }, + "query_id": "test_query282", + "ranked_ids": [ + "negative_test_query344_00001", + "negative_test_query282_00007", + "apositive_test_query282_00000", + "apositive_test_query282_00001", + "apositive_test_query344_00000", + "negative_test_query282_00011", + "negative_test_query282_00016", + "negative_test_query282_00000", + "negative_test_query282_00010", + "negative_test_query344_00008", + "negative_test_query344_00000", + "negative_test_query282_00004", + "negative_test_query282_00012", + "negative_test_query282_00009", + "apositive_test_query20_00004", + "negative_test_query282_00003", + "apositive_test_query220_00002", + "negative_test_query344_00004", + "negative_test_query282_00001", + "negative_test_query282_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query283_00004", + "negative_test_query93_00002", + "negative_test_query93_00004", + "negative_test_query283_00007", + "negative_test_query283_00016", + "negative_test_query283_00003", + "negative_test_query93_00010", + "negative_test_query93_00009", + "negative_test_query31_00010", + "negative_test_query93_00006", + "negative_test_query265_00013", + "negative_test_query283_00012", + "negative_test_query283_00014", + "negative_test_query138_00009", + "negative_test_query138_00005", + "negative_test_query283_00000", + "negative_test_query92_00010", + "apositive_test_query93_00002", + "negative_test_query270_00006", + "negative_test_query299_00017" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.6 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "apositive_test_query284_00003", + "negative_test_query284_00009", + "negative_test_query284_00001", + "apositive_test_query284_00001", + "apositive_test_query198_00002", + "negative_test_query14_00014", + "apositive_test_query162_00005", + "negative_test_query284_00012", + "apositive_test_query163_00003", + "negative_test_query114_00006", + "apositive_test_query141_00002", + "apositive_test_query141_00006", + "apositive_test_query163_00002", + "negative_test_query109_00009", + "apositive_test_query79_00004", + "negative_test_query288_00011", + "negative_test_query284_00004", + "apositive_test_query259_00009", + "negative_test_query141_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.375 + }, + "query_id": "test_query285", + "ranked_ids": [ + "apositive_test_query285_00004", + "apositive_test_query285_00003", + "negative_test_query285_00002", + "negative_test_query285_00004", + "apositive_test_query285_00001", + "negative_test_query285_00008", + "apositive_test_query285_00007", + "negative_test_query304_00003", + "negative_test_query285_00005", + "apositive_test_query285_00002", + "apositive_test_query285_00006", + "apositive_test_query285_00005", + "negative_test_query285_00000", + "negative_test_query285_00011", + "negative_test_query304_00009", + "negative_test_query285_00010", + "negative_test_query86_00001", + "negative_test_query285_00001", + "negative_test_query285_00007", + "negative_test_query285_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query286", + "ranked_ids": [ + "negative_test_query257_00005", + "negative_test_query286_00000", + "negative_test_query36_00001", + "apositive_test_query286_00010", + "apositive_test_query286_00013", + "apositive_test_query286_00012", + "negative_test_query257_00001", + "apositive_test_query286_00006", + "negative_test_query52_00001", + "apositive_test_query286_00005", + "apositive_test_query80_00000", + "negative_test_query286_00003", + "negative_test_query54_00004", + "negative_test_query36_00013", + "negative_test_query125_00012", + "apositive_test_query286_00014", + "negative_test_query52_00005", + "apositive_test_query257_00001", + "negative_test_query139_00008", + "apositive_test_query257_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.4 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008", + "apositive_test_query287_00001", + "apositive_test_query133_00002", + "apositive_test_query133_00001", + "apositive_test_query133_00014", + "negative_test_query301_00010", + "apositive_test_query133_00004", + "apositive_test_query133_00006", + "apositive_test_query133_00013", + "apositive_test_query287_00004", + "apositive_test_query133_00012", + "negative_test_query153_00003", + "negative_test_query359_00011", + "negative_test_query133_00002", + "apositive_test_query153_00003", + "apositive_test_query4_00009", + "negative_test_query134_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query288", + "ranked_ids": [ + "apositive_test_query141_00002", + "negative_test_query288_00007", + "negative_test_query288_00006", + "negative_test_query288_00002", + "apositive_test_query210_00002", + "apositive_test_query210_00001", + "apositive_test_query141_00006", + "apositive_test_query288_00002", + "apositive_test_query304_00001", + "negative_test_query148_00014", + "negative_test_query210_00005", + "negative_test_query213_00007", + "negative_test_query288_00010", + "negative_test_query288_00001", + "negative_test_query148_00004", + "negative_test_query17_00002", + "negative_test_query199_00015", + "negative_test_query210_00013", + "negative_test_query199_00010", + "negative_test_query93_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00006", + "negative_test_query289_00001", + "negative_test_query289_00013", + "negative_test_query334_00014", + "apositive_test_query275_00005", + "negative_test_query289_00000", + "apositive_test_query289_00000", + "negative_test_query267_00005", + "negative_test_query305_00011", + "apositive_test_query157_00008", + "apositive_test_query289_00002", + "apositive_test_query289_00004", + "negative_test_query47_00009", + "negative_test_query113_00009", + "negative_test_query7_00006", + "negative_test_query247_00001", + "negative_test_query289_00004", + "negative_test_query250_00016", + "negative_test_query107_00004", + "negative_test_query289_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.375 + }, + "query_id": "test_query29", + "ranked_ids": [ + "negative_test_query29_00009", + "negative_test_query29_00008", + "apositive_test_query29_00004", + "apositive_test_query29_00005", + "apositive_test_query29_00000", + "negative_test_query248_00009", + "negative_test_query39_00011", + "apositive_test_query29_00007", + "apositive_test_query29_00001", + "apositive_test_query29_00003", + "negative_test_query248_00007", + "negative_test_query29_00007", + "negative_test_query213_00012", + "negative_test_query29_00006", + "negative_test_query29_00010", + "apositive_test_query29_00002", + "negative_test_query292_00015", + "apositive_test_query29_00006", + "negative_test_query29_00000", + "negative_test_query29_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.3973220069685352, + "recall@5": 0.5 + }, + "query_id": "test_query290", + "ranked_ids": [ + "negative_test_query237_00010", + "apositive_test_query290_00001", + "negative_test_query290_00005", + "negative_test_query98_00003", + "apositive_test_query290_00002", + "negative_test_query342_00005", + "negative_test_query290_00009", + "apositive_test_query290_00003", + "negative_test_query290_00014", + "apositive_test_query237_00002", + "negative_test_query290_00003", + "negative_test_query290_00000", + "negative_test_query290_00006", + "negative_test_query312_00001", + "negative_test_query290_00008", + "negative_test_query59_00002", + "negative_test_query116_00002", + "negative_test_query290_00002", + "negative_test_query157_00005", + "negative_test_query237_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query242_00001", + "apositive_test_query291_00001", + "apositive_test_query291_00000", + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query242_00005", + "negative_test_query242_00000", + "negative_test_query291_00000", + "apositive_test_query242_00000", + "apositive_test_query291_00002", + "negative_test_query291_00005", + "apositive_test_query291_00008", + "apositive_test_query242_00010", + "negative_test_query291_00004", + "negative_test_query291_00003", + "apositive_test_query242_00013", + "apositive_test_query291_00005", + "apositive_test_query247_00005", + "negative_test_query186_00000", + "negative_test_query291_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query292", + "ranked_ids": [ + "apositive_test_query292_00001", + "negative_test_query292_00010", + "apositive_test_query292_00000", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query292_00000", + "negative_test_query292_00005", + "negative_test_query292_00012", + "negative_test_query292_00007", + "negative_test_query292_00014", + "negative_test_query292_00008", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query298_00000", + "negative_test_query292_00006", + "negative_test_query139_00001", + "negative_test_query205_00000", + "negative_test_query61_00011", + "negative_test_query99_00004", + "negative_test_query151_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query293", + "ranked_ids": [ + "apositive_test_query293_00001", + "apositive_test_query293_00000", + "negative_test_query293_00015", + "negative_test_query27_00008", + "negative_test_query293_00009", + "negative_test_query27_00011", + "negative_test_query27_00002", + "negative_test_query293_00003", + "apositive_test_query293_00002", + "apositive_test_query44_00001", + "apositive_test_query178_00003", + "negative_test_query293_00000", + "negative_test_query293_00006", + "apositive_test_query52_00005", + "negative_test_query293_00016", + "negative_test_query89_00002", + "negative_test_query293_00010", + "negative_test_query167_00013", + "negative_test_query220_00007", + "negative_test_query293_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00011", + "negative_test_query294_00008", + "apositive_test_query294_00001", + "negative_test_query294_00003", + "negative_test_query294_00002", + "negative_test_query294_00013", + "negative_test_query294_00006", + "negative_test_query294_00005", + "negative_test_query314_00001", + "negative_test_query294_00004", + "negative_test_query294_00012", + "negative_test_query294_00016", + "negative_test_query294_00001", + "negative_test_query294_00007", + "negative_test_query277_00005", + "negative_test_query58_00003", + "negative_test_query111_00015", + "apositive_test_query170_00005", + "apositive_test_query177_00004", + "negative_test_query156_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query295", + "ranked_ids": [ + "apositive_test_query295_00001", + "negative_test_query26_00002", + "apositive_test_query295_00000", + "negative_test_query295_00008", + "negative_test_query295_00002", + "apositive_test_query57_00000", + "apositive_test_query295_00002", + "negative_test_query295_00013", + "apositive_test_query223_00001", + "negative_test_query295_00011", + "negative_test_query292_00015", + "negative_test_query292_00016", + "negative_test_query295_00016", + "negative_test_query168_00005", + "negative_test_query115_00005", + "negative_test_query295_00001", + "negative_test_query295_00012", + "negative_test_query295_00004", + "negative_test_query57_00009", + "negative_test_query295_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query296", + "ranked_ids": [ + "apositive_test_query296_00002", + "negative_test_query340_00009", + "apositive_test_query296_00001", + "negative_test_query296_00005", + "negative_test_query296_00001", + "apositive_test_query296_00000", + "apositive_test_query296_00004", + "negative_test_query296_00009", + "apositive_test_query296_00005", + "apositive_test_query296_00003", + "negative_test_query340_00014", + "negative_test_query296_00003", + "negative_test_query296_00008", + "negative_test_query296_00013", + "negative_test_query329_00011", + "negative_test_query0_00004", + "negative_test_query296_00000", + "negative_test_query296_00002", + "negative_test_query296_00012", + "negative_test_query112_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query297_00012", + "negative_test_query297_00013", + "negative_test_query297_00004", + "negative_test_query297_00014", + "negative_test_query297_00003", + "apositive_test_query139_00001", + "negative_test_query297_00001", + "apositive_test_query297_00000", + "negative_test_query191_00003", + "negative_test_query297_00002", + "negative_test_query64_00003", + "negative_test_query191_00011", + "negative_test_query297_00008", + "negative_test_query64_00005", + "negative_test_query70_00005", + "negative_test_query297_00007", + "negative_test_query191_00010", + "negative_test_query329_00007", + "negative_test_query297_00010", + "negative_test_query117_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "apositive_test_query298_00000", + "apositive_test_query298_00002", + "negative_test_query216_00010", + "negative_test_query298_00006", + "negative_test_query202_00011", + "negative_test_query347_00014", + "negative_test_query182_00011", + "apositive_test_query298_00001", + "apositive_test_query347_00001", + "negative_test_query240_00001", + "negative_test_query182_00010", + "negative_test_query298_00003", + "negative_test_query298_00001", + "apositive_test_query56_00002", + "negative_test_query175_00009", + "negative_test_query193_00006", + "negative_test_query298_00004", + "negative_test_query347_00000", + "negative_test_query347_00016", + "apositive_test_query347_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query299_00005", + "apositive_test_query346_00001", + "negative_test_query299_00006", + "apositive_test_query265_00002", + "apositive_test_query325_00002", + "apositive_test_query141_00002", + "negative_test_query300_00000", + "negative_test_query351_00009", + "negative_test_query125_00005", + "apositive_test_query101_00003", + "negative_test_query134_00000", + "negative_test_query305_00007", + "negative_test_query92_00010", + "negative_test_query288_00007", + "negative_test_query210_00008", + "negative_test_query288_00000", + "negative_test_query3_00003", + "negative_test_query58_00005", + "negative_test_query89_00008", + "apositive_test_query227_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query3", + "ranked_ids": [ + "apositive_test_query3_00000", + "apositive_test_query3_00001", + "negative_test_query3_00000", + "apositive_test_query3_00003", + "negative_test_query279_00000", + "negative_test_query230_00007", + "negative_test_query230_00000", + "negative_test_query329_00000", + "negative_test_query83_00007", + "negative_test_query23_00012", + "negative_test_query230_00010", + "negative_test_query3_00005", + "negative_test_query147_00014", + "apositive_test_query177_00006", + "apositive_test_query3_00007", + "apositive_test_query3_00002", + "negative_test_query147_00011", + "negative_test_query3_00007", + "apositive_test_query230_00003", + "apositive_test_query3_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query30", + "ranked_ids": [ + "apositive_test_query71_00000", + "apositive_test_query71_00002", + "apositive_test_query30_00001", + "apositive_test_query306_00002", + "apositive_test_query30_00004", + "apositive_test_query30_00005", + "negative_test_query231_00008", + "negative_test_query71_00010", + "negative_test_query71_00005", + "apositive_test_query209_00005", + "negative_test_query259_00002", + "apositive_test_query30_00003", + "negative_test_query231_00002", + "negative_test_query182_00014", + "negative_test_query201_00005", + "negative_test_query357_00002", + "apositive_test_query30_00000", + "negative_test_query231_00000", + "negative_test_query356_00010", + "negative_test_query97_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query15_00000", + "negative_test_query300_00009", + "negative_test_query252_00005", + "apositive_test_query300_00001", + "negative_test_query266_00010", + "negative_test_query312_00011", + "negative_test_query300_00013", + "negative_test_query190_00004", + "negative_test_query185_00001", + "negative_test_query300_00008", + "negative_test_query300_00006", + "negative_test_query167_00018", + "apositive_test_query300_00002", + "negative_test_query300_00011", + "negative_test_query141_00001", + "negative_test_query141_00000", + "negative_test_query190_00016", + "negative_test_query174_00001", + "negative_test_query300_00005", + "negative_test_query300_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.75369761125927, + "recall@5": 0.75 + }, + "query_id": "test_query301", + "ranked_ids": [ + "apositive_test_query301_00000", + "negative_test_query301_00011", + "apositive_test_query301_00002", + "apositive_test_query301_00001", + "negative_test_query301_00006", + "negative_test_query301_00015", + "negative_test_query2_00008", + "negative_test_query301_00002", + "negative_test_query301_00001", + "negative_test_query301_00000", + "negative_test_query301_00014", + "negative_test_query316_00017", + "negative_test_query301_00013", + "negative_test_query148_00009", + "negative_test_query301_00007", + "apositive_test_query171_00000", + "apositive_test_query86_00002", + "apositive_test_query86_00000", + "negative_test_query21_00018", + "negative_test_query297_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query302", + "ranked_ids": [ + "apositive_test_query302_00000", + "apositive_test_query302_00002", + "apositive_test_query302_00003", + "negative_test_query302_00010", + "negative_test_query302_00012", + "negative_test_query302_00001", + "negative_test_query308_00008", + "apositive_test_query302_00001", + "apositive_test_query302_00005", + "apositive_test_query302_00004", + "apositive_test_query308_00001", + "negative_test_query302_00000", + "apositive_test_query324_00002", + "negative_test_query308_00003", + "apositive_test_query308_00009", + "apositive_test_query308_00008", + "apositive_test_query324_00001", + "negative_test_query308_00002", + "negative_test_query308_00009", + "negative_test_query308_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query35_00003", + "apositive_test_query303_00000", + "negative_test_query303_00017", + "negative_test_query173_00001", + "negative_test_query316_00009", + "negative_test_query206_00004", + "negative_test_query206_00006", + "negative_test_query303_00002", + "negative_test_query241_00006", + "negative_test_query303_00015", + "negative_test_query206_00011", + "negative_test_query206_00009", + "negative_test_query303_00005", + "negative_test_query243_00000", + "negative_test_query307_00006", + "negative_test_query61_00011", + "negative_test_query274_00014", + "negative_test_query196_00007", + "negative_test_query8_00013", + "apositive_test_query86_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query304_00001", + "negative_test_query304_00005", + "negative_test_query214_00002", + "negative_test_query43_00001", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query304_00013", + "negative_test_query234_00010", + "negative_test_query285_00011", + "negative_test_query304_00009", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "apositive_test_query304_00003", + "negative_test_query304_00012", + "negative_test_query304_00008", + "negative_test_query304_00004", + "negative_test_query304_00010", + "negative_test_query304_00007", + "apositive_test_query285_00005", + "negative_test_query43_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query305", + "ranked_ids": [ + "apositive_test_query305_00001", + "negative_test_query305_00014", + "apositive_test_query357_00000", + "negative_test_query323_00003", + "negative_test_query203_00006", + "negative_test_query0_00007", + "negative_test_query0_00003", + "negative_test_query0_00004", + "negative_test_query134_00006", + "apositive_test_query259_00011", + "apositive_test_query127_00005", + "negative_test_query310_00000", + "negative_test_query164_00011", + "negative_test_query224_00015", + "negative_test_query56_00004", + "negative_test_query51_00003", + "negative_test_query65_00013", + "apositive_test_query71_00003", + "negative_test_query305_00005", + "negative_test_query47_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query306", + "ranked_ids": [ + "apositive_test_query306_00002", + "apositive_test_query71_00002", + "apositive_test_query209_00005", + "negative_test_query305_00014", + "negative_test_query71_00012", + "apositive_test_query71_00001", + "negative_test_query204_00001", + "negative_test_query159_00009", + "negative_test_query259_00002", + "apositive_test_query30_00001", + "apositive_test_query119_00006", + "negative_test_query71_00011", + "apositive_test_query71_00003", + "apositive_test_query289_00003", + "negative_test_query71_00013", + "negative_test_query182_00014", + "apositive_test_query30_00002", + "apositive_test_query127_00007", + "negative_test_query209_00008", + "negative_test_query142_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "negative_test_query103_00012", + "apositive_test_query279_00002", + "apositive_test_query3_00003", + "negative_test_query251_00001", + "negative_test_query70_00010", + "negative_test_query144_00004", + "negative_test_query187_00011", + "negative_test_query346_00017", + "apositive_test_query13_00005", + "negative_test_query207_00004", + "apositive_test_query279_00003", + "negative_test_query48_00018", + "negative_test_query163_00010", + "apositive_test_query170_00005", + "negative_test_query217_00003", + "negative_test_query18_00005", + "negative_test_query307_00006", + "negative_test_query360_00003", + "negative_test_query132_00008", + "apositive_test_query177_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.4 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query308_00002", + "apositive_test_query308_00000", + "apositive_test_query308_00001", + "apositive_test_query308_00008", + "negative_test_query308_00008", + "apositive_test_query324_00000", + "negative_test_query159_00013", + "apositive_test_query308_00009", + "negative_test_query308_00003", + "negative_test_query324_00012", + "negative_test_query324_00014", + "apositive_test_query324_00002", + "apositive_test_query324_00001", + "apositive_test_query308_00003", + "negative_test_query308_00009", + "negative_test_query94_00011", + "negative_test_query308_00007", + "apositive_test_query310_00008", + "apositive_test_query7_00000", + "apositive_test_query341_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "negative_test_query123_00007", + "negative_test_query309_00005", + "negative_test_query354_00002", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "negative_test_query309_00015", + "negative_test_query309_00012", + "apositive_test_query309_00000", + "apositive_test_query358_00007", + "negative_test_query309_00003", + "apositive_test_query358_00006", + "negative_test_query354_00008", + "apositive_test_query161_00008", + "negative_test_query113_00013", + "negative_test_query81_00002", + "negative_test_query113_00011", + "negative_test_query42_00009", + "apositive_test_query358_00003", + "negative_test_query309_00006", + "apositive_test_query76_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query31_00011", + "negative_test_query31_00003", + "negative_test_query313_00007", + "apositive_test_query31_00000", + "negative_test_query31_00013", + "negative_test_query31_00015", + "negative_test_query129_00000", + "negative_test_query31_00001", + "negative_test_query31_00006", + "negative_test_query31_00002", + "negative_test_query31_00007", + "negative_test_query31_00010", + "negative_test_query31_00000", + "negative_test_query31_00008", + "apositive_test_query60_00011", + "apositive_test_query249_00009", + "negative_test_query31_00004", + "apositive_test_query213_00005", + "negative_test_query326_00005", + "negative_test_query38_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5 + }, + "query_id": "test_query310", + "ranked_ids": [ + "apositive_test_query310_00008", + "apositive_test_query310_00001", + "apositive_test_query310_00000", + "apositive_test_query310_00009", + "apositive_test_query310_00002", + "apositive_test_query310_00003", + "apositive_test_query218_00005", + "apositive_test_query310_00006", + "negative_test_query329_00003", + "negative_test_query241_00001", + "negative_test_query179_00013", + "negative_test_query74_00002", + "negative_test_query191_00010", + "negative_test_query330_00007", + "negative_test_query148_00003", + "negative_test_query148_00010", + "negative_test_query292_00007", + "negative_test_query36_00015", + "negative_test_query310_00001", + "negative_test_query59_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query311_00005", + "negative_test_query311_00008", + "apositive_test_query311_00000", + "negative_test_query240_00014", + "negative_test_query311_00012", + "negative_test_query240_00008", + "apositive_test_query330_00001", + "negative_test_query202_00017", + "negative_test_query246_00006", + "negative_test_query134_00001", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query218_00004", + "negative_test_query58_00001", + "apositive_test_query5_00002", + "negative_test_query12_00008", + "apositive_test_query330_00002", + "negative_test_query288_00004", + "negative_test_query330_00009", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "negative_test_query312_00006", + "apositive_test_query212_00004", + "apositive_test_query326_00004", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "apositive_test_query312_00000", + "negative_test_query182_00002", + "negative_test_query157_00002", + "apositive_test_query98_00000", + "negative_test_query246_00015", + "negative_test_query191_00000", + "negative_test_query212_00003", + "negative_test_query206_00010", + "negative_test_query343_00016", + "apositive_test_query237_00001", + "negative_test_query242_00001", + "negative_test_query294_00015", + "negative_test_query305_00015", + "apositive_test_query220_00005", + "negative_test_query283_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query46_00011", + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query334_00001", + "negative_test_query46_00005", + "negative_test_query46_00003", + "negative_test_query359_00005", + "negative_test_query313_00010", + "negative_test_query57_00001", + "negative_test_query56_00001", + "apositive_test_query142_00005", + "apositive_test_query146_00009", + "negative_test_query161_00010", + "negative_test_query336_00001", + "negative_test_query111_00013", + "apositive_test_query108_00001", + "negative_test_query313_00002", + "negative_test_query158_00000", + "negative_test_query119_00005", + "apositive_test_query155_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.11764705882352941 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query3_00009", + "apositive_test_query314_00003", + "apositive_test_query314_00016", + "apositive_test_query314_00001", + "negative_test_query314_00001", + "apositive_test_query314_00007", + "apositive_test_query276_00007", + "apositive_test_query314_00005", + "apositive_test_query314_00006", + "apositive_test_query314_00002", + "apositive_test_query314_00011", + "apositive_test_query314_00004", + "apositive_test_query314_00010", + "negative_test_query37_00006", + "negative_test_query314_00002", + "apositive_test_query314_00012", + "apositive_test_query314_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query315_00002", + "apositive_test_query315_00000", + "negative_test_query333_00007", + "apositive_test_query319_00004", + "negative_test_query85_00013", + "apositive_test_query82_00004", + "apositive_test_query147_00004", + "negative_test_query76_00000", + "negative_test_query345_00006", + "apositive_test_query319_00005", + "apositive_test_query82_00005", + "apositive_test_query315_00001", + "negative_test_query85_00014", + "apositive_test_query85_00003", + "negative_test_query85_00005", + "negative_test_query315_00009", + "negative_test_query269_00004", + "apositive_test_query319_00001", + "apositive_test_query82_00001", + "negative_test_query319_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query316", + "ranked_ids": [ + "apositive_test_query316_00000", + "negative_test_query301_00014", + "negative_test_query316_00011", + "negative_test_query316_00017", + "negative_test_query316_00000", + "negative_test_query316_00004", + "apositive_test_query316_00001", + "negative_test_query316_00008", + "negative_test_query316_00001", + "negative_test_query316_00016", + "negative_test_query316_00005", + "apositive_test_query63_00007", + "negative_test_query301_00005", + "negative_test_query101_00007", + "negative_test_query316_00002", + "apositive_test_query216_00000", + "apositive_test_query72_00000", + "negative_test_query228_00011", + "negative_test_query322_00000", + "negative_test_query158_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00004", + "negative_test_query317_00005", + "negative_test_query317_00008", + "negative_test_query317_00013", + "negative_test_query317_00014", + "negative_test_query317_00000", + "apositive_test_query317_00000", + "negative_test_query317_00010", + "apositive_test_query317_00001", + "negative_test_query317_00006", + "negative_test_query317_00012", + "negative_test_query331_00009", + "negative_test_query116_00014", + "negative_test_query55_00014", + "negative_test_query317_00015", + "negative_test_query116_00012", + "negative_test_query271_00002", + "negative_test_query283_00013", + "negative_test_query317_00016", + "negative_test_query317_00017" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00000", + "apositive_test_query318_00002", + "apositive_test_query318_00001", + "negative_test_query318_00015", + "negative_test_query318_00012", + "negative_test_query73_00012", + "apositive_test_query318_00003", + "negative_test_query318_00010", + "negative_test_query318_00003", + "negative_test_query318_00000", + "negative_test_query318_00002", + "negative_test_query318_00005", + "negative_test_query318_00004", + "negative_test_query73_00005", + "negative_test_query73_00004", + "negative_test_query318_00014", + "negative_test_query318_00008", + "negative_test_query318_00009", + "negative_test_query322_00010", + "negative_test_query73_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00001", + "apositive_test_query319_00003", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00004", + "apositive_test_query82_00001", + "apositive_test_query82_00004", + "apositive_test_query82_00008", + "apositive_test_query319_00008", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query319_00009", + "negative_test_query340_00013", + "negative_test_query82_00009", + "negative_test_query319_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query196_00000", + "apositive_test_query32_00002", + "negative_test_query189_00014", + "apositive_test_query136_00003", + "negative_test_query196_00001", + "apositive_test_query32_00013", + "apositive_test_query357_00000", + "negative_test_query55_00011", + "apositive_test_query17_00000", + "negative_test_query78_00007", + "negative_test_query55_00014", + "apositive_test_query192_00000", + "negative_test_query321_00001", + "negative_test_query341_00010", + "negative_test_query211_00016", + "negative_test_query339_00006", + "negative_test_query211_00005", + "negative_test_query68_00009", + "negative_test_query83_00001", + "apositive_test_query336_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query320", + "ranked_ids": [ + "apositive_test_query320_00003", + "negative_test_query344_00010", + "negative_test_query318_00011", + "negative_test_query100_00001", + "apositive_test_query238_00000", + "negative_test_query212_00006", + "negative_test_query131_00009", + "negative_test_query91_00007", + "negative_test_query320_00005", + "negative_test_query230_00001", + "negative_test_query193_00006", + "negative_test_query258_00012", + "negative_test_query53_00014", + "negative_test_query43_00000", + "negative_test_query320_00004", + "negative_test_query216_00014", + "apositive_test_query267_00001", + "negative_test_query320_00008", + "negative_test_query304_00005", + "negative_test_query290_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "apositive_test_query321_00000", + "negative_test_query321_00010", + "negative_test_query321_00000", + "negative_test_query321_00002", + "negative_test_query34_00012", + "apositive_test_query185_00009", + "negative_test_query103_00011", + "negative_test_query126_00005", + "negative_test_query141_00009", + "negative_test_query136_00004", + "negative_test_query86_00014", + "negative_test_query352_00005", + "apositive_test_query217_00006", + "negative_test_query197_00004", + "negative_test_query226_00002", + "negative_test_query15_00013", + "negative_test_query126_00011", + "negative_test_query322_00002", + "negative_test_query210_00016", + "negative_test_query103_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query322", + "ranked_ids": [ + "apositive_test_query322_00000", + "apositive_test_query322_00005", + "apositive_test_query322_00001", + "negative_test_query255_00010", + "apositive_test_query322_00002", + "apositive_test_query322_00006", + "negative_test_query322_00006", + "negative_test_query322_00003", + "negative_test_query318_00014", + "negative_test_query322_00010", + "negative_test_query322_00002", + "negative_test_query318_00002", + "negative_test_query73_00010", + "negative_test_query322_00005", + "apositive_test_query322_00004", + "apositive_test_query228_00001", + "apositive_test_query322_00003", + "negative_test_query322_00009", + "apositive_test_query254_00006", + "negative_test_query73_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query323", + "ranked_ids": [ + "apositive_test_query323_00001", + "negative_test_query323_00003", + "apositive_test_query323_00000", + "negative_test_query323_00012", + "apositive_test_query323_00006", + "apositive_test_query323_00002", + "apositive_test_query323_00004", + "apositive_test_query130_00001", + "negative_test_query323_00007", + "negative_test_query323_00001", + "negative_test_query75_00002", + "negative_test_query209_00004", + "negative_test_query255_00002", + "negative_test_query201_00009", + "negative_test_query75_00007", + "negative_test_query323_00000", + "negative_test_query345_00011", + "negative_test_query15_00006", + "negative_test_query75_00004", + "negative_test_query323_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.7328286204777911, + "recall@5": 1.0 + }, + "query_id": "test_query324", + "ranked_ids": [ + "negative_test_query308_00003", + "apositive_test_query324_00002", + "apositive_test_query324_00000", + "apositive_test_query324_00001", + "negative_test_query324_00005", + "negative_test_query324_00004", + "negative_test_query324_00003", + "apositive_test_query308_00001", + "negative_test_query324_00015", + "negative_test_query324_00006", + "negative_test_query324_00016", + "negative_test_query308_00007", + "apositive_test_query308_00003", + "negative_test_query324_00014", + "apositive_test_query308_00002", + "negative_test_query218_00003", + "negative_test_query308_00009", + "apositive_test_query308_00009", + "negative_test_query324_00007", + "apositive_test_query308_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query15_00015", + "negative_test_query325_00007", + "negative_test_query15_00017", + "negative_test_query325_00002", + "negative_test_query293_00015", + "negative_test_query15_00005", + "negative_test_query15_00016", + "negative_test_query325_00013", + "apositive_test_query19_00002", + "apositive_test_query293_00002", + "negative_test_query176_00003", + "apositive_test_query178_00003", + "apositive_test_query246_00001", + "negative_test_query293_00000", + "negative_test_query266_00014", + "negative_test_query19_00008", + "negative_test_query191_00011", + "negative_test_query297_00008", + "negative_test_query325_00008", + "negative_test_query169_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query326", + "ranked_ids": [ + "negative_test_query178_00012", + "negative_test_query107_00011", + "negative_test_query17_00011", + "negative_test_query156_00013", + "apositive_test_query326_00000", + "apositive_test_query326_00003", + "apositive_test_query52_00002", + "apositive_test_query54_00002", + "apositive_test_query326_00001", + "apositive_test_query212_00004", + "negative_test_query265_00007", + "apositive_test_query326_00004", + "apositive_test_query355_00002", + "apositive_test_query23_00000", + "negative_test_query183_00001", + "negative_test_query81_00003", + "apositive_test_query294_00001", + "negative_test_query181_00007", + "negative_test_query265_00005", + "negative_test_query45_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query327", + "ranked_ids": [ + "apositive_test_query327_00001", + "negative_test_query327_00014", + "apositive_test_query327_00000", + "negative_test_query321_00014", + "negative_test_query61_00003", + "negative_test_query327_00000", + "negative_test_query327_00003", + "negative_test_query195_00007", + "negative_test_query61_00006", + "negative_test_query327_00013", + "negative_test_query303_00010", + "apositive_test_query327_00002", + "negative_test_query327_00010", + "negative_test_query327_00012", + "negative_test_query271_00007", + "negative_test_query227_00009", + "apositive_test_query271_00000", + "negative_test_query299_00010", + "apositive_test_query271_00001", + "negative_test_query335_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query328", + "ranked_ids": [ + "negative_test_query328_00003", + "negative_test_query328_00000", + "apositive_test_query328_00002", + "negative_test_query24_00006", + "negative_test_query328_00006", + "negative_test_query92_00009", + "negative_test_query328_00007", + "negative_test_query217_00005", + "apositive_test_query328_00007", + "apositive_test_query40_00003", + "negative_test_query138_00013", + "negative_test_query144_00005", + "negative_test_query59_00017", + "negative_test_query222_00016", + "negative_test_query261_00009", + "apositive_test_query328_00001", + "negative_test_query129_00002", + "negative_test_query328_00002", + "apositive_test_query328_00004", + "apositive_test_query227_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query329", + "ranked_ids": [ + "negative_test_query329_00006", + "negative_test_query329_00009", + "apositive_test_query329_00001", + "negative_test_query269_00000", + "negative_test_query269_00012", + "negative_test_query329_00004", + "negative_test_query329_00001", + "negative_test_query329_00010", + "negative_test_query62_00017", + "apositive_test_query329_00002", + "negative_test_query329_00011", + "negative_test_query329_00003", + "negative_test_query329_00002", + "negative_test_query298_00015", + "negative_test_query329_00013", + "negative_test_query35_00000", + "apositive_test_query7_00001", + "negative_test_query183_00006", + "negative_test_query38_00009", + "negative_test_query202_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00004", + "apositive_test_query33_00006", + "apositive_test_query33_00004", + "apositive_test_query20_00010", + "apositive_test_query154_00007", + "apositive_test_query33_00009", + "negative_test_query33_00006", + "negative_test_query154_00006", + "apositive_test_query154_00002", + "negative_test_query33_00007", + "negative_test_query344_00010", + "apositive_test_query33_00003", + "apositive_test_query284_00004", + "negative_test_query356_00006", + "apositive_test_query220_00003", + "negative_test_query216_00004", + "negative_test_query220_00010", + "negative_test_query216_00014", + "apositive_test_query220_00000", + "negative_test_query161_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query330", + "ranked_ids": [ + "apositive_test_query330_00002", + "negative_test_query330_00010", + "negative_test_query252_00007", + "negative_test_query330_00009", + "negative_test_query330_00008", + "negative_test_query27_00013", + "negative_test_query27_00018", + "negative_test_query330_00001", + "negative_test_query288_00006", + "apositive_test_query60_00014", + "negative_test_query311_00009", + "negative_test_query330_00000", + "negative_test_query330_00012", + "apositive_test_query281_00012", + "negative_test_query325_00016", + "apositive_test_query284_00003", + "apositive_test_query288_00002", + "negative_test_query129_00003", + "negative_test_query180_00000", + "negative_test_query27_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.7122630665145961, + "recall@5": 1.0 + }, + "query_id": "test_query331", + "ranked_ids": [ + "negative_test_query331_00013", + "apositive_test_query331_00000", + "apositive_test_query331_00001", + "negative_test_query250_00016", + "apositive_test_query331_00002", + "negative_test_query202_00001", + "negative_test_query303_00001", + "apositive_test_query223_00001", + "negative_test_query331_00000", + "negative_test_query331_00012", + "negative_test_query113_00000", + "negative_test_query312_00012", + "negative_test_query154_00000", + "negative_test_query331_00008", + "negative_test_query110_00007", + "negative_test_query214_00015", + "negative_test_query334_00014", + "negative_test_query251_00009", + "negative_test_query3_00008", + "negative_test_query331_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00001", + "apositive_test_query332_00003", + "negative_test_query202_00010", + "apositive_test_query142_00004", + "negative_test_query12_00005", + "negative_test_query171_00008", + "apositive_test_query203_00002", + "negative_test_query30_00010", + "negative_test_query65_00007", + "negative_test_query180_00011", + "negative_test_query245_00001", + "apositive_test_query348_00008", + "negative_test_query115_00008", + "apositive_test_query162_00005", + "negative_test_query332_00009", + "negative_test_query244_00006", + "negative_test_query202_00002", + "apositive_test_query332_00002", + "negative_test_query145_00010", + "negative_test_query332_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00002", + "negative_test_query333_00008", + "apositive_test_query322_00003", + "negative_test_query216_00013", + "negative_test_query322_00009", + "negative_test_query333_00005", + "negative_test_query73_00008", + "negative_test_query246_00000", + "negative_test_query318_00005", + "negative_test_query67_00000", + "negative_test_query73_00005", + "apositive_test_query322_00000", + "negative_test_query73_00014", + "apositive_test_query333_00000", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00005", + "negative_test_query136_00003", + "negative_test_query318_00010", + "apositive_test_query228_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query37_00000", + "negative_test_query334_00010", + "negative_test_query155_00010", + "apositive_test_query276_00009", + "negative_test_query334_00012", + "negative_test_query111_00017", + "negative_test_query213_00009", + "negative_test_query107_00001", + "negative_test_query155_00013", + "negative_test_query111_00005", + "apositive_test_query57_00006", + "negative_test_query111_00006", + "negative_test_query111_00010", + "negative_test_query170_00004", + "apositive_test_query26_00001", + "negative_test_query209_00003", + "negative_test_query164_00005", + "negative_test_query102_00015", + "negative_test_query334_00016", + "negative_test_query223_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query335", + "ranked_ids": [ + "apositive_test_query335_00000", + "apositive_test_query335_00003", + "negative_test_query335_00006", + "negative_test_query335_00013", + "negative_test_query335_00000", + "apositive_test_query335_00002", + "apositive_test_query335_00001", + "negative_test_query335_00010", + "negative_test_query335_00004", + "negative_test_query335_00007", + "apositive_test_query335_00004", + "negative_test_query335_00003", + "negative_test_query335_00009", + "negative_test_query335_00001", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query92_00002", + "negative_test_query45_00010", + "negative_test_query335_00011", + "negative_test_query210_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "negative_test_query336_00009", + "apositive_test_query50_00005", + "negative_test_query336_00004", + "negative_test_query336_00007", + "apositive_test_query50_00015", + "apositive_test_query186_00002", + "negative_test_query247_00006", + "negative_test_query108_00005", + "apositive_test_query50_00012", + "apositive_test_query38_00005", + "negative_test_query336_00014", + "negative_test_query247_00000", + "negative_test_query247_00008", + "apositive_test_query242_00012", + "negative_test_query247_00003", + "negative_test_query336_00015", + "negative_test_query105_00011", + "negative_test_query186_00012", + "apositive_test_query186_00000", + "apositive_test_query113_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00010", + "apositive_test_query337_00000", + "negative_test_query200_00007", + "negative_test_query200_00001", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query337_00007", + "apositive_test_query38_00003", + "negative_test_query200_00005", + "negative_test_query337_00005", + "negative_test_query200_00000", + "negative_test_query200_00009", + "negative_test_query200_00015", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query337_00000", + "negative_test_query159_00013", + "negative_test_query337_00004", + "negative_test_query337_00001", + "negative_test_query38_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query338", + "ranked_ids": [ + "apositive_test_query338_00002", + "apositive_test_query338_00008", + "apositive_test_query338_00003", + "apositive_test_query338_00005", + "apositive_test_query338_00001", + "negative_test_query114_00004", + "negative_test_query114_00007", + "apositive_test_query338_00006", + "negative_test_query81_00004", + "negative_test_query95_00004", + "apositive_test_query338_00000", + "negative_test_query119_00003", + "negative_test_query338_00003", + "apositive_test_query81_00000", + "negative_test_query76_00005", + "negative_test_query81_00006", + "apositive_test_query338_00004", + "apositive_test_query114_00005", + "negative_test_query53_00008", + "apositive_test_query114_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query339_00001", + "negative_test_query339_00008", + "apositive_test_query339_00000", + "negative_test_query339_00012", + "negative_test_query339_00009", + "apositive_test_query96_00001", + "apositive_test_query96_00002", + "negative_test_query192_00011", + "negative_test_query339_00001", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query339_00016", + "negative_test_query96_00004", + "negative_test_query96_00016", + "negative_test_query96_00003", + "negative_test_query339_00004", + "negative_test_query339_00017", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query339_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00000", + "apositive_test_query352_00002", + "apositive_test_query34_00004", + "negative_test_query34_00002", + "negative_test_query352_00004", + "negative_test_query34_00008", + "apositive_test_query352_00003", + "negative_test_query352_00010", + "apositive_test_query352_00001", + "apositive_test_query352_00004", + "negative_test_query34_00013", + "negative_test_query34_00000", + "negative_test_query352_00000", + "apositive_test_query34_00000", + "apositive_test_query34_00001", + "negative_test_query352_00008", + "negative_test_query352_00013", + "negative_test_query34_00009", + "negative_test_query34_00014", + "negative_test_query352_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00001", + "apositive_test_query340_00004", + "negative_test_query340_00014", + "negative_test_query340_00008", + "negative_test_query340_00007", + "apositive_test_query340_00000", + "negative_test_query340_00012", + "negative_test_query340_00011", + "negative_test_query88_00013", + "negative_test_query88_00006", + "apositive_test_query340_00002", + "negative_test_query88_00007", + "apositive_test_query174_00002", + "negative_test_query340_00001", + "negative_test_query88_00000", + "negative_test_query340_00006", + "apositive_test_query340_00003", + "apositive_test_query174_00012", + "negative_test_query23_00012", + "negative_test_query88_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query96_00005", + "negative_test_query196_00009", + "apositive_test_query341_00003", + "negative_test_query214_00005", + "negative_test_query341_00003", + "negative_test_query316_00008", + "apositive_test_query341_00005", + "negative_test_query32_00003", + "negative_test_query341_00009", + "apositive_test_query341_00002", + "apositive_test_query341_00001", + "negative_test_query73_00014", + "negative_test_query341_00000", + "apositive_test_query32_00007", + "negative_test_query341_00002", + "negative_test_query96_00013", + "negative_test_query96_00007", + "negative_test_query101_00003", + "negative_test_query196_00006", + "negative_test_query341_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query342_00009", + "negative_test_query212_00009", + "negative_test_query200_00009", + "negative_test_query342_00015", + "negative_test_query68_00005", + "negative_test_query134_00013", + "apositive_test_query342_00000", + "negative_test_query342_00016", + "negative_test_query49_00002", + "negative_test_query98_00017", + "negative_test_query84_00007", + "negative_test_query136_00003", + "negative_test_query78_00015", + "negative_test_query77_00004", + "negative_test_query245_00001", + "negative_test_query98_00011", + "negative_test_query324_00015", + "negative_test_query325_00015", + "negative_test_query342_00003", + "negative_test_query77_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query343", + "ranked_ids": [ + "negative_test_query103_00010", + "apositive_test_query343_00000", + "apositive_test_query279_00003", + "negative_test_query307_00006", + "negative_test_query225_00008", + "apositive_test_query307_00008", + "apositive_test_query136_00004", + "apositive_test_query133_00005", + "apositive_test_query287_00003", + "apositive_test_query343_00001", + "negative_test_query360_00017", + "apositive_test_query185_00001", + "negative_test_query103_00008", + "negative_test_query343_00008", + "apositive_test_query185_00008", + "apositive_test_query279_00002", + "negative_test_query227_00010", + "apositive_test_query217_00002", + "apositive_test_query217_00004", + "apositive_test_query28_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query282_00008", + "negative_test_query83_00005", + "negative_test_query344_00009", + "negative_test_query85_00010", + "negative_test_query282_00004", + "negative_test_query282_00001", + "negative_test_query344_00002", + "negative_test_query344_00004", + "apositive_test_query221_00002", + "apositive_test_query83_00006", + "negative_test_query344_00012", + "negative_test_query223_00016", + "negative_test_query344_00005", + "negative_test_query345_00012", + "negative_test_query83_00008", + "negative_test_query231_00015", + "negative_test_query344_00006", + "negative_test_query188_00000", + "negative_test_query282_00000", + "negative_test_query344_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query345_00003", + "negative_test_query47_00008", + "negative_test_query90_00003", + "negative_test_query9_00002", + "apositive_test_query345_00004", + "negative_test_query184_00009", + "negative_test_query345_00011", + "apositive_test_query0_00008", + "negative_test_query345_00001", + "negative_test_query345_00007", + "negative_test_query345_00010", + "apositive_test_query9_00000", + "negative_test_query130_00004", + "negative_test_query221_00002", + "negative_test_query259_00007", + "negative_test_query47_00003", + "apositive_test_query345_00000", + "apositive_test_query203_00005", + "apositive_test_query345_00001", + "apositive_test_query90_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query346_00009", + "negative_test_query346_00001", + "negative_test_query346_00002", + "negative_test_query157_00005", + "apositive_test_query346_00001", + "negative_test_query49_00007", + "negative_test_query346_00010", + "negative_test_query346_00012", + "apositive_test_query346_00000", + "negative_test_query346_00014", + "negative_test_query346_00003", + "negative_test_query346_00017", + "negative_test_query346_00008", + "negative_test_query346_00006", + "negative_test_query174_00001", + "negative_test_query19_00012", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "negative_test_query15_00000", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query347", + "ranked_ids": [ + "apositive_test_query347_00000", + "negative_test_query298_00003", + "negative_test_query347_00000", + "negative_test_query347_00001", + "negative_test_query347_00007", + "negative_test_query31_00003", + "negative_test_query347_00002", + "negative_test_query347_00005", + "negative_test_query347_00008", + "negative_test_query347_00014", + "negative_test_query298_00006", + "negative_test_query347_00006", + "negative_test_query347_00009", + "negative_test_query347_00010", + "negative_test_query347_00012", + "negative_test_query347_00016", + "apositive_test_query347_00001", + "negative_test_query251_00007", + "negative_test_query347_00011", + "negative_test_query347_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query348_00005", + "apositive_test_query348_00001", + "apositive_test_query348_00006", + "apositive_test_query348_00000", + "apositive_test_query348_00004", + "apositive_test_query348_00009", + "negative_test_query348_00008", + "negative_test_query348_00006", + "apositive_test_query115_00002", + "apositive_test_query227_00002", + "apositive_test_query115_00004", + "negative_test_query115_00002", + "negative_test_query115_00006", + "negative_test_query359_00014", + "negative_test_query297_00017", + "apositive_test_query115_00001", + "negative_test_query341_00008", + "negative_test_query115_00012", + "negative_test_query351_00010", + "negative_test_query31_00017" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "negative_test_query253_00007", + "negative_test_query253_00000", + "negative_test_query312_00016", + "negative_test_query349_00015", + "apositive_test_query253_00005", + "negative_test_query74_00001", + "negative_test_query349_00003", + "apositive_test_query74_00003", + "negative_test_query253_00009", + "apositive_test_query253_00009", + "negative_test_query124_00013", + "apositive_test_query355_00007", + "apositive_test_query253_00008", + "apositive_test_query74_00002", + "negative_test_query13_00002", + "negative_test_query157_00009", + "apositive_test_query137_00002", + "negative_test_query194_00001", + "negative_test_query349_00012", + "apositive_test_query42_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6096199500078984, + "recall@5": 0.75 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query139_00000", + "apositive_test_query35_00000", + "apositive_test_query35_00001", + "apositive_test_query35_00003", + "negative_test_query205_00005", + "apositive_test_query146_00007", + "negative_test_query198_00011", + "negative_test_query124_00007", + "negative_test_query8_00009", + "negative_test_query139_00016", + "negative_test_query350_00005", + "negative_test_query163_00000", + "negative_test_query298_00008", + "negative_test_query35_00002", + "negative_test_query35_00012", + "negative_test_query101_00006", + "negative_test_query123_00006", + "negative_test_query231_00012", + "negative_test_query269_00006", + "negative_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query350_00015", + "negative_test_query350_00014", + "negative_test_query350_00010", + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query14_00001", + "negative_test_query149_00005", + "negative_test_query179_00001", + "negative_test_query192_00004", + "negative_test_query63_00000", + "negative_test_query331_00012", + "negative_test_query350_00001", + "negative_test_query350_00017", + "negative_test_query350_00006", + "negative_test_query192_00005", + "negative_test_query224_00010", + "negative_test_query350_00011", + "negative_test_query192_00013", + "negative_test_query316_00006", + "apositive_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query351_00000", + "apositive_test_query299_00000", + "negative_test_query156_00006", + "negative_test_query351_00002", + "negative_test_query92_00005", + "apositive_test_query311_00000", + "negative_test_query351_00006", + "negative_test_query351_00014", + "apositive_test_query60_00003", + "apositive_test_query113_00001", + "apositive_test_query335_00004", + "apositive_test_query60_00000", + "apositive_test_query60_00010", + "apositive_test_query218_00004", + "apositive_test_query60_00012", + "negative_test_query234_00009", + "negative_test_query53_00011", + "negative_test_query8_00003", + "apositive_test_query193_00009", + "apositive_test_query60_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.6 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query34_00001", + "apositive_test_query352_00003", + "apositive_test_query352_00000", + "apositive_test_query352_00001", + "negative_test_query34_00002", + "negative_test_query352_00004", + "negative_test_query34_00008", + "negative_test_query352_00013", + "negative_test_query352_00010", + "apositive_test_query34_00004", + "apositive_test_query352_00002", + "apositive_test_query352_00004", + "negative_test_query34_00000", + "negative_test_query352_00000", + "negative_test_query34_00013", + "apositive_test_query34_00002", + "negative_test_query34_00012", + "negative_test_query352_00008", + "apositive_test_query34_00000", + "negative_test_query34_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query204_00000", + "apositive_test_query353_00011", + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00012", + "negative_test_query353_00002", + "apositive_test_query353_00002", + "apositive_test_query204_00002", + "apositive_test_query204_00009", + "apositive_test_query353_00000", + "apositive_test_query353_00005", + "negative_test_query353_00003", + "apositive_test_query204_00007", + "apositive_test_query353_00009", + "apositive_test_query51_00002", + "apositive_test_query204_00015", + "apositive_test_query353_00007", + "negative_test_query175_00012", + "negative_test_query71_00008", + "negative_test_query108_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.625 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "apositive_test_query354_00006", + "apositive_test_query354_00001", + "apositive_test_query354_00004", + "negative_test_query309_00013", + "negative_test_query354_00004", + "negative_test_query194_00002", + "negative_test_query309_00014", + "negative_test_query358_00003", + "apositive_test_query320_00006", + "negative_test_query280_00007", + "negative_test_query331_00014", + "apositive_test_query16_00003", + "apositive_test_query16_00001", + "negative_test_query148_00009", + "apositive_test_query147_00002", + "apositive_test_query354_00005", + "negative_test_query309_00000", + "negative_test_query354_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query355_00001", + "negative_test_query355_00009", + "negative_test_query355_00007", + "apositive_test_query355_00002", + "negative_test_query355_00008", + "negative_test_query355_00005", + "negative_test_query355_00002", + "negative_test_query159_00017", + "negative_test_query264_00017", + "apositive_test_query355_00005", + "negative_test_query355_00003", + "negative_test_query87_00004", + "negative_test_query215_00001", + "apositive_test_query355_00003", + "negative_test_query264_00003", + "negative_test_query68_00000", + "negative_test_query104_00001", + "apositive_test_query355_00006", + "apositive_test_query38_00006", + "apositive_test_query100_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query356_00000", + "apositive_test_query356_00002", + "apositive_test_query356_00004", + "negative_test_query253_00001", + "negative_test_query15_00006", + "negative_test_query102_00014", + "apositive_test_query66_00003", + "apositive_test_query30_00005", + "negative_test_query47_00011", + "negative_test_query102_00010", + "apositive_test_query20_00001", + "negative_test_query280_00004", + "apositive_test_query356_00008", + "apositive_test_query33_00000", + "negative_test_query334_00000", + "apositive_test_query154_00009", + "apositive_test_query127_00007", + "apositive_test_query295_00000", + "apositive_test_query203_00006", + "apositive_test_query88_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00002", + "apositive_test_query221_00000", + "apositive_test_query221_00002", + "apositive_test_query117_00004", + "apositive_test_query119_00000", + "apositive_test_query314_00000", + "negative_test_query300_00007", + "apositive_test_query119_00001", + "negative_test_query58_00009", + "apositive_test_query119_00009", + "apositive_test_query119_00010", + "apositive_test_query221_00004", + "negative_test_query58_00008", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query119_00005", + "apositive_test_query119_00012", + "apositive_test_query357_00004", + "negative_test_query46_00004", + "apositive_test_query357_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.3 + }, + "query_id": "test_query358", + "ranked_ids": [ + "negative_test_query309_00005", + "apositive_test_query358_00006", + "apositive_test_query358_00007", + "negative_test_query123_00007", + "apositive_test_query358_00004", + "negative_test_query11_00000", + "apositive_test_query309_00000", + "negative_test_query309_00006", + "apositive_test_query11_00007", + "apositive_test_query161_00000", + "apositive_test_query358_00003", + "apositive_test_query358_00009", + "apositive_test_query358_00000", + "negative_test_query132_00001", + "apositive_test_query301_00003", + "negative_test_query358_00001", + "negative_test_query279_00010", + "negative_test_query354_00008", + "negative_test_query309_00015", + "negative_test_query42_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00003", + "apositive_test_query0_00006", + "apositive_test_query0_00004", + "negative_test_query359_00002", + "negative_test_query305_00000", + "negative_test_query0_00001", + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "negative_test_query0_00007", + "negative_test_query184_00016", + "negative_test_query268_00014", + "apositive_test_query259_00008", + "negative_test_query359_00007", + "apositive_test_query0_00002", + "negative_test_query259_00006", + "apositive_test_query0_00007", + "apositive_test_query259_00004", + "apositive_test_query359_00002", + "apositive_test_query259_00005", + "negative_test_query130_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query36", + "ranked_ids": [ + "negative_test_query112_00015", + "negative_test_query112_00004", + "apositive_test_query36_00003", + "apositive_test_query286_00006", + "negative_test_query112_00016", + "negative_test_query36_00014", + "negative_test_query112_00010", + "negative_test_query216_00002", + "apositive_test_query286_00005", + "negative_test_query260_00009", + "apositive_test_query106_00003", + "negative_test_query296_00013", + "negative_test_query6_00003", + "negative_test_query36_00010", + "negative_test_query286_00000", + "negative_test_query36_00001", + "negative_test_query260_00010", + "negative_test_query112_00013", + "apositive_test_query136_00013", + "negative_test_query80_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00001", + "apositive_test_query310_00006", + "apositive_test_query360_00000", + "negative_test_query189_00013", + "apositive_test_query152_00002", + "negative_test_query360_00005", + "apositive_test_query142_00004", + "apositive_test_query153_00011", + "apositive_test_query69_00014", + "negative_test_query240_00010", + "negative_test_query148_00018", + "negative_test_query360_00004", + "negative_test_query175_00009", + "negative_test_query312_00004", + "negative_test_query216_00011", + "negative_test_query189_00014", + "negative_test_query281_00000", + "negative_test_query246_00001", + "negative_test_query5_00014", + "negative_test_query360_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query37", + "ranked_ids": [ + "apositive_test_query37_00001", + "negative_test_query123_00001", + "apositive_test_query177_00000", + "negative_test_query209_00001", + "negative_test_query231_00006", + "apositive_test_query158_00002", + "negative_test_query155_00001", + "negative_test_query123_00011", + "negative_test_query131_00011", + "negative_test_query223_00010", + "negative_test_query20_00003", + "apositive_test_query152_00001", + "negative_test_query157_00010", + "negative_test_query152_00008", + "apositive_test_query209_00002", + "negative_test_query57_00009", + "apositive_test_query158_00007", + "apositive_test_query213_00006", + "apositive_test_query108_00000", + "negative_test_query209_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.25 + }, + "query_id": "test_query38", + "ranked_ids": [ + "negative_test_query267_00016", + "apositive_test_query38_00003", + "negative_test_query159_00010", + "negative_test_query38_00010", + "apositive_test_query38_00000", + "negative_test_query267_00015", + "negative_test_query47_00016", + "negative_test_query267_00001", + "negative_test_query337_00000", + "apositive_test_query38_00007", + "apositive_test_query337_00000", + "apositive_test_query38_00002", + "apositive_test_query247_00005", + "negative_test_query175_00005", + "negative_test_query337_00017", + "negative_test_query38_00006", + "negative_test_query159_00011", + "negative_test_query264_00012", + "negative_test_query79_00002", + "negative_test_query94_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query39", + "ranked_ids": [ + "apositive_test_query39_00001", + "apositive_test_query39_00003", + "negative_test_query18_00004", + "negative_test_query39_00000", + "negative_test_query39_00006", + "negative_test_query39_00012", + "apositive_test_query39_00004", + "apositive_test_query39_00002", + "negative_test_query331_00010", + "negative_test_query39_00009", + "negative_test_query39_00003", + "negative_test_query39_00007", + "apositive_test_query39_00000", + "negative_test_query66_00008", + "negative_test_query39_00004", + "negative_test_query232_00005", + "negative_test_query232_00007", + "negative_test_query232_00009", + "negative_test_query232_00017", + "negative_test_query29_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.35714285714285715 + }, + "query_id": "test_query4", + "ranked_ids": [ + "apositive_test_query4_00003", + "apositive_test_query4_00000", + "apositive_test_query4_00010", + "apositive_test_query4_00004", + "apositive_test_query4_00012", + "apositive_test_query4_00001", + "negative_test_query266_00009", + "negative_test_query174_00001", + "negative_test_query4_00002", + "apositive_test_query19_00000", + "negative_test_query312_00011", + "apositive_test_query4_00005", + "negative_test_query4_00001", + "apositive_test_query174_00000", + "apositive_test_query174_00001", + "negative_test_query266_00004", + "negative_test_query149_00015", + "negative_test_query19_00004", + "negative_test_query15_00000", + "negative_test_query266_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00004", + "negative_test_query40_00006", + "negative_test_query40_00010", + "negative_test_query40_00007", + "negative_test_query40_00001", + "negative_test_query40_00005", + "negative_test_query40_00013", + "negative_test_query40_00000", + "negative_test_query40_00003", + "apositive_test_query40_00002", + "negative_test_query66_00005", + "negative_test_query214_00015", + "negative_test_query40_00011", + "negative_test_query347_00003", + "negative_test_query40_00012", + "negative_test_query328_00006", + "negative_test_query106_00009", + "negative_test_query40_00009", + "negative_test_query324_00001", + "apositive_test_query223_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query41", + "ranked_ids": [ + "apositive_test_query41_00002", + "negative_test_query41_00000", + "negative_test_query282_00003", + "negative_test_query41_00004", + "negative_test_query41_00007", + "apositive_test_query41_00000", + "apositive_test_query41_00001", + "negative_test_query41_00002", + "negative_test_query41_00012", + "negative_test_query41_00010", + "negative_test_query41_00006", + "apositive_test_query41_00003", + "apositive_test_query344_00000", + "apositive_test_query249_00012", + "negative_test_query144_00016", + "negative_test_query226_00011", + "negative_test_query41_00011", + "negative_test_query282_00011", + "negative_test_query98_00000", + "negative_test_query41_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query42_00002", + "negative_test_query42_00008", + "apositive_test_query42_00003", + "negative_test_query42_00003", + "negative_test_query42_00000", + "negative_test_query137_00005", + "apositive_test_query42_00001", + "apositive_test_query42_00005", + "negative_test_query42_00001", + "apositive_test_query42_00000", + "apositive_test_query42_00002", + "negative_test_query42_00007", + "negative_test_query42_00005", + "negative_test_query42_00010", + "negative_test_query42_00006", + "apositive_test_query42_00004", + "apositive_test_query276_00012", + "negative_test_query250_00010", + "negative_test_query42_00011", + "apositive_test_query162_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query43_00000", + "apositive_test_query185_00002", + "negative_test_query48_00017", + "apositive_test_query161_00005", + "negative_test_query43_00009", + "apositive_test_query146_00004", + "apositive_test_query43_00002", + "apositive_test_query146_00003", + "apositive_test_query218_00002", + "negative_test_query333_00001", + "apositive_test_query161_00002", + "negative_test_query179_00006", + "negative_test_query43_00000", + "apositive_test_query276_00001", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query153_00008", + "apositive_test_query153_00014", + "negative_test_query319_00002", + "negative_test_query82_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query44", + "ranked_ids": [ + "apositive_test_query44_00001", + "negative_test_query27_00002", + "negative_test_query27_00011", + "negative_test_query330_00009", + "negative_test_query293_00015", + "negative_test_query293_00003", + "negative_test_query293_00009", + "negative_test_query27_00008", + "negative_test_query44_00002", + "apositive_test_query141_00005", + "negative_test_query293_00000", + "apositive_test_query293_00000", + "negative_test_query10_00012", + "apositive_test_query245_00000", + "apositive_test_query20_00003", + "negative_test_query141_00007", + "negative_test_query21_00009", + "negative_test_query330_00012", + "negative_test_query190_00011", + "negative_test_query180_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query45_00013", + "negative_test_query45_00007", + "apositive_test_query25_00000", + "negative_test_query45_00004", + "negative_test_query45_00001", + "negative_test_query182_00016", + "apositive_test_query45_00000", + "negative_test_query328_00005", + "negative_test_query45_00005", + "negative_test_query45_00003", + "negative_test_query45_00006", + "negative_test_query143_00009", + "negative_test_query143_00014", + "negative_test_query342_00005", + "negative_test_query143_00011", + "apositive_test_query25_00005", + "negative_test_query45_00016", + "negative_test_query143_00004", + "apositive_test_query165_00000", + "negative_test_query45_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "negative_test_query56_00001", + "negative_test_query39_00011", + "negative_test_query108_00003", + "negative_test_query46_00001", + "negative_test_query46_00011", + "apositive_test_query142_00001", + "negative_test_query105_00006", + "negative_test_query213_00012", + "negative_test_query238_00001", + "negative_test_query313_00010", + "negative_test_query359_00005", + "negative_test_query46_00003", + "apositive_test_query146_00013", + "negative_test_query168_00003", + "negative_test_query334_00004", + "negative_test_query79_00012", + "apositive_test_query79_00005", + "apositive_test_query276_00008", + "apositive_test_query186_00001", + "negative_test_query46_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query331_00002", + "apositive_test_query47_00001", + "negative_test_query47_00013", + "negative_test_query47_00006", + "negative_test_query144_00018", + "negative_test_query325_00007", + "apositive_test_query201_00002", + "negative_test_query41_00009", + "negative_test_query47_00012", + "negative_test_query195_00005", + "negative_test_query331_00014", + "negative_test_query331_00005", + "negative_test_query47_00002", + "apositive_test_query201_00004", + "apositive_test_query201_00006", + "apositive_test_query20_00005", + "negative_test_query47_00011", + "negative_test_query331_00013", + "negative_test_query345_00002", + "negative_test_query328_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "apositive_test_query185_00012", + "negative_test_query307_00005", + "apositive_test_query28_00013", + "negative_test_query48_00016", + "negative_test_query48_00009", + "apositive_test_query48_00000", + "apositive_test_query28_00010", + "negative_test_query48_00012", + "negative_test_query48_00011", + "apositive_test_query185_00004", + "negative_test_query103_00012", + "negative_test_query48_00018", + "negative_test_query48_00007", + "negative_test_query307_00003", + "negative_test_query48_00004", + "negative_test_query307_00001", + "apositive_test_query343_00001", + "negative_test_query49_00001", + "negative_test_query48_00000", + "apositive_test_query307_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query49", + "ranked_ids": [ + "apositive_test_query49_00001", + "negative_test_query176_00016", + "negative_test_query342_00005", + "apositive_test_query29_00006", + "negative_test_query169_00000", + "negative_test_query49_00001", + "negative_test_query49_00002", + "negative_test_query328_00009", + "negative_test_query49_00007", + "apositive_test_query19_00001", + "negative_test_query15_00008", + "apositive_test_query54_00009", + "negative_test_query52_00011", + "negative_test_query156_00007", + "negative_test_query72_00017", + "negative_test_query106_00004", + "negative_test_query156_00010", + "apositive_test_query204_00003", + "apositive_test_query353_00004", + "negative_test_query122_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8529278650606567, + "recall@5": 1.0 + }, + "query_id": "test_query5", + "ranked_ids": [ + "apositive_test_query5_00000", + "negative_test_query246_00013", + "negative_test_query5_00005", + "apositive_test_query5_00001", + "apositive_test_query5_00002", + "negative_test_query67_00010", + "negative_test_query246_00002", + "negative_test_query5_00006", + "negative_test_query275_00001", + "negative_test_query5_00004", + "negative_test_query5_00003", + "negative_test_query5_00007", + "negative_test_query5_00000", + "negative_test_query246_00008", + "negative_test_query5_00001", + "negative_test_query5_00013", + "negative_test_query67_00003", + "negative_test_query5_00010", + "negative_test_query5_00014", + "negative_test_query301_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00014", + "apositive_test_query50_00003", + "apositive_test_query186_00005", + "apositive_test_query50_00009", + "negative_test_query336_00011", + "apositive_test_query50_00013", + "apositive_test_query50_00008", + "apositive_test_query50_00005", + "apositive_test_query50_00006", + "apositive_test_query50_00015", + "apositive_test_query50_00001", + "apositive_test_query50_00002", + "negative_test_query50_00001", + "apositive_test_query131_00000", + "negative_test_query336_00004", + "apositive_test_query50_00012", + "negative_test_query66_00002", + "negative_test_query131_00001", + "negative_test_query119_00002", + "negative_test_query223_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.3 + }, + "query_id": "test_query51", + "ranked_ids": [ + "apositive_test_query51_00001", + "negative_test_query51_00006", + "apositive_test_query51_00000", + "apositive_test_query51_00006", + "negative_test_query97_00011", + "negative_test_query97_00002", + "negative_test_query51_00000", + "negative_test_query97_00001", + "apositive_test_query51_00007", + "apositive_test_query51_00008", + "negative_test_query97_00007", + "apositive_test_query51_00005", + "negative_test_query97_00006", + "negative_test_query267_00007", + "apositive_test_query51_00009", + "negative_test_query51_00004", + "negative_test_query204_00001", + "negative_test_query209_00008", + "negative_test_query231_00012", + "apositive_test_query267_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query257_00005", + "negative_test_query215_00000", + "apositive_test_query286_00014", + "negative_test_query52_00007", + "negative_test_query257_00009", + "negative_test_query54_00004", + "apositive_test_query286_00013", + "apositive_test_query286_00006", + "apositive_test_query257_00004", + "negative_test_query257_00003", + "apositive_test_query257_00001", + "apositive_test_query257_00003", + "apositive_test_query257_00000", + "negative_test_query52_00005", + "apositive_test_query257_00002", + "apositive_test_query69_00007", + "apositive_test_query257_00005", + "apositive_test_query52_00001", + "negative_test_query257_00007", + "negative_test_query52_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00016", + "negative_test_query161_00007", + "negative_test_query354_00001", + "negative_test_query150_00015", + "negative_test_query354_00011", + "negative_test_query258_00000", + "negative_test_query258_00013", + "negative_test_query53_00000", + "negative_test_query3_00008", + "apositive_test_query217_00004", + "negative_test_query169_00003", + "negative_test_query258_00018", + "apositive_test_query28_00006", + "negative_test_query348_00001", + "negative_test_query230_00009", + "negative_test_query150_00014", + "negative_test_query258_00008", + "apositive_test_query248_00005", + "negative_test_query132_00015", + "negative_test_query180_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query54_00006", + "negative_test_query140_00006", + "negative_test_query44_00014", + "negative_test_query126_00008", + "negative_test_query151_00008", + "negative_test_query56_00011", + "negative_test_query51_00006", + "apositive_test_query86_00000", + "negative_test_query44_00007", + "negative_test_query200_00006", + "negative_test_query97_00002", + "negative_test_query301_00006", + "apositive_test_query323_00006", + "apositive_test_query335_00002", + "apositive_test_query51_00007", + "negative_test_query136_00004", + "apositive_test_query156_00000", + "apositive_test_query204_00002", + "apositive_test_query51_00006", + "apositive_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query55", + "ranked_ids": [ + "apositive_test_query55_00000", + "negative_test_query55_00002", + "negative_test_query55_00005", + "apositive_test_query55_00001", + "negative_test_query55_00016", + "negative_test_query55_00013", + "negative_test_query55_00008", + "negative_test_query55_00007", + "apositive_test_query55_00002", + "negative_test_query139_00015", + "negative_test_query55_00015", + "negative_test_query55_00010", + "negative_test_query55_00012", + "negative_test_query61_00008", + "negative_test_query55_00000", + "negative_test_query55_00011", + "negative_test_query55_00003", + "negative_test_query234_00011", + "negative_test_query55_00006", + "negative_test_query55_00009" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query56", + "ranked_ids": [ + "negative_test_query56_00003", + "negative_test_query105_00006", + "negative_test_query359_00005", + "apositive_test_query56_00005", + "negative_test_query119_00005", + "negative_test_query56_00005", + "negative_test_query111_00006", + "apositive_test_query56_00006", + "negative_test_query46_00011", + "negative_test_query56_00002", + "negative_test_query46_00014", + "apositive_test_query238_00001", + "negative_test_query186_00007", + "negative_test_query56_00000", + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query56_00004", + "negative_test_query56_00006", + "apositive_test_query46_00001", + "negative_test_query102_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query57", + "ranked_ids": [ + "apositive_test_query57_00003", + "negative_test_query295_00003", + "apositive_test_query57_00005", + "negative_test_query46_00006", + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query57_00006", + "apositive_test_query57_00000", + "apositive_test_query57_00008", + "negative_test_query57_00001", + "apositive_test_query203_00000", + "negative_test_query198_00016", + "negative_test_query65_00010", + "apositive_test_query259_00006", + "apositive_test_query338_00002", + "negative_test_query334_00001", + "negative_test_query57_00010", + "apositive_test_query146_00000", + "negative_test_query111_00010", + "apositive_test_query74_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "negative_test_query58_00010", + "negative_test_query240_00016", + "negative_test_query58_00015", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001", + "negative_test_query308_00004", + "negative_test_query346_00000", + "negative_test_query240_00002", + "negative_test_query11_00004", + "negative_test_query240_00008", + "negative_test_query27_00017", + "negative_test_query27_00016", + "negative_test_query311_00012", + "negative_test_query240_00005", + "negative_test_query240_00004", + "negative_test_query57_00004", + "apositive_test_query311_00003", + "apositive_test_query311_00004", + "negative_test_query240_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "apositive_test_query59_00000", + "negative_test_query59_00012", + "negative_test_query59_00010", + "negative_test_query116_00000", + "negative_test_query59_00006", + "negative_test_query59_00011", + "negative_test_query59_00014", + "negative_test_query59_00009", + "negative_test_query59_00018", + "negative_test_query227_00008", + "negative_test_query59_00017", + "apositive_test_query104_00000", + "negative_test_query116_00013", + "negative_test_query59_00015", + "negative_test_query116_00003", + "negative_test_query66_00008", + "negative_test_query59_00005", + "negative_test_query81_00003", + "negative_test_query280_00003", + "negative_test_query202_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query6", + "ranked_ids": [ + "apositive_test_query6_00002", + "apositive_test_query6_00000", + "apositive_test_query6_00001", + "negative_test_query6_00002", + "negative_test_query6_00004", + "negative_test_query6_00012", + "negative_test_query179_00007", + "apositive_test_query109_00001", + "apositive_test_query305_00003", + "apositive_test_query242_00004", + "negative_test_query296_00013", + "apositive_test_query189_00000", + "negative_test_query252_00004", + "negative_test_query6_00001", + "negative_test_query6_00011", + "negative_test_query112_00010", + "negative_test_query58_00007", + "negative_test_query6_00016", + "negative_test_query296_00006", + "negative_test_query112_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.25 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00007", + "apositive_test_query60_00003", + "apositive_test_query60_00001", + "negative_test_query243_00004", + "apositive_test_query60_00013", + "apositive_test_query60_00000", + "apositive_test_query285_00001", + "apositive_test_query218_00004", + "apositive_test_query60_00009", + "apositive_test_query285_00007", + "negative_test_query304_00002", + "negative_test_query304_00003", + "negative_test_query60_00001", + "apositive_test_query128_00001", + "negative_test_query60_00003", + "apositive_test_query60_00006", + "apositive_test_query60_00005", + "negative_test_query218_00007", + "apositive_test_query60_00011", + "negative_test_query31_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7365896932159578, + "recall@5": 0.75 + }, + "query_id": "test_query61", + "ranked_ids": [ + "apositive_test_query61_00002", + "negative_test_query61_00014", + "apositive_test_query61_00003", + "negative_test_query173_00003", + "apositive_test_query61_00001", + "negative_test_query61_00006", + "negative_test_query129_00005", + "negative_test_query61_00005", + "negative_test_query61_00003", + "negative_test_query61_00015", + "negative_test_query201_00004", + "negative_test_query61_00000", + "apositive_test_query101_00002", + "negative_test_query211_00005", + "negative_test_query210_00012", + "negative_test_query61_00010", + "negative_test_query61_00004", + "apositive_test_query173_00004", + "negative_test_query327_00016", + "negative_test_query61_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00004", + "negative_test_query62_00000", + "negative_test_query62_00002", + "negative_test_query62_00010", + "negative_test_query62_00003", + "negative_test_query62_00001", + "negative_test_query62_00013", + "negative_test_query62_00008", + "negative_test_query62_00011", + "negative_test_query62_00006", + "negative_test_query352_00005", + "negative_test_query15_00011", + "negative_test_query16_00001", + "negative_test_query219_00006", + "negative_test_query287_00007", + "apositive_test_query62_00001", + "negative_test_query88_00015", + "negative_test_query250_00003", + "negative_test_query38_00009", + "negative_test_query222_00007" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00001", + "negative_test_query224_00007", + "negative_test_query63_00005", + "negative_test_query63_00003", + "negative_test_query63_00001", + "negative_test_query297_00006", + "negative_test_query63_00002", + "negative_test_query321_00002", + "apositive_test_query63_00004", + "negative_test_query63_00010", + "negative_test_query78_00006", + "negative_test_query331_00010", + "negative_test_query264_00015", + "negative_test_query63_00004", + "apositive_test_query63_00008", + "apositive_test_query288_00004", + "negative_test_query241_00011", + "negative_test_query63_00009", + "apositive_test_query63_00006", + "apositive_test_query141_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query64_00015", + "negative_test_query205_00009", + "negative_test_query64_00013", + "negative_test_query211_00009", + "negative_test_query149_00017", + "negative_test_query120_00003", + "negative_test_query64_00010", + "apositive_test_query34_00003", + "negative_test_query352_00001", + "negative_test_query120_00000", + "negative_test_query63_00009", + "apositive_test_query160_00003", + "apositive_test_query160_00000", + "negative_test_query310_00005", + "negative_test_query64_00016", + "negative_test_query149_00001", + "negative_test_query64_00009", + "negative_test_query234_00010", + "negative_test_query63_00000", + "negative_test_query214_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "apositive_test_query65_00001", + "negative_test_query65_00004", + "apositive_test_query65_00000", + "apositive_test_query134_00001", + "negative_test_query30_00011", + "apositive_test_query260_00000", + "negative_test_query260_00013", + "negative_test_query6_00008", + "apositive_test_query62_00000", + "apositive_test_query356_00004", + "negative_test_query230_00001", + "negative_test_query47_00005", + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query23_00006", + "negative_test_query261_00003", + "negative_test_query6_00003", + "negative_test_query260_00012", + "negative_test_query268_00012", + "negative_test_query65_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.5 + }, + "query_id": "test_query66", + "ranked_ids": [ + "apositive_test_query66_00004", + "negative_test_query66_00011", + "apositive_test_query66_00005", + "apositive_test_query66_00003", + "apositive_test_query20_00005", + "apositive_test_query66_00002", + "negative_test_query220_00001", + "apositive_test_query20_00001", + "negative_test_query220_00010", + "apositive_test_query220_00003", + "negative_test_query161_00008", + "negative_test_query154_00009", + "negative_test_query188_00014", + "negative_test_query161_00003", + "negative_test_query9_00012", + "apositive_test_query188_00000", + "negative_test_query344_00007", + "apositive_test_query188_00003", + "apositive_test_query220_00000", + "negative_test_query188_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query67", + "ranked_ids": [ + "negative_test_query67_00005", + "negative_test_query67_00010", + "negative_test_query67_00001", + "negative_test_query67_00008", + "negative_test_query246_00016", + "negative_test_query67_00007", + "apositive_test_query246_00000", + "apositive_test_query67_00001", + "apositive_test_query286_00010", + "negative_test_query246_00009", + "negative_test_query67_00015", + "negative_test_query123_00008", + "negative_test_query67_00011", + "negative_test_query260_00013", + "negative_test_query6_00008", + "apositive_test_query67_00000", + "negative_test_query65_00008", + "apositive_test_query286_00004", + "apositive_test_query286_00007", + "negative_test_query55_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query68", + "ranked_ids": [ + "apositive_test_query68_00002", + "negative_test_query68_00001", + "negative_test_query68_00013", + "negative_test_query126_00008", + "negative_test_query151_00008", + "apositive_test_query68_00000", + "negative_test_query330_00009", + "negative_test_query68_00014", + "apositive_test_query252_00001", + "negative_test_query68_00016", + "negative_test_query68_00007", + "apositive_test_query68_00001", + "negative_test_query212_00000", + "apositive_test_query330_00001", + "apositive_test_query5_00002", + "negative_test_query311_00005", + "apositive_test_query326_00005", + "apositive_test_query89_00003", + "negative_test_query288_00004", + "negative_test_query330_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00002", + "apositive_test_query257_00003", + "apositive_test_query69_00011", + "apositive_test_query69_00000", + "apositive_test_query69_00013", + "negative_test_query169_00002", + "apositive_test_query355_00001", + "apositive_test_query69_00009", + "apositive_test_query69_00008", + "apositive_test_query69_00005", + "negative_test_query18_00010", + "apositive_test_query69_00010", + "apositive_test_query174_00016", + "apositive_test_query69_00012", + "apositive_test_query281_00016", + "apositive_test_query52_00001", + "apositive_test_query107_00007", + "negative_test_query277_00001", + "apositive_test_query54_00003", + "apositive_test_query225_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "apositive_test_query212_00002", + "negative_test_query7_00005", + "apositive_test_query212_00001", + "negative_test_query159_00009", + "apositive_test_query119_00006", + "apositive_test_query326_00007", + "negative_test_query306_00008", + "negative_test_query84_00007", + "negative_test_query7_00001", + "apositive_test_query159_00001", + "negative_test_query264_00013", + "negative_test_query38_00010", + "negative_test_query79_00002", + "apositive_test_query38_00003", + "negative_test_query327_00011", + "negative_test_query184_00006", + "negative_test_query290_00006", + "negative_test_query231_00011", + "negative_test_query159_00004", + "negative_test_query159_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00004", + "negative_test_query70_00011", + "negative_test_query227_00013", + "negative_test_query227_00006", + "negative_test_query70_00006", + "apositive_test_query70_00001", + "negative_test_query70_00003", + "apositive_test_query70_00002", + "negative_test_query227_00003", + "negative_test_query227_00000", + "negative_test_query348_00007", + "negative_test_query227_00012", + "negative_test_query227_00010", + "negative_test_query227_00014", + "negative_test_query70_00000", + "negative_test_query70_00012", + "apositive_test_query170_00001", + "apositive_test_query227_00000", + "negative_test_query283_00014", + "negative_test_query59_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query71", + "ranked_ids": [ + "apositive_test_query71_00003", + "negative_test_query71_00001", + "negative_test_query259_00002", + "negative_test_query71_00013", + "negative_test_query65_00002", + "negative_test_query71_00009", + "apositive_test_query71_00000", + "apositive_test_query71_00001", + "negative_test_query71_00000", + "negative_test_query71_00005", + "negative_test_query71_00011", + "apositive_test_query30_00005", + "apositive_test_query71_00002", + "apositive_test_query209_00005", + "negative_test_query71_00002", + "negative_test_query231_00014", + "negative_test_query186_00000", + "negative_test_query247_00002", + "negative_test_query313_00001", + "negative_test_query71_00015" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "apositive_test_query72_00000", + "negative_test_query72_00001", + "negative_test_query72_00016", + "negative_test_query72_00007", + "negative_test_query72_00008", + "negative_test_query185_00000", + "negative_test_query72_00015", + "negative_test_query72_00003", + "negative_test_query72_00013", + "negative_test_query17_00009", + "negative_test_query55_00005", + "negative_test_query92_00001", + "apositive_test_query175_00001", + "negative_test_query228_00000", + "negative_test_query285_00009", + "negative_test_query265_00003", + "negative_test_query233_00009", + "negative_test_query72_00006", + "negative_test_query72_00011", + "negative_test_query185_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query73", + "ranked_ids": [ + "apositive_test_query322_00003", + "negative_test_query128_00004", + "negative_test_query21_00005", + "apositive_test_query254_00004", + "negative_test_query73_00001", + "negative_test_query73_00006", + "negative_test_query322_00009", + "apositive_test_query322_00008", + "negative_test_query73_00008", + "apositive_test_query333_00000", + "negative_test_query73_00013", + "negative_test_query228_00013", + "negative_test_query233_00015", + "apositive_test_query333_00005", + "negative_test_query216_00013", + "apositive_test_query73_00000", + "negative_test_query73_00010", + "apositive_test_query333_00002", + "negative_test_query333_00005", + "negative_test_query246_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query74_00000", + "apositive_test_query74_00001", + "apositive_test_query74_00003", + "negative_test_query74_00014", + "negative_test_query74_00001", + "negative_test_query95_00009", + "negative_test_query44_00006", + "apositive_test_query74_00002", + "negative_test_query74_00000", + "apositive_test_query114_00001", + "negative_test_query85_00005", + "apositive_test_query113_00001", + "negative_test_query74_00013", + "negative_test_query74_00015", + "negative_test_query203_00006", + "apositive_test_query60_00006", + "negative_test_query129_00008", + "apositive_test_query238_00003", + "negative_test_query238_00013", + "negative_test_query74_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.5714285714285714 + }, + "query_id": "test_query75", + "ranked_ids": [ + "apositive_test_query75_00005", + "apositive_test_query75_00000", + "apositive_test_query75_00001", + "negative_test_query130_00012", + "apositive_test_query75_00003", + "negative_test_query89_00000", + "negative_test_query75_00006", + "negative_test_query75_00004", + "negative_test_query75_00003", + "apositive_test_query75_00006", + "negative_test_query319_00004", + "negative_test_query82_00004", + "negative_test_query75_00012", + "negative_test_query323_00011", + "negative_test_query112_00018", + "negative_test_query260_00013", + "negative_test_query99_00006", + "negative_test_query67_00017", + "negative_test_query209_00004", + "negative_test_query6_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query76", + "ranked_ids": [ + "apositive_test_query76_00006", + "apositive_test_query76_00004", + "negative_test_query76_00007", + "negative_test_query91_00004", + "apositive_test_query76_00002", + "negative_test_query147_00007", + "apositive_test_query76_00001", + "negative_test_query181_00008", + "negative_test_query76_00010", + "negative_test_query76_00009", + "negative_test_query53_00001", + "negative_test_query76_00002", + "negative_test_query76_00011", + "negative_test_query76_00008", + "negative_test_query279_00009", + "negative_test_query258_00008", + "negative_test_query309_00011", + "negative_test_query53_00009", + "apositive_test_query76_00000", + "negative_test_query147_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query216_00006", + "negative_test_query98_00014", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query148_00013", + "negative_test_query329_00015", + "negative_test_query342_00007", + "negative_test_query77_00013", + "negative_test_query144_00003", + "negative_test_query14_00010", + "negative_test_query150_00011", + "negative_test_query215_00004", + "negative_test_query216_00012", + "negative_test_query263_00002", + "negative_test_query287_00012", + "negative_test_query301_00003", + "negative_test_query315_00003", + "negative_test_query326_00009", + "negative_test_query47_00010", + "negative_test_query87_00016" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query78", + "ranked_ids": [ + "apositive_test_query78_00001", + "negative_test_query214_00010", + "negative_test_query290_00006", + "negative_test_query290_00013", + "apositive_test_query38_00003", + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query72_00005", + "negative_test_query78_00016", + "apositive_test_query326_00000", + "apositive_test_query78_00000", + "negative_test_query78_00007", + "negative_test_query200_00009", + "negative_test_query236_00018", + "negative_test_query329_00002", + "negative_test_query144_00000", + "negative_test_query272_00001", + "negative_test_query274_00014", + "negative_test_query192_00009", + "negative_test_query7_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "apositive_test_query79_00004", + "negative_test_query79_00002", + "negative_test_query158_00000", + "negative_test_query79_00013", + "apositive_test_query221_00004", + "apositive_test_query119_00005", + "apositive_test_query295_00001", + "apositive_test_query79_00000", + "apositive_test_query306_00010", + "negative_test_query56_00002", + "negative_test_query303_00016", + "negative_test_query79_00004", + "apositive_test_query113_00002", + "negative_test_query38_00007", + "negative_test_query53_00015", + "negative_test_query157_00000", + "apositive_test_query157_00008", + "negative_test_query313_00013", + "negative_test_query79_00009", + "negative_test_query105_00011" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "negative_test_query196_00007", + "negative_test_query8_00013", + "negative_test_query8_00001", + "negative_test_query8_00004", + "negative_test_query8_00014", + "negative_test_query205_00007", + "negative_test_query8_00011", + "apositive_test_query192_00003", + "apositive_test_query8_00000", + "negative_test_query192_00006", + "negative_test_query8_00015", + "negative_test_query8_00008", + "negative_test_query192_00003", + "negative_test_query8_00012", + "apositive_test_query243_00005", + "apositive_test_query192_00002", + "negative_test_query8_00000", + "apositive_test_query243_00010", + "negative_test_query329_00005", + "negative_test_query96_00010" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query179_00005", + "negative_test_query80_00004", + "apositive_test_query80_00002", + "apositive_test_query179_00001", + "negative_test_query80_00001", + "negative_test_query80_00003", + "apositive_test_query189_00004", + "negative_test_query289_00005", + "negative_test_query102_00009", + "negative_test_query36_00010", + "negative_test_query80_00012", + "apositive_test_query179_00003", + "negative_test_query80_00009", + "apositive_test_query136_00013", + "negative_test_query273_00005", + "negative_test_query287_00005", + "negative_test_query85_00007", + "apositive_test_query36_00000", + "negative_test_query179_00006", + "negative_test_query270_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query81", + "ranked_ids": [ + "apositive_test_query81_00000", + "apositive_test_query81_00004", + "negative_test_query81_00005", + "apositive_test_query81_00001", + "negative_test_query81_00006", + "negative_test_query81_00000", + "apositive_test_query81_00005", + "apositive_test_query81_00006", + "apositive_test_query81_00008", + "negative_test_query81_00008", + "negative_test_query81_00007", + "negative_test_query81_00004", + "negative_test_query81_00009", + "negative_test_query81_00010", + "negative_test_query81_00001", + "apositive_test_query81_00003", + "negative_test_query207_00008", + "negative_test_query246_00017", + "negative_test_query27_00006", + "negative_test_query145_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010", + "negative_test_query82_00010", + "apositive_test_query319_00001", + "apositive_test_query319_00003", + "apositive_test_query82_00003", + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00004", + "apositive_test_query82_00001", + "apositive_test_query82_00004", + "apositive_test_query82_00008", + "apositive_test_query319_00008", + "apositive_test_query319_00006", + "apositive_test_query82_00006", + "negative_test_query319_00009", + "negative_test_query340_00013", + "negative_test_query82_00009", + "negative_test_query319_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query83_00004", + "apositive_test_query83_00002", + "apositive_test_query229_00001", + "apositive_test_query168_00001", + "apositive_test_query95_00000", + "negative_test_query348_00006", + "negative_test_query168_00007", + "apositive_test_query168_00000", + "apositive_test_query83_00003", + "apositive_test_query146_00014", + "apositive_test_query164_00000", + "negative_test_query28_00005", + "negative_test_query48_00006", + "apositive_test_query83_00000", + "negative_test_query309_00016", + "apositive_test_query217_00006", + "negative_test_query0_00006", + "negative_test_query279_00012", + "negative_test_query357_00002", + "negative_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query84", + "ranked_ids": [ + "apositive_test_query84_00001", + "negative_test_query84_00017", + "apositive_test_query84_00000", + "negative_test_query84_00012", + "negative_test_query84_00000", + "negative_test_query84_00007", + "negative_test_query84_00006", + "negative_test_query84_00002", + "negative_test_query84_00008", + "negative_test_query84_00005", + "negative_test_query84_00001", + "negative_test_query84_00016", + "negative_test_query84_00009", + "negative_test_query43_00002", + "negative_test_query84_00013", + "negative_test_query224_00010", + "negative_test_query350_00017", + "apositive_test_query155_00002", + "negative_test_query268_00009", + "apositive_test_query249_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query85_00000", + "negative_test_query85_00000", + "negative_test_query85_00012", + "apositive_test_query85_00002", + "negative_test_query85_00003", + "apositive_test_query109_00004", + "negative_test_query112_00008", + "apositive_test_query85_00001", + "apositive_test_query85_00003", + "negative_test_query85_00009", + "negative_test_query85_00011", + "apositive_test_query284_00004", + "negative_test_query106_00003", + "negative_test_query85_00001", + "apositive_test_query286_00006", + "negative_test_query44_00008", + "apositive_test_query81_00008", + "apositive_test_query133_00013", + "apositive_test_query286_00004", + "apositive_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5855700749881525, + "recall@5": 0.5 + }, + "query_id": "test_query86", + "ranked_ids": [ + "apositive_test_query86_00003", + "negative_test_query310_00007", + "apositive_test_query86_00001", + "negative_test_query86_00013", + "negative_test_query86_00001", + "apositive_test_query86_00000", + "negative_test_query86_00012", + "negative_test_query299_00007", + "negative_test_query65_00013", + "negative_test_query86_00014", + "negative_test_query86_00008", + "apositive_test_query86_00002", + "negative_test_query86_00007", + "negative_test_query86_00000", + "negative_test_query86_00002", + "negative_test_query86_00011", + "negative_test_query210_00012", + "negative_test_query245_00012", + "negative_test_query188_00013", + "negative_test_query86_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query87", + "ranked_ids": [ + "apositive_test_query87_00000", + "negative_test_query87_00013", + "apositive_test_query87_00001", + "negative_test_query87_00004", + "negative_test_query87_00008", + "negative_test_query87_00000", + "negative_test_query87_00007", + "negative_test_query87_00011", + "negative_test_query87_00014", + "negative_test_query87_00015", + "negative_test_query87_00006", + "negative_test_query87_00010", + "negative_test_query87_00016", + "apositive_test_query302_00001", + "negative_test_query275_00006", + "negative_test_query315_00006", + "negative_test_query87_00002", + "negative_test_query200_00011", + "negative_test_query182_00017", + "negative_test_query263_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "negative_test_query88_00005", + "apositive_test_query88_00001", + "negative_test_query88_00013", + "negative_test_query88_00012", + "negative_test_query325_00006", + "apositive_test_query250_00002", + "apositive_test_query147_00002", + "negative_test_query88_00003", + "negative_test_query88_00016", + "negative_test_query88_00017", + "negative_test_query105_00014", + "negative_test_query340_00012", + "negative_test_query88_00002", + "negative_test_query184_00013", + "negative_test_query88_00008", + "negative_test_query88_00010", + "apositive_test_query223_00000", + "negative_test_query240_00009", + "negative_test_query334_00006", + "negative_test_query203_00000" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query110_00011", + "negative_test_query326_00006", + "negative_test_query89_00004", + "apositive_test_query89_00004", + "negative_test_query89_00008", + "negative_test_query89_00000", + "negative_test_query212_00000", + "negative_test_query68_00004", + "negative_test_query89_00005", + "negative_test_query156_00013", + "apositive_test_query255_00000", + "apositive_test_query75_00006", + "negative_test_query130_00003", + "negative_test_query89_00007", + "negative_test_query278_00009", + "negative_test_query278_00000", + "negative_test_query325_00012", + "apositive_test_query68_00001", + "apositive_test_query89_00000", + "negative_test_query233_00005" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00008", + "apositive_test_query90_00001", + "negative_test_query9_00006", + "negative_test_query90_00004", + "negative_test_query9_00018", + "negative_test_query9_00004", + "negative_test_query9_00016", + "negative_test_query90_00009", + "apositive_test_query9_00000", + "negative_test_query221_00002", + "negative_test_query9_00005", + "negative_test_query9_00001", + "negative_test_query9_00011", + "negative_test_query9_00015", + "negative_test_query9_00013", + "negative_test_query305_00009", + "negative_test_query221_00011", + "negative_test_query90_00007", + "negative_test_query221_00007", + "negative_test_query9_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query310_00007", + "negative_test_query231_00002", + "apositive_test_query90_00000", + "negative_test_query90_00012", + "negative_test_query90_00011", + "negative_test_query90_00010", + "negative_test_query90_00002", + "negative_test_query90_00017", + "negative_test_query227_00012", + "negative_test_query127_00001", + "negative_test_query90_00000", + "negative_test_query227_00001", + "apositive_test_query304_00000", + "negative_test_query90_00015", + "negative_test_query231_00014", + "negative_test_query90_00006", + "negative_test_query9_00008", + "negative_test_query261_00002", + "negative_test_query227_00006", + "apositive_test_query14_00002" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query91", + "ranked_ids": [ + "apositive_test_query91_00000", + "apositive_test_query91_00002", + "negative_test_query292_00009", + "negative_test_query279_00011", + "negative_test_query319_00002", + "negative_test_query358_00004", + "apositive_test_query342_00001", + "negative_test_query315_00013", + "negative_test_query82_00002", + "negative_test_query350_00004", + "apositive_test_query11_00006", + "apositive_test_query91_00006", + "negative_test_query130_00010", + "negative_test_query91_00010", + "apositive_test_query157_00001", + "negative_test_query118_00015", + "negative_test_query102_00002", + "apositive_test_query157_00000", + "negative_test_query303_00011", + "negative_test_query59_00006" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.5 + }, + "query_id": "test_query92", + "ranked_ids": [ + "apositive_test_query92_00005", + "negative_test_query92_00003", + "apositive_test_query92_00003", + "apositive_test_query92_00002", + "negative_test_query283_00001", + "apositive_test_query92_00000", + "apositive_test_query92_00001", + "negative_test_query105_00016", + "apositive_test_query92_00004", + "apositive_test_query198_00001", + "negative_test_query92_00013", + "negative_test_query1_00012", + "negative_test_query283_00009", + "apositive_test_query100_00001", + "negative_test_query100_00014", + "apositive_test_query99_00003", + "apositive_test_query258_00000", + "negative_test_query230_00006", + "negative_test_query122_00018", + "negative_test_query100_00012" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query93_00004", + "apositive_test_query93_00000", + "apositive_test_query93_00003", + "apositive_test_query243_00006", + "negative_test_query219_00000", + "negative_test_query195_00006", + "negative_test_query335_00003", + "negative_test_query280_00012", + "negative_test_query93_00003", + "negative_test_query195_00001", + "negative_test_query92_00012", + "negative_test_query108_00003", + "apositive_test_query335_00003", + "negative_test_query277_00003", + "negative_test_query321_00001", + "negative_test_query335_00011", + "negative_test_query5_00015", + "apositive_test_query323_00003", + "apositive_test_query160_00007", + "negative_test_query129_00003" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.375 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00006", + "apositive_test_query127_00001", + "apositive_test_query94_00001", + "apositive_test_query94_00005", + "apositive_test_query94_00002", + "apositive_test_query127_00011", + "negative_test_query94_00007", + "negative_test_query94_00009", + "negative_test_query94_00002", + "apositive_test_query94_00004", + "apositive_test_query127_00010", + "negative_test_query94_00003", + "apositive_test_query127_00004", + "apositive_test_query94_00006", + "apositive_test_query127_00015", + "apositive_test_query94_00007", + "negative_test_query94_00005", + "apositive_test_query127_00008", + "apositive_test_query94_00000", + "negative_test_query155_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9197207891481876, + "recall@5": 1.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "apositive_test_query95_00001", + "negative_test_query95_00013", + "apositive_test_query95_00000", + "apositive_test_query253_00007", + "negative_test_query95_00005", + "negative_test_query95_00015", + "negative_test_query95_00012", + "negative_test_query95_00010", + "negative_test_query95_00002", + "negative_test_query95_00008", + "negative_test_query12_00015", + "negative_test_query95_00003", + "negative_test_query95_00014", + "negative_test_query12_00017", + "negative_test_query95_00006", + "negative_test_query244_00010", + "negative_test_query334_00012", + "apositive_test_query177_00000", + "negative_test_query57_00007", + "apositive_test_query292_00001" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query339_00000", + "apositive_test_query96_00001", + "apositive_test_query96_00002", + "negative_test_query192_00011", + "negative_test_query339_00001", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query339_00016", + "negative_test_query339_00006", + "negative_test_query96_00004", + "negative_test_query96_00016", + "negative_test_query339_00008", + "negative_test_query96_00003", + "apositive_test_query339_00001", + "negative_test_query339_00009", + "negative_test_query339_00017", + "negative_test_query339_00004", + "negative_test_query339_00014", + "negative_test_query96_00015", + "negative_test_query96_00013" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.5 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "apositive_test_query97_00003", + "apositive_test_query97_00002", + "apositive_test_query97_00001", + "negative_test_query97_00009", + "negative_test_query97_00012", + "negative_test_query113_00006", + "negative_test_query97_00006", + "apositive_test_query51_00000", + "negative_test_query84_00008", + "negative_test_query97_00011", + "apositive_test_query97_00005", + "negative_test_query209_00008", + "negative_test_query113_00009", + "negative_test_query16_00014", + "apositive_test_query289_00003", + "apositive_test_query51_00002", + "apositive_test_query267_00000", + "apositive_test_query201_00006", + "negative_test_query51_00008" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00007", + "apositive_test_query342_00000", + "negative_test_query98_00016", + "negative_test_query98_00017", + "negative_test_query98_00004", + "negative_test_query231_00014", + "negative_test_query98_00018", + "negative_test_query98_00011", + "apositive_test_query14_00003", + "negative_test_query197_00007", + "apositive_test_query14_00002", + "negative_test_query98_00005", + "negative_test_query129_00003", + "negative_test_query23_00013", + "negative_test_query212_00006", + "apositive_test_query201_00004", + "negative_test_query39_00005", + "negative_test_query345_00002", + "negative_test_query300_00000", + "negative_test_query316_00014" + ], + "valid": true + }, + { + "algorithm": "prp_sliding_p1", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query99", + "ranked_ids": [ + "apositive_test_query193_00008", + "negative_test_query296_00013", + "negative_test_query193_00002", + "negative_test_query174_00002", + "negative_test_query193_00001", + "negative_test_query330_00008", + "negative_test_query129_00001", + "apositive_test_query193_00012", + "negative_test_query193_00005", + "negative_test_query27_00018", + "apositive_test_query153_00012", + "apositive_test_query253_00003", + "apositive_test_query193_00010", + "apositive_test_query333_00003", + "apositive_test_query99_00001", + "apositive_test_query99_00007", + "apositive_test_query347_00002", + "negative_test_query256_00002", + "negative_test_query129_00003", + "negative_test_query27_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00002", + "negative_test_query221_00006", + "apositive_test_query0_00006", + "negative_test_query359_00002", + "apositive_test_query153_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query1", + "ranked_ids": [ + "negative_test_query61_00013", + "negative_test_query1_00009", + "apositive_test_query348_00003", + "apositive_test_query1_00000", + "negative_test_query61_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query10", + "ranked_ids": [ + "negative_test_query10_00000", + "negative_test_query10_00010", + "apositive_test_query10_00002", + "negative_test_query10_00011", + "negative_test_query10_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.6 + }, + "query_id": "test_query100", + "ranked_ids": [ + "negative_test_query100_00013", + "negative_test_query15_00000", + "apositive_test_query100_00004", + "apositive_test_query100_00000", + "apositive_test_query100_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query336_00001", + "apositive_test_query73_00002", + "apositive_test_query291_00000", + "negative_test_query189_00014", + "negative_test_query61_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query108_00009", + "negative_test_query209_00006", + "apositive_test_query238_00000", + "negative_test_query47_00010", + "apositive_test_query47_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "negative_test_query103_00003", + "apositive_test_query185_00003", + "negative_test_query185_00003", + "negative_test_query185_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "apositive_test_query104_00000", + "negative_test_query104_00018", + "apositive_test_query48_00000", + "negative_test_query104_00002", + "negative_test_query104_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query105", + "ranked_ids": [ + "apositive_test_query105_00000", + "negative_test_query105_00005", + "negative_test_query105_00011", + "negative_test_query97_00007", + "negative_test_query79_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query106", + "ranked_ids": [ + "negative_test_query36_00001", + "apositive_test_query106_00005", + "apositive_test_query106_00003", + "negative_test_query286_00000", + "apositive_test_query286_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "negative_test_query160_00001", + "apositive_test_query357_00003", + "negative_test_query46_00012", + "apositive_test_query142_00004", + "apositive_test_query36_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.375 + }, + "query_id": "test_query108", + "ranked_ids": [ + "negative_test_query311_00006", + "apositive_test_query108_00000", + "apositive_test_query108_00003", + "apositive_test_query108_00007", + "negative_test_query108_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.6 + }, + "query_id": "test_query109", + "ranked_ids": [ + "negative_test_query190_00003", + "apositive_test_query109_00003", + "apositive_test_query109_00000", + "apositive_test_query109_00001", + "apositive_test_query319_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.375 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00000", + "apositive_test_query11_00006", + "apositive_test_query11_00001", + "negative_test_query11_00004", + "negative_test_query11_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "negative_test_query89_00006", + "negative_test_query110_00014", + "negative_test_query110_00013", + "negative_test_query110_00002", + "negative_test_query110_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query111_00015", + "negative_test_query134_00006", + "negative_test_query277_00006", + "negative_test_query277_00010", + "negative_test_query223_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "apositive_test_query286_00005", + "apositive_test_query36_00003", + "negative_test_query112_00015", + "negative_test_query112_00004", + "negative_test_query112_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query113", + "ranked_ids": [ + "negative_test_query38_00003", + "apositive_test_query113_00000", + "negative_test_query67_00014", + "negative_test_query7_00016", + "apositive_test_query113_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query114", + "ranked_ids": [ + "apositive_test_query114_00003", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "apositive_test_query114_00001", + "negative_test_query262_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query115", + "ranked_ids": [ + "apositive_test_query115_00002", + "apositive_test_query115_00004", + "negative_test_query121_00008", + "negative_test_query115_00003", + "apositive_test_query133_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "apositive_test_query49_00000", + "apositive_test_query116_00000", + "negative_test_query116_00003", + "negative_test_query116_00015", + "negative_test_query59_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00006", + "negative_test_query194_00002", + "apositive_test_query117_00003", + "apositive_test_query354_00003", + "negative_test_query117_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "negative_test_query23_00005", + "apositive_test_query155_00005", + "apositive_test_query174_00010", + "negative_test_query84_00012", + "negative_test_query296_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query119", + "ranked_ids": [ + "negative_test_query158_00001", + "negative_test_query158_00000", + "apositive_test_query221_00004", + "apositive_test_query105_00000", + "negative_test_query56_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query12_00016", + "negative_test_query12_00003", + "negative_test_query175_00007", + "negative_test_query12_00014", + "negative_test_query12_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00010", + "negative_test_query1_00009", + "apositive_test_query352_00000", + "negative_test_query120_00008", + "negative_test_query120_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00002", + "apositive_test_query307_00010", + "negative_test_query121_00008", + "negative_test_query115_00009", + "negative_test_query313_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query195_00008", + "negative_test_query61_00015", + "negative_test_query17_00012", + "negative_test_query134_00007", + "apositive_test_query175_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query123", + "ranked_ids": [ + "negative_test_query123_00013", + "apositive_test_query123_00002", + "apositive_test_query123_00000", + "negative_test_query161_00009", + "negative_test_query123_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "apositive_test_query194_00000", + "apositive_test_query13_00004", + "negative_test_query42_00006", + "negative_test_query347_00010", + "negative_test_query349_00017" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.3191471544989227, + "recall@5": 0.5 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query125_00014", + "negative_test_query140_00017", + "negative_test_query360_00016", + "apositive_test_query125_00000", + "apositive_test_query125_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query126", + "ranked_ids": [ + "apositive_test_query126_00000", + "negative_test_query151_00008", + "negative_test_query126_00000", + "negative_test_query126_00008", + "apositive_test_query151_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00004", + "apositive_test_query127_00003", + "apositive_test_query127_00016", + "apositive_test_query127_00000", + "apositive_test_query127_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.25 + }, + "query_id": "test_query128", + "ranked_ids": [ + "apositive_test_query128_00000", + "negative_test_query21_00017", + "negative_test_query256_00011", + "negative_test_query283_00006", + "apositive_test_query128_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.25 + }, + "query_id": "test_query129", + "ranked_ids": [ + "apositive_test_query129_00000", + "negative_test_query156_00017", + "negative_test_query117_00006", + "apositive_test_query129_00001", + "negative_test_query211_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.38461538461538464 + }, + "query_id": "test_query13", + "ranked_ids": [ + "apositive_test_query13_00002", + "apositive_test_query13_00006", + "apositive_test_query13_00012", + "apositive_test_query13_00003", + "apositive_test_query13_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query130", + "ranked_ids": [ + "apositive_test_query90_00001", + "apositive_test_query284_00003", + "apositive_test_query130_00005", + "negative_test_query130_00001", + "apositive_test_query323_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.3973220069685352, + "recall@5": 0.5 + }, + "query_id": "test_query131", + "ranked_ids": [ + "negative_test_query186_00012", + "apositive_test_query131_00000", + "negative_test_query345_00009", + "negative_test_query184_00012", + "apositive_test_query131_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "negative_test_query132_00006", + "negative_test_query132_00018", + "negative_test_query123_00012", + "negative_test_query81_00001", + "negative_test_query81_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query287_00001", + "apositive_test_query133_00002", + "apositive_test_query133_00001", + "apositive_test_query133_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query134", + "ranked_ids": [ + "apositive_test_query276_00004", + "apositive_test_query213_00001", + "apositive_test_query134_00002", + "negative_test_query218_00009", + "negative_test_query134_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.5012658353418871, + "recall@5": 1.0 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00007", + "negative_test_query135_00003", + "negative_test_query135_00017", + "apositive_test_query135_00000", + "apositive_test_query135_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.26666666666666666 + }, + "query_id": "test_query136", + "ranked_ids": [ + "apositive_test_query136_00004", + "negative_test_query136_00003", + "apositive_test_query136_00014", + "apositive_test_query136_00011", + "apositive_test_query136_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query137", + "ranked_ids": [ + "negative_test_query74_00001", + "negative_test_query137_00003", + "negative_test_query13_00002", + "apositive_test_query137_00002", + "negative_test_query253_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query138", + "ranked_ids": [ + "apositive_test_query138_00000", + "apositive_test_query138_00001", + "negative_test_query138_00005", + "negative_test_query138_00003", + "negative_test_query138_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query139_00001", + "apositive_test_query35_00001", + "apositive_test_query35_00000", + "apositive_test_query139_00000", + "negative_test_query101_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query14", + "ranked_ids": [ + "apositive_test_query216_00002", + "negative_test_query14_00001", + "apositive_test_query77_00000", + "negative_test_query199_00009", + "negative_test_query179_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00003", + "negative_test_query140_00005", + "negative_test_query140_00001", + "negative_test_query140_00016", + "apositive_test_query18_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query141", + "ranked_ids": [ + "apositive_test_query288_00003", + "apositive_test_query141_00005", + "apositive_test_query141_00006", + "negative_test_query105_00014", + "negative_test_query176_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query142", + "ranked_ids": [ + "negative_test_query142_00004", + "negative_test_query102_00001", + "apositive_test_query142_00001", + "negative_test_query337_00016", + "negative_test_query46_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query143", + "ranked_ids": [ + "negative_test_query143_00006", + "negative_test_query90_00014", + "negative_test_query143_00011", + "negative_test_query56_00005", + "apositive_test_query25_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query144_00012", + "negative_test_query198_00016", + "negative_test_query144_00010", + "negative_test_query41_00001", + "negative_test_query41_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query145", + "ranked_ids": [ + "apositive_test_query225_00002", + "negative_test_query305_00002", + "negative_test_query211_00013", + "apositive_test_query213_00004", + "negative_test_query187_00014" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.0625 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query146_00010", + "apositive_test_query314_00004", + "apositive_test_query161_00005", + "negative_test_query99_00001", + "negative_test_query146_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query147", + "ranked_ids": [ + "negative_test_query147_00013", + "negative_test_query180_00012", + "apositive_test_query147_00004", + "negative_test_query181_00008", + "negative_test_query258_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query124_00010", + "negative_test_query148_00012", + "negative_test_query210_00005", + "negative_test_query148_00014", + "negative_test_query210_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8503449055347546, + "recall@5": 1.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "apositive_test_query149_00001", + "negative_test_query149_00008", + "negative_test_query149_00011", + "negative_test_query225_00006", + "apositive_test_query149_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00010", + "negative_test_query266_00008", + "negative_test_query15_00013", + "apositive_test_query242_00000", + "negative_test_query266_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query150", + "ranked_ids": [ + "negative_test_query150_00010", + "apositive_test_query20_00007", + "apositive_test_query112_00000", + "negative_test_query306_00004", + "apositive_test_query20_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query126_00017", + "apositive_test_query151_00002", + "negative_test_query151_00004", + "negative_test_query201_00008", + "negative_test_query190_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query312_00014", + "negative_test_query237_00004", + "apositive_test_query152_00009", + "negative_test_query237_00011", + "apositive_test_query194_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.1875 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00001", + "apositive_test_query153_00003", + "negative_test_query112_00007", + "negative_test_query250_00006", + "apositive_test_query153_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query154", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query358_00008", + "negative_test_query68_00015", + "apositive_test_query20_00007", + "negative_test_query311_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00009", + "negative_test_query155_00012", + "negative_test_query123_00014", + "negative_test_query155_00006", + "apositive_test_query155_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "apositive_test_query210_00001", + "negative_test_query330_00006", + "apositive_test_query156_00000", + "negative_test_query15_00010", + "negative_test_query140_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query157", + "ranked_ids": [ + "apositive_test_query152_00001", + "apositive_test_query152_00005", + "negative_test_query237_00004", + "apositive_test_query157_00001", + "apositive_test_query152_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query158", + "ranked_ids": [ + "negative_test_query105_00013", + "negative_test_query102_00015", + "apositive_test_query186_00003", + "negative_test_query158_00006", + "negative_test_query26_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query159", + "ranked_ids": [ + "apositive_test_query38_00003", + "negative_test_query264_00013", + "apositive_test_query212_00002", + "negative_test_query159_00011", + "apositive_test_query159_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query16", + "ranked_ids": [ + "apositive_test_query16_00003", + "negative_test_query308_00004", + "negative_test_query208_00012", + "negative_test_query6_00015", + "apositive_test_query286_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query160", + "ranked_ids": [ + "apositive_test_query160_00009", + "negative_test_query214_00013", + "negative_test_query330_00005", + "apositive_test_query160_00000", + "negative_test_query160_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query161", + "ranked_ids": [ + "apositive_test_query161_00006", + "negative_test_query155_00012", + "apositive_test_query3_00006", + "apositive_test_query314_00006", + "apositive_test_query276_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.25 + }, + "query_id": "test_query162", + "ranked_ids": [ + "apositive_test_query162_00005", + "apositive_test_query162_00010", + "negative_test_query177_00002", + "apositive_test_query162_00009", + "apositive_test_query194_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query197_00002", + "negative_test_query163_00003", + "negative_test_query163_00011", + "negative_test_query163_00008", + "negative_test_query197_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query164_00000", + "negative_test_query158_00008", + "negative_test_query131_00014", + "negative_test_query164_00015", + "negative_test_query344_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query149_00012", + "negative_test_query165_00007", + "negative_test_query86_00006", + "negative_test_query86_00010", + "negative_test_query122_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.5 + }, + "query_id": "test_query166", + "ranked_ids": [ + "negative_test_query166_00012", + "apositive_test_query166_00005", + "apositive_test_query166_00003", + "apositive_test_query166_00002", + "negative_test_query166_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query167_00014", + "negative_test_query167_00012", + "negative_test_query167_00011", + "negative_test_query278_00010", + "negative_test_query278_00016" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.592512031964586, + "recall@5": 0.75 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query83_00003", + "apositive_test_query168_00000", + "apositive_test_query168_00001", + "apositive_test_query190_00001", + "apositive_test_query168_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "apositive_test_query80_00003", + "apositive_test_query69_00000", + "negative_test_query104_00001", + "apositive_test_query286_00008", + "apositive_test_query266_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00000", + "negative_test_query17_00014", + "negative_test_query17_00016", + "negative_test_query10_00005", + "negative_test_query17_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query170", + "ranked_ids": [ + "apositive_test_query170_00005", + "apositive_test_query170_00006", + "apositive_test_query297_00000", + "apositive_test_query170_00004", + "negative_test_query227_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query171", + "ranked_ids": [ + "negative_test_query171_00013", + "negative_test_query171_00005", + "negative_test_query171_00003", + "apositive_test_query171_00001", + "negative_test_query171_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query172", + "ranked_ids": [ + "negative_test_query172_00007", + "apositive_test_query172_00002", + "apositive_test_query172_00003", + "negative_test_query192_00010", + "apositive_test_query254_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query173_00011", + "apositive_test_query173_00002", + "negative_test_query64_00013", + "negative_test_query64_00017", + "negative_test_query303_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00004", + "apositive_test_query174_00001", + "apositive_test_query174_00005", + "apositive_test_query174_00012", + "apositive_test_query174_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00004", + "apositive_test_query50_00015", + "negative_test_query175_00010", + "negative_test_query175_00013", + "apositive_test_query175_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query284_00006", + "apositive_test_query344_00003", + "negative_test_query9_00007", + "negative_test_query284_00002", + "negative_test_query35_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query177_00006", + "negative_test_query172_00008", + "negative_test_query124_00012", + "negative_test_query177_00008", + "negative_test_query152_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query178", + "ranked_ids": [ + "apositive_test_query178_00004", + "negative_test_query178_00000", + "negative_test_query178_00007", + "negative_test_query178_00012", + "negative_test_query178_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "negative_test_query179_00014", + "apositive_test_query316_00001", + "negative_test_query64_00003", + "negative_test_query260_00012", + "negative_test_query191_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query18", + "ranked_ids": [ + "apositive_test_query18_00001", + "apositive_test_query18_00003", + "negative_test_query18_00011", + "negative_test_query18_00006", + "negative_test_query215_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query180", + "ranked_ids": [ + "apositive_test_query180_00002", + "negative_test_query180_00004", + "apositive_test_query180_00001", + "negative_test_query176_00001", + "negative_test_query180_00016" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00002", + "negative_test_query181_00006", + "negative_test_query181_00007", + "negative_test_query181_00012", + "negative_test_query181_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "negative_test_query182_00017", + "apositive_test_query182_00000", + "negative_test_query77_00002", + "negative_test_query182_00004", + "negative_test_query182_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00014", + "negative_test_query183_00012", + "negative_test_query183_00015", + "negative_test_query183_00009", + "negative_test_query7_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "negative_test_query216_00014", + "negative_test_query216_00002", + "negative_test_query216_00008", + "negative_test_query41_00013", + "negative_test_query216_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query185", + "ranked_ids": [ + "negative_test_query185_00002", + "negative_test_query185_00005", + "negative_test_query185_00000", + "apositive_test_query185_00002", + "negative_test_query72_00017" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "negative_test_query247_00012", + "negative_test_query186_00005", + "negative_test_query47_00009", + "apositive_test_query50_00010", + "negative_test_query247_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query62_00009", + "negative_test_query245_00006", + "apositive_test_query20_00010", + "negative_test_query279_00002", + "negative_test_query356_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "apositive_test_query220_00003", + "negative_test_query188_00008", + "negative_test_query188_00010", + "apositive_test_query188_00000", + "negative_test_query112_00017" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00001", + "apositive_test_query189_00003", + "apositive_test_query189_00001", + "apositive_test_query85_00002", + "negative_test_query189_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query19", + "ranked_ids": [ + "apositive_test_query4_00000", + "negative_test_query4_00001", + "negative_test_query19_00003", + "apositive_test_query4_00010", + "negative_test_query229_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query82_00008", + "apositive_test_query319_00008", + "apositive_test_query85_00000", + "negative_test_query85_00006", + "apositive_test_query311_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "negative_test_query96_00006", + "negative_test_query191_00011", + "negative_test_query191_00010", + "negative_test_query274_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.4 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00000", + "negative_test_query192_00001", + "negative_test_query322_00008", + "apositive_test_query192_00002", + "negative_test_query192_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query193", + "ranked_ids": [ + "apositive_test_query193_00004", + "apositive_test_query99_00003", + "apositive_test_query193_00001", + "apositive_test_query193_00000", + "apositive_test_query99_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query194_00002", + "apositive_test_query162_00006", + "apositive_test_query124_00000", + "negative_test_query262_00007", + "apositive_test_query194_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query195", + "ranked_ids": [ + "negative_test_query195_00002", + "apositive_test_query195_00001", + "negative_test_query222_00003", + "negative_test_query195_00008", + "apositive_test_query195_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query196", + "ranked_ids": [ + "negative_test_query179_00001", + "apositive_test_query32_00012", + "negative_test_query96_00013", + "negative_test_query325_00004", + "apositive_test_query32_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query197", + "ranked_ids": [ + "negative_test_query163_00001", + "apositive_test_query197_00000", + "negative_test_query197_00016", + "negative_test_query163_00012", + "negative_test_query163_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query239_00010", + "negative_test_query198_00007", + "negative_test_query239_00009", + "negative_test_query239_00011", + "negative_test_query239_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query199_00014", + "negative_test_query139_00014", + "negative_test_query199_00015", + "negative_test_query72_00005", + "negative_test_query213_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query44_00008", + "negative_test_query15_00000", + "negative_test_query2_00012", + "negative_test_query2_00009", + "apositive_test_query57_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query20_00010", + "apositive_test_query154_00006", + "apositive_test_query20_00007", + "negative_test_query33_00008", + "apositive_test_query66_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query200_00004", + "negative_test_query337_00009", + "negative_test_query264_00006", + "negative_test_query200_00010", + "negative_test_query200_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query201", + "ranked_ids": [ + "negative_test_query231_00012", + "apositive_test_query201_00002", + "negative_test_query47_00012", + "negative_test_query71_00013", + "apositive_test_query0_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query202_00004", + "negative_test_query235_00001", + "apositive_test_query61_00003", + "negative_test_query188_00011", + "negative_test_query15_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.3 + }, + "query_id": "test_query203", + "ranked_ids": [ + "apositive_test_query203_00000", + "apositive_test_query30_00005", + "apositive_test_query203_00009", + "negative_test_query203_00006", + "apositive_test_query203_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query204", + "ranked_ids": [ + "apositive_test_query204_00008", + "negative_test_query353_00001", + "apositive_test_query204_00009", + "apositive_test_query204_00002", + "apositive_test_query353_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query205_00015", + "negative_test_query205_00007", + "apositive_test_query205_00000", + "negative_test_query35_00007", + "negative_test_query31_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query206_00014", + "negative_test_query206_00015", + "negative_test_query206_00002", + "negative_test_query206_00013", + "negative_test_query206_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00000", + "apositive_test_query207_00004", + "apositive_test_query207_00002", + "apositive_test_query207_00005", + "negative_test_query207_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "negative_test_query128_00003", + "negative_test_query208_00000", + "negative_test_query208_00010", + "negative_test_query211_00014", + "negative_test_query208_00018" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query209", + "ranked_ids": [ + "negative_test_query209_00007", + "apositive_test_query209_00001", + "apositive_test_query26_00007", + "negative_test_query231_00006", + "negative_test_query223_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "apositive_test_query21_00000", + "negative_test_query128_00006", + "negative_test_query128_00004", + "negative_test_query21_00005", + "negative_test_query330_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query156_00015", + "negative_test_query17_00002", + "negative_test_query210_00013", + "negative_test_query335_00004", + "negative_test_query288_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query211", + "ranked_ids": [ + "apositive_test_query211_00001", + "negative_test_query211_00015", + "negative_test_query211_00012", + "negative_test_query211_00000", + "apositive_test_query101_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query212", + "ranked_ids": [ + "negative_test_query167_00016", + "apositive_test_query212_00004", + "apositive_test_query326_00000", + "apositive_test_query98_00000", + "negative_test_query77_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00004", + "apositive_test_query213_00000", + "negative_test_query213_00011", + "apositive_test_query158_00001", + "apositive_test_query213_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "negative_test_query214_00010", + "negative_test_query214_00001", + "negative_test_query103_00009", + "apositive_test_query17_00001", + "apositive_test_query214_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00012", + "negative_test_query215_00004", + "negative_test_query215_00000", + "apositive_test_query80_00005", + "negative_test_query215_00017" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.8 + }, + "query_id": "test_query216", + "ranked_ids": [ + "negative_test_query216_00000", + "apositive_test_query216_00002", + "apositive_test_query216_00000", + "apositive_test_query216_00001", + "apositive_test_query216_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "apositive_test_query307_00000", + "apositive_test_query28_00008", + "negative_test_query185_00003", + "negative_test_query217_00011", + "apositive_test_query217_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query218", + "ranked_ids": [ + "apositive_test_query218_00004", + "negative_test_query134_00003", + "apositive_test_query218_00005", + "negative_test_query218_00002", + "negative_test_query43_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query219", + "ranked_ids": [ + "apositive_test_query219_00000", + "negative_test_query219_00015", + "negative_test_query219_00014", + "negative_test_query219_00005", + "negative_test_query24_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "apositive_test_query22_00000", + "negative_test_query22_00015", + "negative_test_query22_00004", + "negative_test_query22_00008", + "negative_test_query22_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.5 + }, + "query_id": "test_query220", + "ranked_ids": [ + "apositive_test_query220_00003", + "negative_test_query188_00013", + "negative_test_query161_00010", + "apositive_test_query220_00001", + "apositive_test_query220_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query142_00004", + "negative_test_query277_00010", + "apositive_test_query92_00003", + "negative_test_query65_00013", + "negative_test_query244_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query222", + "ranked_ids": [ + "apositive_test_query275_00004", + "apositive_test_query222_00001", + "apositive_test_query275_00001", + "apositive_test_query275_00006", + "negative_test_query275_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query47_00001", + "apositive_test_query259_00011", + "negative_test_query6_00007", + "negative_test_query109_00006", + "apositive_test_query260_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query224", + "ranked_ids": [ + "negative_test_query224_00010", + "negative_test_query196_00001", + "apositive_test_query224_00000", + "apositive_test_query32_00002", + "negative_test_query98_00018" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00003", + "negative_test_query145_00000", + "apositive_test_query225_00007", + "apositive_test_query225_00005", + "apositive_test_query169_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query226", + "ranked_ids": [ + "negative_test_query226_00009", + "apositive_test_query226_00005", + "apositive_test_query226_00004", + "apositive_test_query226_00001", + "apositive_test_query226_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00003", + "negative_test_query227_00002", + "negative_test_query59_00004", + "apositive_test_query348_00004", + "negative_test_query115_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query228", + "ranked_ids": [ + "negative_test_query228_00003", + "negative_test_query165_00011", + "negative_test_query73_00016", + "apositive_test_query333_00001", + "apositive_test_query322_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query229_00002", + "apositive_test_query229_00000", + "apositive_test_query229_00003", + "negative_test_query229_00005", + "apositive_test_query229_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.6 + }, + "query_id": "test_query23", + "ranked_ids": [ + "negative_test_query66_00010", + "negative_test_query23_00014", + "apositive_test_query23_00001", + "apositive_test_query23_00002", + "apositive_test_query23_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query230_00007", + "negative_test_query3_00006", + "apositive_test_query3_00000", + "negative_test_query230_00000", + "negative_test_query230_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query231", + "ranked_ids": [ + "negative_test_query9_00017", + "negative_test_query345_00011", + "negative_test_query221_00006", + "apositive_test_query130_00000", + "negative_test_query75_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "apositive_test_query232_00000", + "negative_test_query232_00016", + "negative_test_query232_00012", + "negative_test_query232_00007", + "negative_test_query232_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query233", + "ranked_ids": [ + "apositive_test_query233_00002", + "apositive_test_query233_00000", + "negative_test_query233_00010", + "negative_test_query233_00001", + "negative_test_query233_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query234", + "ranked_ids": [ + "apositive_test_query304_00001", + "negative_test_query234_00011", + "negative_test_query10_00008", + "apositive_test_query218_00002", + "negative_test_query304_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query32_00000", + "negative_test_query196_00000", + "apositive_test_query32_00006", + "negative_test_query235_00013", + "negative_test_query24_00017" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "negative_test_query236_00000", + "negative_test_query222_00000", + "negative_test_query236_00018", + "negative_test_query236_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "apositive_test_query237_00001", + "negative_test_query237_00005", + "negative_test_query237_00004", + "negative_test_query237_00002", + "negative_test_query312_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "negative_test_query102_00007", + "negative_test_query108_00007", + "negative_test_query223_00005", + "negative_test_query203_00003", + "negative_test_query13_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "negative_test_query239_00015", + "negative_test_query239_00003", + "negative_test_query239_00014", + "negative_test_query239_00000", + "negative_test_query239_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "negative_test_query24_00008", + "apositive_test_query24_00000", + "negative_test_query24_00009", + "negative_test_query24_00002", + "negative_test_query24_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "negative_test_query240_00012", + "apositive_test_query240_00000", + "negative_test_query240_00016", + "negative_test_query240_00006", + "negative_test_query240_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "apositive_test_query241_00000", + "negative_test_query241_00010", + "negative_test_query191_00011", + "negative_test_query206_00009", + "negative_test_query205_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.2 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query242_00002", + "apositive_test_query247_00004", + "apositive_test_query291_00000", + "apositive_test_query242_00001", + "apositive_test_query242_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.38461538461538464 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query243_00002", + "apositive_test_query243_00000", + "apositive_test_query243_00005", + "apositive_test_query243_00012", + "apositive_test_query243_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query244", + "ranked_ids": [ + "apositive_test_query244_00002", + "negative_test_query244_00001", + "negative_test_query244_00005", + "apositive_test_query244_00000", + "negative_test_query244_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query245_00003", + "apositive_test_query141_00006", + "apositive_test_query141_00002", + "negative_test_query202_00004", + "negative_test_query336_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query246_00002", + "negative_test_query246_00016", + "negative_test_query218_00004", + "negative_test_query246_00012", + "apositive_test_query68_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "negative_test_query291_00003", + "apositive_test_query247_00004", + "apositive_test_query291_00000", + "apositive_test_query242_00002", + "apositive_test_query291_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.25 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00008", + "apositive_test_query248_00007", + "apositive_test_query248_00005", + "negative_test_query248_00002", + "negative_test_query248_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query249", + "ranked_ids": [ + "apositive_test_query249_00004", + "apositive_test_query281_00001", + "apositive_test_query249_00000", + "negative_test_query122_00016", + "negative_test_query249_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query25", + "ranked_ids": [ + "apositive_test_query25_00000", + "apositive_test_query25_00003", + "negative_test_query143_00000", + "apositive_test_query25_00008", + "negative_test_query25_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query250", + "ranked_ids": [ + "apositive_test_query250_00000", + "apositive_test_query203_00000", + "negative_test_query250_00009", + "negative_test_query250_00013", + "negative_test_query250_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query251", + "ranked_ids": [ + "negative_test_query322_00002", + "apositive_test_query251_00001", + "negative_test_query234_00006", + "negative_test_query326_00000", + "negative_test_query14_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query252", + "ranked_ids": [ + "negative_test_query252_00001", + "apositive_test_query252_00003", + "negative_test_query44_00011", + "negative_test_query318_00006", + "negative_test_query252_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.4468535299918563, + "recall@5": 0.3 + }, + "query_id": "test_query253", + "ranked_ids": [ + "apositive_test_query13_00012", + "negative_test_query172_00008", + "apositive_test_query253_00004", + "apositive_test_query253_00007", + "apositive_test_query253_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query254", + "ranked_ids": [ + "apositive_test_query254_00004", + "negative_test_query316_00013", + "negative_test_query350_00000", + "negative_test_query254_00008", + "apositive_test_query254_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "negative_test_query326_00006", + "negative_test_query89_00004", + "apositive_test_query25_00004", + "negative_test_query255_00006", + "negative_test_query255_00017" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query256", + "ranked_ids": [ + "negative_test_query256_00003", + "apositive_test_query256_00001", + "negative_test_query105_00008", + "apositive_test_query142_00002", + "negative_test_query135_00016" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.5 + }, + "query_id": "test_query257", + "ranked_ids": [ + "negative_test_query215_00000", + "apositive_test_query257_00004", + "apositive_test_query257_00000", + "apositive_test_query257_00002", + "negative_test_query257_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "apositive_test_query258_00000", + "negative_test_query228_00005", + "negative_test_query339_00008", + "negative_test_query230_00006", + "negative_test_query319_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.25 + }, + "query_id": "test_query259", + "ranked_ids": [ + "apositive_test_query259_00004", + "apositive_test_query259_00006", + "apositive_test_query259_00008", + "apositive_test_query359_00002", + "negative_test_query259_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.25 + }, + "query_id": "test_query26", + "ranked_ids": [ + "apositive_test_query186_00003", + "apositive_test_query26_00005", + "apositive_test_query158_00003", + "apositive_test_query26_00004", + "negative_test_query102_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "negative_test_query6_00004", + "negative_test_query260_00011", + "apositive_test_query6_00002", + "apositive_test_query109_00001", + "apositive_test_query6_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query261", + "ranked_ids": [ + "negative_test_query261_00000", + "negative_test_query261_00007", + "negative_test_query261_00002", + "negative_test_query116_00004", + "negative_test_query261_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.36331756126716835, + "recall@5": 0.5 + }, + "query_id": "test_query262", + "ranked_ids": [ + "negative_test_query312_00018", + "negative_test_query106_00010", + "apositive_test_query262_00003", + "apositive_test_query262_00000", + "negative_test_query262_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query263", + "ranked_ids": [ + "negative_test_query135_00004", + "negative_test_query190_00009", + "negative_test_query213_00005", + "negative_test_query263_00005", + "negative_test_query357_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "apositive_test_query264_00000", + "negative_test_query159_00017", + "negative_test_query264_00008", + "negative_test_query264_00016", + "negative_test_query337_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query265", + "ranked_ids": [ + "negative_test_query274_00008", + "apositive_test_query265_00001", + "apositive_test_query265_00000", + "negative_test_query190_00013", + "negative_test_query265_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "apositive_test_query266_00000", + "negative_test_query266_00008", + "negative_test_query176_00018", + "negative_test_query266_00001", + "negative_test_query106_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "negative_test_query359_00005", + "apositive_test_query157_00008", + "negative_test_query295_00002", + "apositive_test_query113_00002", + "negative_test_query98_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query268", + "ranked_ids": [ + "apositive_test_query268_00000", + "negative_test_query268_00000", + "negative_test_query47_00017", + "apositive_test_query51_00003", + "negative_test_query268_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query269", + "ranked_ids": [ + "negative_test_query269_00009", + "negative_test_query329_00014", + "negative_test_query269_00007", + "negative_test_query269_00010", + "negative_test_query269_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "negative_test_query27_00004", + "apositive_test_query16_00002", + "negative_test_query27_00009", + "negative_test_query33_00004", + "negative_test_query164_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7039180890341347, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query270", + "ranked_ids": [ + "apositive_test_query270_00001", + "negative_test_query270_00000", + "apositive_test_query270_00000", + "negative_test_query270_00006", + "negative_test_query270_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8772153153380493, + "recall@5": 1.0 + }, + "query_id": "test_query271", + "ranked_ids": [ + "apositive_test_query271_00000", + "apositive_test_query10_00002", + "negative_test_query112_00014", + "apositive_test_query271_00001", + "negative_test_query78_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00016", + "negative_test_query41_00008", + "negative_test_query272_00008", + "negative_test_query272_00015", + "negative_test_query272_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query18_00011", + "negative_test_query273_00008", + "negative_test_query273_00004", + "apositive_test_query273_00000", + "apositive_test_query281_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.44149241373678083, + "recall@5": 0.5 + }, + "query_id": "test_query274", + "ranked_ids": [ + "negative_test_query274_00000", + "apositive_test_query274_00001", + "apositive_test_query274_00000", + "negative_test_query78_00007", + "negative_test_query34_00014" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query275_00000", + "negative_test_query263_00009", + "apositive_test_query275_00007", + "apositive_test_query275_00006", + "apositive_test_query275_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query276", + "ranked_ids": [ + "apositive_test_query153_00006", + "negative_test_query276_00001", + "apositive_test_query276_00004", + "negative_test_query22_00004", + "negative_test_query199_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8318724637288826, + "recall@5": 0.75 + }, + "query_id": "test_query277", + "ranked_ids": [ + "apositive_test_query277_00003", + "apositive_test_query277_00000", + "apositive_test_query277_00001", + "negative_test_query277_00007", + "negative_test_query277_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query278_00002", + "apositive_test_query263_00005", + "negative_test_query278_00011", + "negative_test_query140_00006", + "apositive_test_query4_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.4 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query177_00004", + "apositive_test_query279_00004", + "apositive_test_query279_00003", + "apositive_test_query162_00009", + "apositive_test_query358_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query28", + "ranked_ids": [ + "negative_test_query222_00005", + "negative_test_query72_00005", + "apositive_test_query28_00008", + "negative_test_query72_00004", + "negative_test_query48_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.5 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00000", + "apositive_test_query280_00005", + "negative_test_query280_00003", + "negative_test_query280_00004", + "apositive_test_query280_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6608397947263839, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query273_00002", + "apositive_test_query281_00010", + "apositive_test_query281_00004", + "apositive_test_query281_00012", + "apositive_test_query281_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query282", + "ranked_ids": [ + "negative_test_query344_00001", + "negative_test_query282_00007", + "apositive_test_query282_00000", + "negative_test_query282_00004", + "negative_test_query282_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query283_00012", + "negative_test_query283_00004", + "negative_test_query93_00004", + "apositive_test_query93_00002", + "negative_test_query283_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "negative_test_query284_00009", + "apositive_test_query284_00003", + "negative_test_query284_00012", + "negative_test_query284_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5 + }, + "query_id": "test_query285", + "ranked_ids": [ + "apositive_test_query285_00004", + "apositive_test_query285_00002", + "apositive_test_query285_00007", + "apositive_test_query285_00001", + "negative_test_query304_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.2 + }, + "query_id": "test_query286", + "ranked_ids": [ + "apositive_test_query286_00010", + "apositive_test_query286_00013", + "negative_test_query257_00005", + "apositive_test_query286_00012", + "negative_test_query286_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query133_00000", + "apositive_test_query287_00001", + "apositive_test_query133_00003", + "apositive_test_query287_00002", + "apositive_test_query133_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query288", + "ranked_ids": [ + "apositive_test_query141_00002", + "apositive_test_query141_00006", + "negative_test_query93_00000", + "negative_test_query199_00010", + "negative_test_query288_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00006", + "negative_test_query7_00006", + "negative_test_query334_00014", + "negative_test_query47_00009", + "apositive_test_query289_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.5 + }, + "query_id": "test_query29", + "ranked_ids": [ + "apositive_test_query29_00007", + "apositive_test_query29_00004", + "apositive_test_query29_00000", + "apositive_test_query29_00005", + "negative_test_query29_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query290", + "ranked_ids": [ + "apositive_test_query290_00001", + "negative_test_query98_00003", + "negative_test_query312_00001", + "negative_test_query237_00013", + "apositive_test_query290_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query291_00001", + "apositive_test_query242_00001", + "apositive_test_query247_00004", + "negative_test_query242_00000", + "apositive_test_query242_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query292", + "ranked_ids": [ + "apositive_test_query292_00000", + "negative_test_query292_00010", + "negative_test_query106_00007", + "negative_test_query292_00004", + "negative_test_query292_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query293", + "ranked_ids": [ + "apositive_test_query293_00002", + "apositive_test_query293_00001", + "negative_test_query293_00015", + "negative_test_query220_00007", + "apositive_test_query178_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00013", + "negative_test_query294_00008", + "negative_test_query156_00003", + "apositive_test_query177_00004", + "negative_test_query294_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query295", + "ranked_ids": [ + "negative_test_query295_00008", + "negative_test_query26_00002", + "apositive_test_query295_00001", + "negative_test_query295_00002", + "negative_test_query168_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query296", + "ranked_ids": [ + "apositive_test_query296_00002", + "apositive_test_query296_00003", + "negative_test_query340_00009", + "apositive_test_query296_00001", + "apositive_test_query296_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query117_00006", + "negative_test_query297_00012", + "negative_test_query297_00004", + "negative_test_query297_00010", + "apositive_test_query297_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "apositive_test_query298_00000", + "apositive_test_query298_00002", + "negative_test_query298_00006", + "negative_test_query216_00010", + "negative_test_query298_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query299_00005", + "negative_test_query299_00006", + "negative_test_query351_00009", + "apositive_test_query141_00002", + "apositive_test_query346_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query3", + "ranked_ids": [ + "apositive_test_query3_00000", + "apositive_test_query3_00001", + "negative_test_query3_00000", + "apositive_test_query3_00003", + "negative_test_query83_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query30", + "ranked_ids": [ + "apositive_test_query71_00000", + "apositive_test_query71_00002", + "apositive_test_query30_00004", + "negative_test_query231_00008", + "apositive_test_query30_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query300_00010", + "negative_test_query15_00000", + "negative_test_query252_00005", + "negative_test_query141_00001", + "negative_test_query300_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6366824387328317, + "recall@5": 0.5 + }, + "query_id": "test_query301", + "ranked_ids": [ + "apositive_test_query301_00001", + "apositive_test_query301_00000", + "negative_test_query301_00011", + "negative_test_query301_00015", + "negative_test_query301_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query302", + "ranked_ids": [ + "apositive_test_query324_00002", + "apositive_test_query308_00009", + "negative_test_query308_00003", + "apositive_test_query302_00001", + "apositive_test_query308_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query35_00003", + "apositive_test_query86_00001", + "negative_test_query8_00013", + "negative_test_query196_00007", + "negative_test_query274_00014" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.44149241373678083, + "recall@5": 0.5 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "apositive_test_query304_00001", + "negative_test_query43_00001", + "apositive_test_query285_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query305", + "ranked_ids": [ + "apositive_test_query127_00005", + "negative_test_query305_00014", + "negative_test_query305_00005", + "apositive_test_query305_00001", + "apositive_test_query259_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query306", + "ranked_ids": [ + "negative_test_query71_00013", + "apositive_test_query71_00002", + "apositive_test_query119_00006", + "negative_test_query305_00014", + "apositive_test_query209_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "negative_test_query251_00001", + "apositive_test_query279_00002", + "apositive_test_query177_00004", + "negative_test_query217_00003", + "negative_test_query132_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query324_00002", + "apositive_test_query308_00000", + "apositive_test_query341_00005", + "apositive_test_query7_00000", + "apositive_test_query310_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "negative_test_query123_00007", + "negative_test_query309_00003", + "apositive_test_query358_00006", + "apositive_test_query309_00000", + "negative_test_query309_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query31_00011", + "negative_test_query38_00003", + "negative_test_query31_00001", + "negative_test_query31_00003", + "apositive_test_query31_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.5 + }, + "query_id": "test_query310", + "ranked_ids": [ + "apositive_test_query310_00002", + "apositive_test_query310_00009", + "apositive_test_query310_00000", + "apositive_test_query310_00001", + "apositive_test_query310_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query311_00008", + "negative_test_query311_00005", + "negative_test_query240_00008", + "negative_test_query134_00001", + "negative_test_query218_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "apositive_test_query220_00005", + "apositive_test_query152_00004", + "apositive_test_query212_00004", + "negative_test_query283_00016", + "negative_test_query312_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query56_00001", + "negative_test_query336_00001", + "negative_test_query359_00005", + "negative_test_query46_00005", + "negative_test_query57_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query314_00016", + "apositive_test_query276_00001", + "apositive_test_query314_00012", + "apositive_test_query314_00001", + "apositive_test_query314_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query315_00000", + "apositive_test_query315_00002", + "negative_test_query269_00004", + "negative_test_query333_00007", + "apositive_test_query319_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3065735963827292, + "recall@5": 0.5 + }, + "query_id": "test_query316", + "ranked_ids": [ + "negative_test_query316_00002", + "negative_test_query301_00014", + "apositive_test_query316_00000", + "negative_test_query158_00008", + "negative_test_query322_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00008", + "negative_test_query317_00013", + "negative_test_query317_00004", + "apositive_test_query317_00000", + "negative_test_query331_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.787702056960637, + "recall@5": 0.75 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00000", + "apositive_test_query318_00002", + "negative_test_query322_00010", + "negative_test_query318_00015", + "apositive_test_query318_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "apositive_test_query319_00004", + "negative_test_query319_00008", + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query32_00002", + "negative_test_query196_00001", + "negative_test_query339_00006", + "apositive_test_query336_00001", + "negative_test_query83_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query320", + "ranked_ids": [ + "apositive_test_query320_00003", + "negative_test_query320_00004", + "negative_test_query320_00005", + "negative_test_query320_00008", + "negative_test_query91_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "apositive_test_query321_00000", + "negative_test_query321_00002", + "negative_test_query321_00010", + "negative_test_query197_00004", + "negative_test_query126_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.830419897363192, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query322", + "ranked_ids": [ + "apositive_test_query322_00005", + "apositive_test_query322_00002", + "negative_test_query322_00009", + "apositive_test_query322_00006", + "apositive_test_query322_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query323", + "ranked_ids": [ + "apositive_test_query130_00001", + "apositive_test_query323_00001", + "apositive_test_query323_00002", + "negative_test_query323_00006", + "negative_test_query15_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query324", + "ranked_ids": [ + "apositive_test_query308_00001", + "negative_test_query218_00003", + "negative_test_query324_00005", + "apositive_test_query308_00002", + "negative_test_query324_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query176_00003", + "apositive_test_query293_00002", + "negative_test_query325_00008", + "negative_test_query325_00007", + "negative_test_query325_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8539316501572937, + "recall@5": 0.4444444444444444 + }, + "query_id": "test_query326", + "ranked_ids": [ + "apositive_test_query326_00000", + "apositive_test_query326_00004", + "apositive_test_query326_00003", + "negative_test_query178_00012", + "apositive_test_query326_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query327", + "ranked_ids": [ + "negative_test_query303_00010", + "negative_test_query321_00014", + "apositive_test_query327_00000", + "negative_test_query61_00003", + "negative_test_query327_00014" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query328", + "ranked_ids": [ + "negative_test_query328_00002", + "apositive_test_query328_00001", + "apositive_test_query227_00001", + "negative_test_query328_00000", + "negative_test_query328_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query329", + "ranked_ids": [ + "negative_test_query62_00017", + "negative_test_query269_00000", + "negative_test_query329_00001", + "negative_test_query329_00004", + "negative_test_query329_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.4 + }, + "query_id": "test_query33", + "ranked_ids": [ + "apositive_test_query33_00006", + "negative_test_query154_00004", + "apositive_test_query33_00004", + "apositive_test_query33_00009", + "apositive_test_query33_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query330", + "ranked_ids": [ + "negative_test_query330_00009", + "negative_test_query330_00010", + "negative_test_query325_00016", + "negative_test_query27_00011", + "negative_test_query180_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.41618115554873086, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query331", + "ranked_ids": [ + "negative_test_query331_00000", + "negative_test_query331_00013", + "apositive_test_query331_00000", + "negative_test_query3_00008", + "apositive_test_query331_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00001", + "apositive_test_query332_00002", + "negative_test_query202_00010", + "apositive_test_query332_00003", + "negative_test_query202_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00002", + "negative_test_query333_00008", + "apositive_test_query322_00003", + "negative_test_query73_00014", + "negative_test_query216_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "apositive_test_query57_00006", + "negative_test_query111_00017", + "negative_test_query155_00013", + "negative_test_query107_00001", + "negative_test_query334_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.6 + }, + "query_id": "test_query335", + "ranked_ids": [ + "apositive_test_query335_00001", + "apositive_test_query335_00000", + "negative_test_query210_00005", + "apositive_test_query335_00003", + "negative_test_query45_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "negative_test_query336_00009", + "negative_test_query336_00004", + "negative_test_query336_00007", + "negative_test_query336_00014", + "apositive_test_query50_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00010", + "negative_test_query337_00001", + "negative_test_query38_00008", + "negative_test_query337_00004", + "negative_test_query159_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query338", + "ranked_ids": [ + "apositive_test_query338_00005", + "apositive_test_query338_00002", + "apositive_test_query338_00006", + "apositive_test_query338_00008", + "apositive_test_query338_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6240505200038379, + "recall@5": 1.0 + }, + "query_id": "test_query339", + "ranked_ids": [ + "negative_test_query339_00004", + "apositive_test_query339_00001", + "negative_test_query339_00009", + "negative_test_query339_00008", + "apositive_test_query339_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00000", + "negative_test_query352_00012", + "negative_test_query34_00014", + "apositive_test_query34_00000", + "negative_test_query34_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.4 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00001", + "negative_test_query340_00012", + "negative_test_query340_00008", + "apositive_test_query340_00000", + "apositive_test_query174_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query341", + "ranked_ids": [ + "apositive_test_query341_00005", + "negative_test_query214_00005", + "negative_test_query341_00005", + "negative_test_query196_00006", + "apositive_test_query32_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query68_00005", + "negative_test_query212_00009", + "negative_test_query342_00015", + "negative_test_query77_00002", + "negative_test_query134_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query343", + "ranked_ids": [ + "apositive_test_query343_00001", + "apositive_test_query343_00000", + "negative_test_query103_00010", + "apositive_test_query217_00002", + "apositive_test_query185_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query223_00016", + "apositive_test_query83_00006", + "negative_test_query282_00008", + "negative_test_query344_00005", + "negative_test_query83_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.2 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query345_00007", + "negative_test_query345_00003", + "negative_test_query47_00008", + "apositive_test_query345_00001", + "negative_test_query184_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query346_00001", + "negative_test_query346_00002", + "negative_test_query346_00009", + "negative_test_query346_00008", + "apositive_test_query346_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6713860725233041, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query347", + "ranked_ids": [ + "apositive_test_query347_00001", + "negative_test_query298_00003", + "negative_test_query251_00007", + "apositive_test_query347_00000", + "negative_test_query347_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query115_00002", + "apositive_test_query348_00004", + "negative_test_query31_00017", + "negative_test_query115_00012", + "apositive_test_query348_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "negative_test_query74_00001", + "negative_test_query253_00007", + "negative_test_query349_00015", + "apositive_test_query253_00005", + "apositive_test_query137_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query35_00003", + "negative_test_query350_00005", + "apositive_test_query139_00000", + "apositive_test_query35_00000", + "negative_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query350_00010", + "negative_test_query350_00014", + "negative_test_query350_00015", + "negative_test_query149_00005", + "negative_test_query350_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query351_00000", + "apositive_test_query299_00000", + "negative_test_query92_00005", + "apositive_test_query60_00009", + "apositive_test_query60_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.6 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query34_00001", + "apositive_test_query352_00002", + "apositive_test_query352_00003", + "apositive_test_query352_00004", + "negative_test_query34_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query353_00005", + "negative_test_query353_00003", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.625 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00001", + "apositive_test_query354_00003", + "apositive_test_query354_00000", + "apositive_test_query354_00004", + "apositive_test_query354_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query355_00001", + "negative_test_query355_00009", + "apositive_test_query355_00002", + "negative_test_query355_00007", + "apositive_test_query355_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query356_00000", + "apositive_test_query356_00002", + "apositive_test_query66_00003", + "apositive_test_query356_00004", + "apositive_test_query20_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "negative_test_query58_00009", + "apositive_test_query119_00002", + "apositive_test_query117_00004", + "apositive_test_query119_00012", + "apositive_test_query221_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2 + }, + "query_id": "test_query358", + "ranked_ids": [ + "apositive_test_query358_00007", + "negative_test_query42_00009", + "negative_test_query309_00015", + "negative_test_query123_00007", + "apositive_test_query358_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "negative_test_query359_00002", + "apositive_test_query0_00004", + "apositive_test_query0_00006", + "negative_test_query130_00000", + "negative_test_query305_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query36", + "ranked_ids": [ + "negative_test_query112_00004", + "apositive_test_query286_00006", + "negative_test_query112_00016", + "negative_test_query112_00015", + "apositive_test_query286_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00001", + "apositive_test_query360_00000", + "negative_test_query5_00014", + "apositive_test_query69_00014", + "negative_test_query246_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query231_00006", + "apositive_test_query213_00006", + "apositive_test_query37_00001", + "apositive_test_query108_00000", + "apositive_test_query177_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query38", + "ranked_ids": [ + "negative_test_query38_00010", + "apositive_test_query38_00003", + "negative_test_query94_00011", + "negative_test_query79_00002", + "negative_test_query267_00016" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query39", + "ranked_ids": [ + "apositive_test_query39_00001", + "negative_test_query39_00009", + "negative_test_query18_00004", + "apositive_test_query39_00003", + "apositive_test_query39_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.35714285714285715 + }, + "query_id": "test_query4", + "ranked_ids": [ + "apositive_test_query4_00010", + "apositive_test_query4_00003", + "apositive_test_query4_00004", + "apositive_test_query4_00000", + "apositive_test_query4_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00011", + "apositive_test_query40_00002", + "negative_test_query40_00006", + "negative_test_query40_00005", + "negative_test_query40_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5654495432396527, + "recall@5": 0.75 + }, + "query_id": "test_query41", + "ranked_ids": [ + "negative_test_query41_00000", + "apositive_test_query41_00002", + "negative_test_query41_00007", + "apositive_test_query41_00001", + "apositive_test_query41_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query42_00002", + "negative_test_query42_00000", + "negative_test_query42_00003", + "apositive_test_query42_00003", + "negative_test_query42_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query43_00000", + "negative_test_query43_00009", + "apositive_test_query43_00002", + "negative_test_query333_00001", + "apositive_test_query161_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query44", + "ranked_ids": [ + "negative_test_query27_00011", + "negative_test_query27_00002", + "apositive_test_query44_00001", + "negative_test_query27_00008", + "negative_test_query141_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query45", + "ranked_ids": [ + "apositive_test_query45_00000", + "negative_test_query45_00013", + "negative_test_query45_00001", + "apositive_test_query165_00000", + "negative_test_query45_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "negative_test_query46_00011", + "negative_test_query56_00001", + "apositive_test_query79_00005", + "apositive_test_query186_00001", + "apositive_test_query276_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query331_00002", + "negative_test_query331_00013", + "apositive_test_query201_00002", + "negative_test_query47_00013", + "apositive_test_query201_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "apositive_test_query48_00000", + "apositive_test_query185_00012", + "negative_test_query48_00007", + "negative_test_query307_00005", + "apositive_test_query343_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query49", + "ranked_ids": [ + "apositive_test_query49_00001", + "negative_test_query122_00016", + "negative_test_query169_00000", + "apositive_test_query29_00006", + "negative_test_query106_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.7122630665145961, + "recall@5": 1.0 + }, + "query_id": "test_query5", + "ranked_ids": [ + "negative_test_query67_00010", + "apositive_test_query5_00002", + "apositive_test_query5_00000", + "negative_test_query246_00013", + "apositive_test_query5_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.29411764705882354 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00014", + "apositive_test_query50_00009", + "apositive_test_query50_00005", + "apositive_test_query50_00006", + "apositive_test_query50_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.3 + }, + "query_id": "test_query51", + "ranked_ids": [ + "apositive_test_query51_00000", + "apositive_test_query51_00006", + "apositive_test_query51_00007", + "apositive_test_query267_00001", + "negative_test_query231_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "apositive_test_query257_00004", + "apositive_test_query257_00001", + "negative_test_query54_00004", + "negative_test_query215_00000", + "negative_test_query257_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00016", + "negative_test_query169_00003", + "negative_test_query258_00008", + "negative_test_query258_00018", + "negative_test_query180_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query140_00006", + "apositive_test_query51_00006", + "negative_test_query54_00006", + "negative_test_query44_00014", + "apositive_test_query335_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7653606369886217, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query55", + "ranked_ids": [ + "apositive_test_query55_00000", + "apositive_test_query55_00001", + "negative_test_query55_00005", + "negative_test_query55_00016", + "negative_test_query55_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query56", + "ranked_ids": [ + "apositive_test_query56_00005", + "negative_test_query102_00001", + "negative_test_query105_00006", + "apositive_test_query46_00001", + "negative_test_query56_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query57", + "ranked_ids": [ + "apositive_test_query57_00003", + "negative_test_query295_00003", + "apositive_test_query74_00000", + "apositive_test_query57_00005", + "apositive_test_query57_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "negative_test_query134_00001", + "negative_test_query58_00010", + "negative_test_query240_00016", + "negative_test_query240_00015", + "negative_test_query11_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "negative_test_query59_00012", + "negative_test_query59_00010", + "negative_test_query116_00003", + "apositive_test_query59_00000", + "negative_test_query59_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query6", + "ranked_ids": [ + "apositive_test_query6_00002", + "negative_test_query6_00002", + "negative_test_query296_00006", + "negative_test_query112_00004", + "negative_test_query6_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.1875 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00007", + "apositive_test_query60_00003", + "negative_test_query31_00004", + "negative_test_query218_00007", + "apositive_test_query60_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7365896932159578, + "recall@5": 0.75 + }, + "query_id": "test_query61", + "ranked_ids": [ + "apositive_test_query61_00002", + "negative_test_query61_00015", + "apositive_test_query61_00003", + "negative_test_query173_00003", + "apositive_test_query61_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00000", + "negative_test_query62_00004", + "negative_test_query250_00003", + "negative_test_query62_00002", + "negative_test_query62_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00001", + "apositive_test_query141_00001", + "negative_test_query63_00002", + "apositive_test_query63_00008", + "negative_test_query321_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query64_00013", + "negative_test_query205_00009", + "negative_test_query214_00004", + "negative_test_query211_00009", + "negative_test_query64_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query65", + "ranked_ids": [ + "apositive_test_query65_00001", + "negative_test_query65_00004", + "apositive_test_query134_00001", + "negative_test_query195_00009", + "negative_test_query335_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query66", + "ranked_ids": [ + "apositive_test_query66_00004", + "negative_test_query220_00010", + "apositive_test_query66_00003", + "apositive_test_query220_00003", + "negative_test_query154_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query67", + "ranked_ids": [ + "apositive_test_query67_00000", + "negative_test_query67_00005", + "negative_test_query67_00011", + "negative_test_query67_00008", + "negative_test_query246_00009" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.38364913903790027, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query68", + "ranked_ids": [ + "negative_test_query126_00008", + "negative_test_query68_00001", + "negative_test_query330_00011", + "apositive_test_query68_00002", + "apositive_test_query68_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.21052631578947367 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00010", + "apositive_test_query69_00000", + "apositive_test_query69_00011", + "apositive_test_query69_00002", + "apositive_test_query225_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "negative_test_query306_00008", + "apositive_test_query119_00006", + "apositive_test_query212_00002", + "apositive_test_query212_00001", + "negative_test_query84_00007" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00004", + "apositive_test_query70_00002", + "negative_test_query70_00011", + "negative_test_query227_00003", + "negative_test_query227_00012" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query71", + "ranked_ids": [ + "negative_test_query71_00013", + "apositive_test_query71_00003", + "negative_test_query71_00009", + "apositive_test_query71_00000", + "negative_test_query247_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "apositive_test_query72_00000", + "negative_test_query72_00015", + "negative_test_query72_00011", + "negative_test_query185_00000", + "negative_test_query72_00016" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query73", + "ranked_ids": [ + "negative_test_query322_00009", + "apositive_test_query322_00003", + "negative_test_query73_00008", + "apositive_test_query333_00002", + "negative_test_query73_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.3973220069685352, + "recall@5": 0.5 + }, + "query_id": "test_query74", + "ranked_ids": [ + "negative_test_query74_00000", + "apositive_test_query74_00000", + "negative_test_query74_00013", + "negative_test_query74_00015", + "apositive_test_query74_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query75", + "ranked_ids": [ + "negative_test_query130_00012", + "apositive_test_query75_00005", + "apositive_test_query75_00000", + "apositive_test_query75_00001", + "negative_test_query260_00013" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query76", + "ranked_ids": [ + "apositive_test_query76_00006", + "negative_test_query76_00010", + "apositive_test_query76_00000", + "negative_test_query76_00011", + "apositive_test_query76_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query98_00014", + "negative_test_query77_00013", + "negative_test_query87_00016", + "negative_test_query47_00010", + "negative_test_query329_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5307212739772434, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query78", + "ranked_ids": [ + "negative_test_query290_00013", + "apositive_test_query78_00001", + "apositive_test_query78_00000", + "negative_test_query214_00010", + "negative_test_query290_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query79", + "ranked_ids": [ + "apositive_test_query221_00004", + "apositive_test_query79_00000", + "apositive_test_query79_00004", + "apositive_test_query119_00005", + "apositive_test_query295_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "negative_test_query196_00007", + "negative_test_query8_00013", + "apositive_test_query192_00002", + "negative_test_query8_00014", + "apositive_test_query192_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query80_00004", + "apositive_test_query80_00002", + "apositive_test_query179_00001", + "negative_test_query179_00005", + "negative_test_query80_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query81", + "ranked_ids": [ + "apositive_test_query81_00000", + "negative_test_query81_00007", + "negative_test_query81_00005", + "apositive_test_query81_00001", + "apositive_test_query81_00004" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "apositive_test_query319_00004", + "negative_test_query319_00008", + "apositive_test_query319_00000", + "apositive_test_query82_00000", + "negative_test_query319_00010" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6992148198508501, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query83_00002", + "apositive_test_query83_00004", + "apositive_test_query229_00001", + "apositive_test_query83_00003", + "apositive_test_query168_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6934264036172708, + "recall@5": 1.0 + }, + "query_id": "test_query84", + "ranked_ids": [ + "negative_test_query84_00013", + "apositive_test_query84_00001", + "apositive_test_query84_00000", + "negative_test_query84_00012", + "negative_test_query84_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7365896932159578, + "recall@5": 0.75 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query85_00000", + "negative_test_query85_00012", + "apositive_test_query85_00002", + "apositive_test_query109_00004", + "apositive_test_query85_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.9047172294870751, + "recall@5": 1.0 + }, + "query_id": "test_query86", + "ranked_ids": [ + "apositive_test_query86_00003", + "negative_test_query86_00013", + "apositive_test_query86_00000", + "apositive_test_query86_00002", + "apositive_test_query86_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query87", + "ranked_ids": [ + "apositive_test_query87_00001", + "apositive_test_query87_00000", + "negative_test_query263_00002", + "negative_test_query87_00006", + "negative_test_query200_00011" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "negative_test_query340_00012", + "negative_test_query88_00012", + "negative_test_query203_00000", + "negative_test_query88_00002", + "apositive_test_query88_00001" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query89", + "ranked_ids": [ + "apositive_test_query89_00004", + "negative_test_query89_00008", + "negative_test_query110_00011", + "negative_test_query89_00004", + "negative_test_query89_00005" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query221_00002", + "negative_test_query90_00004", + "negative_test_query9_00003", + "apositive_test_query90_00001", + "negative_test_query9_00008" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query231_00002", + "negative_test_query310_00007", + "negative_test_query90_00010", + "apositive_test_query14_00002", + "negative_test_query227_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query91", + "ranked_ids": [ + "apositive_test_query91_00006", + "apositive_test_query91_00000", + "negative_test_query292_00009", + "negative_test_query91_00010", + "apositive_test_query91_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6164336326286644, + "recall@5": 0.5 + }, + "query_id": "test_query92", + "ranked_ids": [ + "apositive_test_query92_00003", + "negative_test_query100_00014", + "negative_test_query92_00003", + "apositive_test_query92_00005", + "apositive_test_query92_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.42857142857142855 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query93_00004", + "apositive_test_query93_00003", + "apositive_test_query93_00000", + "negative_test_query93_00003", + "apositive_test_query335_00003" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00015", + "negative_test_query94_00003", + "apositive_test_query127_00001", + "negative_test_query155_00008", + "apositive_test_query127_00006" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "apositive_test_query95_00001", + "apositive_test_query95_00000", + "negative_test_query95_00003", + "apositive_test_query253_00007", + "negative_test_query12_00015" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query339_00000", + "negative_test_query96_00013", + "negative_test_query339_00006", + "negative_test_query96_00015", + "apositive_test_query96_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6548086577531307, + "recall@5": 0.5 + }, + "query_id": "test_query97", + "ranked_ids": [ + "apositive_test_query97_00002", + "negative_test_query51_00009", + "apositive_test_query97_00005", + "apositive_test_query97_00001", + "apositive_test_query267_00000" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00016", + "negative_test_query98_00017", + "apositive_test_query342_00000", + "apositive_test_query14_00002", + "negative_test_query345_00002" + ], + "valid": true + }, + { + "algorithm": "setwise_hs_s10", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query99", + "ranked_ids": [ + "negative_test_query330_00008", + "negative_test_query193_00002", + "negative_test_query27_00011", + "apositive_test_query193_00008", + "apositive_test_query193_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query0", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.8687949224876582, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query0", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00006", + "apositive_test_query0_00003", + "negative_test_query359_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query1", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query1", + "ranked_ids": [ + "negative_test_query245_00006", + "apositive_test_query348_00003", + "negative_test_query120_00006", + "negative_test_query61_00002", + "negative_test_query27_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query10", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4776237035032179, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query10", + "ranked_ids": [ + "negative_test_query10_00010", + "apositive_test_query10_00001", + "negative_test_query104_00011", + "negative_test_query10_00000", + "apositive_test_query10_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query100", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query100", + "ranked_ids": [ + "negative_test_query100_00010", + "negative_test_query290_00015", + "negative_test_query100_00009", + "apositive_test_query181_00000", + "negative_test_query100_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query101", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query101", + "ranked_ids": [ + "apositive_test_query196_00000", + "apositive_test_query336_00001", + "negative_test_query61_00002", + "negative_test_query173_00013", + "negative_test_query241_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query102", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query102", + "ranked_ids": [ + "negative_test_query209_00006", + "negative_test_query108_00009", + "negative_test_query219_00004", + "negative_test_query131_00009", + "negative_test_query205_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query103", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query103", + "ranked_ids": [ + "negative_test_query103_00005", + "apositive_test_query103_00005", + "negative_test_query185_00003", + "negative_test_query307_00004", + "negative_test_query307_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query104", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query104", + "ranked_ids": [ + "negative_test_query104_00010", + "negative_test_query104_00016", + "negative_test_query48_00010", + "negative_test_query104_00004", + "apositive_test_query48_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query105", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query105", + "ranked_ids": [ + "negative_test_query313_00013", + "negative_test_query105_00005", + "negative_test_query267_00011", + "negative_test_query119_00005", + "negative_test_query168_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query106", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query106", + "ranked_ids": [ + "negative_test_query326_00008", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "apositive_test_query42_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query107", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query107", + "ranked_ids": [ + "negative_test_query146_00000", + "negative_test_query296_00001", + "negative_test_query31_00013", + "apositive_test_query142_00002", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query108", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.125 + }, + "query_id": "test_query108", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query108_00005", + "negative_test_query57_00009", + "negative_test_query311_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query109", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query109", + "ranked_ids": [ + "negative_test_query109_00003", + "negative_test_query6_00006", + "negative_test_query109_00000", + "negative_test_query51_00004", + "negative_test_query109_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query11", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query11", + "ranked_ids": [ + "apositive_test_query11_00001", + "negative_test_query226_00010", + "negative_test_query309_00002", + "negative_test_query358_00005", + "negative_test_query11_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query110", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query110", + "ranked_ids": [ + "apositive_test_query11_00002", + "negative_test_query110_00014", + "apositive_test_query330_00001", + "negative_test_query17_00008", + "negative_test_query110_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query111", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query111", + "ranked_ids": [ + "negative_test_query243_00001", + "negative_test_query277_00010", + "negative_test_query111_00018", + "negative_test_query292_00010", + "negative_test_query277_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query112", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query112", + "ranked_ids": [ + "apositive_test_query286_00005", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query112_00015", + "apositive_test_query106_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query113", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query113", + "ranked_ids": [ + "negative_test_query38_00003", + "negative_test_query113_00003", + "apositive_test_query113_00001", + "negative_test_query105_00000", + "apositive_test_query186_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query114", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query114", + "ranked_ids": [ + "negative_test_query262_00007", + "apositive_test_query114_00004", + "negative_test_query80_00008", + "negative_test_query106_00009", + "negative_test_query114_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query115", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query115", + "ranked_ids": [ + "negative_test_query205_00004", + "apositive_test_query115_00000", + "apositive_test_query121_00001", + "apositive_test_query115_00002", + "negative_test_query139_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query116", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query116", + "ranked_ids": [ + "negative_test_query116_00000", + "apositive_test_query116_00000", + "negative_test_query59_00007", + "negative_test_query116_00007", + "apositive_test_query49_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query117", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query117", + "ranked_ids": [ + "apositive_test_query117_00003", + "negative_test_query63_00001", + "apositive_test_query117_00007", + "negative_test_query117_00003", + "negative_test_query175_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query118", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query118", + "ranked_ids": [ + "apositive_test_query340_00002", + "negative_test_query88_00007", + "apositive_test_query174_00003", + "apositive_test_query174_00005", + "apositive_test_query174_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query119", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query119", + "ranked_ids": [ + "negative_test_query119_00005", + "apositive_test_query79_00004", + "negative_test_query157_00007", + "negative_test_query359_00005", + "negative_test_query152_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query12", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query12", + "ranked_ids": [ + "negative_test_query234_00014", + "negative_test_query12_00003", + "negative_test_query12_00014", + "negative_test_query12_00007", + "negative_test_query12_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query120", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query120", + "ranked_ids": [ + "negative_test_query120_00008", + "apositive_test_query352_00000", + "negative_test_query303_00018", + "negative_test_query120_00002", + "negative_test_query120_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query121", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query121", + "ranked_ids": [ + "apositive_test_query121_00001", + "negative_test_query121_00012", + "negative_test_query121_00007", + "apositive_test_query121_00000", + "negative_test_query115_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query122", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query122", + "ranked_ids": [ + "negative_test_query64_00017", + "negative_test_query86_00001", + "apositive_test_query173_00001", + "negative_test_query134_00007", + "apositive_test_query122_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query123", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query123", + "ranked_ids": [ + "apositive_test_query123_00002", + "negative_test_query123_00002", + "negative_test_query123_00008", + "negative_test_query123_00006", + "negative_test_query269_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query124", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query124", + "ranked_ids": [ + "negative_test_query124_00003", + "negative_test_query42_00006", + "apositive_test_query13_00004", + "negative_test_query93_00010", + "apositive_test_query162_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query125", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query125", + "ranked_ids": [ + "negative_test_query301_00012", + "apositive_test_query286_00009", + "negative_test_query125_00013", + "negative_test_query125_00014", + "negative_test_query140_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query126", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query126", + "ranked_ids": [ + "negative_test_query131_00009", + "apositive_test_query330_00001", + "apositive_test_query5_00002", + "apositive_test_query151_00003", + "negative_test_query126_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query127", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query127", + "ranked_ids": [ + "apositive_test_query127_00009", + "negative_test_query267_00009", + "apositive_test_query127_00012", + "apositive_test_query94_00005", + "apositive_test_query127_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query128", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query128", + "ranked_ids": [ + "apositive_test_query128_00001", + "negative_test_query75_00012", + "apositive_test_query216_00002", + "negative_test_query128_00011", + "apositive_test_query21_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query129", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.125 + }, + "query_id": "test_query129", + "ranked_ids": [ + "negative_test_query63_00006", + "apositive_test_query249_00003", + "negative_test_query16_00007", + "negative_test_query187_00012", + "apositive_test_query129_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query13", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.23076923076923078 + }, + "query_id": "test_query13", + "ranked_ids": [ + "negative_test_query13_00004", + "apositive_test_query13_00010", + "apositive_test_query13_00008", + "apositive_test_query13_00012", + "negative_test_query13_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query130", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query130", + "ranked_ids": [ + "negative_test_query90_00014", + "negative_test_query130_00001", + "negative_test_query289_00004", + "negative_test_query198_00003", + "negative_test_query284_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query131", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query131", + "ranked_ids": [ + "apositive_test_query131_00001", + "negative_test_query131_00002", + "negative_test_query313_00008", + "negative_test_query168_00013", + "negative_test_query184_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query132", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query132", + "ranked_ids": [ + "apositive_test_query3_00004", + "negative_test_query132_00012", + "negative_test_query132_00018", + "negative_test_query81_00005", + "negative_test_query304_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query133", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query133", + "ranked_ids": [ + "apositive_test_query129_00002", + "negative_test_query12_00014", + "apositive_test_query129_00007", + "apositive_test_query174_00011", + "negative_test_query206_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query134", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query134", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query108_00003", + "apositive_test_query238_00001", + "apositive_test_query276_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query135", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query135", + "ranked_ids": [ + "negative_test_query135_00001", + "negative_test_query297_00017", + "negative_test_query135_00007", + "negative_test_query135_00008", + "negative_test_query135_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query136", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query136", + "ranked_ids": [ + "apositive_test_query136_00002", + "apositive_test_query136_00011", + "apositive_test_query136_00000", + "apositive_test_query136_00010", + "apositive_test_query136_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query137", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query137", + "ranked_ids": [ + "negative_test_query149_00013", + "apositive_test_query137_00002", + "negative_test_query74_00001", + "apositive_test_query137_00000", + "negative_test_query262_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query138", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query138", + "ranked_ids": [ + "negative_test_query138_00015", + "apositive_test_query244_00005", + "negative_test_query138_00005", + "negative_test_query138_00014", + "negative_test_query138_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query139", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query139", + "ranked_ids": [ + "apositive_test_query139_00001", + "negative_test_query262_00002", + "negative_test_query198_00011", + "negative_test_query101_00006", + "apositive_test_query354_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query14", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query14", + "ranked_ids": [ + "apositive_test_query64_00000", + "apositive_test_query26_00006", + "apositive_test_query296_00005", + "negative_test_query14_00001", + "negative_test_query179_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query140", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query140", + "ranked_ids": [ + "negative_test_query140_00009", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00003", + "negative_test_query140_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query141", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query141", + "ranked_ids": [ + "negative_test_query245_00008", + "negative_test_query245_00016", + "negative_test_query308_00005", + "negative_test_query57_00002", + "apositive_test_query288_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query142", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query142", + "ranked_ids": [ + "negative_test_query200_00014", + "negative_test_query337_00016", + "negative_test_query142_00007", + "apositive_test_query142_00002", + "negative_test_query105_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query143", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query143", + "ranked_ids": [ + "negative_test_query90_00014", + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query143_00011", + "negative_test_query143_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query144", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query144", + "ranked_ids": [ + "negative_test_query166_00001", + "negative_test_query144_00012", + "negative_test_query144_00006", + "negative_test_query144_00001", + "negative_test_query144_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query145", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query145", + "ranked_ids": [ + "negative_test_query211_00013", + "negative_test_query230_00002", + "negative_test_query53_00012", + "negative_test_query112_00007", + "negative_test_query187_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query146", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.0625 + }, + "query_id": "test_query146", + "ranked_ids": [ + "apositive_test_query146_00004", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "negative_test_query146_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query147", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query147", + "ranked_ids": [ + "negative_test_query17_00009", + "negative_test_query147_00013", + "negative_test_query147_00008", + "negative_test_query309_00014", + "negative_test_query200_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query148", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query148", + "ranked_ids": [ + "negative_test_query210_00006", + "negative_test_query321_00013", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query148_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query149", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query149", + "ranked_ids": [ + "negative_test_query149_00001", + "negative_test_query149_00013", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "negative_test_query149_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query15", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query15", + "ranked_ids": [ + "negative_test_query15_00014", + "negative_test_query15_00007", + "negative_test_query15_00004", + "negative_test_query325_00011", + "negative_test_query15_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query150", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query150", + "ranked_ids": [ + "negative_test_query301_00007", + "negative_test_query346_00008", + "negative_test_query139_00005", + "negative_test_query254_00003", + "negative_test_query350_00018" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query151", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query151", + "ranked_ids": [ + "negative_test_query190_00014", + "apositive_test_query151_00003", + "negative_test_query126_00006", + "negative_test_query151_00004", + "negative_test_query293_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query152", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query152", + "ranked_ids": [ + "negative_test_query349_00001", + "negative_test_query262_00010", + "negative_test_query342_00005", + "negative_test_query237_00011", + "negative_test_query349_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query153", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.125 + }, + "query_id": "test_query153", + "ranked_ids": [ + "apositive_test_query153_00001", + "apositive_test_query153_00003", + "negative_test_query278_00006", + "negative_test_query112_00007", + "negative_test_query112_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query154", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query154", + "ranked_ids": [ + "negative_test_query245_00006", + "apositive_test_query33_00008", + "apositive_test_query20_00003", + "apositive_test_query358_00008", + "negative_test_query68_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query155", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query155", + "ranked_ids": [ + "negative_test_query155_00006", + "negative_test_query155_00005", + "negative_test_query99_00001", + "apositive_test_query155_00005", + "negative_test_query155_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query156", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query156", + "ranked_ids": [ + "negative_test_query49_00005", + "apositive_test_query210_00001", + "negative_test_query283_00009", + "apositive_test_query156_00000", + "negative_test_query156_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query157", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query157", + "ranked_ids": [ + "negative_test_query271_00003", + "apositive_test_query237_00004", + "apositive_test_query152_00004", + "negative_test_query237_00000", + "negative_test_query152_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query158", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.2 + }, + "query_id": "test_query158", + "ranked_ids": [ + "apositive_test_query158_00000", + "negative_test_query26_00005", + "negative_test_query26_00010", + "apositive_test_query158_00007", + "negative_test_query102_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query159", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query159", + "ranked_ids": [ + "negative_test_query264_00013", + "apositive_test_query38_00003", + "negative_test_query159_00016", + "apositive_test_query212_00003", + "negative_test_query122_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query16", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query16", + "ranked_ids": [ + "negative_test_query16_00009", + "negative_test_query308_00004", + "negative_test_query16_00011", + "negative_test_query16_00008", + "negative_test_query16_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query160", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07142857142857142 + }, + "query_id": "test_query160", + "ranked_ids": [ + "apositive_test_query64_00000", + "negative_test_query271_00010", + "negative_test_query304_00007", + "negative_test_query330_00005", + "apositive_test_query160_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query161", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query161", + "ranked_ids": [ + "apositive_test_query3_00006", + "negative_test_query260_00004", + "apositive_test_query314_00004", + "apositive_test_query161_00008", + "apositive_test_query301_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query162", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.08333333333333333 + }, + "query_id": "test_query162", + "ranked_ids": [ + "apositive_test_query194_00009", + "negative_test_query162_00001", + "negative_test_query343_00012", + "negative_test_query177_00002", + "apositive_test_query162_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query163", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query163", + "ranked_ids": [ + "negative_test_query58_00015", + "negative_test_query197_00002", + "negative_test_query265_00015", + "negative_test_query163_00011", + "negative_test_query202_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query164", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query164", + "ranked_ids": [ + "apositive_test_query146_00014", + "negative_test_query164_00015", + "negative_test_query245_00005", + "negative_test_query321_00011", + "negative_test_query100_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query165", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query165", + "ranked_ids": [ + "negative_test_query327_00016", + "negative_test_query165_00007", + "negative_test_query86_00006", + "negative_test_query264_00016", + "negative_test_query165_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query166", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query166", + "ranked_ids": [ + "apositive_test_query224_00000", + "apositive_test_query166_00002", + "negative_test_query166_00001", + "negative_test_query165_00015", + "negative_test_query166_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query167", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query167", + "ranked_ids": [ + "negative_test_query278_00010", + "apositive_test_query288_00005", + "negative_test_query141_00010", + "negative_test_query167_00008", + "negative_test_query167_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query168", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5585075862632192, + "recall@5": 0.5 + }, + "query_id": "test_query168", + "ranked_ids": [ + "apositive_test_query168_00000", + "negative_test_query168_00007", + "negative_test_query260_00011", + "apositive_test_query168_00001", + "negative_test_query168_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query169", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query169", + "ranked_ids": [ + "negative_test_query10_00013", + "negative_test_query72_00013", + "negative_test_query49_00017", + "apositive_test_query330_00001", + "apositive_test_query69_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query17", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query17", + "ranked_ids": [ + "negative_test_query17_00016", + "negative_test_query17_00005", + "negative_test_query17_00014", + "negative_test_query164_00006", + "negative_test_query17_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query170", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query170", + "ranked_ids": [ + "negative_test_query170_00001", + "negative_test_query227_00010", + "negative_test_query227_00002", + "apositive_test_query170_00000", + "apositive_test_query297_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query171", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query171", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query277_00007", + "negative_test_query346_00005", + "negative_test_query24_00007", + "negative_test_query171_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query172", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query172", + "ranked_ids": [ + "apositive_test_query124_00002", + "negative_test_query349_00001", + "negative_test_query173_00006", + "negative_test_query8_00000", + "negative_test_query350_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query173", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query173", + "ranked_ids": [ + "negative_test_query64_00017", + "apositive_test_query173_00004", + "apositive_test_query173_00001", + "negative_test_query173_00006", + "negative_test_query222_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query174", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query174", + "ranked_ids": [ + "apositive_test_query174_00003", + "negative_test_query19_00012", + "apositive_test_query174_00005", + "apositive_test_query174_00007", + "apositive_test_query174_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query175", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query175", + "ranked_ids": [ + "apositive_test_query175_00000", + "negative_test_query204_00000", + "apositive_test_query50_00015", + "negative_test_query175_00013", + "negative_test_query175_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query176", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query176", + "ranked_ids": [ + "negative_test_query144_00018", + "apositive_test_query92_00002", + "negative_test_query24_00014", + "negative_test_query195_00006", + "negative_test_query284_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query177", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query177", + "ranked_ids": [ + "negative_test_query124_00001", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "negative_test_query177_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query178", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query178", + "ranked_ids": [ + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query274_00001", + "negative_test_query321_00017", + "negative_test_query178_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query179", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query179", + "ranked_ids": [ + "apositive_test_query316_00001", + "negative_test_query191_00003", + "negative_test_query297_00000", + "negative_test_query206_00004", + "negative_test_query241_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query18", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query18", + "ranked_ids": [ + "negative_test_query88_00002", + "negative_test_query215_00000", + "negative_test_query257_00003", + "negative_test_query140_00003", + "negative_test_query18_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query180", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query180", + "ranked_ids": [ + "negative_test_query246_00003", + "negative_test_query207_00007", + "negative_test_query180_00002", + "negative_test_query180_00007", + "apositive_test_query262_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query181", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query181", + "ranked_ids": [ + "negative_test_query181_00005", + "negative_test_query181_00010", + "negative_test_query348_00007", + "apositive_test_query119_00011", + "negative_test_query181_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query182", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query182", + "ranked_ids": [ + "negative_test_query182_00011", + "negative_test_query182_00018", + "negative_test_query342_00004", + "negative_test_query237_00006", + "negative_test_query182_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query183", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query183", + "ranked_ids": [ + "negative_test_query183_00008", + "negative_test_query7_00009", + "apositive_test_query7_00001", + "negative_test_query183_00010", + "negative_test_query7_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query184", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query184", + "ranked_ids": [ + "apositive_test_query33_00004", + "negative_test_query156_00001", + "negative_test_query216_00008", + "negative_test_query216_00002", + "apositive_test_query316_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query185", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query185", + "ranked_ids": [ + "negative_test_query350_00007", + "negative_test_query185_00002", + "apositive_test_query307_00008", + "apositive_test_query103_00000", + "negative_test_query48_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query186", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query186", + "ranked_ids": [ + "apositive_test_query79_00005", + "negative_test_query105_00001", + "negative_test_query29_00003", + "negative_test_query313_00005", + "apositive_test_query50_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query187", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query187", + "ranked_ids": [ + "negative_test_query245_00006", + "apositive_test_query33_00008", + "negative_test_query13_00005", + "negative_test_query245_00015", + "apositive_test_query358_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query188", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.2 + }, + "query_id": "test_query188", + "ranked_ids": [ + "negative_test_query112_00017", + "negative_test_query20_00006", + "apositive_test_query20_00009", + "negative_test_query344_00007", + "apositive_test_query188_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query189", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query189", + "ranked_ids": [ + "negative_test_query189_00004", + "apositive_test_query189_00002", + "negative_test_query189_00002", + "negative_test_query55_00003", + "negative_test_query140_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query19", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query19", + "ranked_ids": [ + "negative_test_query106_00007", + "negative_test_query292_00004", + "apositive_test_query174_00007", + "negative_test_query229_00002", + "negative_test_query141_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query190", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query190", + "ranked_ids": [ + "apositive_test_query234_00001", + "negative_test_query190_00013", + "negative_test_query233_00008", + "negative_test_query265_00004", + "negative_test_query205_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query191", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query191", + "ranked_ids": [ + "apositive_test_query191_00000", + "negative_test_query191_00003", + "negative_test_query274_00002", + "negative_test_query241_00007", + "apositive_test_query191_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query192", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.4 + }, + "query_id": "test_query192", + "ranked_ids": [ + "apositive_test_query192_00002", + "negative_test_query192_00002", + "apositive_test_query192_00004", + "negative_test_query120_00007", + "negative_test_query350_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query193", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.15384615384615385 + }, + "query_id": "test_query193", + "ranked_ids": [ + "negative_test_query193_00006", + "apositive_test_query193_00001", + "apositive_test_query99_00003", + "apositive_test_query193_00009", + "negative_test_query146_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query194", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query194", + "ranked_ids": [ + "negative_test_query262_00007", + "negative_test_query177_00007", + "negative_test_query95_00016", + "apositive_test_query194_00001", + "apositive_test_query194_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query195", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query195", + "ranked_ids": [ + "apositive_test_query195_00002", + "apositive_test_query251_00000", + "negative_test_query109_00008", + "negative_test_query199_00003", + "negative_test_query195_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query196", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query196", + "ranked_ids": [ + "apositive_test_query32_00004", + "apositive_test_query196_00003", + "apositive_test_query32_00011", + "negative_test_query139_00011", + "negative_test_query173_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query197", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query197", + "ranked_ids": [ + "apositive_test_query197_00000", + "apositive_test_query189_00001", + "negative_test_query163_00001", + "apositive_test_query34_00000", + "negative_test_query163_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query198", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query198", + "ranked_ids": [ + "negative_test_query293_00011", + "negative_test_query198_00004", + "negative_test_query239_00007", + "negative_test_query293_00010", + "negative_test_query198_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query199", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query199", + "ranked_ids": [ + "negative_test_query350_00007", + "negative_test_query199_00006", + "negative_test_query185_00002", + "negative_test_query199_00002", + "negative_test_query221_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query2", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query2", + "ranked_ids": [ + "negative_test_query44_00008", + "negative_test_query2_00012", + "negative_test_query197_00004", + "negative_test_query348_00005", + "negative_test_query104_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query20", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38356636737133565, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query20", + "ranked_ids": [ + "apositive_test_query33_00008", + "apositive_test_query20_00009", + "apositive_test_query20_00003", + "negative_test_query66_00002", + "apositive_test_query66_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query200", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query200", + "ranked_ids": [ + "negative_test_query200_00015", + "negative_test_query38_00005", + "negative_test_query264_00007", + "negative_test_query264_00006", + "negative_test_query200_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query201", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query201", + "ranked_ids": [ + "negative_test_query71_00002", + "negative_test_query0_00004", + "negative_test_query47_00013", + "negative_test_query209_00008", + "apositive_test_query30_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query202", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query202", + "ranked_ids": [ + "negative_test_query235_00001", + "apositive_test_query61_00003", + "apositive_test_query206_00000", + "negative_test_query202_00004", + "apositive_test_query309_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query203", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query203", + "ranked_ids": [ + "negative_test_query130_00001", + "apositive_test_query30_00005", + "apositive_test_query284_00003", + "negative_test_query35_00004", + "negative_test_query203_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query204", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.11764705882352941 + }, + "query_id": "test_query204", + "ranked_ids": [ + "negative_test_query204_00000", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00008", + "negative_test_query353_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query205", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query205", + "ranked_ids": [ + "negative_test_query31_00015", + "negative_test_query205_00007", + "negative_test_query115_00000", + "apositive_test_query256_00002", + "negative_test_query205_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query206", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query206", + "ranked_ids": [ + "negative_test_query64_00016", + "negative_test_query206_00001", + "negative_test_query206_00018", + "negative_test_query206_00003", + "negative_test_query299_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query207", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query207", + "ranked_ids": [ + "apositive_test_query207_00005", + "negative_test_query207_00007", + "negative_test_query207_00000", + "negative_test_query207_00003", + "negative_test_query207_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query208", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query208", + "ranked_ids": [ + "negative_test_query128_00001", + "negative_test_query73_00008", + "apositive_test_query128_00002", + "negative_test_query128_00003", + "negative_test_query208_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query209", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query209", + "ranked_ids": [ + "negative_test_query313_00012", + "apositive_test_query119_00008", + "negative_test_query102_00016", + "apositive_test_query158_00004", + "negative_test_query26_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query21", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query21", + "ranked_ids": [ + "negative_test_query21_00009", + "negative_test_query330_00000", + "negative_test_query234_00006", + "apositive_test_query128_00003", + "apositive_test_query128_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query210", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query210", + "ranked_ids": [ + "negative_test_query288_00006", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query156_00015", + "negative_test_query202_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query211", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query211", + "ranked_ids": [ + "negative_test_query159_00006", + "negative_test_query211_00012", + "apositive_test_query101_00000", + "negative_test_query211_00000", + "negative_test_query211_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query212", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query212", + "ranked_ids": [ + "apositive_test_query326_00000", + "apositive_test_query98_00000", + "apositive_test_query11_00007", + "negative_test_query282_00010", + "negative_test_query344_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query213", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query213", + "ranked_ids": [ + "apositive_test_query213_00002", + "negative_test_query138_00016", + "negative_test_query224_00015", + "negative_test_query213_00004", + "negative_test_query213_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query214", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query214", + "ranked_ids": [ + "negative_test_query214_00015", + "apositive_test_query234_00001", + "apositive_test_query214_00000", + "negative_test_query214_00010", + "apositive_test_query160_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query215", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.5, + "recall@5": 1.0 + }, + "query_id": "test_query215", + "ranked_ids": [ + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query215_00000", + "negative_test_query257_00001", + "negative_test_query215_00018" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query216", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query216", + "ranked_ids": [ + "apositive_test_query33_00004", + "negative_test_query216_00008", + "apositive_test_query216_00002", + "negative_test_query216_00009", + "negative_test_query246_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query217", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query217", + "ranked_ids": [ + "negative_test_query28_00002", + "negative_test_query230_00002", + "negative_test_query53_00012", + "apositive_test_query217_00000", + "negative_test_query185_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query218", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query218", + "ranked_ids": [ + "apositive_test_query218_00004", + "negative_test_query218_00003", + "apositive_test_query60_00012", + "negative_test_query218_00000", + "negative_test_query274_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query219", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query219", + "ranked_ids": [ + "negative_test_query165_00010", + "negative_test_query308_00006", + "negative_test_query24_00007", + "negative_test_query187_00015", + "negative_test_query219_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query22", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query22", + "ranked_ids": [ + "negative_test_query22_00000", + "negative_test_query22_00012", + "negative_test_query22_00007", + "negative_test_query22_00008", + "negative_test_query66_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query220", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query220", + "ranked_ids": [ + "negative_test_query188_00007", + "apositive_test_query20_00009", + "negative_test_query344_00007", + "apositive_test_query220_00001", + "apositive_test_query188_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query221", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query221", + "ranked_ids": [ + "apositive_test_query174_00006", + "negative_test_query278_00004", + "negative_test_query277_00010", + "negative_test_query184_00001", + "negative_test_query171_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query222", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query222", + "ranked_ids": [ + "negative_test_query222_00014", + "apositive_test_query289_00002", + "apositive_test_query275_00008", + "negative_test_query275_00004", + "apositive_test_query275_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query223", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query223", + "ranked_ids": [ + "negative_test_query51_00003", + "apositive_test_query259_00008", + "negative_test_query323_00005", + "negative_test_query47_00007", + "negative_test_query359_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query224", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query224", + "ranked_ids": [ + "apositive_test_query224_00000", + "negative_test_query224_00011", + "negative_test_query98_00018", + "negative_test_query224_00014", + "apositive_test_query32_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query225", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query225", + "ranked_ids": [ + "apositive_test_query225_00000", + "apositive_test_query145_00001", + "negative_test_query225_00006", + "apositive_test_query225_00003", + "negative_test_query149_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query226", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query226", + "ranked_ids": [ + "negative_test_query226_00009", + "apositive_test_query226_00002", + "negative_test_query226_00010", + "negative_test_query226_00007", + "apositive_test_query226_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query227", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query227", + "ranked_ids": [ + "negative_test_query227_00006", + "apositive_test_query348_00004", + "negative_test_query321_00004", + "apositive_test_query227_00001", + "apositive_test_query70_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query228", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.38685280723454163, + "recall@5": 0.5 + }, + "query_id": "test_query228", + "ranked_ids": [ + "apositive_test_query322_00003", + "apositive_test_query228_00000", + "negative_test_query73_00016", + "negative_test_query165_00011", + "apositive_test_query254_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query229", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query229", + "ranked_ids": [ + "apositive_test_query146_00004", + "negative_test_query355_00001", + "apositive_test_query229_00000", + "negative_test_query94_00010", + "negative_test_query44_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query23", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query23", + "ranked_ids": [ + "apositive_test_query225_00000", + "negative_test_query356_00008", + "apositive_test_query146_00015", + "negative_test_query292_00013", + "apositive_test_query54_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query230", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query230", + "ranked_ids": [ + "negative_test_query99_00003", + "apositive_test_query3_00006", + "negative_test_query230_00010", + "apositive_test_query3_00000", + "negative_test_query3_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query231", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query231", + "ranked_ids": [ + "negative_test_query201_00011", + "negative_test_query221_00006", + "negative_test_query345_00011", + "negative_test_query306_00006", + "apositive_test_query201_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query232", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query232", + "ranked_ids": [ + "negative_test_query232_00013", + "negative_test_query232_00014", + "negative_test_query232_00011", + "negative_test_query232_00016", + "negative_test_query232_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query233", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query233", + "ranked_ids": [ + "negative_test_query130_00001", + "negative_test_query233_00006", + "negative_test_query233_00001", + "negative_test_query233_00005", + "apositive_test_query233_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query234", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query234", + "ranked_ids": [ + "negative_test_query234_00010", + "apositive_test_query234_00003", + "apositive_test_query304_00002", + "negative_test_query234_00011", + "negative_test_query214_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query235", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query235", + "ranked_ids": [ + "apositive_test_query235_00003", + "apositive_test_query32_00004", + "negative_test_query235_00013", + "negative_test_query235_00011", + "negative_test_query235_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query236", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query236", + "ranked_ids": [ + "apositive_test_query236_00000", + "negative_test_query236_00002", + "negative_test_query236_00005", + "negative_test_query236_00000", + "negative_test_query236_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query237", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query237", + "ranked_ids": [ + "apositive_test_query237_00001", + "negative_test_query237_00005", + "negative_test_query237_00006", + "apositive_test_query152_00004", + "negative_test_query237_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query238", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query238", + "ranked_ids": [ + "negative_test_query238_00011", + "negative_test_query13_00003", + "negative_test_query111_00013", + "negative_test_query142_00011", + "negative_test_query102_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query239", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query239", + "ranked_ids": [ + "apositive_test_query239_00000", + "negative_test_query239_00007", + "negative_test_query239_00003", + "negative_test_query167_00011", + "negative_test_query198_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query24", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query24", + "ranked_ids": [ + "apositive_test_query106_00002", + "negative_test_query24_00002", + "negative_test_query24_00004", + "apositive_test_query24_00000", + "negative_test_query24_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query240", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query240", + "ranked_ids": [ + "negative_test_query240_00010", + "negative_test_query240_00016", + "negative_test_query240_00018", + "negative_test_query297_00005", + "negative_test_query27_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query241", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query241", + "ranked_ids": [ + "negative_test_query206_00004", + "negative_test_query241_00006", + "negative_test_query173_00013", + "negative_test_query241_00005", + "negative_test_query191_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query242", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query242", + "ranked_ids": [ + "apositive_test_query247_00005", + "apositive_test_query242_00001", + "negative_test_query291_00003", + "negative_test_query291_00005", + "apositive_test_query242_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query243", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query243", + "ranked_ids": [ + "apositive_test_query173_00004", + "apositive_test_query160_00003", + "negative_test_query243_00000", + "negative_test_query205_00007", + "negative_test_query243_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query244", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query244", + "ranked_ids": [ + "negative_test_query244_00010", + "negative_test_query244_00009", + "apositive_test_query244_00006", + "negative_test_query244_00008", + "negative_test_query244_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query245", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query245", + "ranked_ids": [ + "negative_test_query293_00003", + "negative_test_query202_00004", + "negative_test_query135_00013", + "negative_test_query122_00002", + "negative_test_query159_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query246", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query246", + "ranked_ids": [ + "negative_test_query6_00006", + "negative_test_query252_00003", + "negative_test_query134_00001", + "negative_test_query218_00004", + "negative_test_query58_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query247", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query247", + "ranked_ids": [ + "apositive_test_query247_00005", + "apositive_test_query242_00001", + "negative_test_query291_00003", + "negative_test_query291_00005", + "negative_test_query186_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query248", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.25 + }, + "query_id": "test_query248", + "ranked_ids": [ + "negative_test_query248_00001", + "apositive_test_query248_00003", + "negative_test_query248_00006", + "negative_test_query205_00017", + "apositive_test_query248_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query249", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query249", + "ranked_ids": [ + "negative_test_query249_00001", + "negative_test_query325_00015", + "negative_test_query301_00013", + "negative_test_query326_00005", + "apositive_test_query249_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query25", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.18181818181818182 + }, + "query_id": "test_query25", + "ranked_ids": [ + "negative_test_query328_00000", + "negative_test_query45_00014", + "apositive_test_query25_00008", + "negative_test_query25_00001", + "apositive_test_query25_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query250", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query250", + "ranked_ids": [ + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query192_00009", + "negative_test_query57_00009", + "negative_test_query64_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query251", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query251", + "ranked_ids": [ + "negative_test_query234_00006", + "negative_test_query251_00003", + "negative_test_query143_00001", + "negative_test_query56_00005", + "negative_test_query170_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query252", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query252", + "ranked_ids": [ + "apositive_test_query110_00001", + "negative_test_query252_00009", + "negative_test_query44_00011", + "negative_test_query321_00004", + "negative_test_query252_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query253", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query253", + "ranked_ids": [ + "negative_test_query124_00001", + "apositive_test_query162_00008", + "negative_test_query177_00000", + "apositive_test_query74_00002", + "apositive_test_query13_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query254", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5296347172140421, + "recall@5": 0.2727272727272727 + }, + "query_id": "test_query254", + "ranked_ids": [ + "negative_test_query21_00009", + "apositive_test_query254_00003", + "apositive_test_query254_00005", + "apositive_test_query254_00010", + "apositive_test_query333_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query255", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.6309297535714575, + "recall@5": 1.0 + }, + "query_id": "test_query255", + "ranked_ids": [ + "negative_test_query246_00003", + "apositive_test_query255_00000", + "negative_test_query278_00000", + "negative_test_query255_00006", + "negative_test_query326_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query256", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.36331756126716835, + "recall@5": 0.5 + }, + "query_id": "test_query256", + "ranked_ids": [ + "negative_test_query183_00008", + "negative_test_query7_00009", + "apositive_test_query256_00002", + "apositive_test_query256_00003", + "apositive_test_query216_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query257", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query257", + "ranked_ids": [ + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00003", + "apositive_test_query286_00014", + "apositive_test_query257_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query258", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query258", + "ranked_ids": [ + "negative_test_query320_00002", + "negative_test_query38_00002", + "negative_test_query162_00004", + "negative_test_query173_00010", + "negative_test_query167_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query259", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.08333333333333333 + }, + "query_id": "test_query259", + "ranked_ids": [ + "apositive_test_query259_00008", + "negative_test_query30_00009", + "negative_test_query213_00004", + "negative_test_query259_00006", + "negative_test_query47_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query26", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.125 + }, + "query_id": "test_query26", + "ranked_ids": [ + "apositive_test_query26_00004", + "negative_test_query26_00005", + "negative_test_query26_00008", + "negative_test_query119_00006", + "negative_test_query158_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query260", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query260", + "ranked_ids": [ + "negative_test_query6_00009", + "negative_test_query6_00012", + "apositive_test_query6_00000", + "negative_test_query252_00004", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query261", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query261", + "ranked_ids": [ + "apositive_test_query261_00000", + "negative_test_query261_00009", + "negative_test_query261_00007", + "negative_test_query261_00005", + "negative_test_query118_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query262", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query262", + "ranked_ids": [ + "negative_test_query326_00008", + "apositive_test_query177_00008", + "negative_test_query147_00001", + "negative_test_query162_00007", + "apositive_test_query42_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query263", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query263", + "ranked_ids": [ + "apositive_test_query263_00002", + "apositive_test_query242_00011", + "apositive_test_query291_00010", + "negative_test_query263_00005", + "negative_test_query307_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query264", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query264", + "ranked_ids": [ + "negative_test_query264_00004", + "negative_test_query264_00011", + "negative_test_query264_00007", + "apositive_test_query38_00003", + "negative_test_query264_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query265", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query265", + "ranked_ids": [ + "negative_test_query265_00002", + "apositive_test_query234_00001", + "negative_test_query165_00007", + "negative_test_query165_00016", + "negative_test_query190_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query266", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query266", + "ranked_ids": [ + "negative_test_query266_00000", + "negative_test_query266_00009", + "negative_test_query266_00007", + "negative_test_query266_00001", + "negative_test_query141_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query267", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query267", + "ranked_ids": [ + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "negative_test_query30_00000", + "apositive_test_query250_00000", + "negative_test_query186_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query268", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.2 + }, + "query_id": "test_query268", + "ranked_ids": [ + "negative_test_query195_00009", + "negative_test_query335_00002", + "apositive_test_query268_00004", + "negative_test_query268_00012", + "negative_test_query268_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query269", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query269", + "ranked_ids": [ + "negative_test_query269_00009", + "negative_test_query329_00004", + "apositive_test_query296_00005", + "apositive_test_query269_00000", + "negative_test_query329_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query27", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query27", + "ranked_ids": [ + "apositive_test_query127_00010", + "negative_test_query27_00009", + "negative_test_query76_00009", + "negative_test_query27_00014", + "negative_test_query130_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query270", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query270", + "ranked_ids": [ + "negative_test_query270_00006", + "negative_test_query270_00010", + "negative_test_query270_00016", + "negative_test_query270_00009", + "apositive_test_query270_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query271", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query271", + "ranked_ids": [ + "negative_test_query271_00007", + "negative_test_query227_00009", + "negative_test_query120_00003", + "apositive_test_query271_00001", + "apositive_test_query10_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query272", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query272", + "ranked_ids": [ + "negative_test_query272_00017", + "negative_test_query332_00013", + "apositive_test_query251_00000", + "negative_test_query328_00000", + "negative_test_query272_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query273", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query273", + "ranked_ids": [ + "negative_test_query273_00009", + "negative_test_query273_00006", + "apositive_test_query18_00001", + "negative_test_query273_00008", + "apositive_test_query18_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query274", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query274", + "ranked_ids": [ + "negative_test_query274_00012", + "negative_test_query218_00011", + "negative_test_query34_00014", + "negative_test_query352_00012", + "apositive_test_query274_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query275", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.08333333333333333 + }, + "query_id": "test_query275", + "ranked_ids": [ + "apositive_test_query110_00001", + "negative_test_query275_00007", + "apositive_test_query93_00003", + "negative_test_query149_00016", + "apositive_test_query275_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query276", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.07692307692307693 + }, + "query_id": "test_query276", + "ranked_ids": [ + "negative_test_query146_00000", + "apositive_test_query153_00006", + "apositive_test_query146_00015", + "negative_test_query251_00012", + "apositive_test_query276_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query277", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query277", + "ranked_ids": [ + "negative_test_query277_00009", + "apositive_test_query359_00001", + "negative_test_query305_00003", + "apositive_test_query277_00002", + "negative_test_query345_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query278", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query278", + "ranked_ids": [ + "negative_test_query140_00006", + "negative_test_query177_00001", + "negative_test_query15_00007", + "negative_test_query278_00009", + "negative_test_query260_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query279", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.4 + }, + "query_id": "test_query279", + "ranked_ids": [ + "apositive_test_query279_00004", + "apositive_test_query279_00003", + "apositive_test_query3_00000", + "apositive_test_query76_00002", + "negative_test_query279_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query28", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query28", + "ranked_ids": [ + "negative_test_query28_00002", + "apositive_test_query217_00000", + "apositive_test_query217_00003", + "negative_test_query185_00003", + "negative_test_query307_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query280", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query280", + "ranked_ids": [ + "apositive_test_query280_00005", + "negative_test_query280_00012", + "negative_test_query9_00012", + "negative_test_query280_00010", + "negative_test_query106_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query281", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5531464700081437, + "recall@5": 0.10526315789473684 + }, + "query_id": "test_query281", + "ranked_ids": [ + "apositive_test_query281_00004", + "apositive_test_query281_00011", + "apositive_test_query273_00000", + "apositive_test_query273_00008", + "negative_test_query273_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query282", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query282", + "ranked_ids": [ + "apositive_test_query282_00000", + "negative_test_query282_00003", + "negative_test_query282_00006", + "negative_test_query282_00004", + "negative_test_query282_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query283", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query283", + "ranked_ids": [ + "negative_test_query270_00006", + "negative_test_query93_00009", + "negative_test_query138_00005", + "negative_test_query283_00014", + "negative_test_query93_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query284", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.4 + }, + "query_id": "test_query284", + "ranked_ids": [ + "apositive_test_query284_00000", + "negative_test_query284_00001", + "apositive_test_query163_00003", + "negative_test_query14_00014", + "apositive_test_query284_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query285", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.25 + }, + "query_id": "test_query285", + "ranked_ids": [ + "negative_test_query285_00002", + "negative_test_query86_00001", + "apositive_test_query285_00001", + "negative_test_query304_00003", + "apositive_test_query285_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query286", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.13333333333333333 + }, + "query_id": "test_query286", + "ranked_ids": [ + "apositive_test_query286_00014", + "apositive_test_query257_00000", + "negative_test_query139_00008", + "negative_test_query52_00001", + "apositive_test_query286_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query287", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query287", + "ranked_ids": [ + "apositive_test_query153_00003", + "apositive_test_query133_00000", + "apositive_test_query133_00004", + "negative_test_query301_00010", + "apositive_test_query133_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query288", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query288", + "ranked_ids": [ + "negative_test_query288_00006", + "apositive_test_query210_00001", + "negative_test_query210_00005", + "negative_test_query148_00014", + "negative_test_query199_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query289", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.4 + }, + "query_id": "test_query289", + "ranked_ids": [ + "negative_test_query289_00004", + "apositive_test_query289_00002", + "negative_test_query113_00009", + "apositive_test_query289_00000", + "negative_test_query7_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query29", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.125 + }, + "query_id": "test_query29", + "ranked_ids": [ + "negative_test_query292_00015", + "apositive_test_query29_00003", + "negative_test_query213_00012", + "negative_test_query29_00005", + "negative_test_query29_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query290", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query290", + "ranked_ids": [ + "negative_test_query290_00005", + "negative_test_query98_00003", + "negative_test_query342_00005", + "negative_test_query290_00000", + "apositive_test_query290_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query291", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query291", + "ranked_ids": [ + "apositive_test_query247_00005", + "apositive_test_query242_00001", + "negative_test_query291_00003", + "negative_test_query291_00005", + "negative_test_query291_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query292", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query292", + "ranked_ids": [ + "negative_test_query292_00015", + "apositive_test_query292_00001", + "negative_test_query292_00000", + "negative_test_query61_00011", + "negative_test_query292_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query293", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query293", + "ranked_ids": [ + "negative_test_query293_00011", + "negative_test_query293_00015", + "apositive_test_query293_00002", + "negative_test_query293_00010", + "negative_test_query293_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query294", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query294", + "ranked_ids": [ + "negative_test_query294_00007", + "negative_test_query294_00001", + "negative_test_query294_00004", + "negative_test_query294_00012", + "negative_test_query294_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query295", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query295", + "ranked_ids": [ + "negative_test_query292_00015", + "negative_test_query295_00012", + "negative_test_query57_00009", + "negative_test_query295_00001", + "negative_test_query295_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query296", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query296", + "ranked_ids": [ + "negative_test_query329_00011", + "apositive_test_query296_00004", + "negative_test_query296_00001", + "negative_test_query0_00004", + "apositive_test_query296_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query297", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query297", + "ranked_ids": [ + "negative_test_query297_00004", + "negative_test_query297_00007", + "apositive_test_query139_00001", + "negative_test_query297_00012", + "negative_test_query191_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query298", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.38364913903790027, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query298", + "ranked_ids": [ + "negative_test_query182_00011", + "negative_test_query193_00006", + "negative_test_query182_00010", + "apositive_test_query298_00000", + "apositive_test_query298_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query299", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query299", + "ranked_ids": [ + "negative_test_query288_00000", + "negative_test_query58_00005", + "negative_test_query89_00008", + "apositive_test_query265_00002", + "apositive_test_query227_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query3", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.09090909090909091 + }, + "query_id": "test_query3", + "ranked_ids": [ + "negative_test_query3_00007", + "negative_test_query230_00010", + "apositive_test_query3_00000", + "negative_test_query3_00000", + "negative_test_query3_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query30", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query30", + "ranked_ids": [ + "negative_test_query231_00008", + "negative_test_query71_00010", + "negative_test_query182_00014", + "apositive_test_query30_00005", + "negative_test_query231_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query300", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query300", + "ranked_ids": [ + "negative_test_query300_00011", + "negative_test_query300_00005", + "negative_test_query300_00009", + "negative_test_query141_00001", + "negative_test_query300_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query301", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.15101961822780524, + "recall@5": 0.25 + }, + "query_id": "test_query301", + "ranked_ids": [ + "apositive_test_query86_00000", + "negative_test_query301_00007", + "negative_test_query2_00008", + "negative_test_query301_00013", + "apositive_test_query301_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query302", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query302", + "ranked_ids": [ + "negative_test_query308_00005", + "apositive_test_query302_00004", + "negative_test_query308_00002", + "negative_test_query302_00010", + "apositive_test_query308_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query303", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.43067655807339306, + "recall@5": 1.0 + }, + "query_id": "test_query303", + "ranked_ids": [ + "negative_test_query243_00000", + "negative_test_query303_00017", + "negative_test_query35_00003", + "apositive_test_query303_00000", + "negative_test_query173_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query304", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query304", + "ranked_ids": [ + "apositive_test_query285_00001", + "negative_test_query304_00003", + "negative_test_query234_00010", + "apositive_test_query285_00005", + "apositive_test_query234_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query305", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query305", + "ranked_ids": [ + "negative_test_query51_00003", + "negative_test_query0_00004", + "negative_test_query224_00015", + "negative_test_query164_00011", + "negative_test_query47_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query306", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query306", + "ranked_ids": [ + "negative_test_query209_00008", + "negative_test_query182_00014", + "apositive_test_query289_00003", + "apositive_test_query119_00006", + "negative_test_query204_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query307", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query307", + "ranked_ids": [ + "negative_test_query307_00006", + "negative_test_query103_00012", + "negative_test_query48_00018", + "negative_test_query163_00010", + "apositive_test_query279_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query308", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.27727342735504823, + "recall@5": 0.2 + }, + "query_id": "test_query308", + "ranked_ids": [ + "apositive_test_query310_00008", + "apositive_test_query324_00000", + "negative_test_query324_00014", + "apositive_test_query308_00002", + "apositive_test_query308_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query309", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query309", + "ranked_ids": [ + "apositive_test_query309_00000", + "negative_test_query309_00003", + "apositive_test_query358_00003", + "apositive_test_query76_00002", + "apositive_test_query161_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query31", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query31", + "ranked_ids": [ + "negative_test_query38_00003", + "apositive_test_query60_00011", + "negative_test_query31_00004", + "negative_test_query31_00007", + "negative_test_query31_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query310", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4912596920895758, + "recall@5": 0.3 + }, + "query_id": "test_query310", + "ranked_ids": [ + "negative_test_query179_00013", + "apositive_test_query310_00008", + "negative_test_query74_00002", + "apositive_test_query310_00002", + "apositive_test_query310_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query311", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query311", + "ranked_ids": [ + "negative_test_query330_00009", + "negative_test_query288_00004", + "negative_test_query330_00011", + "negative_test_query240_00014", + "apositive_test_query330_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query312", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query312", + "ranked_ids": [ + "negative_test_query242_00001", + "apositive_test_query237_00001", + "apositive_test_query220_00005", + "apositive_test_query98_00000", + "negative_test_query294_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query313", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query313", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "negative_test_query336_00001", + "negative_test_query119_00005", + "negative_test_query334_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query314", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7860137352654724, + "recall@5": 0.23529411764705882 + }, + "query_id": "test_query314", + "ranked_ids": [ + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query314_00001", + "apositive_test_query314_00011", + "apositive_test_query314_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query315", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.20210734650054757, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query315", + "ranked_ids": [ + "apositive_test_query319_00005", + "apositive_test_query82_00005", + "negative_test_query315_00009", + "apositive_test_query315_00002", + "negative_test_query333_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query316", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query316", + "ranked_ids": [ + "apositive_test_query316_00001", + "negative_test_query301_00005", + "negative_test_query316_00000", + "negative_test_query316_00002", + "negative_test_query316_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query317", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query317", + "ranked_ids": [ + "negative_test_query317_00010", + "negative_test_query331_00009", + "negative_test_query55_00014", + "negative_test_query317_00017", + "negative_test_query317_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query318", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5413996682199069, + "recall@5": 0.5 + }, + "query_id": "test_query318", + "ranked_ids": [ + "apositive_test_query318_00002", + "negative_test_query318_00002", + "negative_test_query318_00004", + "negative_test_query73_00004", + "apositive_test_query318_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query319", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.30078518014914984, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query319", + "ranked_ids": [ + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query32", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query32", + "ranked_ids": [ + "apositive_test_query196_00000", + "apositive_test_query336_00001", + "negative_test_query211_00016", + "negative_test_query55_00011", + "negative_test_query55_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query320", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query320", + "ranked_ids": [ + "negative_test_query258_00012", + "negative_test_query53_00014", + "negative_test_query193_00006", + "negative_test_query320_00005", + "negative_test_query131_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query321", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.38685280723454163, + "recall@5": 1.0 + }, + "query_id": "test_query321", + "ranked_ids": [ + "negative_test_query321_00002", + "negative_test_query321_00010", + "negative_test_query136_00004", + "negative_test_query210_00016", + "apositive_test_query321_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query322", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query322", + "ranked_ids": [ + "negative_test_query73_00008", + "negative_test_query318_00002", + "apositive_test_query322_00003", + "apositive_test_query322_00005", + "negative_test_query322_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query323", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query323", + "ranked_ids": [ + "negative_test_query255_00002", + "negative_test_query323_00006", + "negative_test_query209_00004", + "apositive_test_query323_00006", + "negative_test_query75_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query324", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query324", + "ranked_ids": [ + "negative_test_query218_00003", + "apositive_test_query324_00000", + "negative_test_query324_00014", + "apositive_test_query308_00002", + "negative_test_query324_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query325", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query325", + "ranked_ids": [ + "negative_test_query293_00015", + "apositive_test_query293_00002", + "negative_test_query325_00007", + "negative_test_query15_00016", + "negative_test_query15_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query326", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query326", + "ranked_ids": [ + "negative_test_query265_00007", + "apositive_test_query326_00000", + "negative_test_query178_00012", + "apositive_test_query326_00001", + "negative_test_query17_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query327", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query327", + "ranked_ids": [ + "negative_test_query271_00007", + "negative_test_query327_00000", + "negative_test_query227_00009", + "negative_test_query327_00014", + "negative_test_query327_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query328", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query328", + "ranked_ids": [ + "apositive_test_query40_00003", + "apositive_test_query328_00004", + "negative_test_query328_00003", + "negative_test_query222_00016", + "negative_test_query328_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query329", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query329", + "ranked_ids": [ + "negative_test_query329_00006", + "negative_test_query329_00011", + "apositive_test_query7_00001", + "negative_test_query35_00000", + "negative_test_query329_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query33", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1 + }, + "query_id": "test_query33", + "ranked_ids": [ + "negative_test_query154_00006", + "apositive_test_query33_00004", + "apositive_test_query154_00007", + "negative_test_query154_00004", + "apositive_test_query284_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query330", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query330", + "ranked_ids": [ + "negative_test_query330_00001", + "negative_test_query288_00006", + "negative_test_query330_00009", + "negative_test_query330_00000", + "negative_test_query330_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query331", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4776237035032179, + "recall@5": 0.6666666666666666 + }, + "query_id": "test_query331", + "ranked_ids": [ + "negative_test_query214_00015", + "apositive_test_query331_00002", + "negative_test_query331_00008", + "negative_test_query303_00001", + "apositive_test_query331_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query332", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query332", + "ranked_ids": [ + "apositive_test_query332_00001", + "negative_test_query202_00002", + "negative_test_query332_00002", + "apositive_test_query348_00008", + "negative_test_query115_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query333", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query333", + "ranked_ids": [ + "apositive_test_query333_00000", + "apositive_test_query254_00005", + "negative_test_query73_00008", + "negative_test_query136_00003", + "negative_test_query128_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query334", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query334", + "ranked_ids": [ + "negative_test_query209_00003", + "negative_test_query111_00017", + "negative_test_query213_00009", + "negative_test_query334_00010", + "negative_test_query170_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query335", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.2 + }, + "query_id": "test_query335", + "ranked_ids": [ + "negative_test_query335_00009", + "apositive_test_query335_00002", + "negative_test_query210_00006", + "negative_test_query321_00013", + "negative_test_query335_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query336", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query336", + "ranked_ids": [ + "apositive_test_query50_00012", + "apositive_test_query38_00005", + "apositive_test_query113_00005", + "negative_test_query336_00015", + "negative_test_query247_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query337", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query337", + "ranked_ids": [ + "negative_test_query200_00015", + "apositive_test_query38_00003", + "apositive_test_query38_00000", + "negative_test_query267_00001", + "negative_test_query337_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query338", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query338", + "ranked_ids": [ + "negative_test_query119_00003", + "apositive_test_query338_00001", + "negative_test_query81_00004", + "apositive_test_query114_00005", + "negative_test_query81_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query339", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.23719771276929622, + "recall@5": 0.5 + }, + "query_id": "test_query339", + "ranked_ids": [ + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004", + "apositive_test_query339_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query34", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query34", + "ranked_ids": [ + "apositive_test_query352_00000", + "negative_test_query34_00014", + "negative_test_query352_00012", + "negative_test_query34_00008", + "negative_test_query352_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query340", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3391602052736161, + "recall@5": 0.2 + }, + "query_id": "test_query340", + "ranked_ids": [ + "apositive_test_query340_00002", + "negative_test_query88_00007", + "negative_test_query340_00001", + "negative_test_query340_00006", + "negative_test_query88_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query341", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query341", + "ranked_ids": [ + "negative_test_query341_00005", + "negative_test_query96_00007", + "negative_test_query341_00009", + "negative_test_query101_00003", + "negative_test_query341_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query342", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query342", + "ranked_ids": [ + "negative_test_query342_00016", + "negative_test_query68_00005", + "negative_test_query325_00015", + "negative_test_query136_00003", + "negative_test_query324_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query343", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query343", + "ranked_ids": [ + "negative_test_query307_00006", + "negative_test_query227_00010", + "apositive_test_query279_00003", + "apositive_test_query307_00008", + "negative_test_query225_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query344", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query344", + "ranked_ids": [ + "negative_test_query282_00004", + "negative_test_query282_00001", + "negative_test_query344_00002", + "negative_test_query344_00004", + "negative_test_query344_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query345", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query345", + "ranked_ids": [ + "negative_test_query90_00003", + "negative_test_query9_00002", + "negative_test_query130_00004", + "negative_test_query345_00011", + "negative_test_query345_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query346", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query346", + "ranked_ids": [ + "negative_test_query19_00012", + "negative_test_query346_00008", + "negative_test_query346_00010", + "negative_test_query15_00008", + "negative_test_query346_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query347", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.18154179253735267, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query347", + "ranked_ids": [ + "negative_test_query347_00006", + "negative_test_query251_00007", + "negative_test_query347_00015", + "negative_test_query347_00002", + "apositive_test_query347_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query348", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2 + }, + "query_id": "test_query348", + "ranked_ids": [ + "apositive_test_query348_00004", + "negative_test_query297_00017", + "apositive_test_query115_00002", + "negative_test_query351_00010", + "apositive_test_query348_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query349", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query349", + "ranked_ids": [ + "apositive_test_query137_00002", + "apositive_test_query42_00004", + "negative_test_query74_00001", + "apositive_test_query253_00008", + "apositive_test_query74_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query35", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.3903800499921017, + "recall@5": 0.25 + }, + "query_id": "test_query35", + "ranked_ids": [ + "apositive_test_query35_00003", + "negative_test_query262_00002", + "negative_test_query198_00011", + "negative_test_query101_00006", + "negative_test_query205_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query350", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query350", + "ranked_ids": [ + "negative_test_query120_00007", + "negative_test_query350_00013", + "negative_test_query63_00000", + "negative_test_query192_00005", + "apositive_test_query33_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query351", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query351", + "ranked_ids": [ + "apositive_test_query218_00004", + "negative_test_query8_00003", + "negative_test_query53_00011", + "negative_test_query156_00006", + "apositive_test_query60_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query352", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.4 + }, + "query_id": "test_query352", + "ranked_ids": [ + "apositive_test_query352_00000", + "negative_test_query34_00014", + "negative_test_query34_00008", + "negative_test_query352_00010", + "apositive_test_query352_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query353", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.06666666666666667 + }, + "query_id": "test_query353", + "ranked_ids": [ + "apositive_test_query204_00012", + "negative_test_query353_00002", + "apositive_test_query204_00000", + "apositive_test_query353_00002", + "apositive_test_query204_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query354", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6843515475204855, + "recall@5": 0.375 + }, + "query_id": "test_query354", + "ranked_ids": [ + "apositive_test_query354_00006", + "apositive_test_query354_00004", + "negative_test_query354_00004", + "negative_test_query309_00013", + "apositive_test_query354_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query355", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query355", + "ranked_ids": [ + "apositive_test_query100_00001", + "negative_test_query355_00003", + "negative_test_query264_00003", + "apositive_test_query355_00005", + "negative_test_query355_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query356", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query356", + "ranked_ids": [ + "apositive_test_query30_00005", + "apositive_test_query356_00004", + "negative_test_query253_00001", + "negative_test_query15_00006", + "negative_test_query102_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query357", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query357", + "ranked_ids": [ + "apositive_test_query119_00002", + "apositive_test_query221_00000", + "apositive_test_query32_00002", + "negative_test_query196_00001", + "apositive_test_query119_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query358", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.3156484524795145, + "recall@5": 0.2 + }, + "query_id": "test_query358", + "ranked_ids": [ + "apositive_test_query309_00000", + "apositive_test_query161_00000", + "apositive_test_query358_00000", + "apositive_test_query358_00003", + "apositive_test_query11_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query359", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query359", + "ranked_ids": [ + "apositive_test_query0_00010", + "apositive_test_query0_00000", + "apositive_test_query0_00006", + "apositive_test_query0_00003", + "negative_test_query359_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query36", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query36", + "ranked_ids": [ + "apositive_test_query286_00005", + "negative_test_query112_00013", + "negative_test_query80_00007", + "negative_test_query112_00015", + "apositive_test_query106_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query360", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query360", + "ranked_ids": [ + "apositive_test_query360_00000", + "negative_test_query360_00009", + "negative_test_query189_00013", + "apositive_test_query152_00002", + "negative_test_query240_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query37", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query37", + "ranked_ids": [ + "negative_test_query57_00009", + "negative_test_query20_00003", + "apositive_test_query177_00000", + "apositive_test_query158_00007", + "negative_test_query152_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query38", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.5087403079104241, + "recall@5": 0.25 + }, + "query_id": "test_query38", + "ranked_ids": [ + "apositive_test_query38_00002", + "apositive_test_query247_00005", + "apositive_test_query38_00007", + "negative_test_query267_00016", + "negative_test_query267_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query39", + "metrics": { + "mrr@5": 0.2, + "ndcg@5": 0.13120507751234178, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query39", + "ranked_ids": [ + "negative_test_query232_00007", + "negative_test_query39_00003", + "negative_test_query232_00017", + "negative_test_query331_00010", + "apositive_test_query39_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query4", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query4", + "ranked_ids": [ + "negative_test_query266_00009", + "apositive_test_query4_00010", + "negative_test_query4_00002", + "apositive_test_query174_00000", + "apositive_test_query4_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query40", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query40", + "ranked_ids": [ + "negative_test_query40_00012", + "negative_test_query40_00010", + "negative_test_query40_00006", + "negative_test_query40_00005", + "negative_test_query214_00015" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query41", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query41", + "ranked_ids": [ + "negative_test_query282_00003", + "negative_test_query41_00000", + "negative_test_query98_00000", + "negative_test_query41_00011", + "negative_test_query41_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query42", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.25 + }, + "query_id": "test_query42", + "ranked_ids": [ + "negative_test_query250_00010", + "apositive_test_query42_00003", + "negative_test_query42_00000", + "apositive_test_query42_00004", + "apositive_test_query162_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query43", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.125 + }, + "query_id": "test_query43", + "ranked_ids": [ + "apositive_test_query146_00004", + "apositive_test_query314_00009", + "negative_test_query146_00002", + "apositive_test_query43_00000", + "negative_test_query43_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query44", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query44", + "ranked_ids": [ + "negative_test_query293_00015", + "negative_test_query21_00009", + "negative_test_query330_00009", + "negative_test_query141_00007", + "negative_test_query190_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query45", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query45", + "ranked_ids": [ + "negative_test_query143_00004", + "negative_test_query45_00009", + "negative_test_query45_00003", + "negative_test_query143_00011", + "negative_test_query342_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query46", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query46", + "ranked_ids": [ + "apositive_test_query79_00005", + "negative_test_query213_00012", + "negative_test_query108_00003", + "negative_test_query168_00003", + "negative_test_query79_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query47", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query47", + "ranked_ids": [ + "apositive_test_query331_00002", + "negative_test_query47_00013", + "negative_test_query325_00007", + "negative_test_query144_00018", + "negative_test_query47_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query48", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 1.0, + "recall@5": 1.0 + }, + "query_id": "test_query48", + "ranked_ids": [ + "apositive_test_query48_00000", + "negative_test_query103_00012", + "negative_test_query48_00018", + "apositive_test_query307_00002", + "negative_test_query48_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query49", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query49", + "ranked_ids": [ + "negative_test_query342_00005", + "negative_test_query52_00011", + "negative_test_query15_00008", + "negative_test_query156_00010", + "apositive_test_query204_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query5", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query5", + "ranked_ids": [ + "negative_test_query5_00004", + "negative_test_query5_00007", + "negative_test_query275_00001", + "negative_test_query5_00014", + "negative_test_query5_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query50", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.7227265726449519, + "recall@5": 0.17647058823529413 + }, + "query_id": "test_query50", + "ranked_ids": [ + "apositive_test_query50_00006", + "apositive_test_query50_00012", + "apositive_test_query50_00015", + "apositive_test_query186_00005", + "apositive_test_query131_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query51", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1 + }, + "query_id": "test_query51", + "ranked_ids": [ + "negative_test_query51_00000", + "negative_test_query97_00001", + "apositive_test_query51_00006", + "negative_test_query51_00004", + "negative_test_query209_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query52", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query52", + "ranked_ids": [ + "negative_test_query215_00000", + "negative_test_query257_00003", + "apositive_test_query257_00003", + "apositive_test_query286_00014", + "apositive_test_query257_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query53", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query53", + "ranked_ids": [ + "negative_test_query258_00018", + "negative_test_query132_00015", + "negative_test_query150_00014", + "negative_test_query258_00013", + "negative_test_query169_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query54", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query54", + "ranked_ids": [ + "negative_test_query44_00007", + "negative_test_query54_00006", + "apositive_test_query86_00000", + "apositive_test_query335_00002", + "apositive_test_query51_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query55", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query55", + "ranked_ids": [ + "negative_test_query55_00010", + "negative_test_query55_00003", + "negative_test_query55_00012", + "negative_test_query55_00011", + "negative_test_query234_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query56", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query56", + "ranked_ids": [ + "negative_test_query46_00014", + "negative_test_query56_00006", + "apositive_test_query46_00001", + "apositive_test_query238_00001", + "negative_test_query56_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query57", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query57", + "ranked_ids": [ + "negative_test_query313_00012", + "negative_test_query334_00013", + "apositive_test_query57_00008", + "apositive_test_query203_00000", + "negative_test_query334_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query58", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query58", + "ranked_ids": [ + "apositive_test_query311_00004", + "negative_test_query308_00004", + "negative_test_query58_00015", + "negative_test_query27_00017", + "negative_test_query240_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query59", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query59", + "ranked_ids": [ + "negative_test_query116_00000", + "negative_test_query59_00011", + "negative_test_query59_00014", + "negative_test_query59_00015", + "negative_test_query202_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query6", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.23463936301137822, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query6", + "ranked_ids": [ + "negative_test_query179_00007", + "negative_test_query6_00012", + "apositive_test_query6_00000", + "negative_test_query252_00004", + "apositive_test_query109_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query60", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.48522855511632257, + "recall@5": 0.125 + }, + "query_id": "test_query60", + "ranked_ids": [ + "apositive_test_query60_00011", + "negative_test_query31_00004", + "apositive_test_query218_00004", + "apositive_test_query60_00006", + "apositive_test_query285_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query61", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.19519002499605084, + "recall@5": 0.25 + }, + "query_id": "test_query61", + "ranked_ids": [ + "apositive_test_query173_00004", + "negative_test_query327_00016", + "apositive_test_query61_00003", + "negative_test_query61_00002", + "negative_test_query129_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query62", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.2640681225725909, + "recall@5": 0.5 + }, + "query_id": "test_query62", + "ranked_ids": [ + "negative_test_query62_00003", + "negative_test_query62_00002", + "negative_test_query62_00000", + "apositive_test_query62_00001", + "negative_test_query219_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query63", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.4703652827859579, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query63", + "ranked_ids": [ + "apositive_test_query63_00006", + "negative_test_query321_00002", + "negative_test_query297_00006", + "negative_test_query63_00003", + "apositive_test_query63_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query64", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query64", + "ranked_ids": [ + "negative_test_query120_00000", + "apositive_test_query160_00003", + "negative_test_query149_00001", + "negative_test_query120_00003", + "negative_test_query64_00016" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query65", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query65", + "ranked_ids": [ + "negative_test_query195_00009", + "negative_test_query335_00002", + "negative_test_query65_00000", + "negative_test_query268_00012", + "apositive_test_query356_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query66", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query66", + "ranked_ids": [ + "negative_test_query9_00012", + "apositive_test_query66_00004", + "apositive_test_query20_00005", + "negative_test_query188_00006", + "apositive_test_query66_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query67", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query67", + "ranked_ids": [ + "apositive_test_query286_00004", + "apositive_test_query286_00010", + "negative_test_query123_00008", + "negative_test_query67_00007", + "negative_test_query260_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query68", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.2960819109658652, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query68", + "ranked_ids": [ + "negative_test_query330_00009", + "apositive_test_query68_00000", + "negative_test_query288_00004", + "negative_test_query330_00011", + "negative_test_query68_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query69", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.639945385422766, + "recall@5": 0.15789473684210525 + }, + "query_id": "test_query69", + "ranked_ids": [ + "apositive_test_query69_00010", + "apositive_test_query257_00003", + "apositive_test_query69_00005", + "negative_test_query18_00010", + "apositive_test_query69_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query7", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query7", + "ranked_ids": [ + "apositive_test_query159_00001", + "negative_test_query264_00013", + "apositive_test_query38_00003", + "negative_test_query159_00010", + "negative_test_query7_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query70", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query70", + "ranked_ids": [ + "negative_test_query227_00006", + "apositive_test_query70_00001", + "negative_test_query227_00010", + "apositive_test_query227_00000", + "negative_test_query348_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query71", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query71", + "ranked_ids": [ + "negative_test_query71_00002", + "negative_test_query71_00015", + "apositive_test_query30_00005", + "negative_test_query71_00009", + "negative_test_query71_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query72", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query72", + "ranked_ids": [ + "negative_test_query17_00009", + "negative_test_query72_00015", + "negative_test_query72_00006", + "negative_test_query72_00003", + "negative_test_query72_00007" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query73", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query73", + "ranked_ids": [ + "negative_test_query73_00013", + "apositive_test_query333_00000", + "negative_test_query73_00008", + "negative_test_query233_00015", + "negative_test_query128_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query74", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.16812753627111746, + "recall@5": 0.25 + }, + "query_id": "test_query74", + "ranked_ids": [ + "apositive_test_query60_00006", + "negative_test_query74_00001", + "apositive_test_query113_00001", + "apositive_test_query74_00002", + "negative_test_query74_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query75", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query75", + "ranked_ids": [ + "negative_test_query99_00006", + "apositive_test_query75_00001", + "negative_test_query209_00004", + "negative_test_query67_00017", + "negative_test_query75_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query76", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.36005461457723403, + "recall@5": 0.2857142857142857 + }, + "query_id": "test_query76", + "ranked_ids": [ + "negative_test_query76_00009", + "apositive_test_query76_00000", + "negative_test_query147_00007", + "apositive_test_query76_00002", + "negative_test_query181_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query77", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query77", + "ranked_ids": [ + "negative_test_query329_00015", + "negative_test_query77_00013", + "negative_test_query342_00007", + "negative_test_query98_00014", + "negative_test_query301_00003" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query78", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query78", + "ranked_ids": [ + "negative_test_query101_00008", + "negative_test_query61_00007", + "negative_test_query214_00010", + "apositive_test_query38_00003", + "apositive_test_query326_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query79", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query79", + "ranked_ids": [ + "negative_test_query38_00007", + "negative_test_query313_00013", + "apositive_test_query79_00004", + "apositive_test_query295_00001", + "negative_test_query157_00000" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query8", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query8", + "ranked_ids": [ + "apositive_test_query192_00002", + "negative_test_query329_00005", + "negative_test_query8_00000", + "negative_test_query8_00014", + "negative_test_query96_00010" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query80", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query80", + "ranked_ids": [ + "negative_test_query289_00005", + "apositive_test_query179_00003", + "negative_test_query287_00005", + "negative_test_query80_00009", + "negative_test_query80_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query81", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.34519134224686937, + "recall@5": 0.2222222222222222 + }, + "query_id": "test_query81", + "ranked_ids": [ + "negative_test_query81_00005", + "apositive_test_query81_00003", + "negative_test_query81_00004", + "negative_test_query145_00004", + "apositive_test_query81_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query82", + "metrics": { + "mrr@5": 0.25, + "ndcg@5": 0.14606834984270645, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query82", + "ranked_ids": [ + "negative_test_query319_00003", + "negative_test_query82_00003", + "apositive_test_query319_00004", + "apositive_test_query82_00004", + "apositive_test_query319_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query83", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query83", + "ranked_ids": [ + "apositive_test_query146_00014", + "apositive_test_query168_00000", + "negative_test_query279_00012", + "negative_test_query168_00007", + "negative_test_query287_00004" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query84", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query84", + "ranked_ids": [ + "negative_test_query84_00013", + "negative_test_query268_00009", + "negative_test_query84_00006", + "negative_test_query84_00005", + "negative_test_query84_00017" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query85", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.4144299250118475, + "recall@5": 0.5 + }, + "query_id": "test_query85", + "ranked_ids": [ + "apositive_test_query286_00004", + "apositive_test_query85_00002", + "apositive_test_query81_00008", + "apositive_test_query85_00000", + "negative_test_query85_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query86", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.24630238874073, + "recall@5": 0.25 + }, + "query_id": "test_query86", + "ranked_ids": [ + "negative_test_query86_00001", + "apositive_test_query86_00000", + "negative_test_query86_00003", + "negative_test_query86_00000", + "negative_test_query86_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query87", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query87", + "ranked_ids": [ + "apositive_test_query87_00001", + "negative_test_query87_00002", + "negative_test_query182_00017", + "negative_test_query200_00011", + "negative_test_query315_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query88", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.6131471927654584, + "recall@5": 0.5 + }, + "query_id": "test_query88", + "ranked_ids": [ + "apositive_test_query88_00001", + "negative_test_query88_00002", + "negative_test_query88_00010", + "negative_test_query88_00012", + "negative_test_query105_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query89", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query89", + "ranked_ids": [ + "negative_test_query89_00005", + "negative_test_query68_00004", + "negative_test_query233_00005", + "apositive_test_query255_00000", + "negative_test_query89_00008" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query9", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query9", + "ranked_ids": [ + "negative_test_query9_00018", + "negative_test_query9_00013", + "negative_test_query9_00005", + "negative_test_query9_00016", + "negative_test_query9_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query90", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query90", + "ranked_ids": [ + "negative_test_query227_00006", + "negative_test_query90_00012", + "negative_test_query227_00001", + "negative_test_query127_00001", + "negative_test_query90_00002" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query91", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query91", + "ranked_ids": [ + "negative_test_query358_00004", + "apositive_test_query342_00001", + "apositive_test_query11_00006", + "negative_test_query279_00011", + "negative_test_query303_00011" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query92", + "metrics": { + "mrr@5": 0.3333333333333333, + "ndcg@5": 0.16958010263680806, + "recall@5": 0.16666666666666666 + }, + "query_id": "test_query92", + "ranked_ids": [ + "apositive_test_query100_00001", + "negative_test_query122_00018", + "apositive_test_query92_00001", + "apositive_test_query99_00003", + "negative_test_query283_00009" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query93", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.14285714285714285 + }, + "query_id": "test_query93", + "ranked_ids": [ + "apositive_test_query243_00006", + "apositive_test_query93_00003", + "apositive_test_query335_00003", + "negative_test_query108_00003", + "negative_test_query280_00012" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query94", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.5147714448836774, + "recall@5": 0.375 + }, + "query_id": "test_query94", + "ranked_ids": [ + "apositive_test_query127_00010", + "apositive_test_query94_00002", + "apositive_test_query94_00006", + "negative_test_query94_00002", + "apositive_test_query94_00005" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query95", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query95", + "ranked_ids": [ + "negative_test_query244_00010", + "apositive_test_query292_00001", + "negative_test_query95_00008", + "apositive_test_query177_00000", + "negative_test_query95_00006" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query96", + "metrics": { + "mrr@5": 1.0, + "ndcg@5": 0.46927872602275644, + "recall@5": 0.3333333333333333 + }, + "query_id": "test_query96", + "ranked_ids": [ + "apositive_test_query96_00002", + "negative_test_query339_00005", + "negative_test_query339_00007", + "negative_test_query96_00004", + "apositive_test_query339_00001" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query97", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query97", + "ranked_ids": [ + "negative_test_query51_00009", + "negative_test_query113_00009", + "negative_test_query113_00006", + "negative_test_query209_00008", + "negative_test_query16_00014" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query98", + "metrics": { + "mrr@5": 0.0, + "ndcg@5": 0.0, + "recall@5": 0.0 + }, + "query_id": "test_query98", + "ranked_ids": [ + "negative_test_query98_00018", + "negative_test_query98_00011", + "negative_test_query98_00005", + "apositive_test_query14_00002", + "negative_test_query23_00013" + ], + "valid": true + }, + { + "algorithm": "answer_confidence", + "fixture_id": "askubuntu-bm25-test-test_query99", + "metrics": { + "mrr@5": 0.5, + "ndcg@5": 0.21398626473452756, + "recall@5": 0.1111111111111111 + }, + "query_id": "test_query99", + "ranked_ids": [ + "apositive_test_query193_00010", + "apositive_test_query99_00001", + "apositive_test_query333_00003", + "apositive_test_query253_00003", + "apositive_test_query153_00012" + ], + "valid": true + } + ], + "query_id_filter": [], + "retry_sources": [ + "benchmark-results/live/askubuntu-bm25-top20-default-live.all-valid.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry-acurank-k5-b1.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry-rankgpt-w5.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry-single.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry2-acurank-k5-b1.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry2-rankgpt-w5.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry2-single.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry3-acurank-k5-b1.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry4-acurank-k5-b1.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry5-acurank-k5-b1.query_results.jsonl", + "benchmark-results/live/askubuntu-bm25-top20-default-live.v2.retry6-acurank-k5-b1.query_results.jsonl" + ], + "row_attempt_counts": { + "acurank_k5_b1": 403, + "original_bm25": 361, + "prp_sliding_p1": 361, + "rankgpt_sw_w5": 365, + "single_call_listwise@20": 374, + "tourrank_r2": 372 + }, + "row_attempt_notes": "tourrank_r2/prp_sliding_p1 rows are reused from the previous all-valid run because their method settings did not change in v2.", + "schema_version": 1, + "seed": 13, + "set_size": 3, + "stride": 10, + "timeout": 60.0, + "top_k": 5, + "tourrank_rounds": 2, + "tourrank_stage_policy": "paper_top_100_else_single_group_halving", + "window_size": 20 +} diff --git a/benchmarks/bm25.py b/benchmarks/bm25.py new file mode 100644 index 0000000..c87b72a --- /dev/null +++ b/benchmarks/bm25.py @@ -0,0 +1,127 @@ +"""Pure-Python Okapi BM25 for building first-stage candidate runs. + +This exists so benchmark tooling can produce BM25 hard negatives without an +external dependency such as Pyserini. Defaults match Pyserini's Lucene BM25 +(k1=0.9, b=0.4) so runs stay comparable with the documented +``bm25_top20_reranking`` setup; the tokenizer is a simple lowercase +alphanumeric split, so absolute scores are not identical to Lucene's. +""" + +from __future__ import annotations + +import math +import re +from collections import Counter, defaultdict +from collections.abc import Collection, Mapping +from dataclasses import dataclass + +DEFAULT_K1 = 0.9 +DEFAULT_B = 0.4 + +_TOKEN_PATTERN = re.compile(r"[a-z0-9]+") + + +def tokenize(text: str) -> list[str]: + return _TOKEN_PATTERN.findall(text.lower()) + + +@dataclass(frozen=True) +class BM25Index: + k1: float + b: float + document_ids: tuple[str, ...] + document_lengths: tuple[int, ...] + average_document_length: float + # term -> ((document position, term frequency), ...) + postings: Mapping[str, tuple[tuple[int, int], ...]] + inverse_document_frequency: Mapping[str, float] + + +def build_bm25_index( + documents: Mapping[str, str], + *, + k1: float = DEFAULT_K1, + b: float = DEFAULT_B, +) -> BM25Index: + if not documents: + raise ValueError("documents must not be empty") + if k1 <= 0: + raise ValueError("k1 must be greater than 0") + if not 0 <= b <= 1: + raise ValueError("b must be between 0 and 1") + + document_ids = tuple(sorted(documents)) + document_lengths: list[int] = [] + postings: dict[str, list[tuple[int, int]]] = defaultdict(list) + for position, document_id in enumerate(document_ids): + tokens = tokenize(documents[document_id]) + document_lengths.append(len(tokens)) + for term, frequency in sorted(Counter(tokens).items()): + postings[term].append((position, frequency)) + + total_documents = len(document_ids) + inverse_document_frequency = { + # Lucene-style BM25 idf: always positive, robust for df close to N. + term: math.log( + 1.0 + + (total_documents - len(term_postings) + 0.5) / (len(term_postings) + 0.5) + ) + for term, term_postings in postings.items() + } + return BM25Index( + k1=k1, + b=b, + document_ids=document_ids, + document_lengths=tuple(document_lengths), + average_document_length=sum(document_lengths) / total_documents, + postings={ + term: tuple(term_postings) for term, term_postings in postings.items() + }, + inverse_document_frequency=inverse_document_frequency, + ) + + +def bm25_search( + index: BM25Index, + query: str, + *, + top_k: int, + exclude: Collection[str] = (), +) -> list[tuple[str, float]]: + """Return up to ``top_k`` ``(document_id, score)`` pairs, best first. + + Documents sharing no term with the query are never returned, so the + result can be shorter than ``top_k`` (or empty). Ties break on document + id so runs are deterministic. + """ + if top_k < 1: + raise ValueError("top_k must be greater than 0") + excluded_ids = frozenset(exclude) + scores: dict[int, float] = defaultdict(float) + average_length = index.average_document_length + for term in tokenize(query): + term_postings = index.postings.get(term) + if term_postings is None: + continue + idf = index.inverse_document_frequency[term] + for position, frequency in term_postings: + length_norm = ( + 1.0 + - index.b + + index.b * (index.document_lengths[position] / average_length) + ) + scores[position] += ( + idf + * frequency + * (index.k1 + 1.0) + / (frequency + index.k1 * length_norm) + ) + ranked = sorted( + ( + (index.document_ids[position], score) + for position, score in scores.items() + if index.document_ids[position] not in excluded_ids + ), + key=lambda pair: (-pair[1], pair[0]), + ) + return ranked[:top_k] diff --git a/docs/benchmarks/answer_confidence_askubuntu.md b/docs/benchmarks/answer_confidence_askubuntu.md new file mode 100644 index 0000000..9f0f346 --- /dev/null +++ b/docs/benchmarks/answer_confidence_askubuntu.md @@ -0,0 +1,128 @@ +# answer_confidence를 README 표준 벤치마크에 올리는 러닝북 + +이 문서는 `answer_confidence` reranker를 README의 표준 벤치마크 표 +(AskUbuntuDupQuestions 361쿼리, Pyserini BM25 top-20, `@5` 평가, +`gpt-5.4-nano`)에서 **기존 메소드들과 같은 조건으로** 측정하는 절차를 +기록합니다. 기존 7개 메소드의 결과는 +`benchmark-results/live/askubuntu-bm25-top20-default-live.v3.merged.json`에 +커밋되어 있으므로 다시 돌릴 필요가 없습니다. `answer_confidence`만 같은 +candidate/qrels/모델로 돌린 뒤 병합합니다. + +## 왜 별도 학습 데이터가 필요한가 (정직성 요구사항) + +- `answer_confidence`는 학습된 scorer artifact가 전제조건입니다. 학습 + 라벨은 "모델 답변이 gold answer와 일치했는가"인데, **AskUbuntu에는 + qrels만 있고 free-text gold answer가 없어 artifact를 AskUbuntu로 학습할 + 수 없습니다.** +- 따라서 artifact는 QA 데이터(SQuAD v1.1)로 학습하며, 결과는 **도메인 + 밖 artifact 측정**입니다. 표에 올릴 때 artifact 학습 도메인을 반드시 + 함께 표기합니다(스펙 + `docs/specs/spec_confidence_aware_reranking.md`의 보고 규칙). +- 학습 데이터는 질문마다 gold 문맥 1개 + BM25 hard negative 문맥 + 1개(기본)로 구성합니다. 추론 시 estimator는 관련/무관 후보 문서 양쪽에서 + 나온 답변을 채점하므로, 학습 분포도 같아야 합니다. + +## 전제조건 + +- `.benchmark-cache/askubuntu-bm25/` 캐시(corpus/queries/qrels). 리포에는 + gitignore되어 없습니다. 이 캐시는 MTEB `AskUbuntuDupQuestions`에서 + export한 upstream artifact이며, README 벤치마크를 만든 환경에 있습니다. +- 후보 run은 리포에 커밋되어 있습니다: + `benchmark-results/pyserini/askubuntu-bm25-top20.trec`. +- 모델 접근: 기존 표와 같은 `gpt-5.4-nano` Azure 배포(.env), 또는 로컬 + 스모크용 OpenAI 호환 endpoint(`RANKSMITH_OPENAI_BASE_URL`). +- `confidence-train` extra 설치 (torch/transformers/lightgbm/scikit-learn). + +## 1단계 — 학습 데이터 빌드 (LLM 호출 없음, 1회) + +```bash +uv run python scripts/build_answer_confidence_training_data.py \ + --download \ + --output .benchmark-cache/answer-confidence/answer_train.jsonl +``` + +- 기본값: 질문 250개 × (gold 1 + hard negative 1) = 500행. 스펙 기록상 + 100행은 과적합(랜덤 이하), 500행은 유효했습니다. +- 결정적(seed 13)이며 `answer_train.report.json`에 통계와 SQuAD 파일 + sha256이 남습니다. + +## 2단계 — artifact 학습 (LLM 호출: 행 수만큼, 기본 500회) + +**벤치마크와 같은 배포로 generation하세요.** estimator는 추론 시점에 그 +모델이 만든 답변을 채점하므로, 학습 답변도 같은 모델이어야 공정합니다. + +```bash +# Azure(gpt-5.4-nano)는 .env에서 읽습니다. 로컬 스모크는 +# RANKSMITH_OPENAI_BASE_URL(+_MODEL)로 대체할 수 있습니다. +uv run python scripts/train_answer_confidence.py \ + --input .benchmark-cache/answer-confidence/answer_train.jsonl \ + --workdir .benchmark-cache/answer-confidence/work \ + --artifact .benchmark-cache/answer-confidence/answer_confidence.joblib \ + --allow-truncation \ + --allow-live +``` + +- 끝에 test split 지표가 출력되고, **test roc_auc < 0.6이면 스크립트가 + 실패합니다.** 그 artifact로는 벤치마크를 돌리지 마세요(행 수를 늘리거나 + `generated.jsonl` 라벨을 점검). +- artifact는 커밋하지 않습니다(`.gitignore` 정책 유지). + +## 3단계 — answer_confidence 단독 run (LLM 호출: 361 × 20 = 7,220회) + +기존 optional method 관행대로 별도 run으로 실행합니다. + +```bash +uv run python scripts/compare_reranking.py \ + --dataset benchmark-cache \ + --dataset-name askubuntu-bm25 \ + --cache-dir .benchmark-cache/askubuntu-bm25 \ + --split test \ + --candidates benchmark-results/pyserini/askubuntu-bm25-top20.trec \ + --candidate-count 20 \ + --algorithm answer_confidence \ + --answer-confidence-artifact .benchmark-cache/answer-confidence/answer_confidence.joblib \ + --top-k 5 \ + --window-size 20 \ + --stride 10 \ + --output benchmark-results/askubuntu-bm25-top20-answer-confidence.json \ + --allow-live +``` + +- 플래그(`--dataset-name`, `--candidate-count`, `--top-k`)는 커밋된 + v3 결과와 정확히 같아야 4단계 병합 검증을 통과합니다. +- `--algorithm`은 이제 반복 지정으로 여러 메소드를 한 번에 같은 케이스에서 + 돌릴 수도 있습니다(예: `--algorithm original_bm25 --algorithm + answer_confidence` 재검증 run). + +## 4단계 — 커밋된 기존 결과와 병합 + +```bash +uv run python scripts/merge_benchmark_reports.py \ + benchmark-results/live/askubuntu-bm25-top20-default-live.v3.merged.json \ + benchmark-results/askubuntu-bm25-top20-answer-confidence.json \ + --output benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json +``` + +병합 스크립트는 벤치마크 정체성(dataset, candidate 파일명, candidate 수, +case 수, top_k)과 **알고리즘별 query_id 집합의 완전 일치**를 검증하고, +동일하지 않으면 거부합니다. 수동 병합으로 생기던 조건 불일치 사고를 +막기 위한 것입니다. + +## 보고 규칙 + +- README 표에 올릴 때 method 이름에 artifact 도메인을 병기합니다. 예: + `answer_confidence` *(scorer: SQuAD v1.1, 도메인 밖)*. +- Nominal LLM calls/query = 20 (후보당 answer 1회). 여기에 더해 로컬 + encoder+scorer 추론이 문서당 1회 있습니다(LLM 아님). +- artifact 학습 generation 비용(행당 1회)은 reranking 호출과 분리해서 + 보고합니다. +- 스모크/부분 run(예: `--max-cases`)은 품질 수치로 보고하지 않습니다 + (`docs/benchmarks/bm25_top20_reranking.md` 규칙 동일). + +## 이 러닝북이 검증된 범위 + +전 단계(빌드 → generation → 학습 → artifact → 단독 run → 다중 +알고리즘 run → 병합)는 실제 SQuAD 다운로드 + 결정적 fake +OpenAI-호환 서버 + 로컬 소형 encoder로 기계적으로 관통 확인됐습니다 +(2026-07-16, 원격 개발 환경). LLM 품질에 좌우되는 수치는 이 러닝북을 +실행하는 환경에서 나옵니다. diff --git a/docs/benchmarks/bm25_top20_reranking.md b/docs/benchmarks/bm25_top20_reranking.md index bc8ac47..49f4138 100644 --- a/docs/benchmarks/bm25_top20_reranking.md +++ b/docs/benchmarks/bm25_top20_reranking.md @@ -53,6 +53,10 @@ Optional methods: | `acurank_b4` | `N=20`, `target_rank=10`, `w=20`, adaptive budget 4 | legacy stronger AcuRank top-10-boundary reference | 5 | | `tourrank_r10` | `N=20`, `r=10` | high-cost setwise reference | 40 | | `prp_sliding_p3` | `N=20`, `passes=3` | stronger but expensive PRP | 114 | +| `answer_confidence` | `N=20`, 학습된 scorer artifact 필요 | confidence-change reranker (도메인 밖 artifact 라벨링 필수) | 20 | + +`answer_confidence`는 학습된 artifact가 전제조건이며 절차와 보고 규칙이 +별도 러닝북에 있습니다: `docs/benchmarks/answer_confidence_askubuntu.md`. `prp_sliding_p1`은 asymptotic하게 O(n)이지만, 현재 `PairwiseStrategy`는 인접 pair마다 양방향 비교를 수행합니다. 그래서 20개 후보에서는 `2 * (20 - 1) = 38` calls/query로 추정합니다. diff --git a/docs/specs/TEMPLATE.md b/docs/specs/TEMPLATE.md index 470d75d..18eed64 100644 --- a/docs/specs/TEMPLATE.md +++ b/docs/specs/TEMPLATE.md @@ -31,7 +31,7 @@ - **성공 케이스 (Happy Paths)**: 정상적으로 동작해야 하는 시나리오, 기댓값, 필수 모의(Mock) 데이터 구조를 명시합니다. - **엣지/실패 케이스 (Edge & Failure Cases)**: 경계값, 특이한 입력, 또는 에러가 발생해야 하는 시나리오와 예상 Exception을 명시합니다. - **공통 Reranking Smoke/Benchmark**: - - reranking algorithm을 추가하거나 기존 algorithm의 순위 생성 로직을 바꾸는 경우, `tests/fixtures/reranking_smoke_fixture.jsonl`과 `src/ranksmith/_metrics.py`를 재사용해 실제 query/documents/qrels 기반 smoke test를 추가하거나 갱신합니다. + - reranking algorithm을 추가하거나 기존 algorithm의 순위 생성 로직을 바꾸는 경우, `tests/fixtures/reranking_smoke_fixture.jsonl`과 `benchmarks/metrics.py`를 재사용해 실제 query/documents/qrels 기반 smoke test를 추가하거나 갱신합니다. - LLM provider를 사용하는 algorithm이면 `scripts/compare_reranking.py`의 비교 대상에 포함하고, live 실행은 명시적 opt-in으로만 수행합니다. - 완료 판단에는 synthetic provider 테스트와 실제 fixture 기반 metric 검증을 모두 포함합니다. live provider metric은 credential/cost 때문에 별도 기록으로 둡니다. diff --git a/docs/specs/spec_confidence_aware_reranking.md b/docs/specs/spec_confidence_aware_reranking.md new file mode 100644 index 0000000..85e127a --- /dev/null +++ b/docs/specs/spec_confidence_aware_reranking.md @@ -0,0 +1,206 @@ +# Spec: Confidence-Aware Reranking (초안) + +## 1. 개요 (Overview) +- **작업 목적**: 구현 완료된 confidence 신호(`ranksmith.confidence`)를 실제 reranking 결정에 연결한다. runtime readiness 스펙이 "CBDR-ready confidence signal"까지 완성해 두었으므로, 이 스펙은 그 신호를 소비하는 첫 모듈을 정의한다. +- **Reference**: + - `docs/wiki/references/structural_confidence.md` (Trust in One Round — 구현된 confidence 신호의 근간) + - `docs/wiki/references/cbdr_parametric_confidence_rag.md` (CBDR — 개념 로드맵. white-box 전제라 방식 직접 이식 불가) + - `docs/wiki/references/llm_confidence_reranker.md` (LCR — training-free 정렬 규칙의 후보) + - `docs/specs/spec_confidence_runtime_readiness.md` (소비할 signal contract) +- **상태**: `[ ] Draft` | `[x] In Progress` | `[ ] Completed` + +Q004-1(경로 A), Q004-2(새 Strategy)가 확정됐고, 순위 규칙은 논문 충실도를 위해 **confidence 변화(answer_confidence) 기반**으로 최종 결정됐다(초기 signed-judgment 방식을 교체). `AnswerConfidenceRerankStrategy` / `AsyncAnswerConfidenceRerankStrategy`를 구현했다. 남은 것은 쓸 만한 scorer artifact(현재는 스모크용)와 벤치마크. + +## 확정 설계 (구현됨) — confidence 변화 기반 +CBDR 논문의 핵심은 confidence 변화다: `Inc(Q, D) = Conf(질문+문서) − Conf(질문)`. 문서가 답변 확신을 얼마나 높이는가로 순위를 매긴다. 우리는 이를 black-box(구조적 confidence)로 구현한다. + +- **형태**: 새 Strategy. `AzureOpenAIReranker(strategy=AnswerConfidenceRerankStrategy(estimator=...))`. +- **answer 획득**: 문서당 LLM `answer(query, document)` 1회 — 그 문서로 질문에 답하게 함. confidence 측정에는 LLM을 쓰지 않는다(로컬 encoder+scorer). LCR처럼 문서당 반복 샘플링하지 않는다. +- **confidence**: `estimator.score(AnswerConfidenceInput(context=document, answer=answer)).score` — "이 답변이 맞을 확률"(structural confidence, 로컬). +- **순위 규칙**: `C_i = Conf(문서_i + 답변_i)` 내림차순. **기준선 생략의 근거**: CBDR은 `Inc = C_i − C_0`로 정렬하는데, 한 질문 안에서 `C_0`(질문만)는 모든 문서에 동일한 상수라 **순위에 영향이 없다**. 게다가 `answer_confidence`는 빈 context를 스코어링할 수 없어(`_require_non_empty`) `C_0` 자체를 못 구한다. 따라서 `C_i`로 정렬 = 질문 내 confidence 변화 정렬. +- **비용**: 문서 N개 → answer N회(리랭킹 신호) + 로컬 confidence N회(LLM 0회). 기준선 호출 없음. +- **async**: answer 호출은 `asyncio.gather`로 동시 실행, scoring은 순차(CPU/encoder). + +### 통합 지점 (구현 완료) +- `src/ranksmith/model.py`: `ModelClient.answer` / `AsyncModelClient.answer` 추가(generation 파이프라인의 answer 프롬프트와 일치 — 학습/추론 일관성). +- `src/ranksmith/parsing.py`: `parse_answer_response` 추가. +- `src/ranksmith/strategies/confidence.py`: `AnswerConfidenceRerankStrategy` / `AsyncAnswerConfidenceRerankStrategy`. +- estimator는 Protocol로 타입해 strategies가 torch를 강제 import하지 않는다. + +### 검증 (2026-07-05) +- 단위 테스트 8개(synthetic answer client + fake estimator): confidence 내림차순 정렬, 동점 보존, top_k, answer 부재/비-answer estimator/invalid JSON fast fail, sync/async. +- 엔드투엔드 + 벤치마크(LM Studio qwen3.5-9b answer + SQuAD로 학습한 answer_confidence artifact, macOS env 2개). held-out 15개 질문(각: 정답 문맥 1 + 무관 distractor 3), 정답 문맥 복원율: + + | 학습 데이터 | test roc_auc | rerank accuracy@1 | MRR | + | --- | --- | --- | --- | + | 100 샘플 | 0.333 | 0.000 | 0.294 | + | 500 샘플 | 0.875 | **0.800** | **0.878** | + | 랜덤 기준 | 0.5 | 0.250 | 0.521 | + + **데이터 규모가 관건이다.** 100샘플에선 과적합으로 랜덤 이하(정답 문맥이 오히려 바닥). 500샘플에선 강한 판별력(정답 문맥 12/15가 1위). 접근 자체는 유효하다. + +- **Baseline 비교 (같은 15개 세팅) — 정직한 결과: 기존 Listwise한테 진다.** + + | 전략 | accuracy@1 | MRR | LLM 호출/쿼리 | + | --- | --- | --- | --- | + | ConfidenceRerank (500) | 0.800 | 0.878 | 4.0 | + | `ListwiseStrategy` | **1.000** | **1.000** | **1.0** | + + 이 세팅에서 Listwise가 완벽하면서 호출은 1/4. confidence 리랭커는 더 나쁘고 4배 비싸다. **현재까지 이 방법이 기존 전략보다 나은 세팅을 하나도 입증하지 못했다.** 이길 가능성이 있는 곳은 listwise window를 초과하는 대규모 후보 등 — 미측정. + +- **측정 조건(정직한 한계)**: distractor가 무관 랜덤 SQuAD 문맥이라 모델이 NO_ANSWER를 내는 쉬운 세팅(그래서 Listwise가 자명하게 만점). BM25 hard negative(주제 유사) 같은 현실 세팅은 더 어렵다. 평가셋 15개로 신뢰구간이 넓다. README 성능 claim은 하지 않는다. + +### 표준 벤치마크 편입 (구현됨) + 특이사항 +`scripts/compare_reranking.py`에 `answer_confidence` 알고리즘을 추가했다(opt-in). +`--algorithm answer_confidence --answer-confidence-artifact --allow-live`로 +다른 전략과 같은 qrels 기반 지표(NDCG/Recall/MRR)로 비교할 수 있다. + +**특이사항 (반드시 인지):** +- **이 벤치마크는 IR(qrels)이라 answer_confidence scorer를 학습할 수 없다.** 학습 + 라벨은 "모델 답변이 gold answer와 일치했나"인데, IR 벤치마크(AskUbuntu/SciFact + qrels)에는 free-text gold answer가 없다. 따라서 artifact는 **별도 QA 데이터로 + 학습해서 경로로 넘겨야** 한다. +- **다른 도메인(예: SQuAD)에서 학습한 artifact를 넘기면 도메인 shift를 측정하는 + 것이지 공정한 비교가 아니다.** 결과에 그 사실을 라벨링해야 한다. (CLI help에도 + 명시.) +- 스크립트에 `RANKSMITH_OPENAI_BASE_URL`(+`_MODEL`/`_API_KEY`) 경로를 추가해 + LM Studio 등 OpenAI 호환 엔드포인트로도 돌릴 수 있다(로컬/테스트용). Azure는 + 그 env가 없을 때의 기본 경로. `mypy src`만 CI 게이트이고 scripts는 타입체크 + 대상이 아니다. +- **README의 표준 벤치마크(AskUbuntu 361쿼리, BM25 top-20, gpt-5.4-nano)에 + answer_confidence를 공정하게 올리는 건 이 개발 환경에선 불가**: (1) AskUbuntu + corpus 캐시(`.benchmark-cache/askubuntu-bm25`)가 gitignore라 리포에 없고, + (2) Azure가 VNet 차단이며, (3) AskUbuntu는 gold answer가 없어 answer_confidence + artifact 자체를 학습할 수 없다. → **AskUbuntu corpus + Azure 접근이 있는 다른 + 환경에서 PR을 받아 실행**해야 한다. 그 환경에서도 artifact는 QA 데이터로 별도 + 학습해야 하며, SQuAD 등 도메인 밖 artifact를 쓰면 참고 수치(도메인 shift)일 뿐 + README 표에 넣을 공정 비교가 아니다. +- **(2026-07-16 갱신) 그 실행을 턴키로 만드는 도구가 추가됐다** — 절차 전체는 + `docs/benchmarks/answer_confidence_askubuntu.md` 러닝북 참고: + - `scripts/build_answer_confidence_training_data.py`: SQuAD v1.1 train에서 + 질문당 gold 문맥 + BM25 hard negative 문맥 쌍을 만드는 결정적 빌더 + (`benchmarks/bm25.py` 순수 파이썬 BM25 포함). 추론 분포(관련/무관 후보 + 양쪽의 답변 채점)와 학습 분포를 일치시킨다. + - `scripts/train_answer_confidence.py`: generation(라이브 모델) → labeling → + training → artifact 내보내기 턴키. test roc_auc < 0.6이면 fast fail + (100샘플 과적합 사고 재발 방지 게이트). + - `scripts/compare_reranking.py`: `--algorithm` 반복 지정으로 여러 메소드를 + 동일 케이스에서 한 번에 실행 가능. answer_confidence 선택 시 artifact + 경로/존재를 실행 전에 검증(fast fail). + - `scripts/merge_benchmark_reports.py`: 커밋된 v3.merged.json 같은 기존 + 결과와 새 단독 run을 병합. 벤치마크 정체성과 알고리즘별 query_id 집합 + 일치를 검증해 조건 불일치 병합을 거부한다. + - 전 경로(빌드→생성→학습→벤치마크→병합)는 실제 SQuAD 다운로드 + 결정적 + fake OpenAI 호환 서버 + 로컬 소형 encoder로 기계 검증 완료. LLM 품질 + 수치만 실행 환경 몫이다. + +#### 예비 진단 (LM Studio, SciFact, 도메인 밖 artifact — README에 넣지 말 것) +official 스크립트를 LM Studio(qwen3.5-9b)로 SciFact 15케이스(oracle+random, +후보 10)에 돌린 참고 수치. artifact는 SQuAD 학습(도메인 밖), distractor는 무관 +랜덤(쉬운 세팅): + +| 전략 | NDCG@5 | MRR@5 | Recall@5 | 호출/쿼리 | +| --- | --- | --- | --- | --- | +| answer_confidence (SQuAD art.) | 0.311 | 0.212 | 0.656 | 10 | +| rankgpt_sliding_window | 1.000 | 1.000 | 1.000 | 1 | + +listwise가 완벽하고 answer_confidence는 훨씬 나쁘며 10배 비싸다. 단 위의 불리한 +조건(도메인 밖 + 쉬운 distractor + 작은 로컬 모델) 때문이며, 공정 비교가 아니다. + +### 남은 작업 +- ~~README 표 편입 실행~~: 2026-07-20 완료. AskUbuntu 361쿼리, BM25 top-20, + gpt-5.4-nano 환경에서 러닝북(빌드→학습→벤치마크→병합) 전체를 실행했다. + `answer_confidence` NDCG@5=0.1722 / MRR@5=0.2862 / Recall@5=0.1435, + 361/361 valid, invalid_rate 0.000. BM25 baseline(NDCG@5=0.3520)보다 낮다 — + SQuAD 학습 → AskUbuntu 도메인 밖이므로 예상된 결과이며, 이기도록 튜닝하지 + 않고 측정된 그대로 README에 반영했다. 근거: + `benchmark-results/live/askubuntu-bm25-top20-default-live.v4.merged.json`. + 실행 환경 노트: 이 macOS 환경에서 torch로 BERT를 인코딩한 뒤 같은 프로세스에서 + LightGBM `.fit()`을 호출하면 세그폴트가 재현됐다(torch/lightgbm의 OpenMP + 런타임 충돌로 추정, `OMP_NUM_THREADS=1`/`KMP_DUPLICATE_LIB_OK=TRUE`로도 + 해결되지 않음). feature 추출과 LightGBM 학습을 별도 프로세스로 분리해 + 우회했다 — 동일 하드웨어/라이브러리 조합에서 이 러닝북을 재현하려는 + 사람을 위해 기록. +- IR에 맞는 변형: qrels로 학습 가능한 judgment_confidence 리랭커(초기 구현 후 + 교체됨)를 되살리면 이 IR 벤치마크로 공정 비교 가능. +- 학습된 artifact는 커밋하지 않는다(`.gitignore`). 배포 시 사용자가 자기 도메인 + 데이터로 학습. + +### 범위에 대한 사실 정리 +- CBDR 논문의 CBDR 메커니즘 자체는 "retrieval을 할지 결정하는 트리거"다. ranksmith에는 retriever가 없으므로 **retrieval 트리거는 이 스펙의 범위 밖**이며 caller 애플리케이션 책임이다. +- CBDR 논문의 confidence 추출(LLM hidden state 직접 접근)과 reranker fine-tuning은 ranksmith의 폐쇄형 모델 / training-free 전제와 충돌한다. 이 스펙이 논문에서 가져오는 것은 **"confidence 신호로 문서 우선순위를 조정한다"는 설계 패턴**뿐이다. + +## 2. 요구 사항 및 제약 (Requirements & Constraints) + +> **초안 단계 기록 (superseded).** 아래 §2~§7은 설계 초안 시점의 요구/계획 +> 기록이다. 최종 확정·구현된 설계는 상단 "확정 설계 (구현됨)" 섹션이 정본이며, +> 초안과의 주요 차이는: ① confidence 소스가 `judgment_confidence`가 아니라 +> **`answer_confidence`**(문서로 답변 생성 → 답변 신뢰도 채점)로 바뀌었고, +> ② LCR binning/threshold/환원 gate는 **미채택** — 구현은 confidence 원점수 +> 내림차순 + 동점 시 원래 순서 유지다(채택 여부는 실측 후 재검토, Q004 참조). + +### 포함 범위 +- 기존 reranking 결과(`list[RerankResult]`) 또는 첫 단계 순서를 입력으로 받아, 후보별 confidence로 순위를 보정하는 모듈 1개. +- confidence 소스는 **경로 A로 확정**(Q004-1 resolved). 호출 비용 때문에 경로 B(LCR/MSCP)는 채택하지 않는다. + - ~~**경로 A 초안**: `StructuralConfidenceEstimator.score_batch(...)`의 `judgment_confidence` 신호.~~ → **교체됨(구현)**: 문서당 `ModelClient.answer` 1회로 답변을 만들고 `estimator.score(AnswerConfidenceInput(context, answer))`로 채점하는 `answer_confidence` 신호. candidate identity 결합은 Strategy 내부 책임. + - ~~경로 B (LCR/MSCP)~~: 문서당 K회+ 호출 비용으로 미채택. +- ~~정렬 규칙은 LCR 방식을 기본 후보로 한다: `T_upper`/`T_lower` binning + StableSort, query-level gate.~~ → **교체됨(구현)**: confidence 원점수 내림차순 + 동점 시 원래 순서 유지. binning/gate는 미채택(실측 후 재검토). +- ~~기존 순위로의 환원성 보장: gate 조건에서 원래 순위를 그대로 반환.~~ → gate 미채택으로 해당 없음. fast-fail 원칙은 유지(아래 제약). + +### 제외 범위 +- retrieval 트리거 (CBDR 본래 의미) — caller 책임. +- reranker/scorer fine-tuning — 경로 A의 artifact 학습은 기존 `confidence_training` 스펙 소관. +- 벤치마크 수치 주장 — 측정 전에는 README에 성능 claim을 쓰지 않는다. + +### 제약 +- fast fail 원칙 유지. silent fallback 금지 — confidence 계산 실패 시 원래 순위로 조용히 돌아가지 않고 에러를 낸다. +- ~~경로 B 채택 시 sampling 전용 요청 경로가 선행 결정 사항이다.~~ → 경로 B 미채택으로 해당 없음. +- 경로 A는 학습된 scorer artifact가 선행 조건이다. 리포에 커밋된 artifact는 없다(정책) — `scripts/build_answer_confidence_training_data.py` → `scripts/train_answer_confidence.py`로 만들어야 한다. + +## 3. 상세 설계 (Architecture & Design) — 초안 기록 (superseded) + +> 확정 메커니즘은 상단 "확정 설계 (구현됨)"가 정본이다. 아래는 초안 시점 골격. + +### 동작 메커니즘 (초안 골격 — binning은 미채택됨) +1. 입력: `query`, 순서 있는 후보 목록(기존 순위 = PrevRank), confidence 파라미터. +2. 후보별 confidence 계산 (경로 A: score_batch 순서 보존 결과를 zip / 경로 B: MSCP). +3. ~~binning: `c >= T_upper → +1`, `c <= T_lower → −1`, 그 외 0.~~ +4. ~~StableSort by (bin desc, PrevRank asc). gate 조건이면 PrevRank 그대로.~~ +5. 출력: `list[RerankResult]`, metadata에 confidence 점수 포함(구현: `answer_confidence` 키). + +### 통합 지점 (초안 후보 → 형태 1로 확정) +- **형태 1: 새 Strategy — 채택됨** (`AnswerConfidenceRerankStrategy`, `docs/wiki/08_custom_strategy_extension.md` 패턴). 다른 Strategy와 조합하려면 caller가 2단계 호출. +- ~~형태 2: `list[RerankResult]`를 받는 post-processor 함수~~ — 미채택 (Q004-2 resolved). + +## 5. 에러 핸들링 (Error Handling) +- confidence 소스 실패(artifact 불일치 등): 해당 confidence 에러로 fast fail. +- ~~threshold 검증: `0 <= T_lower <= T_upper <= 1` 위반 시 `ValueError`.~~ → threshold 미채택으로 해당 없음. 구현된 검증: estimator `task_type` 불일치·invalid answer JSON·빈/공백 문서·잘못된 `top_k`는 `RerankError` 계열로 fast fail. +- 빈 후보 목록: 빈 결과 반환 (기존 Strategy들과 동일). + +## 6. 테스트 계획 (Test Plan) — 초안 (superseded) +> 실제 수행된 검증은 상단 "검증 (2026-07-05)" 및 `tests/test_confidence_rerank.py`, +> `tests/test_model_architecture.py`, `tests/test_public_protocols.py`의 +> answer 경로 테스트가 정본이다. ~~binning 경계값·환원성·StableSort bin 테스트~~는 +> binning 미채택으로 함께 폐기됐다. + +## 7. 열린 결정 (`docs/wiki/05_open_questions.md` Q004) — 전부 해소됨 +1. ~~confidence 소스~~ — **경로 A(structural + LightGBM)로 확정**. 호출 비용으로 경로 B 미채택. +2. ~~형태~~ — **새 Strategy로 확정** (`AnswerConfidenceRerankStrategy` / async 변형, 구현 완료). +3. ~~score → 순위 반영 규칙~~ — **원점수 내림차순 + 동점 시 원래 순서 유지로 확정**. binning/threshold는 미채택(실측 후 재검토). +4. ~~artifact 학습 데이터~~ — **gold answer가 있는 QA 데이터로 확정**. SQuAD v1.1 경로가 도구화됨(`scripts/build_answer_confidence_training_data.py`), ≥30 샘플 요건은 기본 500행으로 충족. + +## 스모크 검증 기록 (2026-07-05) +경로 A 파이프라인을 LM Studio(qwen3.5-9b) + libomp 설치 환경에서 부분 관통했다. +- generation: `generate_judgment_confidence_dataset`가 fixture 15개를 엔드투엔드 생성 (LM Studio provider). +- 추론 절반: `FrozenAutoEncoder`(bert-base-uncased) 로드 + 70차원 `structural-v1` 특징 추출 확인. +- 학습: `split.py`의 `MIN_TOTAL_SAMPLES = 30` 가드에 막힘 — 리포 fixture는 15개뿐. (이후 SciFact 실데이터 56개, 그리고 SQuAD 빌더 기본 500행으로 해소 — Q004 참조.) + +## 작업 태스크 추적 (Task Checklist) +### Phase 0: 결정 +- [x] Q004-1 confidence 소스 = 경로 A +- [x] Q004 나머지 (형태 = 새 Strategy, 순위 규칙 = 원점수 내림차순, 학습 데이터 = QA/SQuAD) +### Phase 1 이후 +- [x] ≥30개 실제 라벨 데이터로 scorer artifact 생성 (SciFact 56개 스모크 → SQuAD 500행 도구화) +- [x] 구현: `AnswerConfidenceRerankStrategy` / async 변형 + `ModelClient.answer` + 단위 테스트 +- [x] 표준 벤치마크 편입 도구 + 러닝북 (`docs/benchmarks/answer_confidence_askubuntu.md`) +- [x] README 표 편입 실행 (AskUbuntu 캐시 + gpt-5.4-nano 환경에서 러닝북 실행, 도메인 라벨 필수) — 2026-07-20 완료. `answer_confidence` NDCG@5=0.1722, MRR@5=0.2862, Recall@5=0.1435, 361/361 valid, invalid_rate 0.000. BM25 baseline보다 낮음(예상대로, SQuAD 학습 → AskUbuntu 도메인 밖). diff --git a/docs/specs/spec_confidence_generation_pipeline.md b/docs/specs/spec_confidence_generation_pipeline.md index 87307d1..dc4367b 100644 --- a/docs/specs/spec_confidence_generation_pipeline.md +++ b/docs/specs/spec_confidence_generation_pipeline.md @@ -227,10 +227,11 @@ Normalization: - collapse internal whitespace No-answer policy: -- `no_answer_value`는 non-empty string이어야 한다. -- closed model answer가 `no_answer_value`와 정확히 같으면 항상 `label=0`이다. -- `gold_answer`가 우연히 `no_answer_value`와 같아도 match로 인정하지 않는다. -- 기본 sentinel은 `"__NO_ANSWER__"`이다. +- sentinel은 `ranksmith.model.NO_ANSWER_VALUE`(`"__NO_ANSWER__"`) 고정 상수다. + (초안의 config 필드 `no_answer_value`는 answer prompt를 리랭커와 공유 상수로 + 통일하면서 제거됐다 — 학습/추론 프롬프트 일관성 요구사항.) +- closed model answer가 sentinel과 정확히 같으면 항상 `label=0`이다. +- `gold_answer`가 우연히 sentinel과 같아도 match로 인정하지 않는다. 제외: - punctuation removal @@ -307,7 +308,9 @@ Generation metadata는 output `metadata.generation` 아래에 nested로 기록 - `include_raw_model_output: bool = True` - `on_usage: Callable[[RerankUsage], None] | None = None` - `source: str | None = None` -- `no_answer_value: str = "__NO_ANSWER__"` + +(sentinel은 config가 아니라 `ranksmith.model.NO_ANSWER_VALUE` 공유 상수를 쓴다. +초안에 있던 `no_answer_value` 필드는 제거됐다.) #### `RelevanceGenerationConfig` @@ -589,7 +592,6 @@ Provider 호출 실패: - `max_items < 1`. - `max_context_chars < 1`. - `max_document_chars < 1`. -- `no_answer_value`가 non-empty string이 아님. - metadata가 JSON object가 아님. - metadata key가 string이 아님. - metadata value가 JSON-serializable이 아님. @@ -623,7 +625,7 @@ Resume 실패: - answer raw JSONL을 읽고 `answer_confidence` canonical JSONL을 생성한다. - answer normalized exact match가 `label=1`을 만든다. - answer mismatch가 `label=0`을 만든다. -- answer가 `no_answer_value`와 같으면 gold answer와 무관하게 `label=0`을 만든다. +- answer가 `NO_ANSWER_VALUE` sentinel과 같으면 gold answer와 무관하게 `label=0`을 만든다. - gold answer list 중 하나와 match하면 `label=1`이다. - relevance raw JSONL을 읽고 `judgment_confidence` canonical JSONL을 생성한다. - `relevance_label > 0` 기본 truth가 relevant를 만든다. @@ -652,7 +654,6 @@ Resume 실패: - invalid `max_items`. - invalid `max_context_chars`. - invalid `max_document_chars`. -- invalid `no_answer_value`. - invalid metadata. - provider empty response. - invalid provider JSON. diff --git a/docs/wiki/00_context.md b/docs/wiki/00_context.md index f645d88..e65e858 100644 --- a/docs/wiki/00_context.md +++ b/docs/wiki/00_context.md @@ -53,6 +53,8 @@ LLM 기반 reranking을 위한 작고 신뢰성 있는 Python 패키지를 만 - `TourRankStageConfig` - `AcuRankStrategy` - `AsyncAcuRankStrategy` +- `AnswerConfidenceRerankStrategy` (experimental) +- `AsyncAnswerConfidenceRerankStrategy` (experimental) - `RerankError` - `RerankInputError` - `RerankParseError` diff --git a/docs/wiki/02_architecture.md b/docs/wiki/02_architecture.md index e5d1b51..18d9f80 100644 --- a/docs/wiki/02_architecture.md +++ b/docs/wiki/02_architecture.md @@ -2,7 +2,7 @@ ## 계층 - ModelProvider: vendor별 JSON completion 호출을 담당한다. -- ModelClient: ranksmith 도메인의 `rank`, `compare`, `select` 계약과 prompt 생성을 담당한다. +- ModelClient: ranksmith 도메인의 `rank`, `compare`, `select`, `answer` 계약과 prompt 생성을 담당한다. - Strategy: reranking에 쓰는 비교 단위다. - Algorithm: 최종 순위를 만드는 절차다. @@ -26,6 +26,7 @@ src/ranksmith/ setwise.py tourrank.py acurank.py + confidence.py # answer-confidence rerank (experimental) providers/ __init__.py # public provider exports azure.py # Azure OpenAI implementation @@ -49,6 +50,8 @@ Pairwise model client call은 `"A"` 또는 `"B"` winner를 담은 JSON 문자열 Selection model client call은 group 안에서 선택된 문서의 1-based index list를 담은 JSON 문자열을 반환한다. +Answer model client call은 주어진 context만으로 질문에 답한 `"answer"` 문자열을 담은 JSON 문자열을 반환한다(문맥에 답이 없으면 `"__NO_ANSWER__"`). + ## Strategy v1 공개 strategy: - `ListwiseStrategy` @@ -61,12 +64,15 @@ v1 공개 strategy: - `AsyncTourRankStrategy` - `AcuRankStrategy` - `AsyncAcuRankStrategy` +- `AnswerConfidenceRerankStrategy` (실험적 — 학습된 estimator artifact 필요) +- `AsyncAnswerConfidenceRerankStrategy` (실험적) 공식 확장 지점: - 새 reranking 방법은 새 Strategy 클래스로 추가한다. - Strategy protocol과 model client/provider contract는 `ranksmith.protocols` 및 root import에서 공개한다. - model client JSON ranking을 직접 다루는 custom Strategy는 `parse_ranking_response()`를 사용해 검증한다. - selection 기반 Strategy는 `parse_selection_response()`를 사용해 selected index를 검증한다. +- answer 기반 Strategy는 `parse_answer_response()`를 사용해 answer 문자열을 검증한다. - 자세한 확장 규칙은 `docs/wiki/08_custom_strategy_extension.md`를 따른다. 향후 strategy 후보: @@ -80,11 +86,11 @@ v1 지원 algorithm: - `tourrank_r` - `acurank` -향후 algorithm 후보: -- `confidence` +benchmark opt-in algorithm: +- `answer_confidence` (`scripts/compare_reranking.py`, 학습된 artifact 필요) ## Confidence -`ranksmith.confidence`는 reranking Strategy나 Algorithm이 아니라, closed model output confidence를 계산하는 utility layer다. +`ranksmith.confidence`(estimator)는 reranking Strategy가 아니라 closed model output confidence를 계산하는 utility layer다. 이 estimator를 소비하는 실험적 reranker가 `AnswerConfidenceRerankStrategy`(`ranksmith.strategies`)이며, 스펙에 기록된 소규모 자체 평가에서 Listwise에 진다(`docs/specs/spec_confidence_aware_reranking.md` — 증거 artifact는 리포에 커밋되어 있지 않다). 현재 범위: - frozen HuggingFace encoder 기반 token-level trajectory 생성 @@ -124,7 +130,7 @@ v1 지원 algorithm: - sync closed model call - resume 가능한 JSONL output -제외: +제외 (package 범위 — SQuAD 학습 데이터 빌더와 turnkey CLI는 package 밖 `scripts/build_answer_confidence_training_data.py`, `scripts/train_answer_confidence.py`로 제공): - async generation - dataset adapter - CLI @@ -149,5 +155,11 @@ Selection JSON: {"selected": [3, 1]} ``` +Answer JSON: + +```json +{"answer": "vitamin C deficiency"} +``` + 잘못된 JSON, 누락 값, 중복 값, 범위 밖 값, 정수가 아닌 값, 잘못된 winner 값은 `RerankParseError`로 실패한다. `true`, `false`는 JSON bool이며 정수로 인정하지 않는다. diff --git a/docs/wiki/04_references_index.md b/docs/wiki/04_references_index.md index 368e4bd..59ff7e6 100644 --- a/docs/wiki/04_references_index.md +++ b/docs/wiki/04_references_index.md @@ -7,6 +7,8 @@ - [AcuRank](references/acurank.md): Paper + Repo / Bayesian, Uncertainty-aware Adaptive Reranking / 요약 완료, 구현 완료 - [A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models](references/setwise_ranking_prompting.md): Paper / Setwise Reranking, Heapsort-based Selection Reranking / 요약 완료, 구현 완료 - [Trust in One Round: Confidence Estimation for Large Language Models via Structural Signals](references/structural_confidence.md): Paper / Black-box confidence, proxy hidden-state trajectory / 요약 완료, Phase 1 inference spec 작성 +- [Rethinking LLM Parametric Knowledge as Confidence (CBDR)](references/cbdr_parametric_confidence_rag.md): Paper / Confidence-change preference, dynamic retrieval trigger / 요약 완료, white-box 전제라 개념 참고용 +- [LLM-Confidence Reranker (LCR)](references/llm_confidence_reranker.md): Paper / Training-free black-box confidence reranking, MSCP / 요약 완료, confidence-aware reranking 스펙 후보 ## 처리 대기 Reference - [Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers](references/Attention%20in%20Large%20Language%20Models%20Yields%20Efficient%20Zero-Shot%20Re-Rankers.pdf): Paper / Attention-based reranking, zero-shot reranking / 요약 대기 @@ -18,7 +20,6 @@ - [FIRST: Faster Improved Listwise Reranking with Single Token Decoding](references/FIRST-%20Faster%20Improved%20Listwise%20Reranking%20with%20Single%20Token%20Decoding.pdf): Paper / Listwise reranking, single-token decoding, efficiency / 요약 대기 - [Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models](references/Found%20in%20the%20Middle-%20Permutation%20Self-Consistency%20Improves%20Listwise%20Ranking%20in%20Large%20Language%20Models.pdf): Paper / Permutation self-consistency, listwise reranking / 요약 대기 - [InstUPR: Instruction-based Unsupervised Passage Reranking with Large Language Models](references/InstUPR-%20Instruction-based%20Unsupervised%20Passage%20Reranking%20with%20Large%20Language%20Models.pdf): Paper / Pointwise/unsupervised passage reranking / 요약 대기 -- [LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems](references/LLM-Confidence%20Reranker-%20A%20Training-Free%20Approach%20for%20Enhancing%20Retrieval-Augmented%20Generation%20Systems.pdf): Paper / Confidence-based reranking, RAG / 요약 대기 - [Large Language Models are Zero-Shot Rankers for Recommender Systems](references/Large%20Language%20Models%20are%20Zero-Shot%20Rankers%20for%20Recommender%20Systems.pdf): Paper / Zero-shot recommendation ranking / 요약 대기 - [Measuring Black-Box Confidence via Reasoning Trajectories: Geometry Coverage and Verbalization](references/Measuring%20Black-Box%20Confidence%20via%20Reasoning%20Trajectories-%20Geometry%20Coverage%20and%20Verbalization.pdf): Paper / Black-box confidence, reasoning trajectory / 요약 대기 - [Parameters vs Context: Fine-Grained Control of Knowledge Reliance in Language Models](references/Parameters%20vs%20Context-%20Fine-Grained%20Control%20of%20Knowledge%20Reliance%20in%20Language%20Models.pdf): Paper / Context confidence gain, token entropy shift / 요약 대기 @@ -26,7 +27,6 @@ - [Rank-K: Test-Time Reasoning for Listwise Reranking](references/Rank-K-%20Test-Time%20Reasoning%20for%20Listwise%20Reranking.pdf): Paper / Test-time reasoning, listwise reranking / 요약 대기 - [Re-Ranking Step by Step: Investigating Pre-Filtering for Re-Ranking with Large Language Models](references/Re-Ranking%20Step%20by%20Step-%20Investigating%20Pre-Filtering%20for%20Re-Ranking%20with%20Large%20Language%20Models.pdf): Paper / Pre-filtering, staged reranking / 요약 대기 - [ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillation](references/ReasoningRank-%20Teaching%20Student%20Models%20to%20Rank%20through%20Reasoning-Based%20Knowledge%20Distillation.pdf): Paper / Reasoning distillation, training-based ranking / 요약 대기 -- [Rethinking LLM Parametric Knowledge as Confidence for Effective and Efficient Retrieval-Augmented Generation](references/Rethinking%20LLM%20Parametric%20Knowledge%20as%20Confidence%20for%20Effective%20and%20Efficient%20Retrieval-Augmented%20Generation.pdf): Paper / Confidence shift, RAG reranker alignment / 요약 대기 - [Self-Calibrated Listwise Reranking with Large Language Models](references/Self-Calibrated%20Listwise%20Reranking%20with%20Large%20Language%20Models.pdf): Paper / Self-calibrated listwise reranking / 요약 대기 - [Stop Overthinking: Unlocking Efficient Listwise Reranking with Minimal Reasoning](references/Stop%20Overthinking-%20Unlocking%20Efficient%20Listwise%20Reranking%20with%20Minimal%20Reasoning.pdf): Paper / Minimal reasoning, listwise reranking efficiency / 요약 대기 - [Think When Needed: Model-Aware Reasoning Routing for LLM-based Ranking](references/Think%20When%20Needed-%20Model-Aware%20Reasoning%20Routing%20for%20LLM-based%20Ranking.pdf): Paper / Reasoning routing, model-aware ranking / 요약 대기 diff --git a/docs/wiki/05_open_questions.md b/docs/wiki/05_open_questions.md index 6e9f4c1..3699806 100644 --- a/docs/wiki/05_open_questions.md +++ b/docs/wiki/05_open_questions.md @@ -49,6 +49,37 @@ Impact: Needed From User: - 없음 +## Q004 Confidence-aware reranking 설계 결정 +Status: resolved + +Context: +- `docs/specs/spec_confidence_aware_reranking.md` 참조. +- CBDR 논문은 white-box hidden state 전제라 방식 직접 이식이 불가하고, 설계 패턴만 가져온다. +- LCR 논문은 training-free지만 temperature 1 sampling K회가 필요해 현재 ModelClient 계약(temperature 0, JSON-only)과 충돌한다. + +Resolved Decision: +1. **confidence 소스 = 경로 A (structural confidence + LightGBM scorer artifact)**. 사용자 결정 근거: 경로 B(LCR/MSCP)는 문서당 K회+ LLM 호출이 필요해 리랭킹 호출 비용을 크게 늘린다. 경로 A는 추론 시 answer 1회 + 로컬 CPU 연산이므로 호출 비용이 낮다. +2. **형태 = 새 Strategy.** `AnswerConfidenceRerankStrategy` / `AsyncAnswerConfidenceRerankStrategy`로 구현 완료 (`src/ranksmith/strategies/confidence.py`). +3. **순위 규칙 = confidence 원점수 내림차순 + 동점 시 원래 순서 유지.** LCR의 binning/이전-순위-환원 gate는 미채택 — 실측에서 필요성이 확인되면 별도 결정으로 재검토. +4. **scorer 학습 데이터 = gold answer가 있는 QA 데이터.** SQuAD v1.1 경로가 `scripts/build_answer_confidence_training_data.py` + `scripts/train_answer_confidence.py`로 도구화됨. 도메인 밖 artifact 라벨링 규칙은 스펙과 `docs/benchmarks/answer_confidence_askubuntu.md` 참조. + +Smoke 실행에서 확인된 사실 (2026-07-05, LM Studio qwen3.5-9b + libomp 설치 후): +- **경로 A 전체 관통 성공** — BEIR/SciFact 실데이터 56개(qrels positive 14 + 어휘 유사 hard negative 42) → LM Studio judgment 생성(정답 일치 44 / 불일치 12) → bert 특징 추출 → LightGBM 학습 + sigmoid 보정 → **실제 scorer artifact(scorer.joblib) 생성** → held-out test 특징에 confidence 점수 산출 확인. +- generation 파이프라인: `generate_judgment_confidence_dataset`가 LM Studio provider로 엔드투엔드 동작. +- 추론 절반: `FrozenAutoEncoder`(bert-base-uncased) 로드 + 70차원 `structural-v1` 특징 생성 확인. + +당시 발견한 문제와 처리 결과: +1. **로더 버그 (`load_lightgbm_scorer`)** — joblib 아티팩트가 metadata_path 지정 시 booster 텍스트 포맷으로 오인되던 문제. main의 PR #8(`d60cafb`)에서 수정 후 이 브랜치에 머지됨. +2. **macOS ARM 환경 제약 (ranksmith 코드 문제 아님)**: torch(인코더)와 lightgbm을 **한 프로세스**에서 같이 쓰면 이중 OpenMP 충돌로 segfault/hang 가능. 우회는 특징 추출(torch)과 학습/예측(lightgbm)의 프로세스 분리. 환경 제약으로 기록 유지. +3. ~~≥30 샘플 요건~~ — SciFact 실데이터 56개로 해소, 이후 SQuAD 빌더가 기본 500행 생산. + +Impact: +- 구현 완료: 전략 2종, `ModelClient.answer`/`AsyncModelClient.answer`, `parse_answer_response`, opt-in 벤치마크 통합. +- 남은 실행 항목(질문 아님): README 표준 벤치마크 편입 실행 — `docs/benchmarks/answer_confidence_askubuntu.md` 러닝북 참조. + +Needed From User: +- 없음 + ## 형식 ```markdown ## Q001 diff --git a/docs/wiki/references/cbdr_parametric_confidence_rag.md b/docs/wiki/references/cbdr_parametric_confidence_rag.md new file mode 100644 index 0000000..24c54ab --- /dev/null +++ b/docs/wiki/references/cbdr_parametric_confidence_rag.md @@ -0,0 +1,49 @@ +# Reference: Rethinking LLM Parametric Knowledge as Confidence (CBDR) + +## Source +- Paper: Rethinking LLM Parametric Knowledge as Confidence for Effective and Efficient Retrieval-Augmented Generation (Anonymous ACL submission, 16pp) +- Local PDF: `docs/wiki/references/Rethinking LLM Parametric Knowledge as Confidence for Effective and Efficient Retrieval-Augmented Generation.pdf` +- Repo: 논문에 명시되지 않음 (anonymous submission, 코드 링크 없음) +- License: 논문에 명시되지 않음 + +## 적용 영역 +- confidence-aware reranking의 개념적 로드맵 (runtime readiness 스펙이 "CBDR-ready signal"이라 부른 출처) +- confidence 변화(Inc) 기반 preference 신호 설계 +- retrieval 트리거 결정 (ranksmith 범위 밖 — 아래 충돌 참조) + +## 핵심 메커니즘 +세 가지 구성 요소: + +1. **Confidence detection model E**: target LLM이 질문 처리 중 만든 내부 hidden state(본문 기준 Mid_Layer, 첫 답변 토큰 생성 직전)를 입력으로 받아 "정답 가능 확신" 확률을 내는 5-layer MLP(2M 파라미터). NQ 질문의 정답/오답 여부를 라벨로 학습. +2. **Confidence-change 기반 reranker fine-tuning**: context 유무에 따른 confidence 변화 `Inc(Q, Ci) = Conf(H_{M,Q+Ci}) − Conf(H_{M,Q})`를 계산해, 상승 Top-5를 positive / 하락 Top-5를 negative로 하는 preference dataset(NQ_Rerank, 7,622 train)을 만들고 bge-reranker-v2-m3를 InfoNCE로 fine-tuning. 즉 "의미 유사도"가 아니라 "downstream LLM의 confidence를 올리는 문서"를 위로 올리는 reranker. +3. **CBDR (Confidence-Based Dynamic Retrieval)**: 원 질문에 대한 confidence가 임계값 β를 넘으면 retrieval/reranking을 생략하고 직접 생성, 아니면 전체 RAG 파이프라인 실행. β=0.98이 실험 최적. + +보고된 결과: fine-tuned reranker가 NQ_Rerank에서 원본 bge 대비 P@1 +5.19pp(91.20), Qwen3-Reranker-8B 대비 +3.95pp. RAG 정확도는 target LLM(Llama3-8B)에서 NQ Top-3 +4.7pp. CBDR β=0.98에서 정확도 67.8%로 전량 검색(β=1.00, 66.9%)보다 높으면서 retrieval cost 7.1pp 절감. Abstract의 "+5.6pp" 헤드라인은 어떤 셀 비교인지 본문에 도출식이 없음. + +주의 — 논문 내부 불일치(원문 병기, 해명 없음): +- hidden state 위치: 본문 §3.1은 Mid_Layer(Layer/2), Algorithm 1은 `hidden_states[−1]`(마지막 층) +- NQ_Confidence 규모: §4.1은 1k/300/500, Appendix G는 2,000/1,000/500 +- E 학습 epoch: Appendix B는 30, Appendix G는 100 + +## ranksmith 매핑 (제안, 미확정) +- Strategy: 직접 매핑 없음 — 이 논문은 개념적 로드맵으로만 사용 +- Algorithm: 없음 +- Public API 영향: 없음 (아래 충돌로 인해 방식 자체를 이식할 수 없음) +- Error 동작: 해당 없음 +- 추가할 테스트: 해당 없음 + +## 현재 설계와 충돌 +- **White-box 전제**: 이 방법은 학습(hidden state 수집에 target LLM forward 74,204회)과 추론(질문마다 hidden state 추출) 모두 target LLM 내부 접근이 필요하다. ranksmith는 폐쇄형 API 모델이 전제라 **직접 적용 불가**. ranksmith의 구현된 대안이 Trust in One Round 방식(출력 텍스트 → frozen proxy encoder → structural confidence)이며, 이 논문에서 가져올 것은 "confidence 신호로 문서 선호를 정한다"는 설계 패턴뿐이다. +- **Reranker fine-tuning 필요**: ranksmith core는 training-free reranking 지향. preference dataset 구축 + fine-tuning 경로는 core 밖이다. +- **CBDR = 검색 트리거**: CBDR 자체는 reranker가 아니라 "retrieval을 할지 결정하는 스위치"다. ranksmith에는 retriever가 없으므로 트리거 부분은 라이브러리 범위 밖이고, caller 애플리케이션 몫이다. +- **target LLM 종속**: 논문 스스로 한계로 명시 — reranker가 특정 downstream LLM preference에 강결합되어 LLM 교체 시 재학습 필요. + +## Do Not Copy +- 공개 코드 없음(복사 대상 자체가 없음). 라이선스 미명시이므로 수식/알고리즘 서술만 설계 입력으로 사용한다. + +## 부족한 정보 +- retriever 종류/corpus, Algorithm 1의 top_k 배수 r, InfoNCE τ와 negative 수 N +- Mid_Layer vs 마지막 층 불일치의 실제 구현 +- "+5.6pp" 헤드라인의 산출 근거 +- RAG accuracy 채점 방식(EM/substring 여부) +- 코드/라이선스 diff --git a/docs/wiki/references/llm_confidence_reranker.md b/docs/wiki/references/llm_confidence_reranker.md new file mode 100644 index 0000000..aebf351 --- /dev/null +++ b/docs/wiki/references/llm_confidence_reranker.md @@ -0,0 +1,47 @@ +# Reference: LLM-Confidence Reranker (LCR) + +## Source +- Paper: LLM-Confidence Reranker: A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems (Zhipeng Song et al., arXiv:2602.13571v1, 2026-02-14, Preprint submitted to Elsevier) +- Local PDF: `docs/wiki/references/LLM-Confidence Reranker- A Training-Free Approach for Enhancing Retrieval-Augmented Generation Systems.pdf` +- Repo: 논문에 명시되지 않음 +- License: 논문에 명시되지 않음 (코드 공개 언급 없음) + +## 적용 영역 +- confidence-based reranking (CBDR 방향의 후보 접근 중 하나) +- black-box confidence 추정 (샘플링 기반, hidden state 불필요) +- 기존 reranker 뒤에 붙는 post-reranking 단계 + +## 핵심 메커니즘 +Training-free, plug-and-play 리랭커. LLM을 순수 black-box(입출력만 접근)로 두고 confidence를 추정한다. + +**MSCP (Maximum Semantic Cluster Proportion)**: query(또는 query+document)에 대해 temperature 1로 K개(실험 K=10) 답변을 multinomial sampling한다. 같은 LLM에게 답변 쌍의 양방향 entailment를 판정시켜 semantic cluster를 만들고(greedy, O(K·M) 호출), 가장 큰 클러스터의 비율 `max|s_m| / K`를 confidence로 쓴다. + +**2단계 LCR 알고리즘** (hyperparameter 3개: `T_query`, `T_upper`, `T_lower`): +1. Binning: 문서별 joint confidence `C(q,d)`를 High(≥T_upper, +1) / Medium(0) / Low(≤T_lower, −1)으로 구간화. +2. Ranking: query confidence `C(q) < T_query`이면 [bin 점수 내림차순, 기존 점수 내림차순]으로 StableSort. `C(q) ≥ T_query`이면 **원래 순위 유지** (LLM이 이미 답을 알면 문서 신호가 약하다는 근거). `T_query=0`이면 알고리즘 전체가 원래 순위로 환원되어 baseline 이하로 떨어지지 않는다. + +**보고된 결과** (NDCG@5, BEIR 6종 + TREC DL19/20, top-10 재정렬): BM25 계열에서 상대 개선 최대 20.6%(YesNo+BEIR 0.1737→0.2095), Contriever 계열 최대 32.4%(YesNo+BEIR 0.2281→0.3021), 전체 평균 약 3.6%. 논문 주장 기준 "+LCR" 구성에서 하락 사례 없음. Qwen2.5-7B-Instruct 기본, 7–9B 4종 모델 모두에서 개선(InternLM 최고, Qwen 최저). MSCP가 Semantic Entropy보다 일관되게 우수. + +## ranksmith 매핑 (제안, 미확정) +- Strategy: 신규 후보 `ConfidenceRerankStrategy` 류의 post-reranking Strategy (기존 Strategy 출력 뒤에 적용하는 형태) +- Algorithm: 미정 +- Public API 영향: 미정 — 아래 "현재 설계와 충돌" 해소가 선행되어야 함 +- Error 동작: fast fail 원칙 유지 (entailment 응답이 3개 라벨 밖이면 파싱 에러) +- 추가할 테스트: MSCP 클러스터링(양방향 entailment, greedy), binning 경계값, T_query=0 환원성(원래 순위 보존), StableSort 안정성 + +## 현재 설계와 충돌 +- **temperature 계약**: ranksmith `ModelClient`는 temperature 0 고정(결정적 JSON)이다. MSCP는 temperature 1 multinomial sampling K회가 필수라서, sampling 전용 요청 경로가 필요하다. +- **호출 수**: 문서당 K=10 샘플 + O(K·M) entailment 판정 + query 자체 confidence. 논문은 총 호출 수 공식을 제시하지 않지만(명시되지 않음) 기존 전략 대비 호출량이 크게 늘어난다. 호출 수 추정 정책(advisor method-guide)과의 정합 필요. +- **출력 계약**: MSCP의 답변 샘플은 자유 텍스트다. 현재 ModelClient의 JSON-only 계약과 다르다. +- **기존 점수 필요**: Stage 2 정렬이 PrevScore(기존 retriever/reranker 점수)를 요구한다. `RerankResult.rank` 또는 `Document.metadata["score"]`를 입력으로 받는 adapter 설계 필요. + +## Do Not Copy +- 논문에 공개 코드가 없으므로 복사 문제는 없으나, 논문 라이선스가 명시되지 않았으므로 수식·알고리즘 설명만 설계 입력으로 사용한다. +- 프롬프트 3종(answer w/o doc, answer w/ doc, entailment)은 논문 p.10에 공개된 것을 참고하되 ranksmith 계약에 맞게 재작성한다. + +## 부족한 정보 +- Table 2–3을 만든 `T_query`/`T_upper`/`T_lower` 실제 값 (논문은 "tuning 후 최적"만 언급; 가이드는 UT≈0.9, LT≈0.1–0.4) +- query당 총 LLM 호출 수 / 지연 / 토큰 비용 (논문 미제시) +- 샘플링 디코딩 파라미터 상세 (T=1 외 top-p/top-k/max tokens 미명시) +- greedy 클러스터링의 대표 샘플 선택 방식 (초기 샘플 vs embedding centroid — 실험에서 무엇을 썼는지 미명시) +- 코드/라이선스 diff --git a/examples/answer_confidence.py b/examples/answer_confidence.py new file mode 100644 index 0000000..56ddcb3 --- /dev/null +++ b/examples/answer_confidence.py @@ -0,0 +1,111 @@ +#!/usr/bin/env python +from __future__ import annotations + +import json +import sys +from pathlib import Path + +# 저장소 루트에서 바로 실행할 수 있도록 src 경로를 추가합니다. +# 패키지를 설치한 사용자는 아래 두 줄이 필요하지 않습니다. +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT / "src")) + +from ranksmith import ( # noqa: E402 + AnswerConfidenceRerankStrategy, + AzureOpenAIReranker, + Document, +) +from ranksmith.confidence.types import AnswerConfidenceInput # noqa: E402 + +NO_ANSWER = "__NO_ANSWER__" + + +class KeywordAnswerClient: + """예제용 deterministic model client. 실제 서비스에서는 ModelClient를 사용합니다. + + 문서가 질문에 답할 정보를 담고 있으면 답을, 아니면 NO_ANSWER를 반환합니다. + """ + + def __init__(self) -> None: + self.answer_calls = 0 + + def answer(self, query: str, context: str) -> str: + del query + self.answer_calls += 1 + if "괴혈병" in context: + return json.dumps({"answer": "괴혈병"}) + if "빈혈" in context: + return json.dumps({"answer": "악성 빈혈"}) + return json.dumps({"answer": NO_ANSWER}) + + +class _Score: + def __init__(self, value: float) -> None: + self.score = value + + +class KeywordAnswerConfidenceEstimator: + """예제용 deterministic estimator. 실제 서비스에서는 + ``StructuralConfidenceEstimator.from_artifact(...)`` 로 학습된 scorer를 씁니다. + + 답변이 있고 문맥에 질병 근거가 뚜렷할수록 confidence가 높습니다. + """ + + task_type = "answer_confidence" + + def score(self, item: AnswerConfidenceInput) -> _Score: + if item.answer == NO_ANSWER: + return _Score(0.1) + if "괴혈병" in item.context: + return _Score(0.9) + if "빈혈" in item.context: + return _Score(0.8) + return _Score(0.5) + + +def main() -> None: + query = "비타민 결핍으로 생기는 질병" + documents = [ + Document( + id="apple", + text="사과는 비타민을 포함하지만 결핍성 질병 설명과는 직접 관련이 낮다.", + ), + Document( + id="vitamin_b12", + text="비타민 B12 결핍은 피로, 신경 증상, 악성 빈혈을 유발할 수 있다.", + ), + Document( + id="sleep", + text="수면 부족은 면역 저하와 관련되지만 비타민 결핍 질병은 아니다.", + ), + Document( + id="vitamin_c", + text=( + "비타민 C 결핍은 괴혈병을 일으키며 잇몸 출혈과 상처 회복 지연을 낳는다." + ), + ), + ] + model_client = KeywordAnswerClient() + estimator = KeywordAnswerConfidenceEstimator() + reranker = AzureOpenAIReranker( + api_key="example-key", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="example-deployment", + model_client=model_client, + strategy=AnswerConfidenceRerankStrategy(estimator=estimator), + ) + + results = reranker.rerank(query, documents) + + print("Answer Confidence rerank example") + print(f"query={query}") + for result in results: + print( + f"rank={result.rank:02d} id={result.document.id} " + f"answer_confidence={result.metadata['answer_confidence']:.2f}" + ) + print(f"answer_calls={model_client.answer_calls}") + + +if __name__ == "__main__": + main() diff --git a/scripts/build_answer_confidence_training_data.py b/scripts/build_answer_confidence_training_data.py new file mode 100644 index 0000000..60e867d --- /dev/null +++ b/scripts/build_answer_confidence_training_data.py @@ -0,0 +1,452 @@ +#!/usr/bin/env python +"""Build answer-confidence training data from SQuAD v1.1. + +The ``answer_confidence`` reranker needs a scorer artifact, and the artifact +needs QA training data with gold answers. IR benchmarks such as AskUbuntu have +qrels but no gold answers, so the artifact must be trained on a QA dataset and +that domain must be reported next to any benchmark number. + +Each sampled SQuAD question produces one gold-context row and +``--negatives-per-question`` BM25 hard-negative-context rows. This mirrors the +reranker's inference distribution: at inference the estimator scores answers +generated from both relevant and non-relevant candidate documents. Labels are +NOT assigned here — the generation pipeline +(``scripts/train_answer_confidence.py``) labels each row by whether the live +model's answer matches ``gold_answer``, so a hard negative that still yields a +correct answer is truthfully labeled positive. + +Output rows follow the ``AnswerGenerationSample`` schema: +``{"id", "query", "context", "gold_answer", "source", "group_id", "metadata"}``. +Contexts are ``"{title}\\n\\n{text}"`` to match how +``scripts/compare_reranking.py`` joins candidate documents at inference. +""" + +from __future__ import annotations + +import argparse +import hashlib +import json +import random +import sys +import urllib.error +import urllib.request +from collections.abc import Mapping, Sequence +from dataclasses import dataclass +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) + +from benchmarks.bm25 import ( # noqa: E402 + DEFAULT_B, + DEFAULT_K1, + BM25Index, + bm25_search, + build_bm25_index, +) + +SQUAD_TRAIN_URL = ( + "https://raw.githubusercontent.com/rajpurkar/SQuAD-explorer/" + "master/dataset/train-v1.1.json" +) +SQUAD_SOURCE_LABEL = "squad-v1.1-train" +# The generation pipeline rejects contexts longer than its max_context_chars +# (default 4000), and the reranking strategy rejects documents longer than its +# max_document_chars (default 4000). Keep the builder aligned with both. +DEFAULT_MAX_CONTEXT_CHARS = 4000 +DEFAULT_QUESTIONS = 250 +DEFAULT_NEGATIVES_PER_QUESTION = 1 +DEFAULT_NEGATIVE_POOL = 10 +DEFAULT_SEED = 13 + + +@dataclass(frozen=True) +class SquadParagraph: + id: str + title: str + text: str + + @property + def context(self) -> str: + return f"{self.title}\n\n{self.text}" + + +@dataclass(frozen=True) +class SquadQuestion: + id: str + title: str + paragraph_id: str + text: str + answers: tuple[str, ...] + + +def main() -> None: + args = _parse_args() + _validate_args(args) + squad_path = _resolve_squad_path(args) + paragraphs, questions = _load_squad(squad_path) + print( + f"Loaded {len(paragraphs)} unique paragraphs and " + f"{len(questions)} questions from {squad_path}.", + file=sys.stderr, + ) + index = build_bm25_index( + {paragraph.id: paragraph.context for paragraph in paragraphs.values()}, + k1=args.bm25_k1, + b=args.bm25_b, + ) + rows, stats = _build_rows( + paragraphs=paragraphs, + questions=questions, + index=index, + question_count=args.questions, + negatives_per_question=args.negatives_per_question, + negative_pool=args.negative_pool, + max_context_chars=args.max_context_chars, + seed=args.seed, + ) + _write_jsonl(args.output, rows) + report = { + "source": SQUAD_SOURCE_LABEL, + "squad_file": str(squad_path), + "squad_sha256": _sha256(squad_path), + "parameters": { + "questions": args.questions, + "negatives_per_question": args.negatives_per_question, + "negative_pool": args.negative_pool, + "max_context_chars": args.max_context_chars, + "seed": args.seed, + "bm25_k1": args.bm25_k1, + "bm25_b": args.bm25_b, + }, + "statistics": stats, + } + report_path = _report_path(args.output) + report_path.write_text( + json.dumps(report, ensure_ascii=False, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + print( + f"Wrote {len(rows)} rows to {args.output} " + f"({stats['gold_rows']} gold + {stats['negative_rows']} hard-negative). " + f"Report: {report_path}", + file=sys.stderr, + ) + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Build answer-confidence training data (gold + BM25 hard-negative " + "contexts) from SQuAD v1.1 train." + ) + ) + parser.add_argument( + "--output", + type=Path, + required=True, + help="Output JSONL path, e.g. .benchmark-cache/answer_train.jsonl.", + ) + parser.add_argument( + "--squad-train", + type=Path, + help=( + "Local train-v1.1.json path. Omit together with --download to " + "fetch the official file." + ), + ) + parser.add_argument( + "--download", + action="store_true", + help=( + f"Download train-v1.1.json from {SQUAD_TRAIN_URL} next to --output " + "when --squad-train is not given. Reuses an existing download." + ), + ) + parser.add_argument( + "--questions", + type=int, + default=DEFAULT_QUESTIONS, + help=( + "Number of SQuAD questions to sample. Each question emits " + "1 + negatives-per-question rows. The training pipeline needs " + ">= 30 rows total and both labels present after generation; " + "the committed spec run found 500 total rows workable and " + "100 too few." + ), + ) + parser.add_argument( + "--negatives-per-question", + type=int, + default=DEFAULT_NEGATIVES_PER_QUESTION, + help="BM25 hard-negative contexts per question.", + ) + parser.add_argument( + "--negative-pool", + type=int, + default=DEFAULT_NEGATIVE_POOL, + help=( + "How many top BM25 paragraphs (excluding the gold paragraph) to " + "consider when picking hard negatives." + ), + ) + parser.add_argument( + "--max-context-chars", + type=int, + default=DEFAULT_MAX_CONTEXT_CHARS, + help=( + "Skip questions whose gold or negative context exceeds this " + "length. Must match the generation pipeline and strategy limits." + ), + ) + parser.add_argument("--seed", type=int, default=DEFAULT_SEED) + parser.add_argument("--bm25-k1", type=float, default=DEFAULT_K1) + parser.add_argument("--bm25-b", type=float, default=DEFAULT_B) + parser.add_argument( + "--overwrite", + action="store_true", + help="Overwrite --output and its report if they already exist.", + ) + return parser.parse_args() + + +def _validate_args(args: argparse.Namespace) -> None: + for name in ("questions", "negatives_per_question", "negative_pool"): + if getattr(args, name) < 1: + raise SystemExit(f"--{name.replace('_', '-')} must be greater than 0.") + if args.max_context_chars < 1: + raise SystemExit("--max-context-chars must be greater than 0.") + if args.negative_pool < args.negatives_per_question: + raise SystemExit("--negative-pool must be at least --negatives-per-question.") + if args.squad_train is not None and args.download: + raise SystemExit("Pass either --squad-train or --download, not both.") + if args.squad_train is None and not args.download: + raise SystemExit( + "SQuAD input is required: pass --squad-train or --download." + ) + if not args.overwrite: + for path in (args.output, _report_path(args.output)): + if path.exists(): + raise SystemExit(f"Refusing to overwrite {path} without --overwrite.") + + +def _report_path(output: Path) -> Path: + return output.with_name(f"{output.stem}.report.json") + + +def _resolve_squad_path(args: argparse.Namespace) -> Path: + if args.squad_train is not None: + if not args.squad_train.is_file(): + raise SystemExit(f"--squad-train file does not exist: {args.squad_train}") + return args.squad_train + target = args.output.parent / "train-v1.1.json" + if target.is_file(): + print(f"Reusing existing download: {target}", file=sys.stderr) + return target + target.parent.mkdir(parents=True, exist_ok=True) + print(f"Downloading {SQUAD_TRAIN_URL} -> {target}", file=sys.stderr) + try: + with urllib.request.urlopen(SQUAD_TRAIN_URL, timeout=120) as response: + data = response.read() + except (urllib.error.URLError, OSError) as exc: + raise SystemExit(f"SQuAD download failed: {exc}") from exc + target.write_bytes(data) + return target + + +def _load_squad( + path: Path, +) -> tuple[dict[str, SquadParagraph], list[SquadQuestion]]: + try: + payload = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise SystemExit(f"Failed to read SQuAD JSON {path}: {exc}") from exc + if not isinstance(payload, Mapping) or not isinstance(payload.get("data"), list): + raise SystemExit(f'{path} is not SQuAD-format JSON (missing "data" list).') + version = payload.get("version") + if version is not None and str(version) != "1.1": + raise SystemExit( + f"Expected SQuAD version 1.1, found {version!r}. SQuAD 2.0 adds " + "unanswerable questions and needs an explicit design decision." + ) + + paragraphs: dict[str, SquadParagraph] = {} + paragraph_id_by_text: dict[str, str] = {} + questions: list[SquadQuestion] = [] + seen_question_ids: set[str] = set() + for article_number, article in enumerate(payload["data"], 1): + title = _required_str(article, "title", f"data[{article_number}]") + for paragraph_number, paragraph in enumerate( + _required_list(article, "paragraphs", f"data[{article_number}]"), 1 + ): + location = f"data[{article_number}].paragraphs[{paragraph_number}]" + context = _required_str(paragraph, "context", location) + paragraph_id = paragraph_id_by_text.get(context) + if paragraph_id is None: + paragraph_id = f"p{len(paragraphs):05d}" + paragraph_id_by_text[context] = paragraph_id + paragraphs[paragraph_id] = SquadParagraph( + id=paragraph_id, + title=title, + text=context, + ) + for qa in _required_list(paragraph, "qas", location): + question_id = _required_str(qa, "id", location) + if question_id in seen_question_ids: + raise SystemExit(f"Duplicate SQuAD question id: {question_id}") + seen_question_ids.add(question_id) + answers = tuple( + dict.fromkeys( + answer["text"] + for answer in _required_list(qa, "answers", location) + if isinstance(answer, Mapping) + and isinstance(answer.get("text"), str) + and answer["text"].strip() != "" + ) + ) + questions.append( + SquadQuestion( + id=question_id, + title=title, + paragraph_id=paragraph_id, + text=_required_str(qa, "question", location), + answers=answers, + ) + ) + if not paragraphs or not questions: + raise SystemExit(f"{path} contains no usable paragraphs or questions.") + return paragraphs, questions + + +def _build_rows( + *, + paragraphs: Mapping[str, SquadParagraph], + questions: Sequence[SquadQuestion], + index: BM25Index, + question_count: int, + negatives_per_question: int, + negative_pool: int, + max_context_chars: int, + seed: int, +) -> tuple[list[dict[str, object]], dict[str, object]]: + order = list(range(len(questions))) + random.Random(seed).shuffle(order) + + rows: list[dict[str, object]] = [] + accepted = 0 + skipped: dict[str, int] = { + "empty_question_or_answers": 0, + "gold_context_too_long": 0, + "not_enough_negatives": 0, + } + max_context_length = 0 + for position in order: + if accepted >= question_count: + break + question = questions[position] + if question.text.strip() == "" or not question.answers: + skipped["empty_question_or_answers"] += 1 + continue + gold = paragraphs[question.paragraph_id] + if len(gold.context) > max_context_chars: + skipped["gold_context_too_long"] += 1 + continue + negative_candidates = [ + paragraphs[document_id] + for document_id, _ in bm25_search( + index, + question.text, + top_k=negative_pool, + exclude=(gold.id,), + ) + if len(paragraphs[document_id].context) <= max_context_chars + ] + if len(negative_candidates) < negatives_per_question: + skipped["not_enough_negatives"] += 1 + continue + + accepted += 1 + gold_answer = list(question.answers) + rows.append( + { + "id": f"{question.id}::gold", + "query": question.text, + "context": gold.context, + "gold_answer": gold_answer, + "source": SQUAD_SOURCE_LABEL, + "group_id": question.title, + "metadata": { + "context_kind": "gold", + "paragraph_id": gold.id, + }, + } + ) + max_context_length = max(max_context_length, len(gold.context)) + for negative_rank, negative in enumerate( + negative_candidates[:negatives_per_question], 1 + ): + rows.append( + { + "id": f"{question.id}::neg{negative_rank}", + "query": question.text, + "context": negative.context, + "gold_answer": gold_answer, + "source": SQUAD_SOURCE_LABEL, + "group_id": question.title, + "metadata": { + "context_kind": "bm25_hard_negative", + "paragraph_id": negative.id, + "bm25_rank": negative_rank, + }, + } + ) + max_context_length = max(max_context_length, len(negative.context)) + + if accepted < question_count: + raise SystemExit( + f"Only {accepted} of the requested {question_count} questions were " + f"usable (skipped: {skipped}). Lower --questions or relax limits." + ) + stats = { + "questions_accepted": accepted, + "gold_rows": accepted, + "negative_rows": len(rows) - accepted, + "total_rows": len(rows), + "skipped": skipped, + "max_context_chars_seen": max_context_length, + "distinct_group_ids": len({str(row["group_id"]) for row in rows}), + } + return rows, stats + + +def _write_jsonl(path: Path, rows: Sequence[Mapping[str, object]]) -> None: + path.parent.mkdir(parents=True, exist_ok=True) + with path.open("w", encoding="utf-8") as handle: + for row in rows: + handle.write(json.dumps(row, ensure_ascii=False, sort_keys=True) + "\n") + + +def _sha256(path: Path) -> str: + digest = hashlib.sha256() + with path.open("rb") as handle: + for chunk in iter(lambda: handle.read(1 << 20), b""): + digest.update(chunk) + return digest.hexdigest() + + +def _required_str(data: Mapping[str, object], key: str, location: str) -> str: + value = data.get(key) + if not isinstance(value, str): + raise SystemExit(f'{location}: "{key}" must be a string.') + return value + + +def _required_list(data: Mapping[str, object], key: str, location: str) -> list[object]: + value = data.get(key) + if not isinstance(value, list): + raise SystemExit(f'{location}: "{key}" must be a list.') + return value + + +if __name__ == "__main__": + main() diff --git a/scripts/compare_reranking.py b/scripts/compare_reranking.py index 91c81d9..a28fa8f 100644 --- a/scripts/compare_reranking.py +++ b/scripts/compare_reranking.py @@ -2,13 +2,53 @@ from __future__ import annotations import argparse +import functools import json import math import os import sys from collections.abc import Mapping, Sequence from pathlib import Path -from typing import Literal, cast +from typing import Any, Literal, cast + + +@functools.lru_cache(maxsize=2) +def _load_answer_confidence_estimator(artifact_path: str) -> Any: + # Loaded once per artifact: from_artifact pulls torch + lightgbm and is slow. + from ranksmith.confidence import StructuralConfidenceEstimator + + return StructuralConfidenceEstimator.from_artifact( + artifact_path, allow_truncation=True + ) + + +def _openai_compatible_provider(base_url: str) -> Any: + # ModelProvider backed by any OpenAI-compatible chat endpoint (LM Studio, + # vLLM, ...). RANKSMITH_OPENAI_MODEL / _API_KEY override the defaults. + from openai import OpenAI + + from ranksmith.model import ModelRequest, ModelResponse + + client = OpenAI( + base_url=base_url, + api_key=os.getenv("RANKSMITH_OPENAI_API_KEY", "local"), + timeout=180, + ) + model = os.getenv("RANKSMITH_OPENAI_MODEL", "local-model") + + class _Provider: + def complete(self, request: ModelRequest) -> ModelResponse: + resp = client.chat.completions.create( + model=model, + messages=[ + {"role": m.role, "content": m.content} for m in request.messages + ], + temperature=0, + ) + return ModelResponse(content=resp.choices[0].message.content or "") + + return _Provider() + ROOT = Path(__file__).resolve().parents[1] sys.path.insert(0, str(ROOT)) @@ -49,6 +89,7 @@ "tourrank_r", "setwise_heapsort", "acurank", + "answer_confidence", ] Dataset = Literal["fixture", "benchmark-cache", "beir-scifact"] DEFAULT_FIXTURE = ROOT / "tests/fixtures/reranking_smoke_fixture.jsonl" @@ -66,6 +107,8 @@ "acurank_b4", "tourrank_r10", "prp_sliding_p3", + # Opt-in only: needs --answer-confidence-artifact, excluded from --algorithm all. + "answer_confidence", ) LEGACY_ALGORITHMS: tuple[Algorithm, ...] = ( "rankgpt_sliding_window", @@ -186,8 +229,14 @@ def _parse_args() -> argparse.Namespace: parser.add_argument("--seed", type=int, default=13) parser.add_argument( "--algorithm", + action="append", choices=("all", *ALGORITHMS, *OPTIONAL_ALGORITHMS, *LEGACY_ALGORITHMS), - default="all", + default=None, + help=( + "Algorithm to run; repeat the flag to run several algorithms on " + "identical cases in one invocation. Defaults to the default " + "method set ('all')." + ), ) parser.add_argument("--window-size", type=int, default=20) parser.add_argument("--stride", type=int, default=10) @@ -195,6 +244,20 @@ def _parse_args() -> argparse.Namespace: parser.add_argument("--tourrank-rounds", type=int, default=2) parser.add_argument("--set-size", type=int, default=3) parser.add_argument("--top-k", type=int, default=5) + parser.add_argument( + "--answer-confidence-artifact", + type=str, + default=None, + help=( + "Path to a trained answer_confidence scorer (.joblib) for the " + "'answer_confidence' algorithm. NOTE: this scorer must be trained " + "separately on QA data with gold answers (this IR benchmark has " + "qrels but no gold answers, so it cannot train one). Passing an " + "artifact trained on a different domain (e.g. SQuAD) measures that " + "domain shift, not a like-for-like comparison — label results " + "accordingly. Requires 'pip install ranksmith[confidence]'." + ), + ) parser.add_argument( "--query-id", action="append", @@ -272,6 +335,17 @@ def _validate_args(args: argparse.Namespace) -> None: "--candidates is required for BEIR benchmark mode. " "Use --candidate-strategy oracle_plus_random only for diagnostics." ) + if "answer_confidence" in (args.algorithm or []): + if args.answer_confidence_artifact is None: + raise SystemExit( + "--algorithm answer_confidence requires " + "--answer-confidence-artifact (a trained scorer .joblib)." + ) + if not Path(args.answer_confidence_artifact).is_file(): + raise SystemExit( + "--answer-confidence-artifact file does not exist: " + f"{args.answer_confidence_artifact}" + ) def _load_cases(args: argparse.Namespace) -> list[BenchmarkCase]: @@ -311,9 +385,16 @@ def _selected_algorithms( args: argparse.Namespace, cases: Sequence[BenchmarkCase], ) -> tuple[Algorithm, ...]: - if args.algorithm != "all": - return (cast(Algorithm, args.algorithm),) - return ALGORITHMS + selected: list[str] = args.algorithm or ["all"] + if "all" in selected: + if len(selected) > 1: + raise SystemExit( + "--algorithm all cannot be combined with other --algorithm values." + ) + return ALGORITHMS + if len(set(selected)) != len(selected): + raise SystemExit("--algorithm values must not repeat.") + return tuple(cast(Algorithm, algorithm) for algorithm in selected) def _evaluate_cases( @@ -344,6 +425,9 @@ def _evaluate_cases( set_size=args.set_size, top_k=args.top_k, timeout=getattr(args, "timeout", None), + answer_confidence_artifact=getattr( + args, "answer_confidence_artifact", None + ), ) evaluation = evaluate_ranked_ids( case=case, @@ -392,9 +476,11 @@ def _rank_case( set_size: int = 3, top_k: int | None = None, timeout: float | None = None, + answer_confidence_artifact: str | None = None, ) -> tuple[str, ...]: from ranksmith import ( AcuRankStrategy, + AnswerConfidenceRerankStrategy, AzureOpenAIReranker, Document, ListwiseStrategy, @@ -434,6 +520,15 @@ def _rank_case( window_size=window_size, max_adaptive_reranker_calls=_acurank_budget_for_algorithm(algorithm), ) + elif algorithm == "answer_confidence": + if not answer_confidence_artifact: + raise ValueError( + "answer_confidence requires --answer-confidence-artifact " + "(a trained answer_confidence scorer)." + ) + strategy = AnswerConfidenceRerankStrategy( + estimator=_load_answer_confidence_estimator(answer_confidence_artifact), + ) else: listwise_window_size, listwise_stride = _listwise_window_stride_for_algorithm( algorithm, @@ -445,21 +540,32 @@ def _rank_case( stride=listwise_stride, ) - reranker = AzureOpenAIReranker( - api_key=_required_env("AZURE_OPENAI_API_KEY"), - azure_endpoint=_required_env("AZURE_OPENAI_ENDPOINT"), - azure_deployment=_required_env( - "AZURE_OPENAI_LLM_DEPLOYMENT", - fallback="AZURE_OPENAI_DEPLOYMENT", - ), - api_version=_env_value( - "AZURE_OPENAI_LLM_API_VERSION", - fallback="AZURE_OPENAI_API_VERSION", - default="2024-08-01-preview", - ), - timeout=timeout or _env_float("AZURE_OPENAI_LLM_TIMEOUT"), - strategy=strategy, - ) + # Local/test escape hatch: point at any OpenAI-compatible endpoint (e.g. + # LM Studio) instead of Azure by setting RANKSMITH_OPENAI_BASE_URL. + base_url = os.getenv("RANKSMITH_OPENAI_BASE_URL") + if base_url: + from ranksmith import ModelClient + + reranker = AzureOpenAIReranker( + model_client=ModelClient(provider=_openai_compatible_provider(base_url)), + strategy=strategy, + ) + else: + reranker = AzureOpenAIReranker( + api_key=_required_env("AZURE_OPENAI_API_KEY"), + azure_endpoint=_required_env("AZURE_OPENAI_ENDPOINT"), + azure_deployment=_required_env( + "AZURE_OPENAI_LLM_DEPLOYMENT", + fallback="AZURE_OPENAI_DEPLOYMENT", + ), + api_version=_env_value( + "AZURE_OPENAI_LLM_API_VERSION", + fallback="AZURE_OPENAI_API_VERSION", + default="2024-08-01-preview", + ), + timeout=timeout or _env_float("AZURE_OPENAI_LLM_TIMEOUT"), + strategy=strategy, + ) documents = [ Document( id=document.id, @@ -730,6 +836,8 @@ def _estimate_provider_calls( if adaptive_budget is None: adaptive_budget = 1 return max(1, math.ceil(document_count / window_size) + adaptive_budget) + if algorithm == "answer_confidence": + return document_count # one answer() call per document window_size, stride = _listwise_window_stride_for_algorithm( algorithm, window_size, diff --git a/scripts/merge_benchmark_reports.py b/scripts/merge_benchmark_reports.py new file mode 100644 index 0000000..7b6333b --- /dev/null +++ b/scripts/merge_benchmark_reports.py @@ -0,0 +1,201 @@ +#!/usr/bin/env python +"""Merge per-method compare_reranking reports into one comparison report. + +The documented benchmark convention runs optional methods (for example +``answer_confidence``) as separate ``scripts/compare_reranking.py`` +invocations against the same cache and candidate file, then merges the +reports (see ``benchmark-results/live/*.merged.json``). This script replaces +the previously manual merge and refuses to merge runs that were not measured +on identical cases: + +- benchmark identity fields must match across inputs + (dataset, candidate file name, candidate count, top_k, case count, ...); +- every algorithm in every input must cover exactly the same query ids; +- the same algorithm may not appear in two inputs. + +Fields that are neither validated nor merged (for example ``timeout``) are +taken from the first input; any differing values are recorded per input under +``merged_from`` so the provenance stays visible. +""" + +from __future__ import annotations + +import argparse +import json +import sys +from collections.abc import Mapping, Sequence +from pathlib import Path +from typing import Any + +MUST_MATCH_FIELDS = ( + "schema_version", + "benchmark_type", + "dataset", + "dataset_name", + "candidate_strategy", + "candidate_count", + "case_count", + "top_k", +) +MERGED_FIELDS = ( + "algorithm", + "aggregate", + "per_query", + "call_estimates", + "method_settings", +) + + +def main() -> None: + args = _parse_args() + if not args.overwrite and args.output.exists(): + raise SystemExit(f"Refusing to overwrite {args.output} without --overwrite.") + reports = [_load_report(path) for path in args.inputs] + merged = _merge_reports(args.inputs, reports) + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text( + json.dumps(merged, ensure_ascii=False, indent=2, sort_keys=True) + "\n", + encoding="utf-8", + ) + print( + f"Merged {len(reports)} reports covering algorithms " + f"{merged['algorithm']} into {args.output}.", + file=sys.stderr, + ) + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Merge compare_reranking reports measured on identical cases " + "into one comparison report." + ) + ) + parser.add_argument( + "inputs", + nargs="+", + type=Path, + help="Two or more compare_reranking report JSON files.", + ) + parser.add_argument("--output", type=Path, required=True) + parser.add_argument("--overwrite", action="store_true") + args = parser.parse_args() + if len(args.inputs) < 2: + parser.error("Provide at least two input reports to merge.") + return args + + +def _load_report(path: Path) -> dict[str, Any]: + try: + report = json.loads(path.read_text(encoding="utf-8")) + except (OSError, json.JSONDecodeError) as exc: + raise SystemExit(f"Failed to read report {path}: {exc}") from exc + if not isinstance(report, dict): + raise SystemExit(f"Report {path} is not a JSON object.") + missing = [ + key + for key in (*MUST_MATCH_FIELDS, *MERGED_FIELDS, "candidates", "seed") + if key not in report + ] + if missing: + raise SystemExit(f"Report {path} is missing required keys: {missing}") + return report + + +def _merge_reports( + paths: Sequence[Path], + reports: Sequence[Mapping[str, Any]], +) -> dict[str, Any]: + first_path, first = paths[0], reports[0] + for path, report in zip(paths[1:], reports[1:], strict=True): + for field in MUST_MATCH_FIELDS: + if report[field] != first[field]: + raise SystemExit( + f"{path} and {first_path} disagree on {field!r}: " + f"{report[field]!r} != {first[field]!r}." + ) + if _candidate_name(report) != _candidate_name(first): + raise SystemExit( + f"{path} and {first_path} used different candidate files: " + f"{report['candidates']!r} != {first['candidates']!r}." + ) + if ( + first["candidate_strategy"] == "oracle_plus_random" + and report["seed"] != first["seed"] + ): + raise SystemExit( + f"{path} and {first_path} used different seeds with " + "oracle_plus_random candidates; the cases differ." + ) + + algorithms: list[str] = [] + aggregate: list[Any] = [] + per_query: list[Any] = [] + call_estimates: dict[str, Any] = {} + method_settings: dict[str, Any] = {} + query_ids_by_algorithm: dict[str, frozenset[str]] = {} + for path, report in zip(paths, reports, strict=True): + for algorithm in report["algorithm"]: + if algorithm in algorithms: + raise SystemExit( + f"Algorithm {algorithm!r} appears in more than one input " + f"(second occurrence in {path}); refusing to merge " + "duplicate measurements." + ) + algorithms.append(algorithm) + aggregate.extend(report["aggregate"]) + per_query.extend(report["per_query"]) + call_estimates.update(report["call_estimates"]) + method_settings.update(report["method_settings"]) + for row in report["per_query"]: + algorithm = str(row["algorithm"]) + query_ids_by_algorithm.setdefault(algorithm, frozenset()) + query_ids_by_algorithm[algorithm] |= {str(row["query_id"])} + + reference_algorithm = algorithms[0] + reference_ids = query_ids_by_algorithm[reference_algorithm] + for algorithm, query_ids in query_ids_by_algorithm.items(): + if query_ids != reference_ids: + missing = sorted(reference_ids - query_ids)[:5] + extra = sorted(query_ids - reference_ids)[:5] + raise SystemExit( + f"Algorithm {algorithm!r} covers different query ids than " + f"{reference_algorithm!r} (missing e.g. {missing}, extra " + f"e.g. {extra}); the runs are not an identical-case " + "comparison." + ) + + merged = dict(first) + merged["algorithm"] = algorithms + merged["aggregate"] = aggregate + merged["per_query"] = per_query + merged["call_estimates"] = call_estimates + merged["method_settings"] = method_settings + merged["checkpoint_output"] = None + merged["merged_from"] = [ + { + "file": path.name, + "algorithm": list(report["algorithm"]), + "divergent_fields": { + key: report[key] + for key in sorted(report) + if key not in MUST_MATCH_FIELDS + and key not in MERGED_FIELDS + and key not in {"checkpoint_output", "merged_from"} + and report[key] != first.get(key) + }, + } + for path, report in zip(paths, reports, strict=True) + ] + return merged + + +def _candidate_name(report: Mapping[str, Any]) -> str | None: + candidates = report["candidates"] + if candidates is None: + return None + return Path(str(candidates)).name + + +if __name__ == "__main__": + main() diff --git a/scripts/train_answer_confidence.py b/scripts/train_answer_confidence.py new file mode 100644 index 0000000..5785a15 --- /dev/null +++ b/scripts/train_answer_confidence.py @@ -0,0 +1,366 @@ +#!/usr/bin/env python +"""Generate answer-confidence labels with a live model and train the scorer. + +Turnkey path from ``scripts/build_answer_confidence_training_data.py`` output +to the ``.joblib`` artifact that ``scripts/compare_reranking.py +--algorithm answer_confidence`` consumes: + +1. Generation: for every row the live model answers ``query`` from + ``context`` (same answer prompt the reranker uses at inference), and the + row is labeled by whether the answer matches ``gold_answer``. +2. Training: features from a frozen encoder + LightGBM + sigmoid calibration + via ``train_confidence_scorer``; test-split metrics are printed. + +For a comparable benchmark row, generate with the SAME deployment the +benchmark uses for live calls (the README table used ``gpt-5.4-nano``): the +estimator should be trained on answers from the model it will score. + +Model access follows scripts/compare_reranking.py: set +``RANKSMITH_OPENAI_BASE_URL`` (+ optional ``RANKSMITH_OPENAI_MODEL`` / +``RANKSMITH_OPENAI_API_KEY``) for any OpenAI-compatible endpoint such as +LM Studio, otherwise Azure OpenAI environment variables are required. +Requires the ``confidence-train`` extra (torch, transformers, lightgbm, +scikit-learn, joblib). +""" + +from __future__ import annotations + +import argparse +import json +import os +import sys +from pathlib import Path +from typing import Any + +ROOT = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(ROOT)) +sys.path.insert(0, str(ROOT / "src")) + +# Quality gate: the committed spec run saw test roc_auc 0.875 with 500 rows +# and 0.333 (worse than random) with 100 rows. Below this bound the artifact +# is not benchmark-worthy. +MIN_TEST_ROC_AUC = 0.6 + + +def main() -> None: + args = _parse_args() + _validate_args(args) + _load_env_file(args.env_file) + generated_path = args.workdir / "generated.jsonl" + if args.skip_generation: + if not generated_path.is_file(): + raise SystemExit( + f"--skip-generation requires an existing {generated_path}." + ) + print(f"Skipping generation, reusing {generated_path}.", file=sys.stderr) + else: + _run_generation(args, generated_path) + _run_training(args, generated_path) + + +def _parse_args() -> argparse.Namespace: + parser = argparse.ArgumentParser( + description=( + "Generate answer-confidence labels with a live model, then train " + "the scorer artifact." + ) + ) + parser.add_argument( + "--input", + type=Path, + required=True, + help=( + "AnswerGenerationSample JSONL, e.g. the output of " + "scripts/build_answer_confidence_training_data.py." + ), + ) + parser.add_argument( + "--workdir", + type=Path, + required=True, + help=( + "Working directory for generated.jsonl and training outputs " + "(features, report, manifests)." + ), + ) + parser.add_argument( + "--artifact", + type=Path, + required=True, + help="Export path for the trained scorer artifact (.joblib).", + ) + parser.add_argument( + "--max-items", + type=int, + help="Optional cap on generated rows (smoke runs).", + ) + parser.add_argument( + "--resume", + action="store_true", + help="Resume an interrupted generation run (skips completed row ids).", + ) + parser.add_argument( + "--overwrite", + action="store_true", + help="Overwrite an existing generated.jsonl instead of failing.", + ) + parser.add_argument( + "--skip-generation", + action="store_true", + help="Reuse an existing generated.jsonl and only run training.", + ) + parser.add_argument( + "--encoder", + default="bert-base-uncased", + help="Hugging Face encoder name or local path for feature extraction.", + ) + parser.add_argument( + "--tokenizer", + help="Tokenizer name or path when it differs from --encoder.", + ) + parser.add_argument( + "--max-length", + type=int, + default=256, + help="Encoder max token length.", + ) + parser.add_argument( + "--allow-truncation", + action="store_true", + help=( + "Allow the encoder to truncate long context+answer pairs. SQuAD " + "contexts regularly exceed 256 tokens, so this is required for " + "the documented runbook; the benchmark loads the artifact with " + "truncation allowed as well." + ), + ) + parser.add_argument("--seed", type=int, default=42) + parser.add_argument( + "--env-file", + type=Path, + default=ROOT / ".env", + help="Path to a .env file. Existing process environment values win.", + ) + parser.add_argument( + "--allow-live", + action="store_true", + help="Required because generation sends live model requests.", + ) + return parser.parse_args() + + +def _validate_args(args: argparse.Namespace) -> None: + if not args.input.is_file(): + raise SystemExit(f"--input file does not exist: {args.input}") + if args.max_items is not None and args.max_items < 1: + raise SystemExit("--max-items must be greater than 0.") + if args.max_length < 1: + raise SystemExit("--max-length must be greater than 0.") + if args.skip_generation and (args.resume or args.overwrite): + raise SystemExit( + "--skip-generation cannot be combined with --resume/--overwrite." + ) + if not args.skip_generation and not args.allow_live: + raise SystemExit("Refusing live model calls without --allow-live.") + if args.artifact.suffix != ".joblib": + raise SystemExit("--artifact must end in .joblib.") + + +def _run_generation(args: argparse.Namespace, generated_path: Path) -> None: + from ranksmith.confidence_generation import ( + AnswerGenerationConfig, + generate_answer_confidence_dataset, + ) + + provider = _CallCountingProvider(_build_provider()) + args.workdir.mkdir(parents=True, exist_ok=True) + result = generate_answer_confidence_dataset( + AnswerGenerationConfig( + input_path=args.input, + output_path=generated_path, + provider=provider, + overwrite=args.overwrite, + resume=args.resume, + max_items=args.max_items, + ) + ) + print( + f"Generation finished: {result.generated_count} generated, " + f"{result.skipped_count} skipped, {result.positive_count} positive, " + f"{result.negative_count} negative " + f"({provider.call_count} model calls).", + file=sys.stderr, + ) + if result.positive_count == 0 or result.negative_count == 0: + raise SystemExit( + "Training needs both labels but generation produced " + f"{result.positive_count} positive / {result.negative_count} " + "negative rows. Check the model quality or the training data mix." + ) + + +def _run_training(args: argparse.Namespace, generated_path: Path) -> None: + from ranksmith.confidence_training import ( + ConfidenceTrainingConfig, + train_confidence_scorer, + ) + + result = train_confidence_scorer( + ConfidenceTrainingConfig( + task_type="answer_confidence", + dataset_path=generated_path, + output_dir=args.workdir / "training", + export_path=args.artifact, + encoder_name=args.encoder, + tokenizer_name=args.tokenizer, + max_length=args.max_length, + allow_truncation=args.allow_truncation, + seed=args.seed, + ) + ) + report = json.loads(result.report_path.read_text(encoding="utf-8")) + test_metrics = report["test"] + print(f"Training report: {result.report_path}", file=sys.stderr) + print( + "Test split (calibrated): " + + ", ".join( + f"{name}={test_metrics[name]:.4f}" + for name in ("roc_auc", "average_precision", "brier_score") + if name in test_metrics + ), + file=sys.stderr, + ) + print(f"Artifact exported: {result.export_path}", file=sys.stderr) + roc_auc = float(test_metrics["roc_auc"]) + if roc_auc < MIN_TEST_ROC_AUC: + raise SystemExit( + f"Artifact quality gate failed: test roc_auc {roc_auc:.4f} < " + f"{MIN_TEST_ROC_AUC}. Do not benchmark with this artifact; add " + "training rows (the spec run needed ~500) or inspect " + "generated.jsonl labels." + ) + + +class _CallCountingProvider: + """Counts provider calls; usage metadata is endpoint-dependent.""" + + def __init__(self, inner: Any) -> None: + self._inner = inner + self.call_count = 0 + + def complete(self, request: Any) -> Any: + self.call_count += 1 + return self._inner.complete(request) + + +def _build_provider() -> Any: + base_url = os.getenv("RANKSMITH_OPENAI_BASE_URL") + if base_url: + return _openai_compatible_provider(base_url) + from ranksmith.providers.azure import AzureAOAIProvider + + return AzureAOAIProvider( + api_key=_required_env("AZURE_OPENAI_API_KEY"), + azure_endpoint=_required_env("AZURE_OPENAI_ENDPOINT"), + azure_deployment=_required_env( + "AZURE_OPENAI_LLM_DEPLOYMENT", + fallback="AZURE_OPENAI_DEPLOYMENT", + ), + api_version=_env_value( + "AZURE_OPENAI_LLM_API_VERSION", + fallback="AZURE_OPENAI_API_VERSION", + default="2024-08-01-preview", + ), + timeout=_env_float("AZURE_OPENAI_LLM_TIMEOUT"), + ) + + +def _openai_compatible_provider(base_url: str) -> Any: + # Same escape hatch as scripts/compare_reranking.py: any OpenAI-compatible + # chat endpoint (LM Studio, vLLM, ...) with RANKSMITH_OPENAI_MODEL / + # RANKSMITH_OPENAI_API_KEY overrides. + from openai import OpenAI + + from ranksmith.model import ModelRequest, ModelResponse + + client = OpenAI( + base_url=base_url, + api_key=os.getenv("RANKSMITH_OPENAI_API_KEY", "local"), + timeout=180, + ) + model = os.getenv("RANKSMITH_OPENAI_MODEL", "local-model") + + class _Provider: + def complete(self, request: ModelRequest) -> ModelResponse: + resp = client.chat.completions.create( + model=model, + messages=[ + {"role": m.role, "content": m.content} for m in request.messages + ], + temperature=0, + ) + return ModelResponse(content=resp.choices[0].message.content or "") + + return _Provider() + + +def _load_env_file(path: Path) -> None: + if not path.exists(): + return + for line in path.read_text(encoding="utf-8").splitlines(): + stripped = line.strip() + if stripped == "" or stripped.startswith("#"): + continue + key, separator, value = stripped.partition("=") + if separator == "": + raise SystemExit(f"Invalid .env line without '=': {line}") + key = key.strip() + if key == "": + raise SystemExit(f"Invalid .env line with empty key: {line}") + os.environ.setdefault(key, _clean_env_value(value)) + + +def _clean_env_value(value: str) -> str: + stripped = value.strip() + if stripped.startswith(("'", '"')): + quote = stripped[0] + end = stripped.find(quote, 1) + if end == -1: + raise SystemExit("Invalid .env quoted value.") + return stripped[1:end] + return stripped.split("#", maxsplit=1)[0].strip() + + +def _required_env(name: str, *, fallback: str | None = None) -> str: + value = _env_value(name, fallback=fallback) + if value is None or value == "": + names = name if fallback is None else f"{name} or {fallback}" + raise SystemExit(f"Missing required environment variable: {names}") + return value + + +def _env_value( + name: str, + *, + fallback: str | None = None, + default: str | None = None, +) -> str | None: + value = os.environ.get(name) + if value is not None and value != "": + return value + if fallback is not None: + fallback_value = os.environ.get(fallback) + if fallback_value is not None and fallback_value != "": + return fallback_value + return default + + +def _env_float(name: str) -> float | None: + value = os.environ.get(name) + if value is None or value == "": + return None + return float(value) + + +if __name__ == "__main__": + main() diff --git a/skills/ranksmith-advisor/SKILL.md b/skills/ranksmith-advisor/SKILL.md index fd34c26..5d5c30e 100644 --- a/skills/ranksmith-advisor/SKILL.md +++ b/skills/ranksmith-advisor/SKILL.md @@ -33,8 +33,10 @@ ranksmith's contracts intact matters — those are the most common failures. - `rank` is 1-based; `original_index` is 0-based against the input list. - ranksmith **fails fast**: never silently truncate documents or repair an invalid ranking. -- `ranksmith.confidence` is **not** a reranking strategy — it is a separate - optional utility. +- `ranksmith.confidence` (the estimator) is a scoring utility, not a reranker. + `AnswerConfidenceRerankStrategy` consumes it and **is** a reranking Strategy, + but it is experimental and loses to Listwise on the committed benchmark — not + a default recommendation. ## Evidence rule diff --git a/skills/ranksmith-advisor/guardrails.md b/skills/ranksmith-advisor/guardrails.md index cbf3fc0..d3f01ff 100644 --- a/skills/ranksmith-advisor/guardrails.md +++ b/skills/ranksmith-advisor/guardrails.md @@ -30,11 +30,17 @@ second copy. are not integers in a ranking. Source: `src/ranksmith/parsing.py`, `src/ranksmith/errors.py`, AGENTS.md "프로젝트 원칙". -6. **Confidence is not reranking.** `ranksmith.confidence`, - `ranksmith.confidence_generation`, and `ranksmith.confidence_training` are - optional utilities (`pip install "ranksmith[confidence]"`), not a Strategy or - Algorithm. Do not present them as a reranker. - Source: `docs/wiki/02_architecture.md` "Confidence". +6. **Confidence: the estimator is a utility; the reranker is experimental.** + `ranksmith.confidence` / `confidence_generation` / `confidence_training` are + optional scoring/data/training utilities (`pip install + "ranksmith[confidence]"`), not rerankers themselves. + `AnswerConfidenceRerankStrategy` **is** a reranking Strategy that consumes a + trained `answer_confidence` estimator (one LLM answer per document + local + scoring). It is **experimental and not a default recommendation**: on the + committed benchmark it loses to `ListwiseStrategy` (acc@1 0.80 vs 1.00) at 4x + the LLM cost. Recommend it only if the user explicitly wants confidence-based + reranking, and always name the Listwise baseline. + Source: `docs/specs/spec_confidence_aware_reranking.md`. 7. **Evidence only.** Quote metrics and call counts from the committed README / benchmark artifacts, label call counts as estimates, and never fabricate diff --git a/skills/ranksmith-advisor/method-guide.md b/skills/ranksmith-advisor/method-guide.md index c5e8f97..848513d 100644 --- a/skills/ranksmith-advisor/method-guide.md +++ b/skills/ranksmith-advisor/method-guide.md @@ -14,6 +14,7 @@ in sync with them; do not invent numbers. | Pairwise preference / reproduce PRP / best MRR | `PairwiseStrategy(...)` | Best MRR@5 in the benchmark, but many calls. | | Adaptive spend near the top-k boundary | `AcuRankStrategy(target_rank=k)` | TrueSkill-based; uses `metadata["score"]` as a first-stage prior when present. | | Setwise top-k extraction with early stopping | `SetwiseStrategy(set_size=...)` | The only strategy with true `top_k` early stopping; lowest quality in the benchmark. | +| Confidence-based reranking (explicitly asked for) | `AnswerConfidenceRerankStrategy(estimator=...)` | Experimental; needs a trained artifact and 1 LLM call/doc. Loses to Listwise on the committed eval — name that baseline. | | Deterministic business logic / a new research method | A custom Strategy class | You own the ranking contract. | | High-throughput (FastAPI, etc.) | The `Async*` variant of any of the above | Same contracts, `await`ed. | @@ -73,6 +74,17 @@ exact provider-call telemetry. seeds the prior. All-or-none — partial or boolean scores fail fast. - Model op: `rank()` over batches. +### AnswerConfidenceRerankStrategy / AsyncAnswerConfidenceRerankStrategy — experimental +- `estimator` (required, a trained `answer_confidence` + `StructuralConfidenceEstimator`), `max_document_chars=4000`. +- Per document: one `answer()` LLM call, then local structural confidence that + the answer is correct; documents sorted by that confidence, descending. +- **Not a default recommendation.** Needs a trained artifact (QA data with gold + answers) and costs 1 LLM call per document. On the committed eval it loses to + `ListwiseStrategy` (acc@1 0.80 vs 1.00) at 4x the cost. Recommend only when the + user explicitly asks for confidence-based reranking; always name Listwise. +- Model op: `answer()` per document. + ## top_k behavior Only `SetwiseStrategy` stops early at `top_k`. Every other strategy computes the full order and slices to `top_k` at the end. `top_k` must be `>= 0`. diff --git a/skills/ranksmith-advisor/snippets.md b/skills/ranksmith-advisor/snippets.md index 2f43f86..a196eab 100644 --- a/skills/ranksmith-advisor/snippets.md +++ b/skills/ranksmith-advisor/snippets.md @@ -19,6 +19,7 @@ Two sources, by audience: | Setwise heapsort | `examples/setwise_heapsort.py` | | TourRank-r | `examples/tourrank.py` | | AcuRank + first-stage scores | `examples/acurank.py` | +| Answer-confidence rerank (experimental) | `examples/answer_confidence.py` | | Custom strategy contracts | `examples/custom_strategy.py` | GitHub: https://github.com/pko89403/ranksmith/tree/main/examples diff --git a/src/ranksmith/__init__.py b/src/ranksmith/__init__.py index 4a353c5..53f640d 100644 --- a/src/ranksmith/__init__.py +++ b/src/ranksmith/__init__.py @@ -16,12 +16,18 @@ ModelRequest, ModelResponse, ) -from ranksmith.parsing import parse_ranking_response, parse_selection_response +from ranksmith.parsing import ( + parse_answer_response, + parse_ranking_response, + parse_selection_response, +) from ranksmith.protocols import AsyncRerankStrategy, RerankStrategy from ranksmith.providers import AsyncAzureAOAIProvider, AzureAOAIProvider from ranksmith.strategies import ( AcuRankStrategy, + AnswerConfidenceRerankStrategy, AsyncAcuRankStrategy, + AsyncAnswerConfidenceRerankStrategy, AsyncListwiseStrategy, AsyncPairwiseStrategy, AsyncSetwiseStrategy, @@ -39,6 +45,7 @@ "AsyncAcuRankStrategy", "AsyncAzureAOAIProvider", "AsyncAzureOpenAIReranker", + "AsyncAnswerConfidenceRerankStrategy", "AsyncListwiseStrategy", "AsyncModelClient", "AsyncModelProvider", @@ -48,6 +55,7 @@ "AsyncTourRankStrategy", "AzureAOAIProvider", "AzureOpenAIReranker", + "AnswerConfidenceRerankStrategy", "Document", "DocumentTooLongError", "ListwiseStrategy", @@ -68,6 +76,7 @@ "RerankUsage", "TourRankStageConfig", "TourRankStrategy", + "parse_answer_response", "parse_ranking_response", "parse_selection_response", ] diff --git a/src/ranksmith/confidence_generation/labeling.py b/src/ranksmith/confidence_generation/labeling.py index ed8e374..3eaf1dc 100644 --- a/src/ranksmith/confidence_generation/labeling.py +++ b/src/ranksmith/confidence_generation/labeling.py @@ -5,22 +5,14 @@ from typing import Literal from ranksmith.confidence_generation.errors import ConfidenceGenerationInputError +from ranksmith.model import NO_ANSWER_VALUE JudgmentValue = Literal["relevant", "not_relevant"] -def normalized_exact_match( - answer: str, - gold_answer: str | list[str], - *, - no_answer_value: str = "__NO_ANSWER__", -) -> bool: - if not isinstance(no_answer_value, str) or not no_answer_value.strip(): - raise ConfidenceGenerationInputError( - "no_answer_value must be a non-empty string" - ) +def normalized_exact_match(answer: str, gold_answer: str | list[str]) -> bool: normalized_answer = _normalize(answer) - if normalized_answer == _normalize(no_answer_value): + if normalized_answer == _normalize(NO_ANSWER_VALUE): return False candidates = [gold_answer] if isinstance(gold_answer, str) else gold_answer diff --git a/src/ranksmith/confidence_generation/pipeline.py b/src/ranksmith/confidence_generation/pipeline.py index b7687d6..622439b 100644 --- a/src/ranksmith/confidence_generation/pipeline.py +++ b/src/ranksmith/confidence_generation/pipeline.py @@ -21,9 +21,7 @@ parse_relevance_output, ) from ranksmith.confidence_generation.prompts import ( - ANSWER_SYSTEM_PROMPT, RELEVANCE_SYSTEM_PROMPT, - build_answer_prompt, build_relevance_prompt, ) from ranksmith.confidence_generation.types import ( @@ -35,7 +33,13 @@ UsageCallback, ) from ranksmith.errors import RerankProviderError -from ranksmith.model import ModelMessage, ModelProvider, ModelRequest +from ranksmith.model import ( + NO_ANSWER_VALUE, + ModelMessage, + ModelProvider, + ModelRequest, + _answer_messages, +) from ranksmith.types import RerankUsage SampleT = TypeVar("SampleT") @@ -181,23 +185,15 @@ def _build_answer_generated_row( sample: AnswerGenerationSample, config: AnswerGenerationConfig, ) -> Mapping[str, Any]: + system, user = _answer_messages(sample.query, sample.context) raw_output = _call_provider( config.provider, - system=ANSWER_SYSTEM_PROMPT, - user=build_answer_prompt( - sample, - no_answer_value=config.no_answer_value, - ), + system=system, + user=user, on_usage=config.on_usage, ) answer = parse_answer_output(raw_output) - label = int( - normalized_exact_match( - answer, - sample.gold_answer, - no_answer_value=config.no_answer_value, - ) - ) + label = int(normalized_exact_match(answer, sample.gold_answer)) return _answer_canonical_row( sample, answer=answer, @@ -219,7 +215,7 @@ def _answer_canonical_row( "generation_task": "answer_oriented", "query": sample.query, "match_policy": "normalized_exact", - "no_answer_value": config.no_answer_value, + "no_answer_value": NO_ANSWER_VALUE, } if config.include_raw_model_output: generation["raw_model_output"] = raw_output diff --git a/src/ranksmith/confidence_generation/prompts.py b/src/ranksmith/confidence_generation/prompts.py index ac5f496..bcd36da 100644 --- a/src/ranksmith/confidence_generation/prompts.py +++ b/src/ranksmith/confidence_generation/prompts.py @@ -1,43 +1,15 @@ from __future__ import annotations -import json - from ranksmith.confidence_generation.types import ( - AnswerGenerationSample, RelevanceGenerationSample, ) -ANSWER_SYSTEM_PROMPT = ( - "You answer questions using only the provided context. " - 'Return only JSON with an "answer" string.' -) - RELEVANCE_SYSTEM_PROMPT = ( "You judge document relevance. " 'Return only JSON with a "judgment" value of "relevant" or "not_relevant".' ) -def build_answer_prompt( - sample: AnswerGenerationSample, - *, - no_answer_value: str, -) -> str: - no_answer_contract = json.dumps( - {"answer": no_answer_value}, - ensure_ascii=False, - separators=(",", ":"), - ) - return ( - f"Question:\n{sample.query}\n\n" - f"Context:\n{sample.context}\n\n" - "Return JSON exactly like this shape:\n" - '{"answer":"..."}\n\n' - "Use only the context. If the context does not contain the answer, " - f"return {no_answer_contract}." - ) - - def build_relevance_prompt(sample: RelevanceGenerationSample) -> str: return ( f"Query:\n{sample.query}\n\n" diff --git a/src/ranksmith/confidence_generation/types.py b/src/ranksmith/confidence_generation/types.py index dcc8e06..54fdb9c 100644 --- a/src/ranksmith/confidence_generation/types.py +++ b/src/ranksmith/confidence_generation/types.py @@ -27,7 +27,6 @@ class AnswerGenerationConfig: include_raw_model_output: bool = True on_usage: UsageCallback | None = None source: str | None = None - no_answer_value: str = "__NO_ANSWER__" def __post_init__(self) -> None: _validate_common_config( @@ -38,13 +37,6 @@ def __post_init__(self) -> None: source=self.source, ) _validate_positive_int("max_context_chars", self.max_context_chars) - if ( - not isinstance(self.no_answer_value, str) - or not self.no_answer_value.strip() - ): - raise ConfidenceGenerationInputError( - "no_answer_value must be a non-empty string" - ) @dataclass(frozen=True) diff --git a/src/ranksmith/model.py b/src/ranksmith/model.py index 969b9c6..b5ed698 100644 --- a/src/ranksmith/model.py +++ b/src/ranksmith/model.py @@ -77,6 +77,10 @@ def select(self, query: str, documents: list[Document], top_m: int) -> str: system, user = _select_messages(query, documents, top_m) return self._complete(system=system, user=user) + def answer(self, query: str, context: str) -> str: + system, user = _answer_messages(query, context) + return self._complete(system=system, user=user) + def _complete(self, *, system: str, user: str) -> str: try: response = self._provider.complete( @@ -125,6 +129,10 @@ async def select(self, query: str, documents: list[Document], top_m: int) -> str system, user = _select_messages(query, documents, top_m) return await self._complete(system=system, user=user) + async def answer(self, query: str, context: str) -> str: + system, user = _answer_messages(query, context) + return await self._complete(system=system, user=user) + async def _complete(self, *, system: str, user: str) -> str: try: response = await self._provider.complete( @@ -201,6 +209,25 @@ def _select_messages( return system, _build_selection_prompt(query, documents, top_m) +NO_ANSWER_VALUE = "__NO_ANSWER__" + + +def _answer_messages(query: str, context: str) -> tuple[str, str]: + system = ( + "You answer questions using only the provided context. " + 'Return only JSON with an "answer" string.' + ) + user = ( + f"Question:\n{query}\n\n" + f"Context:\n{context}\n\n" + "Return JSON exactly like this shape:\n" + '{"answer":"..."}\n\n' + "Use only the context. If the context does not contain the answer, " + f'return {{"answer":"{NO_ANSWER_VALUE}"}}.' + ) + return system, user + + def _build_prompt(query: str, documents: list[Document]) -> str: candidate_count = len(documents) ranking_example = ", ".join(str(index) for index in range(1, candidate_count + 1)) diff --git a/src/ranksmith/parsing.py b/src/ranksmith/parsing.py index 1607f36..c261e81 100644 --- a/src/ranksmith/parsing.py +++ b/src/ranksmith/parsing.py @@ -4,13 +4,33 @@ from ranksmith.errors import RerankInputError, RerankParseError -__all__ = ["parse_ranking_response", "parse_selection_response"] +__all__ = [ + "parse_answer_response", + "parse_ranking_response", + "parse_selection_response", +] def _is_json_int(value: object) -> bool: return type(value) is int +def parse_answer_response(raw_response: str) -> str: + """Parse a non-empty answer string from provider JSON ({"answer": ...}).""" + try: + data = json.loads(raw_response) + except json.JSONDecodeError as exc: + raise RerankParseError("LLM response is not valid JSON.", raw_response) from exc + + answer = data.get("answer") if isinstance(data, dict) else None + if not isinstance(answer, str) or not answer.strip(): + raise RerankParseError( + 'LLM response must contain a non-empty "answer" string.', + raw_response, + ) + return answer + + def parse_ranking_response(raw_response: str, *, expected_count: int) -> list[int]: """Parse and validate a 1-based ranking permutation from provider JSON.""" if expected_count < 0: diff --git a/src/ranksmith/strategies/__init__.py b/src/ranksmith/strategies/__init__.py index 222d0d4..6f8a4c4 100644 --- a/src/ranksmith/strategies/__init__.py +++ b/src/ranksmith/strategies/__init__.py @@ -1,4 +1,8 @@ from ranksmith.strategies.acurank import AcuRankStrategy, AsyncAcuRankStrategy +from ranksmith.strategies.confidence import ( + AnswerConfidenceRerankStrategy, + AsyncAnswerConfidenceRerankStrategy, +) from ranksmith.strategies.listwise import AsyncListwiseStrategy, ListwiseStrategy from ranksmith.strategies.pairwise import AsyncPairwiseStrategy, PairwiseStrategy from ranksmith.strategies.setwise import AsyncSetwiseStrategy, SetwiseStrategy @@ -11,10 +15,12 @@ __all__ = [ "AcuRankStrategy", "AsyncAcuRankStrategy", + "AsyncAnswerConfidenceRerankStrategy", "AsyncListwiseStrategy", "AsyncPairwiseStrategy", "AsyncSetwiseStrategy", "AsyncTourRankStrategy", + "AnswerConfidenceRerankStrategy", "ListwiseStrategy", "PairwiseStrategy", "SetwiseStrategy", diff --git a/src/ranksmith/strategies/confidence.py b/src/ranksmith/strategies/confidence.py new file mode 100644 index 0000000..28901e8 --- /dev/null +++ b/src/ranksmith/strategies/confidence.py @@ -0,0 +1,154 @@ +from __future__ import annotations + +import asyncio +from collections.abc import Sequence +from dataclasses import dataclass +from typing import Protocol + +from ranksmith.confidence.types import AnswerConfidenceInput +from ranksmith.errors import RerankInputError +from ranksmith.model import AsyncModelClient, ModelClient +from ranksmith.parsing import parse_answer_response +from ranksmith.types import Document, RerankResult + +from .common import ( + ensure_capability, + validate_documents_max_chars, + validate_top_k, +) + + +class _ConfidenceResult(Protocol): + @property + def score(self) -> float: ... + + +class AnswerConfidenceScorer(Protocol): + # Read-only property members so frozen estimators (for example + # StructuralConfidenceEstimator with its Literal task_type) satisfy the + # protocol; plain attribute members would demand settable fields. + @property + def task_type(self) -> str: ... + + def score(self, item: AnswerConfidenceInput) -> _ConfidenceResult: ... + + +@dataclass(frozen=True) +class _AnswerConfidenceRerankConfigMixin: + """Confidence-change reranking (CBDR-style), black-box via structural confidence. + + For each candidate the model answers the query from that document, and the + document's confidence is the local structural confidence that the answer is + correct. Documents are ordered by that confidence, descending. + + This ranks by confidence change: CBDR ranks by Inc = C(query+doc) - C(query). + Within a single query the baseline C(query) is a constant, so it does not + affect the order — and answer_confidence cannot score an empty context + anyway — so the reranker ranks by C(query+doc) directly. + """ + + estimator: AnswerConfidenceScorer + max_document_chars: int = 4000 + + def __post_init__(self) -> None: + if self.estimator.task_type != "answer_confidence": + raise RerankInputError('estimator.task_type must be "answer_confidence"') + if self.max_document_chars < 1: + raise ValueError("max_document_chars must be greater than 0") + + def _validate_documents(self, documents: Sequence[Document]) -> None: + validate_documents_max_chars( + documents, + max_document_chars=self.max_document_chars, + ) + # answer_confidence cannot score an empty context; fail before any + # paid answer() call instead of surfacing a non-RerankError from the + # estimator afterwards. + for index, document in enumerate(documents): + if document.text.strip() == "": + raise RerankInputError( + f"documents[{index}].text must not be empty or " + "whitespace-only for answer_confidence." + ) + + def _confidence(self, *, document: Document, answer: str) -> float: + return self.estimator.score( + AnswerConfidenceInput(context=document.text, answer=answer) + ).score + + def _results_from_confidence( + self, + documents: Sequence[Document], + confidences: Sequence[float], + top_k: int | None, + ) -> list[RerankResult]: + ordered_indexes = sorted( + range(len(documents)), + key=lambda index: (-confidences[index], index), + ) + if top_k is not None: + ordered_indexes = ordered_indexes[:top_k] + return [ + RerankResult( + document=documents[original_index], + rank=rank, + original_index=original_index, + metadata={ + "strategy": "answer_confidence", + "algorithm": "answer_confidence", + "answer_confidence": confidences[original_index], + }, + ) + for rank, original_index in enumerate(ordered_indexes, start=1) + ] + + +@dataclass(frozen=True) +class AnswerConfidenceRerankStrategy(_AnswerConfidenceRerankConfigMixin): + def rerank( + self, + *, + query: str, + documents: Sequence[Document], + model_client: ModelClient, + top_k: int | None = None, + ) -> list[RerankResult]: + validate_top_k(top_k) + self._validate_documents(documents) + if not documents: + return [] + + model_client = ensure_capability(model_client, "answer", "answer") + confidences: list[float] = [] + for document in documents: + answer = parse_answer_response(model_client.answer(query, document.text)) + confidences.append(self._confidence(document=document, answer=answer)) + return self._results_from_confidence(documents, confidences, top_k) + + +@dataclass(frozen=True) +class AsyncAnswerConfidenceRerankStrategy(_AnswerConfidenceRerankConfigMixin): + async def rerank( + self, + *, + query: str, + documents: Sequence[Document], + model_client: AsyncModelClient, + top_k: int | None = None, + ) -> list[RerankResult]: + validate_top_k(top_k) + self._validate_documents(documents) + if not documents: + return [] + + model_client = ensure_capability(model_client, "answer", "answer") + # Answer generation is I/O-bound: run the calls concurrently. Scoring is + # a synchronous CPU/encoder step, so it stays sequential afterwards. + raw_answers = await asyncio.gather( + *(model_client.answer(query, document.text) for document in documents) + ) + confidences: list[float] = [] + for document, raw in zip(documents, raw_answers, strict=True): + answer = parse_answer_response(raw) + confidences.append(self._confidence(document=document, answer=answer)) + return self._results_from_confidence(documents, confidences, top_k) diff --git a/tests/test_advisor_references.py b/tests/test_advisor_references.py index 42aa60f..c7b4bc1 100644 --- a/tests/test_advisor_references.py +++ b/tests/test_advisor_references.py @@ -83,6 +83,9 @@ def test_snippets_use_public_ranksmith_symbols() -> None: "initial_pass": True, "max_adaptive_reranker_calls": None, }, + "AnswerConfidenceRerankStrategy": { + "max_document_chars": 4000, + }, } diff --git a/tests/test_bm25.py b/tests/test_bm25.py new file mode 100644 index 0000000..6af6e55 --- /dev/null +++ b/tests/test_bm25.py @@ -0,0 +1,97 @@ +from __future__ import annotations + +import pytest + +from benchmarks.bm25 import bm25_search, build_bm25_index, tokenize + + +def test_tokenize_lowercases_and_splits_on_non_alphanumerics() -> None: + assert tokenize("Vitamin-C deficiency, causes SCURVY!") == [ + "vitamin", + "c", + "deficiency", + "causes", + "scurvy", + ] + + +def test_bm25_ranks_matching_document_first() -> None: + index = build_bm25_index( + { + "d1": "scurvy is caused by vitamin c deficiency", + "d2": "sleep deprivation weakens the immune system", + "d3": "apples contain vitamins and fiber", + } + ) + + results = bm25_search(index, "what causes scurvy", top_k=3) + + assert [document_id for document_id, _ in results][0] == "d1" + assert all(score > 0 for _, score in results) + + +def test_bm25_omits_documents_without_query_terms() -> None: + index = build_bm25_index( + { + "d1": "scurvy is caused by vitamin c deficiency", + "d2": "sleep deprivation weakens the immune system", + } + ) + + results = bm25_search(index, "scurvy", top_k=5) + + assert [document_id for document_id, _ in results] == ["d1"] + + +def test_bm25_returns_empty_for_out_of_vocabulary_query() -> None: + index = build_bm25_index({"d1": "alpha beta"}) + + assert bm25_search(index, "zzz", top_k=3) == [] + + +def test_bm25_excludes_requested_documents() -> None: + index = build_bm25_index( + { + "d1": "vitamin c deficiency", + "d2": "vitamin c deficiency", + } + ) + + results = bm25_search(index, "vitamin c", top_k=2, exclude=("d1",)) + + assert [document_id for document_id, _ in results] == ["d2"] + + +def test_bm25_breaks_score_ties_by_document_id() -> None: + index = build_bm25_index( + { + "d2": "vitamin c deficiency", + "d1": "vitamin c deficiency", + } + ) + + results = bm25_search(index, "vitamin", top_k=2) + + assert [document_id for document_id, _ in results] == ["d1", "d2"] + assert results[0][1] == results[1][1] + + +def test_bm25_respects_top_k() -> None: + index = build_bm25_index({f"d{i}": f"vitamin document {i}" for i in range(10)}) + + assert len(bm25_search(index, "vitamin", top_k=4)) == 4 + + +def test_bm25_rejects_empty_corpus() -> None: + with pytest.raises(ValueError, match="must not be empty"): + build_bm25_index({}) + + +def test_bm25_rejects_invalid_parameters() -> None: + with pytest.raises(ValueError, match="k1"): + build_bm25_index({"d1": "text"}, k1=0) + with pytest.raises(ValueError, match="b"): + build_bm25_index({"d1": "text"}, b=1.5) + index = build_bm25_index({"d1": "text"}) + with pytest.raises(ValueError, match="top_k"): + bm25_search(index, "text", top_k=0) diff --git a/tests/test_build_answer_confidence_training_data.py b/tests/test_build_answer_confidence_training_data.py new file mode 100644 index 0000000..f9fc116 --- /dev/null +++ b/tests/test_build_answer_confidence_training_data.py @@ -0,0 +1,228 @@ +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path + +import pytest + +from ranksmith.confidence_generation.io import load_answer_generation_samples + +ROOT = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location( + "build_answer_confidence_training_data", + ROOT / "scripts" / "build_answer_confidence_training_data.py", +) +assert SPEC is not None +builder = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +# Dataclass processing looks the module up in sys.modules by name. +sys.modules[SPEC.name] = builder +SPEC.loader.exec_module(builder) + + +def _squad_payload() -> dict[str, object]: + return { + "version": "1.1", + "data": [ + { + "title": "Scurvy", + "paragraphs": [ + { + "context": ( + "Scurvy is a disease resulting from a lack of " + "vitamin C. Early symptoms include weakness and " + "gum disease." + ), + "qas": [ + { + "id": "q-scurvy-1", + "question": "What deficiency causes scurvy?", + "answers": [ + {"text": "a lack of vitamin C"}, + {"text": "a lack of vitamin C"}, + {"text": "vitamin C"}, + ], + } + ], + }, + { + "context": ( + "Treatment of scurvy consists of vitamin C " + "supplementation and a diet rich in fresh fruit." + ), + "qas": [ + { + "id": "q-scurvy-2", + "question": "How is scurvy treated?", + "answers": [{"text": "vitamin C supplementation"}], + } + ], + }, + ], + }, + { + "title": "Sleep", + "paragraphs": [ + { + "context": ( + "Sleep deprivation weakens the immune system and " + "impairs memory consolidation in humans." + ), + "qas": [ + { + "id": "q-sleep-1", + "question": "What does sleep deprivation weaken?", + "answers": [{"text": "the immune system"}], + } + ], + }, + { + "context": ( + "Good sleep hygiene includes regular schedules " + "and avoiding caffeine before bed." + ), + "qas": [], + }, + ], + }, + ], + } + + +def _write_squad(path: Path) -> Path: + path.write_text(json.dumps(_squad_payload()), encoding="utf-8") + return path + + +def _run_builder(tmp_path: Path, *extra: str) -> Path: + squad_path = _write_squad(tmp_path / "train-v1.1.json") + output = tmp_path / "answer_train.jsonl" + argv = [ + "build_answer_confidence_training_data.py", + "--squad-train", + str(squad_path), + "--output", + str(output), + "--questions", + "3", + "--negatives-per-question", + "1", + "--negative-pool", + "2", + *extra, + ] + original = sys.argv + sys.argv = argv + try: + builder.main() + finally: + sys.argv = original + return output + + +def test_builder_output_loads_with_the_real_generation_loader( + tmp_path: Path, +) -> None: + output = _run_builder(tmp_path) + + samples = load_answer_generation_samples(output, max_context_chars=4000) + + assert len(samples) == 6 # 3 questions x (1 gold + 1 hard negative) + by_id = {sample.id: sample for sample in samples} + gold = by_id["q-scurvy-1::gold"] + negative = by_id["q-scurvy-1::neg1"] + assert gold.query == negative.query + assert gold.gold_answer == negative.gold_answer + assert gold.gold_answer == ["a lack of vitamin C", "vitamin C"] + assert gold.context.startswith("Scurvy\n\n") + assert gold.context != negative.context + assert gold.metadata["context_kind"] == "gold" + assert negative.metadata["context_kind"] == "bm25_hard_negative" + assert gold.group_id == "Scurvy" + + +def test_builder_hard_negative_is_the_top_non_gold_bm25_paragraph( + tmp_path: Path, +) -> None: + output = _run_builder(tmp_path) + + rows = [ + json.loads(line) for line in output.read_text(encoding="utf-8").splitlines() + ] + negative = next(row for row in rows if row["id"] == "q-scurvy-1::neg1") + + # The other Scurvy paragraph shares "scurvy"/"vitamin" terms, so it must + # beat the Sleep paragraph as the hard negative. + assert negative["context"].startswith("Scurvy\n\n") + assert "Treatment of scurvy" in negative["context"] + + +def test_builder_is_deterministic(tmp_path: Path) -> None: + first = _run_builder(tmp_path).read_bytes() + second = _run_builder(tmp_path, "--overwrite").read_bytes() + + assert first == second + + +def test_builder_writes_a_report(tmp_path: Path) -> None: + output = _run_builder(tmp_path) + + report = json.loads( + (tmp_path / "answer_train.report.json").read_text(encoding="utf-8") + ) + + assert report["source"] == "squad-v1.1-train" + assert report["statistics"]["total_rows"] == 6 + assert report["statistics"]["gold_rows"] == 3 + assert report["statistics"]["negative_rows"] == 3 + assert report["parameters"]["questions"] == 3 + assert len(report["squad_sha256"]) == 64 + assert output.exists() + + +def test_builder_refuses_overwrite_without_flag(tmp_path: Path) -> None: + _run_builder(tmp_path) + + with pytest.raises(SystemExit, match="Refusing to overwrite"): + _run_builder(tmp_path) + + +def test_builder_fails_when_not_enough_questions_qualify(tmp_path: Path) -> None: + with pytest.raises(SystemExit, match="Only .* usable"): + _run_builder(tmp_path, "--overwrite", "--questions", "4") + + +def test_builder_rejects_squad_v2(tmp_path: Path) -> None: + payload = _squad_payload() + payload["version"] = "v2.0" + squad_path = tmp_path / "train-v2.0.json" + squad_path.write_text(json.dumps(payload), encoding="utf-8") + original = sys.argv + sys.argv = [ + "build_answer_confidence_training_data.py", + "--squad-train", + str(squad_path), + "--output", + str(tmp_path / "out.jsonl"), + ] + try: + with pytest.raises(SystemExit, match="version 1.1"): + builder.main() + finally: + sys.argv = original + + +def test_builder_requires_an_input_source(tmp_path: Path) -> None: + original = sys.argv + sys.argv = [ + "build_answer_confidence_training_data.py", + "--output", + str(tmp_path / "out.jsonl"), + ] + try: + with pytest.raises(SystemExit, match="--squad-train or --download"): + builder.main() + finally: + sys.argv = original diff --git a/tests/test_compare_reranking.py b/tests/test_compare_reranking.py index 35528ba..3df75cf 100644 --- a/tests/test_compare_reranking.py +++ b/tests/test_compare_reranking.py @@ -23,7 +23,7 @@ def test_compare_all_uses_top20_bm25_default_methods() -> None: - args = argparse.Namespace(algorithm="all") + args = argparse.Namespace(algorithm=["all"]) cases = [ BenchmarkCase( fixture_id="fixture", @@ -54,7 +54,7 @@ def test_compare_all_uses_top20_bm25_default_methods() -> None: def test_compare_all_is_stable_for_100_candidate_cases() -> None: - args = argparse.Namespace(algorithm="all") + args = argparse.Namespace(algorithm=["all"]) cases = [ BenchmarkCase( fixture_id="fixture", @@ -83,28 +83,28 @@ def test_compare_all_is_stable_for_100_candidate_cases() -> None: def test_compare_optional_prp_sliding_p3_is_preserved() -> None: - args = argparse.Namespace(algorithm="prp_sliding_p3") + args = argparse.Namespace(algorithm=["prp_sliding_p3"]) cases: list[BenchmarkCase] = [] assert compare_reranking._selected_algorithms(args, cases) == ("prp_sliding_p3",) def test_compare_optional_prp_sliding_p1_is_preserved() -> None: - args = argparse.Namespace(algorithm="prp_sliding_p1") + args = argparse.Namespace(algorithm=["prp_sliding_p1"]) cases: list[BenchmarkCase] = [] assert compare_reranking._selected_algorithms(args, cases) == ("prp_sliding_p1",) def test_compare_explicit_tourrank_is_preserved_for_non_100_candidate_cases() -> None: - args = argparse.Namespace(algorithm="tourrank_r") + args = argparse.Namespace(algorithm=["tourrank_r"]) cases: list[BenchmarkCase] = [] assert compare_reranking._selected_algorithms(args, cases) == ("tourrank_r",) def test_compare_explicit_setwise_heapsort_is_preserved() -> None: - args = argparse.Namespace(algorithm="setwise_heapsort") + args = argparse.Namespace(algorithm=["setwise_heapsort"]) cases: list[BenchmarkCase] = [] assert compare_reranking._selected_algorithms(args, cases) == ("setwise_heapsort",) @@ -133,7 +133,7 @@ def test_compare_estimates_tourrank_calls_from_generated_stages() -> None: def test_compare_explicit_acurank_is_preserved() -> None: - args = argparse.Namespace(algorithm="acurank") + args = argparse.Namespace(algorithm=["acurank"]) cases: list[BenchmarkCase] = [] assert compare_reranking._selected_algorithms(args, cases) == ("acurank",) @@ -822,3 +822,116 @@ def test_compare_beir_scifact_dataset_name_remains_legacy_alias() -> None: ) assert compare_reranking._dataset_name(args) == "BEIR/SciFact" + + +def test_compare_default_algorithm_selection_is_the_default_set() -> None: + args = argparse.Namespace(algorithm=None) + + algorithms = compare_reranking._selected_algorithms(args, []) + + assert algorithms == compare_reranking.ALGORITHMS + + +def test_compare_repeated_algorithm_flags_run_in_one_invocation() -> None: + args = argparse.Namespace( + algorithm=["original_bm25", "single_call_listwise@20", "answer_confidence"] + ) + + algorithms = compare_reranking._selected_algorithms(args, []) + + assert algorithms == ( + "original_bm25", + "single_call_listwise@20", + "answer_confidence", + ) + + +def test_compare_rejects_all_combined_with_other_algorithms() -> None: + args = argparse.Namespace(algorithm=["all", "answer_confidence"]) + + with pytest.raises(SystemExit, match="cannot be combined"): + compare_reranking._selected_algorithms(args, []) + + +def test_compare_rejects_duplicate_algorithm_flags() -> None: + args = argparse.Namespace(algorithm=["tourrank_r2", "tourrank_r2"]) + + with pytest.raises(SystemExit, match="must not repeat"): + compare_reranking._selected_algorithms(args, []) + + +def _answer_confidence_args(**overrides: object) -> argparse.Namespace: + defaults: dict[str, object] = { + "window_size": 20, + "stride": 10, + "passes": 10, + "tourrank_rounds": 2, + "set_size": 3, + "top_k": 5, + "candidate_count": 20, + "max_cases": None, + "timeout": None, + "checkpoint_output": None, + "output": None, + "dataset": "fixture", + "cache_dir": None, + "dataset_name": None, + "candidates": None, + "candidate_strategy": "candidate_file", + "algorithm": ["answer_confidence"], + "answer_confidence_artifact": None, + } + defaults.update(overrides) + return argparse.Namespace(**defaults) + + +def test_compare_answer_confidence_requires_artifact_before_running() -> None: + args = _answer_confidence_args() + + with pytest.raises(SystemExit, match="requires"): + compare_reranking._validate_args(args) + + +def test_compare_answer_confidence_requires_existing_artifact_file() -> None: + args = _answer_confidence_args( + answer_confidence_artifact="/nonexistent/scorer.joblib" + ) + + with pytest.raises(SystemExit, match="does not exist"): + compare_reranking._validate_args(args) + + +def test_compare_estimates_one_answer_call_per_document_for_answer_confidence() -> None: + assert ( + compare_reranking._estimate_provider_calls( + 20, + "answer_confidence", + window_size=20, + stride=10, + ) + == 20 + ) + + +def test_compare_rank_case_fast_fails_answer_confidence_without_artifact() -> None: + case = BenchmarkCase( + fixture_id="fixture", + dataset="dataset", + source="source", + license="license", + query_id="q1", + query="query", + documents=(BenchmarkDocument(id="1", title="", text="text"),), + qrels={}, + ) + + with pytest.raises(ValueError, match="answer-confidence-artifact"): + compare_reranking._rank_case( + case=case, + algorithm="answer_confidence", + window_size=20, + stride=10, + passes=10, + tourrank_rounds=2, + answer_confidence_artifact=None, + ) diff --git a/tests/test_confidence_generation_api.py b/tests/test_confidence_generation_api.py index 4c8cead..49bbdac 100644 --- a/tests/test_confidence_generation_api.py +++ b/tests/test_confidence_generation_api.py @@ -74,14 +74,6 @@ def test_answer_generation_config_rejects_invalid_options(tmp_path: Path) -> Non max_context_chars=0, ) - with pytest.raises(generation.ConfidenceGenerationInputError): - generation.AnswerGenerationConfig( - input_path=tmp_path / "in.jsonl", - output_path=tmp_path / "out.jsonl", - provider=FakeProvider(), - no_answer_value=" ", - ) - def test_relevance_generation_config_rejects_invalid_options(tmp_path: Path) -> None: generation = importlib.import_module("ranksmith.confidence_generation") diff --git a/tests/test_confidence_generation_labeling.py b/tests/test_confidence_generation_labeling.py index 8339dc5..86c16f1 100644 --- a/tests/test_confidence_generation_labeling.py +++ b/tests/test_confidence_generation_labeling.py @@ -19,16 +19,8 @@ def test_normalized_exact_match_uses_simple_normalization() -> None: def test_no_answer_value_is_always_mismatch() -> None: - assert not normalized_exact_match( - "__NO_ANSWER__", - "__NO_ANSWER__", - no_answer_value="__NO_ANSWER__", - ) - assert not normalized_exact_match( - " __no_answer__ ", - "__NO_ANSWER__", - no_answer_value="__NO_ANSWER__", - ) + assert not normalized_exact_match("__NO_ANSWER__", "__NO_ANSWER__") + assert not normalized_exact_match(" __no_answer__ ", "__NO_ANSWER__") def test_relevance_truth_defaults_to_gt_zero() -> None: @@ -60,8 +52,3 @@ def test_relevance_truth_rejects_invalid_thresholds() -> None: with pytest.raises(ConfidenceGenerationInputError): relevance_truth(1, threshold=math.nan, operator="gt") - - -def test_normalized_exact_match_rejects_invalid_no_answer_value() -> None: - with pytest.raises(ConfidenceGenerationInputError): - normalized_exact_match("answer", "answer", no_answer_value=" ") diff --git a/tests/test_confidence_generation_pipeline.py b/tests/test_confidence_generation_pipeline.py index f4898e7..555c06d 100644 --- a/tests/test_confidence_generation_pipeline.py +++ b/tests/test_confidence_generation_pipeline.py @@ -12,16 +12,12 @@ RelevanceGenerationConfig, ) from ranksmith.confidence_generation.pipeline import _call_provider -from ranksmith.confidence_generation.prompts import ( - build_answer_prompt, - build_relevance_prompt, -) +from ranksmith.confidence_generation.prompts import build_relevance_prompt from ranksmith.confidence_generation.types import ( - AnswerGenerationSample, RelevanceGenerationSample, ) from ranksmith.errors import RerankProviderError -from ranksmith.model import ModelMessage, ModelRequest, ModelResponse +from ranksmith.model import ModelMessage, ModelRequest, ModelResponse, _answer_messages from ranksmith.types import RerankUsage @@ -48,21 +44,16 @@ def complete(self, request: ModelRequest) -> ModelResponse: raise RuntimeError("boom") -def test_answer_prompt_includes_configured_no_answer_value() -> None: - prompt = build_answer_prompt( - AnswerGenerationSample( - id="a1", - query="Who played Karen?", - context="Nancy Travis played Karen.", - gold_answer="Nancy Travis", - ), - no_answer_value="NO_CONTEXT_ANSWER", +def test_answer_prompt_includes_no_answer_contract() -> None: + _system, prompt = _answer_messages( + "Who played Karen?", + "Nancy Travis played Karen.", ) assert "Question:\nWho played Karen?" in prompt assert "Context:\nNancy Travis played Karen." in prompt assert '{"answer":"..."}' in prompt - assert '{"answer":"NO_CONTEXT_ANSWER"}' in prompt + assert '{"answer":"__NO_ANSWER__"}' in prompt def test_relevance_prompt_includes_relevant_not_relevant_json_contract() -> None: @@ -225,16 +216,7 @@ def test_generate_answer_confidence_dataset_writes_canonical_rows( ), ModelMessage( role="user", - content=build_answer_prompt( - AnswerGenerationSample( - id="a1", - query="Who?", - context="Nancy Travis played Karen.", - gold_answer=["nancy travis"], - metadata={"dataset": "unit"}, - ), - no_answer_value="__NO_ANSWER__", - ), + content=_answer_messages("Who?", "Nancy Travis played Karen.")[1], ), ] diff --git a/tests/test_confidence_rerank.py b/tests/test_confidence_rerank.py new file mode 100644 index 0000000..410bddb --- /dev/null +++ b/tests/test_confidence_rerank.py @@ -0,0 +1,198 @@ +from __future__ import annotations + +import json +from dataclasses import dataclass + +import pytest + +from ranksmith import ( + AnswerConfidenceRerankStrategy, + AsyncAnswerConfidenceRerankStrategy, + Document, +) +from ranksmith.confidence.types import AnswerConfidenceInput +from ranksmith.errors import RerankInputError + + +@dataclass +class _Result: + score: float + + +class FakeEstimator: + """Answer-confidence estimator with scripted per-context scores.""" + + task_type = "answer_confidence" + + def __init__(self, scores: dict[str, float]) -> None: + self._scores = scores + + def score(self, item: AnswerConfidenceInput) -> _Result: + return _Result(score=self._scores[item.context]) + + +class FakeAnswerClient: + """Model client that returns scripted answers per context.""" + + def __init__(self, answers: dict[str, str]) -> None: + self._answers = answers + self.calls: list[str] = [] + + def answer(self, query: str, context: str) -> str: + self.calls.append(context) + return json.dumps({"answer": self._answers[context]}) + + +class AsyncFakeAnswerClient(FakeAnswerClient): + async def answer(self, query: str, context: str) -> str: # type: ignore[override] + self.calls.append(context) + return json.dumps({"answer": self._answers[context]}) + + +DOCS = [Document(text="a"), Document(text="b"), Document(text="c"), Document(text="d")] +ANSWERS = {"a": "A1", "b": "B1", "c": "C1", "d": "D1"} + + +def _strategy() -> AnswerConfidenceRerankStrategy: + estimator = FakeEstimator({"a": 0.9, "b": 0.3, "c": 0.7, "d": 0.1}) + return AnswerConfidenceRerankStrategy(estimator=estimator) + + +def test_orders_by_answer_confidence_descending() -> None: + client = FakeAnswerClient(ANSWERS) + results = _strategy().rerank(query="q", documents=DOCS, model_client=client) + + assert [r.document.text for r in results] == ["a", "c", "b", "d"] + assert [r.rank for r in results] == [1, 2, 3, 4] + assert results[0].metadata["answer_confidence"] == pytest.approx(0.9) + assert results[3].metadata["answer_confidence"] == pytest.approx(0.1) + # one answer call per document, no repeated sampling + assert client.calls == ["a", "b", "c", "d"] + + +def test_ties_preserve_original_order() -> None: + client = FakeAnswerClient(ANSWERS) + estimator = FakeEstimator({"a": 0.5, "b": 0.5, "c": 0.5, "d": 0.5}) + results = AnswerConfidenceRerankStrategy(estimator=estimator).rerank( + query="q", documents=DOCS, model_client=client + ) + assert [r.document.text for r in results] == ["a", "b", "c", "d"] + + +def test_top_k_slices_after_ordering() -> None: + client = FakeAnswerClient(ANSWERS) + results = _strategy().rerank( + query="q", documents=DOCS, model_client=client, top_k=2 + ) + assert [r.document.text for r in results] == ["a", "c"] + + +def test_empty_documents_returns_empty() -> None: + client = FakeAnswerClient({}) + assert _strategy().rerank(query="q", documents=[], model_client=client) == [] + + +def test_rejects_non_answer_estimator() -> None: + class JudgmentEstimator: + task_type = "judgment_confidence" + + def score(self, item: AnswerConfidenceInput) -> _Result: + return _Result(score=0.5) + + with pytest.raises(RerankInputError, match="answer_confidence"): + AnswerConfidenceRerankStrategy(estimator=JudgmentEstimator()) + + +def test_rejects_client_without_answer() -> None: + class NoAnswerClient: + def rank(self, query: str, documents: list[Document]) -> str: + return "{}" + + with pytest.raises(RerankInputError, match="answer"): + _strategy().rerank(query="q", documents=DOCS, model_client=NoAnswerClient()) + + +def test_invalid_answer_json_fast_fails() -> None: + from ranksmith.errors import RerankParseError + + class BadClient(FakeAnswerClient): + def answer(self, query: str, context: str) -> str: + return "not json" + + with pytest.raises(RerankParseError): + _strategy().rerank(query="q", documents=DOCS, model_client=BadClient(ANSWERS)) + + +@pytest.mark.asyncio +async def test_async_orders_by_answer_confidence() -> None: + client = AsyncFakeAnswerClient(ANSWERS) + estimator = FakeEstimator({"a": 0.9, "b": 0.3, "c": 0.7, "d": 0.1}) + results = await AsyncAnswerConfidenceRerankStrategy(estimator=estimator).rerank( + query="q", documents=DOCS, model_client=client + ) + assert [r.document.text for r in results] == ["a", "c", "b", "d"] + assert client.calls == ["a", "b", "c", "d"] + + +def test_metadata_carries_strategy_and_algorithm_keys() -> None: + client = FakeAnswerClient(ANSWERS) + results = _strategy().rerank(query="q", documents=DOCS, model_client=client) + + assert results[0].metadata["strategy"] == "answer_confidence" + assert results[0].metadata["algorithm"] == "answer_confidence" + + +def test_blank_document_fast_fails_before_any_answer_call() -> None: + client = FakeAnswerClient(ANSWERS) + documents = [Document(text="a"), Document(text=" ")] + + with pytest.raises(RerankInputError, match="documents\\[1\\]"): + _strategy().rerank(query="q", documents=documents, model_client=client) + assert client.calls == [] + + +def test_rejects_negative_top_k() -> None: + client = FakeAnswerClient(ANSWERS) + with pytest.raises(RerankInputError, match="top_k"): + _strategy().rerank(query="q", documents=DOCS, model_client=client, top_k=-1) + + +@pytest.mark.asyncio +async def test_async_invalid_answer_json_fast_fails() -> None: + from ranksmith.errors import RerankParseError + + class AsyncBadClient(AsyncFakeAnswerClient): + async def answer(self, query: str, context: str) -> str: + return "not json" + + estimator = FakeEstimator({"a": 0.9, "b": 0.3, "c": 0.7, "d": 0.1}) + with pytest.raises(RerankParseError): + await AsyncAnswerConfidenceRerankStrategy(estimator=estimator).rerank( + query="q", documents=DOCS, model_client=AsyncBadClient(ANSWERS) + ) + + +@pytest.mark.asyncio +async def test_async_rejects_client_without_answer() -> None: + class AsyncNoAnswerClient: + async def rank(self, query: str, documents: list[Document]) -> str: + return "{}" + + estimator = FakeEstimator({"a": 0.9}) + with pytest.raises(RerankInputError, match="answer"): + await AsyncAnswerConfidenceRerankStrategy(estimator=estimator).rerank( + query="q", documents=DOCS, model_client=AsyncNoAnswerClient() + ) + + +@pytest.mark.asyncio +async def test_async_blank_document_fast_fails_before_any_answer_call() -> None: + client = AsyncFakeAnswerClient(ANSWERS) + estimator = FakeEstimator({"a": 0.9}) + documents = [Document(text="a"), Document(text="")] + + with pytest.raises(RerankInputError, match="documents\\[1\\]"): + await AsyncAnswerConfidenceRerankStrategy(estimator=estimator).rerank( + query="q", documents=documents, model_client=client + ) + assert client.calls == [] diff --git a/tests/test_examples.py b/tests/test_examples.py index b7dbfa7..770dbad 100644 --- a/tests/test_examples.py +++ b/tests/test_examples.py @@ -78,6 +78,23 @@ def test_setwise_heapsort_example_runs_without_live_provider() -> None: assert "select_calls=" in result.stdout +def test_answer_confidence_example_runs_without_live_provider() -> None: + result = subprocess.run( + [sys.executable, str(ROOT / "examples" / "answer_confidence.py")], + cwd=ROOT, + capture_output=True, + text=True, + timeout=10, + check=False, + ) + + assert result.returncode == 0, result.stderr + assert "Answer Confidence rerank example" in result.stdout + assert "rank=01 id=vitamin_c answer_confidence=0.90" in result.stdout + assert "rank=02 id=vitamin_b12 answer_confidence=0.80" in result.stdout + assert "answer_calls=4" in result.stdout + + def test_selection_prompt_example_matches_top_m() -> None: prompt = _build_selection_prompt( "query", diff --git a/tests/test_merge_benchmark_reports.py b/tests/test_merge_benchmark_reports.py new file mode 100644 index 0000000..298b1e9 --- /dev/null +++ b/tests/test_merge_benchmark_reports.py @@ -0,0 +1,170 @@ +from __future__ import annotations + +import importlib.util +import json +import sys +from pathlib import Path +from typing import Any + +import pytest + +ROOT = Path(__file__).resolve().parents[1] +SPEC = importlib.util.spec_from_file_location( + "merge_benchmark_reports", + ROOT / "scripts" / "merge_benchmark_reports.py", +) +assert SPEC is not None +merger = importlib.util.module_from_spec(SPEC) +assert SPEC.loader is not None +SPEC.loader.exec_module(merger) + + +def _report( + algorithms: list[str], + *, + query_ids: list[str] | None = None, + **overrides: Any, +) -> dict[str, Any]: + query_ids = query_ids if query_ids is not None else ["q1", "q2"] + report: dict[str, Any] = { + "schema_version": 1, + "benchmark_type": "reranking_with_first_stage_candidates", + "dataset": "benchmark-cache", + "dataset_name": "askubuntu-bm25", + "candidate_strategy": "candidate_file", + "candidates": "benchmark-results/pyserini/askubuntu-bm25-top20.trec", + "candidate_count": 20, + "case_count": len(query_ids), + "top_k": 5, + "seed": 13, + "timeout": None, + "checkpoint_output": None, + "algorithm": algorithms, + "aggregate": [ + { + "algorithm": algorithm, + "case_count": len(query_ids), + "metrics": {"ndcg@5": 0.4}, + } + for algorithm in algorithms + ], + "per_query": [ + {"algorithm": algorithm, "query_id": query_id, "valid": True} + for algorithm in algorithms + for query_id in query_ids + ], + "call_estimates": {algorithm: 1 for algorithm in algorithms}, + "method_settings": {algorithm: {} for algorithm in algorithms}, + } + report.update(overrides) + return report + + +def _write_report(path: Path, report: dict[str, Any]) -> Path: + path.write_text(json.dumps(report), encoding="utf-8") + return path + + +def _run_merge(tmp_path: Path, *reports: dict[str, Any]) -> dict[str, Any]: + paths = [ + _write_report(tmp_path / f"report{index}.json", report) + for index, report in enumerate(reports) + ] + output = tmp_path / "merged.json" + original = sys.argv + sys.argv = [ + "merge_benchmark_reports.py", + *[str(path) for path in paths], + "--output", + str(output), + ] + try: + merger.main() + finally: + sys.argv = original + return json.loads(output.read_text(encoding="utf-8")) + + +def test_merge_combines_disjoint_algorithm_runs(tmp_path: Path) -> None: + merged = _run_merge( + tmp_path, + _report(["original_bm25", "single_call_listwise@20"]), + _report(["answer_confidence"], timeout=120.0), + ) + + assert merged["algorithm"] == [ + "original_bm25", + "single_call_listwise@20", + "answer_confidence", + ] + assert [row["algorithm"] for row in merged["aggregate"]] == merged["algorithm"] + assert len(merged["per_query"]) == 6 + assert merged["call_estimates"] == { + "original_bm25": 1, + "single_call_listwise@20": 1, + "answer_confidence": 1, + } + assert merged["merged_from"][0]["divergent_fields"] == {} + assert merged["merged_from"][1]["divergent_fields"] == {"timeout": 120.0} + + +def test_merge_rejects_duplicate_algorithms(tmp_path: Path) -> None: + with pytest.raises(SystemExit, match="more than one input"): + _run_merge( + tmp_path, + _report(["original_bm25"]), + _report(["original_bm25"]), + ) + + +def test_merge_rejects_mismatched_benchmark_identity(tmp_path: Path) -> None: + with pytest.raises(SystemExit, match="disagree on 'candidate_count'"): + _run_merge( + tmp_path, + _report(["original_bm25"]), + _report(["answer_confidence"], candidate_count=10), + ) + + +def test_merge_rejects_different_candidate_files(tmp_path: Path) -> None: + with pytest.raises(SystemExit, match="different candidate files"): + _run_merge( + tmp_path, + _report(["original_bm25"]), + _report(["answer_confidence"], candidates="runs/other-top20.trec"), + ) + + +def test_merge_accepts_same_candidate_file_under_different_paths( + tmp_path: Path, +) -> None: + merged = _run_merge( + tmp_path, + _report(["original_bm25"]), + _report( + ["answer_confidence"], + candidates="/home/elsewhere/askubuntu-bm25-top20.trec", + ), + ) + + assert merged["algorithm"] == ["original_bm25", "answer_confidence"] + + +def test_merge_rejects_runs_on_different_query_ids(tmp_path: Path) -> None: + with pytest.raises(SystemExit, match="different query ids"): + _run_merge( + tmp_path, + _report(["original_bm25"]), + _report( + ["answer_confidence"], + query_ids=["q1", "q3"], + case_count=2, + ), + ) + + +def test_merge_refuses_existing_output_without_overwrite(tmp_path: Path) -> None: + (tmp_path / "merged.json").write_text("{}", encoding="utf-8") + + with pytest.raises(SystemExit, match="Refusing to overwrite"): + _run_merge(tmp_path, _report(["original_bm25"]), _report(["tourrank_r2"])) diff --git a/tests/test_model_architecture.py b/tests/test_model_architecture.py index f8dd41f..39005c3 100644 --- a/tests/test_model_architecture.py +++ b/tests/test_model_architecture.py @@ -407,3 +407,46 @@ async def run() -> None: ) asyncio.run(run()) + + +def test_model_client_answer_builds_domain_prompt_and_emits_usage() -> None: + captured: list[RerankUsage] = [] + usage = RerankUsage(prompt_tokens=3, completion_tokens=2, total_tokens=5) + provider = FakeModelProvider( + [ModelResponse(content='{"answer": "scurvy"}', usage=usage)] + ) + client = ModelClient(provider=provider, on_usage=captured.append) + + result = client.answer("What causes scurvy?", "Vitamin C deficiency.") + + assert result == '{"answer": "scurvy"}' + assert captured == [usage] + request = provider.requests[0] + assert request.messages[0].role == "system" + assert 'Return only JSON with an "answer" string' in request.messages[0].content + assert "Question:\nWhat causes scurvy?" in request.messages[1].content + assert "Context:\nVitamin C deficiency." in request.messages[1].content + assert '{"answer":"..."}' in request.messages[1].content + assert '{"answer":"__NO_ANSWER__"}' in request.messages[1].content + + +def test_model_client_answer_wraps_errors_and_rejects_empty_content() -> None: + client = ModelClient(provider=FailingModelProvider()) + with pytest.raises(RerankProviderError, match="provider timeout"): + client.answer("query", "context") + + empty_client = ModelClient(provider=FakeModelProvider([ModelResponse(content="")])) + with pytest.raises(RerankProviderError, match="empty response"): + empty_client.answer("query", "context") + + +@pytest.mark.asyncio +async def test_async_model_client_answer_matches_sync_prompt() -> None: + sync_provider = FakeModelProvider([ModelResponse(content='{"answer": "x"}')]) + ModelClient(provider=sync_provider).answer("q", "ctx") + async_provider = AsyncFakeModelProvider([ModelResponse(content='{"answer": "x"}')]) + + result = await AsyncModelClient(provider=async_provider).answer("q", "ctx") + + assert result == '{"answer": "x"}' + assert async_provider.requests[0].messages == sync_provider.requests[0].messages diff --git a/tests/test_public_protocols.py b/tests/test_public_protocols.py index 41507c5..ce50a54 100644 --- a/tests/test_public_protocols.py +++ b/tests/test_public_protocols.py @@ -424,3 +424,29 @@ async def test_custom_async_strategy_unexpected_error_is_not_provider_error() -> await reranker.rerank("query", ["first"]) assert "custom async strategy bug" in str(error.value) + + +def test_parse_answer_response_is_publicly_importable_from_root() -> None: + from ranksmith import parse_answer_response # noqa: PLC0415 + + assert parse_answer_response('{"answer": "scurvy"}') == "scurvy" + + +@pytest.mark.parametrize( + "raw_response", + [ + "[]", + "{}", + '{"answer": 3}', + '{"answer": ""}', + '{"answer": " "}', + '{"Answer": "wrong key"}', + ], +) +def test_parse_answer_response_fast_fails_invalid_payloads( + raw_response: str, +) -> None: + from ranksmith import parse_answer_response # noqa: PLC0415 + + with pytest.raises(RerankParseError, match='non-empty "answer" string'): + parse_answer_response(raw_response)