diff --git a/.claude-plugin/marketplace.json b/.claude-plugin/marketplace.json new file mode 100644 index 0000000..07c1e54 --- /dev/null +++ b/.claude-plugin/marketplace.json @@ -0,0 +1,14 @@ +{ + "name": "ranksmith", + "version": "0.1.0", + "owner": { "name": "ranksmith contributors" }, + "plugins": [ + { + "name": "ranksmith-advisor", + "description": "Guide ranksmith users to the right LLM reranking strategy and generate correct code.", + "version": "0.1.0", + "source": "./", + "category": "development" + } + ] +} diff --git a/.claude-plugin/plugin.json b/.claude-plugin/plugin.json new file mode 100644 index 0000000..e63184f --- /dev/null +++ b/.claude-plugin/plugin.json @@ -0,0 +1,9 @@ +{ + "name": "ranksmith-advisor", + "description": "Guide ranksmith users to the right LLM reranking strategy and generate correct, idiomatic code.", + "version": "0.1.0", + "author": { "name": "ranksmith contributors" }, + "homepage": "https://github.com/pko89403/ranksmith", + "repository": "https://github.com/pko89403/ranksmith", + "license": "MIT" +} diff --git a/.claude/settings.json b/.claude/settings.json new file mode 100644 index 0000000..9b30a90 --- /dev/null +++ b/.claude/settings.json @@ -0,0 +1,14 @@ +{ + "$schema": "https://json.schemastore.org/claude-code-settings.json", + "extraKnownMarketplaces": { + "ranksmith": { + "source": { + "source": "directory", + "path": "." + } + } + }, + "enabledPlugins": { + "ranksmith-advisor@ranksmith": true + } +} diff --git a/README.ko.md b/README.ko.md index a3dd07e..96b9312 100644 --- a/README.ko.md +++ b/README.ko.md @@ -358,6 +358,26 @@ print(result.export_path) - [acurank.py](https://github.com/pko89403/ranksmith/blob/main/examples/acurank.py): first-stage score prior 기반 AcuRank - [custom_strategy.py](https://github.com/pko89403/ranksmith/blob/main/examples/custom_strategy.py): custom Strategy 계약 +## Claude Code Advisor + +이 레포에는 [Claude Code](https://code.claude.com/docs) 플러그인 +`ranksmith-advisor`가 포함되어 있습니다. 사용 사례에 맞는 reranking strategy +선택을 돕고 CI로 검증된 동작 스니펫을 제시하며, 제안 코드가 라이브러리의 실제 +계약을 따르도록 ranksmith 전용 guardrail을 적용합니다(미구현 provider 호출, +`algorithm` 문자열 확장, `confidence`를 reranker로 취급하는 패턴을 차단). + +Claude Code에서 사용: + +```bash +/plugin marketplace add pko89403/ranksmith +/plugin install ranksmith-advisor@ranksmith +``` + +레포 기여자는 자동으로 활성화됩니다. 프로젝트 공유 `.claude/settings.json`이 +로컬 marketplace를 등록하고 플러그인을 enable 하므로 수동 설치가 필요 없습니다. +플러그인 본문은 `skills/ranksmith-advisor/`에 있으며 PyPI 배포물에서는 +제외됩니다. + ## 벤치마크 아래 benchmark는 reranking만 측정합니다. Pyserini BM25는 고정된 first-stage diff --git a/README.md b/README.md index 231c54e..a338d92 100644 --- a/README.md +++ b/README.md @@ -363,6 +363,27 @@ Runnable examples live in the `examples/` directory. - [acurank.py](https://github.com/pko89403/ranksmith/blob/main/examples/acurank.py): AcuRank with first-stage score priors - [custom_strategy.py](https://github.com/pko89403/ranksmith/blob/main/examples/custom_strategy.py): custom strategy contracts +## Claude Code Advisor + +This repo ships a [Claude Code](https://code.claude.com/docs) plugin, +`ranksmith-advisor`, that helps you choose a reranking strategy for your use +case and returns working, CI-verified snippets. It encodes ranksmith-specific +guardrails, so the suggested code follows the library's real contracts (no +calls into unimplemented providers, no `algorithm` string hacks, no treating +`confidence` as a reranker). + +Use it from Claude Code: + +```bash +/plugin marketplace add pko89403/ranksmith +/plugin install ranksmith-advisor@ranksmith +``` + +Repo contributors get it automatically: the project-shared +`.claude/settings.json` registers the local marketplace and enables the plugin, +so no manual install is needed. The plugin content lives under +`skills/ranksmith-advisor/` and is excluded from the PyPI distribution. + ## Benchmarking The benchmark below measures reranking only. Pyserini BM25 provides the fixed diff --git a/docs/specs/spec_ranksmith_advisor_plugin.md b/docs/specs/spec_ranksmith_advisor_plugin.md new file mode 100644 index 0000000..164a0c0 --- /dev/null +++ b/docs/specs/spec_ranksmith_advisor_plugin.md @@ -0,0 +1,180 @@ +# Spec: ranksmith Advisor (Claude Code 플러그인/스킬) + +> **작성 가이드**: 이 문서는 코딩 어시스턴트의 작업 추적용이기도 하지만, **최우선적으로 사람(개발자)이 읽고 이해하기 가장 좋은 형태(가독성)**여야 합니다. + +## 1. 개요 (Overview) +- **작업 목적**: 코딩 어시스턴트(Claude Code)가 `ranksmith` 사용자에게 ① **적절한 리랭킹 메소드 선택 가이드**와 ② **동작하는 코드 스니펫**을 정확하게 제공하도록, ranksmith 도메인 지식을 Claude Code 스킬/플러그인으로 레포에 내장한다. +- **핵심 가치**: 일반 LLM이 이 라이브러리에서 *반복적으로 틀리는* 안티패턴(미구현 provider 호출, `algorithm` 문자열 확장, confidence를 리랭커로 취급 등)을 **가드레일**로 차단한다. 추천 로직은 추론이 아니라 README 표 + 커밋된 벤치마크 근거로 인코딩한다. +- **Reference**: + - Claude Code Skills: https://code.claude.com/docs/en/skills.md + - Claude Code Plugins: https://code.claude.com/docs/en/plugins.md + - Plugins Reference(매니페스트/검증): https://code.claude.com/docs/en/plugins-reference.md + - 근거 문서(레포 내부): `README.md`(Recommended Use Cases·Benchmark), `docs/wiki/00_context.md`, `docs/wiki/02_architecture.md`, `docs/wiki/08_custom_strategy_extension.md`, `AGENTS.md` +- **상태**: `[ ] Draft` | `[ ] In Progress` | `[x] Completed` + - 구현·패키징·자동 검증(로컬 verify.sh 462 passed, CI test 3.10–3.12 green) 완료. advisor 스킬 in-session 스모크 테스트도 통과(추천 + 스니펫 + 가드레일). + +## 2. 요구 사항 및 제약 (Requirements & Constraints) +- **입력 (Inputs)**: 사용자의 자연어 제약조건 — 후보 수, 비용/지연 예산, 품질 목표, 동기/비동기, top_k, first-stage score 보유 여부, 커스텀 로직 필요 여부. +- **출력 (Outputs)**: ① 전략+파라미터 추천(근거·콜 비용 포함), ② 검증된 `examples/`에 기반한 코드 스니펫, ③ 가드레일 준수. +- **스코프 (v1)**: **리랭킹 전략만**. confidence(`ranksmith.confidence` 등)는 "리랭킹 Strategy가 아님 + 별도 유틸"이라는 **짧은 포인터**로만 다룬다. confidence 사용/학습 가이드는 v2 후보. +- **배포 (단계적)**: + - **1단계 — 프로젝트 로컬**: `.claude/skills/ranksmith-advisor/`. 이 레포에서 작업하는 기여자에게 설치 0으로 즉시 동작. + - **2단계 — 배포형 플러그인**: 동일 스킬을 `.claude-plugin/plugin.json` + 루트 `skills/`로 노출하고 `.claude-plugin/marketplace.json`을 추가해, pip로 ranksmith를 설치한 외부 개발자가 자기 프로젝트에 설치 가능하게 한다. +- **제약 사항 (Constraints)**: + - **드리프트 금지**: 스니펫 코드를 복제하지 않고, CI에서 실행·검증되는 `examples/*.py`(`tests/test_examples.py`)를 source of truth로 가리킨다. + - **증거 정책**: AGENTS.md "README/Benchmark Evidence Policy"를 그대로 반영한다. 실행하지 않은 method의 metric을 만들지 않고, 콜 수는 estimate로 명시한다. + - **fast-fail 철학 노출**: 어시스턴트가 조용한 truncation/ranking 보정 코드를 생성하지 않도록 명시한다. + - **작은 표면적**: v1은 단일 스킬 `ranksmith-advisor` + 보조 `.md` 3개. 선택기/생성기 분리는 하지 않는다. + - **패키지 산출물 비포함**: 본 작업은 `src/ranksmith/` public API를 변경하지 않는다. `pyproject.toml`의 wheel `exclude`에 영향이 없도록 레포 루트 도구 디렉터리에만 추가한다(빌드 산출물에 포함되지 않음). + +## 3. 상세 설계 (Architecture & Design) + +### 파일 구조 +```text +# 단일 소스: 플러그인 루트 = 레포 루트 +skills/ranksmith-advisor/ # 스킬 본문(유일 사본) + SKILL.md # 라우팅 + 가드레일 요약 (상시 로드) + method-guide.md # 전략 표 · 콜 비용 모델 · 벤치마크 근거 · 파라미터 + snippets.md # 시나리오 → examples/ 매핑 + 최소 스니펫 + guardrails.md # 하드룰(안티패턴) +.claude-plugin/ + plugin.json # 플러그인 매니페스트 (plugin root = repo root) + marketplace.json # 자체 marketplace (plugin source "./") +.claude/settings.json # 기여자 자동 등록 (extraKnownMarketplaces + enabledPlugins) +tests/test_advisor_references.py # 자동 참조 무결성 테스트 +pyproject.toml # exclude += /.claude /.claude-plugin /skills (sdist 격리) +``` +> **단일 소스 확정**: 스킬 본문은 `skills/ranksmith-advisor/` 한 곳에만 둔다. 레포 전체가 곧 플러그인(plugin root = repo root)이며, 자체 `marketplace.json`이 `source: "./"`로 이 플러그인을 가리킨다. 외부 사용자: `/plugin marketplace add pko89403/ranksmith` → `/plugin install ranksmith-advisor@ranksmith`. 기여자는 `.claude/settings.json`의 `extraKnownMarketplaces`(local `source: directory`, `path "."`) + `enabledPlugins`(`ranksmith-advisor@ranksmith`)로 자동 활성화된다. + +### SKILL.md frontmatter (초안) +```yaml +--- +name: ranksmith-advisor +description: > + Guide ranksmith users to the right reranking strategy and generate correct, + idiomatic code. Use when the user asks how to rerank with ranksmith, which + strategy to pick (RankGPT/listwise, PRP/pairwise, Setwise, TourRank, AcuRank), + how to configure window_size/passes/rounds/target_rank, async usage, custom + strategies, or how to wire AzureOpenAIReranker / ModelClient / ModelProvider. +--- +``` +- `description`이 트리거 품질을 좌우하므로 핵심 키워드(ranksmith, rerank, 전략명, 파라미터명, AzureOpenAIReranker)를 앞에 배치한다. +- `/ranksmith-advisor`로 사용자 직접 호출도 가능하게 둔다(별도 `disable-model-invocation` 미설정). + +### method-guide.md — 결정 절차 (근거 기반) +README "Recommended Use Cases" + Benchmark 표를 절차로 인코딩한다. + +```text +비용 우선 / 기본값 → ListwiseStrategy("rankgpt_sliding_window") window>=N 이면 single-call +품질 우선 / 예산 보통 → TourRankStrategy(rounds=2) ★벤치 NDCG@5·Recall@5 최고 +오프라인 최고 품질 → TourRankStrategy(rounds=10) +MRR 중시 / PRP 재현 → PairwiseStrategy ★벤치 MRR@5 최고 (단 호출 多) +적응형 / top-k 경계 집중 → AcuRankStrategy(target_rank=k) TrueSkill, metadata["score"] 프라이어 +setwise top-k 추출 → SetwiseStrategy(set_size) top-k 조기종료 유일 (벤치 품질 최저) +결정적 비즈니스 로직 → 커스텀 Strategy 클래스 +고처리량(FastAPI) → Async* 변형 +``` + +콜 비용(AskUbuntu BM25 top-20, **estimate**, README 근거): + +| Method | NDCG@5 | MRR@5 | Recall@5 | Nominal calls/query | +| --- | ---: | ---: | ---: | ---: | +| original_bm25 | 0.3520 | 0.5062 | 0.2862 | 0 | +| single_call_listwise@20 | 0.4082 | 0.5541 | 0.3345 | 1 | +| rankgpt_sw_w5 | 0.3973 | 0.5283 | 0.3366 | 9 | +| acurank_k5_b1 | 0.4053 | 0.5491 | 0.3377 | 2 | +| tourrank_r2 | 0.4236 | 0.5725 | 0.3601 | 8 | +| setwise_hs_s10 | 0.3653 | 0.5059 | 0.3005 | 12 | +| prp_sliding_p1 | 0.4065 | 0.5818 | 0.3277 | 38 | + ++ 파라미터 기본값/제약: `window_size=20`, `stride=10`, `max_document_chars=4000`, `passes=10`, `set_size>=3`, `rounds=2`, `target_rank=10`, `tolerance(0 본 작업은 reranking 알고리즘이 아니므로 TEMPLATE의 "공통 Reranking Smoke/Benchmark"는 비적용. 문서/매니페스트 무결성 중심으로 검증한다. + +- **성공 케이스**: + - `skills/ranksmith-advisor/SKILL.md`가 유효한 frontmatter로 로드된다. + - (2단계) `plugin.json`/`marketplace.json`이 스키마상 유효하다(가능 시 `claude plugin validate` 수준 확인). + - snippets.md가 가리키는 모든 `examples/*.py` 경로가 실제 존재한다(참조 무결성). + - method-guide.md의 수치가 README 표와 일치한다. +- **엣지/실패 케이스**: + - 존재하지 않는 전략/파라미터를 추천하지 않는지(가드레일과 충돌 없음) 수기 검토. + - guardrails.md 항목이 현재 코드(`_stubs.py`, `parsing.py`, `errors.py`, 전략 default)와 사실 일치하는지 대조. +- **검증 명령**: `./scripts/verify.sh`(기존 라이브러리 회귀 없음 확인) + 참조 무결성/일치 체크(스크립트화 여부는 구현 시 결정). + +--- + +## 7. 작업 태스크 추적 (Task Checklist) +> **코딩 어시스턴트 필수 지침**: 완료 작업은 `[x]`로 표시하고 필요 시 하위 태스크를 추가한다. + +### Phase 1: 컨텍스트 및 설계 확인 +- [x] 기존 코드베이스·Wiki·README·examples·AGENTS.md 확인 +- [x] Claude Code Skills/Plugins 작성 규격 확인 +- [x] 본 스펙(배포 단계·스코프·구조) 사용자 검토 및 최종 승인 + +### Phase 2: 구현 — 스킬 본문 (단일 소스) +- [x] `skills/ranksmith-advisor/SKILL.md` (라우팅 + 가드레일 요약) +- [x] `skills/ranksmith-advisor/method-guide.md` (결정 절차·콜 비용·파라미터) +- [x] `skills/ranksmith-advisor/snippets.md` (examples 매핑 + 최소 스니펫) +- [x] `skills/ranksmith-advisor/guardrails.md` (하드룰) + +### Phase 3: 구현 — 배포 패키징 +- [x] `.claude-plugin/plugin.json` (plugin root = repo root) +- [x] `.claude-plugin/marketplace.json` (source `"./"`) +- [x] `pyproject.toml` exclude += `/.claude` `/.claude-plugin` `/skills` (sdist 격리) +- [x] `.claude/settings.json` 기여자 자동 등록(`extraKnownMarketplaces` + `enabledPlugins`) + +### Phase 4: 검증 (Verification) +- [x] `tests/test_advisor_references.py`: snippets ↔ `examples/*.py` 참조 무결성 + public symbol 검증 +- [x] guardrails.md ↔ 현재 코드(`_stubs.py`/`parsing.py`/`errors.py`/전략 default) 사실 대조 +- [x] `./scripts/verify.sh` 통과(462 passed, ruff/mypy clean, build OK) +- [x] 실제 Claude Code 세션에서 추천/스니펫 트리거 + 매니페스트 로드 검증 — 이 세션에서 advisor 스킬 호출로 확인: 플러그인 로드 + 참조파일 해석 + 근거 기반 추천(TourRank-r2) + 가드레일 준수 스니펫. (자연어 설명 기반 자동 라우팅은 Claude Code 런타임 동작; 보강 근거 CI 3.10–3.12 green + 매니페스트 3종 JSON 유효) + +### Phase 5: 완료 및 정리 +- [x] `README.md`/`README.ko.md`에 "Claude Code Advisor" 섹션 추가(ko/en 동기화) +- [x] 본 문서 최상단 **상태**를 `Completed`로 변경 diff --git a/pyproject.toml b/pyproject.toml index 8b68024..80cd1a6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -75,6 +75,9 @@ exclude = [ "/.pytest_cache", "/.ruff_cache", "/.githooks", + "/.claude", + "/.claude-plugin", + "/skills", "/AGENTS.md", "/docs", "/dist", diff --git a/skills/ranksmith-advisor/SKILL.md b/skills/ranksmith-advisor/SKILL.md new file mode 100644 index 0000000..6de9647 --- /dev/null +++ b/skills/ranksmith-advisor/SKILL.md @@ -0,0 +1,44 @@ +--- +name: ranksmith-advisor +description: >- + Advise users of the ranksmith Python library on LLM reranking. Use whenever + the user asks how to rerank with ranksmith, which strategy or parameters to + pick (RankGPT/listwise, PRP/pairwise, Setwise heapsort, TourRank, AcuRank), + how to tune window_size/passes/rounds/target_rank, sync vs async, writing a + custom strategy, or wiring AzureOpenAIReranker / ModelClient / ModelProvider. +--- + +# ranksmith Advisor + +Help users of the `ranksmith` library (training-free LLM reranking over closed +APIs) in two ways: **recommend a method** and **generate correct code**. Keeping +ranksmith's contracts intact matters — those are the most common failures. + +## How to use this skill + +1. **"Which strategy / parameters should I use?"** → read `method-guide.md`, ask + for the missing constraints (candidate count, cost/latency budget, quality + target, sync vs async, `top_k`, first-stage scores), then recommend with the + call-cost shape and the committed evidence. +2. **"Write or fix ranksmith code"** → read `snippets.md`, start from the + matching tested example, adapt it, and keep every guardrail. +3. **Always** apply `guardrails.md`. These are hard rules, not suggestions. + +## Non-negotiable guardrails (full list with sources in `guardrails.md`) + +- Only the **Azure** provider actually runs. `OpenAIProvider`, + `AnthropicProvider`, and `GeminiProvider` are reserved stubs that raise + `RerankProviderError`. Never build working code on them. +- A new ranking method is a **new Strategy class**, never a new + `ListwiseStrategy(algorithm="...")` string. +- `rank` is 1-based; `original_index` is 0-based against the input list. +- ranksmith **fails fast**: never silently truncate documents or repair an + invalid ranking. +- `ranksmith.confidence` is **not** a reranking strategy — it is a separate + optional utility. + +## Evidence rule + +Quote benchmark numbers and call counts only from `method-guide.md` (sourced +from the committed README and benchmark artifacts), and label call counts as +estimates. Never invent metrics for a method that was not measured. diff --git a/skills/ranksmith-advisor/guardrails.md b/skills/ranksmith-advisor/guardrails.md new file mode 100644 index 0000000..311fa80 --- /dev/null +++ b/skills/ranksmith-advisor/guardrails.md @@ -0,0 +1,51 @@ +# Guardrails (hard rules) + +Apply these to every recommendation and every line of code you generate. They +encode ranksmith's contracts. The canonical statements live in the repository; +this file points to them instead of re-deriving them, so fix the sources, not a +second copy. + +1. **Azure is the only working provider.** `OpenAIProvider`, + `AnthropicProvider`, `GeminiProvider` (and their `Async*` forms) are reserved + stubs that raise `RerankProviderError`. Generate Azure-backed code + (`AzureOpenAIReranker`, `AzureAOAIProvider`) or a user-supplied `ModelClient`. + Source: `src/ranksmith/providers/_stubs.py`, README "Model Provider Architecture". + +2. **A new ranking method is a new Strategy class**, never a new + `ListwiseStrategy(algorithm="...")` string. + Source: `docs/wiki/08_custom_strategy_extension.md`, `docs/wiki/01_decisions.md` (D006). + +3. **Indexing contract.** `rank` is 1-based (display); `original_index` is + 0-based against the input list, even after reordering. + Source: `docs/wiki/00_context.md` "현재 기본값", README "Result Model". + +4. **Custom strategy contract.** Keyword-only + `rerank(*, query, documents, model_client, top_k=None) -> list[RerankResult]`. + Validate model JSON with `parse_ranking_response()` / + `parse_selection_response()`. Classify provider failures inside the strategy + as `RerankProviderError`. + Source: `docs/wiki/08_custom_strategy_extension.md`, `examples/custom_strategy.py`. + +5. **Fail fast.** Never silently truncate a long document (raise / expect + `DocumentTooLongError`) or repair an invalid ranking. JSON `true` / `false` + are not integers in a ranking. + Source: `src/ranksmith/parsing.py`, `src/ranksmith/errors.py`, AGENTS.md "프로젝트 원칙". + +6. **Confidence is not reranking.** `ranksmith.confidence`, + `ranksmith.confidence_generation`, and `ranksmith.confidence_training` are + optional utilities (`pip install "ranksmith[confidence]"`), not a Strategy or + Algorithm. Do not present them as a reranker. + Source: `docs/wiki/02_architecture.md` "Confidence". + +7. **Evidence only.** Quote metrics and call counts from the committed README / + benchmark artifacts, label call counts as estimates, and never fabricate + numbers for an unmeasured method. + Source: AGENTS.md "README / Benchmark Evidence Policy". + +8. **`top_k` early stopping is Setwise-only.** Other strategies slice to `top_k` + after computing the full order. + Source: `src/ranksmith/strategies/_setwise.py`. + +9. **Public API only.** Use root `ranksmith` exports in user code; do not reach + into private modules (`ranksmith._providers`, `ranksmith.strategies._*`, …). + Source: `docs/wiki/02_architecture.md`. diff --git a/skills/ranksmith-advisor/method-guide.md b/skills/ranksmith-advisor/method-guide.md new file mode 100644 index 0000000..a59f943 --- /dev/null +++ b/skills/ranksmith-advisor/method-guide.md @@ -0,0 +1,83 @@ +# Method selection guide + +Source of truth: the repository `README.md` ("Supported Strategies & Algorithms" +and "Benchmarking") and `docs/wiki/00_context.md` ("현재 기본값"). Keep this file +in sync with them; do not invent numbers. + +## Quick decision + +| If the user wants… | Recommend | Why | +| --- | --- | --- | +| The default, lowest-friction reranker | `ListwiseStrategy("rankgpt_sliding_window")` | Fewest calls. With `window_size >= N` it becomes one-shot listwise. | +| Best quality on a moderate budget | `TourRankStrategy(rounds=2)` | Best NDCG@5 and Recall@5 in the committed benchmark. | +| Quality-focused offline / paper-style | `TourRankStrategy(rounds=10)` | Highest built-in call cost; strongest listwise-style quality. | +| Pairwise preference / reproduce PRP / best MRR | `PairwiseStrategy(...)` | Best MRR@5 in the benchmark, but many calls. | +| Adaptive spend near the top-k boundary | `AcuRankStrategy(target_rank=k)` | TrueSkill-based; uses `metadata["score"]` as a first-stage prior when present. | +| Setwise top-k extraction with early stopping | `SetwiseStrategy(set_size=...)` | The only strategy with true `top_k` early stopping; lowest quality in the benchmark. | +| Deterministic business logic / a new research method | A custom Strategy class | You own the ranking contract. | +| High-throughput (FastAPI, etc.) | The `Async*` variant of any of the above | Same contracts, `await`ed. | + +## Committed benchmark evidence + +AskUbuntuDupQuestions test set, BM25 top-20 candidates, `@5`, 361 queries, +Azure deployment `gpt-5.4-nano`. Call counts are **nominal estimates**, not +exact provider-call telemetry. + +| Method | NDCG@5 | MRR@5 | Recall@5 | Nominal LLM calls/query | +| --- | ---: | ---: | ---: | ---: | +| original_bm25 (no rerank) | 0.3520 | 0.5062 | 0.2862 | 0 | +| single_call_listwise@20 | 0.4082 | 0.5541 | 0.3345 | 1 | +| rankgpt_sw_w5 | 0.3973 | 0.5283 | 0.3366 | 9 | +| acurank_k5_b1 | 0.4053 | 0.5491 | 0.3377 | 2 | +| tourrank_r2 | 0.4236 | 0.5725 | 0.3601 | 8 | +| setwise_hs_s10 | 0.3653 | 0.5059 | 0.3005 | 12 | +| prp_sliding_p1 | 0.4065 | 0.5818 | 0.3277 | 38 | + +`tourrank_r2` led NDCG@5 and Recall@5; `prp_sliding_p1` led MRR@5. + +## Parameters & defaults + +### ListwiseStrategy / AsyncListwiseStrategy — RankGPT +- `algorithm="rankgpt_sliding_window"` (only value), `window_size=20`, + `stride=10`, `max_document_chars=4000`. +- Cost shape: roughly the number of windows; `window_size >= N` → 1 call. +- Model op: `rank()` (a full listwise permutation). + +### PairwiseStrategy / AsyncPairwiseStrategy — PRP +- `algorithm="prp_sliding_k"`, `passes=10` (expensive), `max_document_chars=4000`. +- Async adds `pair_order_parallelism` (default 2) to compare both orders concurrently. +- Cost shape: about `passes * (N-1) * 2` compare calls; lower `passes` to cut cost. +- Model op: `compare()` (A/B winner); each adjacent pair is compared in both directions. + +### SetwiseStrategy / AsyncSetwiseStrategy — Setwise heapsort +- `algorithm="setwise_heapsort"`, `set_size=3` (minimum 3), `max_document_chars=4000`. +- The only strategy that truly early-stops on `top_k` (extracts just the needed top). +- Larger `set_size` → fewer calls but a harder selection prompt. +- Model op: `select()` (best of a set). + +### TourRankStrategy / AsyncTourRankStrategy — TourRank-r +- `algorithm="tourrank_r"`, `rounds=2`, `shuffle_seed=13`, configurable + `stage_configs` (`TourRankStageConfig`), `max_document_chars=4000`. +- Deterministic seeded shuffling. Sync `group_parallelism=1`; async allows a + semaphore-limited or unbounded value. +- Model op: `select()` per group; scores accumulate across rounds. + +### AcuRankStrategy / AsyncAcuRankStrategy — AcuRank +- `algorithm="acurank"`, `target_rank=10`, `window_size=20`, `tolerance=0.01` + (must satisfy `0 < tolerance < 0.5`), `uncertain_threshold=10`, + `initial_pass=True`, `score_metadata_key="score"`, `batch_parallelism=1`, + `max_adaptive_reranker_calls=None`. +- Uses TrueSkill ratings; reranks only uncertain candidates near the top-k + boundary until convergence or budget. Align `target_rank` with your eval cutoff. +- First-stage prior: if every `Document` has a numeric `metadata["score"]`, it + seeds the prior. All-or-none — partial or boolean scores fail fast. +- Model op: `rank()` over batches. + +## top_k behavior +Only `SetwiseStrategy` stops early at `top_k`. Every other strategy computes the +full order and slices to `top_k` at the end. `top_k` must be `>= 0`. + +## Sync vs async +Use `AzureOpenAIReranker` for scripts and batch jobs; `AsyncAzureOpenAIReranker` +for async servers. The strategy class must match the reranker (`TourRankStrategy` +with the sync reranker, `AsyncTourRankStrategy` with the async one, etc.). diff --git a/skills/ranksmith-advisor/snippets.md b/skills/ranksmith-advisor/snippets.md new file mode 100644 index 0000000..2f43f86 --- /dev/null +++ b/skills/ranksmith-advisor/snippets.md @@ -0,0 +1,164 @@ +# Code snippets + +Two sources, by audience: + +- **In-repo / GitHub readers:** the full, runnable, CI-tested programs in + `examples/` are the canonical reference (executed by `tests/test_examples.py`). + Prefer adapting these. +- **Downstream users (plugin installed elsewhere):** the minimal starters below + are copy-paste ready. They use only public `ranksmith` exports, which + `tests/test_advisor_references.py` verifies. + +## Scenario → canonical example + +| Scenario | Example (tested) | +| --- | --- | +| Sync listwise (RankGPT) | `examples/rankgpt_sync.py` | +| Async listwise (servers) | `examples/rankgpt_async.py` | +| Pairwise PRP | `examples/pairwise_prp.py` | +| Setwise heapsort | `examples/setwise_heapsort.py` | +| TourRank-r | `examples/tourrank.py` | +| AcuRank + first-stage scores | `examples/acurank.py` | +| Custom strategy contracts | `examples/custom_strategy.py` | + +GitHub: https://github.com/pko89403/ranksmith/tree/main/examples + +## Minimal starters + +### Sync quick start +```python +from ranksmith import AzureOpenAIReranker, Document + +reranker = AzureOpenAIReranker( + api_key="...", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="gpt-4o-mini", +) +results = reranker.rerank( + query="What is listwise reranking?", + documents=[ + Document(id="a", text="Listwise reranking compares candidates together."), + Document(id="b", text="Vector search retrieves candidate documents."), + ], + top_k=2, +) +for result in results: + print(result.rank, result.original_index, result.document.id) +``` + +### Swap the strategy +```python +from ranksmith import AzureOpenAIReranker, TourRankStrategy + +reranker = AzureOpenAIReranker( + api_key="...", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="gpt-4o-mini", + strategy=TourRankStrategy(rounds=2), +) +``` +Substitute `ListwiseStrategy`, `PairwiseStrategy`, `SetwiseStrategy`, or +`AcuRankStrategy`; see `method-guide.md` for parameters. + +### Async +```python +from ranksmith import AsyncAzureOpenAIReranker + +reranker = AsyncAzureOpenAIReranker( + api_key="...", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="gpt-4o-mini", +) +results = await reranker.rerank("query", documents) +``` +Pair async rerankers with `Async*` strategies only. + +### AcuRank with first-stage scores +```python +from ranksmith import AcuRankStrategy, AzureOpenAIReranker, Document + +documents = [ + Document(id="a", text="...", metadata={"score": 12.5}), + Document(id="b", text="...", metadata={"score": 9.1}), +] +reranker = AzureOpenAIReranker( + api_key="...", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="gpt-4o-mini", + strategy=AcuRankStrategy(target_rank=10, window_size=20), +) +``` +Either every document has a numeric `metadata["score"]` or none do. + +### Custom strategy skeleton +```python +from collections.abc import Sequence + +from ranksmith import Document, RerankResult + + +class MyStrategy: + def rerank( + self, + *, + query: str, + documents: Sequence[Document], + model_client: object, + top_k: int | None = None, + ) -> list[RerankResult]: + del query, model_client + ordered = sorted( + range(len(documents)), + key=lambda i: len(documents[i].text), + reverse=True, + ) + results = [ + RerankResult( + document=documents[i], + rank=rank, + original_index=i, + metadata={"strategy": "my"}, + ) + for rank, i in enumerate(ordered, start=1) + ] + return results if top_k is None else results[:top_k] +``` +Model-backed strategies must validate JSON with `parse_ranking_response()` and +classify provider errors as `RerankProviderError`; see `examples/custom_strategy.py`. + +### Error handling +```python +from ranksmith import ( + DocumentTooLongError, + RerankParseError, + RerankProviderError, + RerankStrategyError, +) + +try: + results = reranker.rerank("query", documents) +except DocumentTooLongError: + ... +except RerankParseError: + ... +except RerankProviderError: + ... +except RerankStrategyError: + ... +``` + +### Reuse one ModelClient across strategies +```python +from ranksmith import AzureAOAIProvider, AzureOpenAIReranker, ModelClient, PairwiseStrategy + +provider = AzureAOAIProvider( + api_key="...", + azure_endpoint="https://example.openai.azure.com", + azure_deployment="gpt-4o-mini", + api_version="2024-08-01-preview", +) +reranker = AzureOpenAIReranker( + model_client=ModelClient(provider=provider), + strategy=PairwiseStrategy(passes=3), +) +``` diff --git a/tests/test_advisor_references.py b/tests/test_advisor_references.py new file mode 100644 index 0000000..96b8d2d --- /dev/null +++ b/tests/test_advisor_references.py @@ -0,0 +1,114 @@ +from __future__ import annotations + +import inspect +import re +from pathlib import Path + +import ranksmith + +ROOT = Path(__file__).resolve().parents[1] +SKILL_DIR = ROOT / "skills" / "ranksmith-advisor" + +_EXAMPLE_RE = re.compile(r"examples/[\w./-]+\.py") +_IMPORT_RE = re.compile(r"from ranksmith import\s+(?:\(([^)]*)\)|([^\n]+))") + + +def _skill_markdown_files() -> list[Path]: + files = sorted(SKILL_DIR.glob("*.md")) + assert files, f"no skill markdown found in {SKILL_DIR}" + return files + + +def _imported_names(body: str) -> list[str]: + # Strip inline comments per line before splitting on commas, so a symbol + # after an inline comment in a multi-line import is still validated. + no_comments = "\n".join(line.split("#")[0] for line in body.splitlines()) + names: list[str] = [] + for raw in no_comments.split(","): + name = raw.strip().split(" as ")[0].strip() + if name: + names.append(name) + return names + + +def test_referenced_examples_exist() -> None: + missing: list[str] = [] + for path in _skill_markdown_files(): + text = path.read_text(encoding="utf-8") + for rel in sorted(set(_EXAMPLE_RE.findall(text))): + if not (ROOT / rel).is_file(): + missing.append(f"{path.name}: {rel}") + assert not missing, f"referenced example files do not exist: {missing}" + + +def test_snippets_use_public_ranksmith_symbols() -> None: + public = set(ranksmith.__all__) + unknown: list[str] = [] + for path in _skill_markdown_files(): + text = path.read_text(encoding="utf-8") + for paren_body, line_body in _IMPORT_RE.findall(text): + for name in _imported_names(paren_body or line_body): + if name not in public: + unknown.append(f"{path.name}: {name}") + assert not unknown, f"snippets import non-public ranksmith symbols: {unknown}" + + +# Defaults the advisor docs (method-guide.md) tell the model to rely on. If a +# default changes in src/ranksmith, this guard fails so the advisor docs are +# updated in lock-step instead of silently drifting. +_DOCUMENTED_DEFAULTS: dict[str, dict[str, object]] = { + "ListwiseStrategy": { + "algorithm": "rankgpt_sliding_window", + "window_size": 20, + "stride": 10, + "max_document_chars": 4000, + }, + "PairwiseStrategy": { + "algorithm": "prp_sliding_k", + "passes": 10, + "max_document_chars": 4000, + }, + "SetwiseStrategy": { + "algorithm": "setwise_heapsort", + "set_size": 3, + "max_document_chars": 4000, + }, + "TourRankStrategy": { + "algorithm": "tourrank_r", + "rounds": 2, + "shuffle_seed": 13, + "group_parallelism": 1, + "max_document_chars": 4000, + }, + "AcuRankStrategy": { + "algorithm": "acurank", + "target_rank": 10, + "window_size": 20, + "tolerance": 0.01, + "uncertain_threshold": 10, + "initial_pass": True, + "score_metadata_key": "score", + "batch_parallelism": 1, + "max_adaptive_reranker_calls": None, + }, +} + + +def test_documented_strategy_defaults_match_source() -> None: + mismatches: list[str] = [] + for cls_name, expected in _DOCUMENTED_DEFAULTS.items(): + cls = getattr(ranksmith, cls_name) + params = inspect.signature(cls.__init__).parameters + for param, want in expected.items(): + if param not in params: + mismatches.append(f"{cls_name}.{param}: param missing from source") + continue + got = params[param].default + if type(got) is not type(want) or got != want: + mismatches.append( + f"{cls_name}.{param}: source default {got!r} != documented {want!r}" + ) + assert not mismatches, ( + "method-guide.md documents strategy defaults that no longer match source; " + f"update the advisor docs and this guard together: {mismatches}" + )