Skip to content

feat: retrival eval add open-eval mode #439

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
e06084 merged 2 commits into from
Jun 16, 2026
Merged

feat: retrival eval add open-eval mode #439

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
81cab1e
feat: retrival eval add open-eval mode
e06084 Jun 16, 2026
e0f95d4
fix comment
e06084 Jun 16, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
15 changes: 15 additions & 0 deletions dingo/config/input_args.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,6 +82,19 @@ class ExecutorResultSaveArgs(BaseModel):
full_field_sample_count: int = 0 # 保留完整字段样本条数，0表示关闭


class OpenEvalArgs(BaseModel):
"""LLM-as-Judge open eval config (Exa-style pointwise grading)."""
enabled: bool = False
model: Optional[str] = None
key: Optional[str] = None
api_url: Optional[str] = None
top_k: int = 5
aggregate: str = "mean"
max_workers: int = 4
prompt_mode: str = "standard"
expected_criteria: Optional[str] = None


class RetrievalArgs(BaseModel):
backend: str = "agentic"
api_url: str = ""
Expand All @@ -103,6 +116,8 @@ class RetrievalArgs(BaseModel):
rate_limit: Optional[float] = None
max_retries: int = 3
max_workers: int = 1
open_eval: Optional[OpenEvalArgs] = None
input_queries: Optional[str] = None


class ExecutorArgs(BaseModel):
Expand Down
Loading
Loading