Skip to content

Add bertscore eval#5

Merged
krisztianfekete merged 1 commit intomainfrom
feature/add-bertscore-eval
Mar 26, 2026
Merged

Add bertscore eval#5
krisztianfekete merged 1 commit intomainfrom
feature/add-bertscore-eval

Conversation

@krisztianfekete
Copy link
Copy Markdown
Collaborator

@krisztianfekete krisztianfekete commented Mar 26, 2026

Example config:

evaluators:
  # Built-in metric
  - name: tool_trajectory_avg_score
    type: builtin

  # Custom code evaluators (local scripts)
  - name: bertscore
    type: remote
    source: github
    ref: evaluators/bertscore/bertscore.py
    threshold: 0.7
    timeout: 300
    executor: local
    config:
      expected: "There are two Helm releases installed in the cluster: kagent in namespace kagent (revision 2, deployed, chart kagent-0.7.14) and kagent-crds in namespace kagent (revision 1, deployed, chart kagent-crds-0.7.14)."
      metric: "f1"

Companion PR: agentevals-dev/agentevals#65

Fixes #4

@krisztianfekete krisztianfekete marked this pull request as ready for review March 26, 2026 15:25
@krisztianfekete krisztianfekete merged commit 63e037b into main Mar 26, 2026
1 check passed
@krisztianfekete krisztianfekete deleted the feature/add-bertscore-eval branch March 26, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add a BERTScore evaluator

2 participants