Skip to content
BranchesTags

Repository files navigation

Coding Agent Bench

Reproducible benchmarks for coding agents and models using Harbor

Features

  • Complete instructions for running popular benchmarks with open models
  • CLI utility to simplify benchmark runs against self-hosted models
  • Deployable queue service for scheduling benchmark runs in OpenShift
  • Leaderboards for popular benchmarks with instructions for reproducing results
  • Full manifests for deploying open models on OpenShift with vLLM

Table of Contents

Leaderboards

✨ Check out our Coding Agent Leaderboard on HuggingFace ✨

SWE-Bench Verified (pass@1, N=500)

Model Harness Score Cost
Opus 4.8 Claude Code 86.8% $395
Opus 4.8 OpenCode 83.4% $320
GPT 5.5 Codex 79.8% $443
Sonnet 4.6 Claude Code 79.6% N/A
RedHatAI/Qwen3.6-35B-A3B-NVFP4 Pi 65.0% $51†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 Qwen Code 63.8% $37†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 Claude Code 63.2% $48†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 OpenClaw 58.8% $33†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 OpenCode 54.8% $67†

SWE-Bench Pro - Ansible Tasks (pass@1, N=96)

Model Harness Score Cost
Opus 4.8 OpenCode 78.1% $151
Opus 4.8 Claude Code 69.8% $186
GPT 5.5 Codex 60.4% $188
GPT 5.5 OpenCode 57.3% $111
Opus 4.6 Claude Code 51.0% $172
Sonnet 4.6 Claude Code 50.0% $184
RedHatAI/Qwen3.6-35B-A3B-NVFP4 Pi 47.9% $13†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 Claude Code 45.6% $10†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 Qwen Code 43.8% $9†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 OpenClaw 40.6% $9†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 OpenCode 37.5% $11†

Terminal Bench 2.0 (pass@1, N=87)

Model Harness Score Cost
RedHatAI/Qwen3.6-35B-A3B-NVFP4 Pi 36.0% $11†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 OpenCode 30.3% $11†
RedHatAI/Qwen3.6-35B-A3B-NVFP4 OpenClaw 20.2% $10†

More coming soon...

† - Cost estimates for OSS models are calculated by ($4 per A100 GPU hour × agent benchmark duration).

CLI Utility

The CLI utility will help you configure and run a benchmark jobs with Harbor for self-hosted models. It automatically constructs and runs the Harbor job command for your specified benchmark, agent, self-hosted model.

Prerequisites

  • Install dependencies with uv

    uv sync

  • Set up a vLLM server, or other Anthropic- and OpenAI-compatible server

  • Select a benchmark from among the options in Harbor Hub

Run a Benchmark

The following is the minimal configuration needed to run a job with the CLI:

uv run coding-agent-bench run \
    --agent <agent> \
    --dataset <benchmark-name> \
    --model-name <model-name> \
    --server-url <server-url>

For example, to run swe-bench/swe-bench-verified in Claude Code against a self-hosted model:

uv run coding-agent-bench run \
    --agent claude-code \
    --dataset scale-ai/swe-bench-pro \
    --model-name my-model \
    --server-url http://my.server.url

If you want to see a preview of Harbor command that would be run for a given set of arguments without actually running the job, add the --dry-run flag.

Note

Additional configuration options are available, use uv run coding-agent-bench run --help to see them.

Use Agent Skills

Pass one or more skill directories or Git sources with the repeatable --skill option (--skills is an alias). Harbor installs the resolved skills into the agent used by the benchmark.

To test a skill from your local filesystem:

uv run coding-agent-bench run \
    --agent opencode \
    --dataset swe-bench/swe-bench-verified \
    --model-name my-model \
    --server-url http://my.server.url \
    --skill ./my-skills

Git sources make skills easy to share and reproduce. Repeat the option to test multiple skill collections, for example Superpowers together with Caveman:

uv run coding-agent-bench run \
    --agent claude-code \
    --dataset swe-bench/swe-bench-verified \
    --model-name my-model \
    --server-url http://my.server.url \
    --skill obra/superpowers \
    --skill juliusbrussee/caveman

Harbor accepts org/name and org/name@ref shorthand and HTTP(S) Git URLs. Repository shorthand loads skills from the repository's skills/ directory. Use a full URL such as https://github.com/org/repo/tree/<ref>/<subdir> to select another directory. Pin a tag or named branch with @ref (or in the full URL) when comparing benchmark runs. Harbor resolves the reference to a commit and records that commit in the job lock file for reproducibility.

Queue Service

The queue service is a FastAPI application that can be deployed on OpenShift to queue and run benchmarks automatically. Benchmark results are stored to MinIO for later review.

sequenceDiagram
    Requestor->>Queue Service: Request Benchmark Run
    Queue Service->>Queue Service: Wait in Queue
    Queue Service->>Openshift Job: Start Benchmark Job
    Openshift Job->>Harbor Orchestrator Pod: Start Harbor Run
    loop For Each Task
        Harbor Orchestrator Pod->>Task Pod: Run Task Pod
        Task Pod->>Harbor Orchestrator Pod: Save Results
    end
    Harbor Orchestrator Pod->>MinIO: Save Benchmark Results
    Harbor Orchestrator Pod->>Openshift Job: Complete
    Queue Service-->>Openshift Job: Poll for completion
    Openshift Job->>Queue Service: Complete
    Queue Service->>Openshift Job: Cleanup
    Queue Service->>Queue Service: Start Next Job
Loading

Set up the service

  1. Log in to your cluster and project:

    oc login --server=<server> --token=<token>
oc project <project>

  2. Create the MinIO service for artifact storage:

    oc apply -f deploy/harbor-minio.yml

    Note: the default username and password are (minioadmin, minioadmin). You can update this in the deployment file if needed.

  3. Create the orchestrator and task service accounts:

    oc apply -f deploy/harbor-orchestrator-sa.yml
oc apply -f deploy/harbor-task-sa.yml

  4. Create a secret file named job-queue-secret with the queue service's API_KEY and any queue or Nebius settings, then apply it:

    apiVersion: v1
kind: Secret
metadata:
  name:  job-queue-secret
stringData:
  API_KEY: <your-api-key>
type: Opaque

    If the intake CronJob is deployed, create its separate poller secret:

    apiVersion: v1
kind: Secret
metadata:
  name: intake-poller-secret
stringData:
  JOB_QUEUE_URL: https://<queue-route-host>
  GOOGLE_SHEET_ID: <sheet-id>
  SENDER_EMAIL: ace-model-evals@redhat.com
  AUTO_APPROVE: 'false'
type: Opaque

  5. Create the queue service:

    oc apply -f deploy/job-queue-service.yml

  6. (Optional) To run jobs against OpenRouter (server_url: openrouter), create an openrouter-api-key secret. Job pods mount it automatically (it is optional, so non-OpenRouter jobs are unaffected):

    apiVersion: v1
kind: Secret
metadata:
  name: openrouter-api-key
stringData:
  OPENROUTER_API_KEY: <your-openrouter-api-key>
type: Opaque

    The queue service itself also needs OPENROUTER_API_KEY in its environment to validate OpenRouter jobs at request time. Add it to job-queue-secret (which the service already loads) or envFrom the openrouter-api-key secret in deploy/job-queue-service.yml.

    The queue listens on HTTPS inside the cluster. OpenShift's service-serving certificate operator creates the job-queue-tls Secret referenced by the Deployment, and the Route uses re-encryption so traffic remains encrypted from the router to the queue pod. Wait for that Secret to appear before troubleshooting pod startup:

    oc get secret job-queue-tls

Get the route for the deployed service:

oc get route job-queue-route --output jsonpath='{.spec.host}'

Set JOB_QUEUE_URL in intake-poller-secret to this HTTPS route before applying deploy/intake-cronjob.yml.

Check that the application is live by visiting the docs:

export JOB_QUEUE_URL="https://$(oc get route job-queue-route --output jsonpath='{.spec.host}')"
open $JOB_QUEUE_URL/docs

Use the service

Queue up a new benchmark task. Skills submitted to the queue must be public Git sources because the OpenShift job cannot access paths on the requestor's local filesystem:

curl -X POST $JOB_QUEUE_URL/jobs \
    -d '{"job_name": "test", "agent": "pi", "dataset": "swe-bench/swe-bench-verified", "model_name": "qwen3.6-27b", "server_url": "<server-url>", "n_tasks": 1, "skills": ["obra/superpowers@<ref>"]}' \
    -H "Content-Type: application/json" \
    -H "X-API-Key: <your-api-key>"
{
    "message":"Job created.",
    "job_id":"b5ef13c8-8909-4bf1-b5b1-43354e9f395c", 
    ... 
}

View the queued/running/completed tasks:

open $JOB_QUEUE_URL/ui

Or list them from the API:

curl $JOB_QUEUE_URL/jobs -H "X-API-Key: <your-api-key>"

Cancel a running or queued job:

curl -X DELETE $JOB_QUEUE_URL/jobs/<job_id> -H "X-API-Key: <your-api-key>"

(Optional) Connect to Nebius

The queue service supports starting and stopping vLLM server instances automatically using Nebius. When enabled, the queue service will create a Nebius VM, SSH into the instance, start the vLLM server for the next model in the queue, run the benchmark job against that model, swap the running model for the next model in the queue, then spin down the Nebius VM when the queue is empty.

To connect the queue service to Nebius, first set up an AI Cloud account.

After setting up the account, install the CLI and login to your account.

Then run the following commands to create a service account in your project:

# Create a service account
export SA_ID=$(nebius iam service-account create \
  --name <service_account_name> \
  --format json | jq -r '.metadata.id')

# Create and attach an authorized key to the service account
nebius iam auth-public-key generate \
  --service-account-id $SA_ID \
  --output ~/.nebius/$SA_ID-credentials.json

Once the service account is created, you can update your job queue secret with the following environment variables needed for Nebius:

apiVersion: v1
kind: Secret
metadata:
  name:  job-queue-secret
stringData:
  API_KEY: <your-api-key>
  NEBIUS_ENABLED: '1'
  NEBIUS_SERVICE_ACCOUNT_CREDS: |
    <service-account-file-content>
  NEBIUS_PARENT_ID: <project-id>
  NEBIUS_TENANT_ID: <tenant-id>
  NEBIUS_SERVICE_ACCOUNT_ID: <service-account-id>
  NEBIUS_SUBNET_ID: <subnet-id>
  NEBIUS_INSTANCE_NAME_PREFIX: job-queue-worker
  NEBIUS_IDLE_TIMEOUT_SECONDS: '600'
  HF_TOKEN: <optional-huggingface-token>
type: Opaque

When creating a job, set server_url to nebius-<resource> to use a managed Nebius instance with the specified GPU resource (e.g. nebius-h200, nebius-b200). Available resources are defined in RESOURCE_CONFIG_REGISTRY.

Harbor Command Examples

Prerequisites:

  • Install Harbor

  • Set up a vLLM server, or other Anthropic- and OpenAI-compatible server

  • Set your benchmark in your environment from among the options in Harbor Hub, e.g.:

    export BENCHMARK='swe-bench/swe-bench-verified'

  • If you need to filter tasks in your benchmark by name, add the -i flag with your glob pattern to your harbor run command, e.g. -i "*ansible*"

Directory:

Harness Model Server Example Status
Claude Code vLLM Link Validated
Codex vLLM Link Testing
OpenClaw vLLM Link Validated
OpenCode vLLM Link Validated
OpenHands vLLM Link Validated
Pi vLLM Link Validated
Qwen Code vLLM Link Validated
Claude Code Anthropic Link Validated
Claude Code VertexAI Link Validated
Codex OpenAI Link Validated

Note

To use with a locally hosted model (e.g. llama.cpp) use a vLLM example and set SERVER_URL=http://host.docker.internal:<server-port>

Claude Code vLLM

Set the following variables in your environ:

export SERVER_URL=
export MODEL_NAME=

Then run:

harbor run --agent claude-code -d $BENCHMARK \
    --ae ANTHROPIC_BASE_URL=$SERVER_URL \
    --ae ANTHROPIC_API_KEY='sk-no-key-required' \
    --ae ANTHROPIC_MODEL=$MODEL_NAME \
    --ae ANTHROPIC_DEFAULT_OPUS_MODEL=$MODEL_NAME \
    --ae ANTHROPIC_DEFAULT_SONNET_MODEL=$MODEL_NAME \
    --ae ANTHROPIC_DEFAULT_HAIKU_MODEL=$MODEL_NAME

Codex vLLM

Set the following variables in your environ:

export SERVER_URL=
export MODEL_NAME=

Use the utility script to create the config.toml file:

uv run scripts/codex_config_toml.py $MODEL_NAME $SERVER_URL

Then run:

harbor run --agent codex -d $BENCHMARK \
    -m vllm/$MODEL_NAME \
    --ae CODEX_HOME=/root/.codex/ \
    --mounts-json '[ { "type": "bind", "source":"/path/to/coding-agent-bench/config.toml", "target": "/root/.codex/config.toml" } ]'

OpenClaw vLLM

Set the following variables in your environ:

export MODEL_NAME=
export SERVER_URL=
export OPENAI_BASE_URL=$SERVER_URL/v1
export OPENAI_API_KEY='NONE'

Then run:

harbor run --agent openclaw -p $DATASET_DIR/swe-bench-verified \
    -m openai/$MODEL_NAME \
    --agent-kwarg thinking=off \
    --n-concurrent 8

OpenCode vLLM

Set the following variables in your environ:

export MODEL_NAME=

Set the content of your OpenCode config in your environ. Remember to replace the <server-url> with your vLLM server url and the <model-name> with your served model name:

export OPENCODE_CONFIG_CONTENT='{"$schema":"https://opencode.ai/config.json","model":"vllm/<model-name>","provider":{"vllm":{"npm":"@ai-sdk/openai-compatible","name":"vLLM","options":{"baseURL":"<server-url>"},"models":{"<model-name>":{"name":"<model-name>","limit":{"context":196500,"output":65500}}}}}}'

Then run:

harbor run --agent opencode -p $DATASET_DIR/swe-bench-verified \
    -m vllm/$MODEL_NAME \
    --ae "OPENCODE_CONFIG_CONTENT=$OPENCODE_CONFIG_CONTENT"

OpenHands vLLM

Set the following variables in your environ:

export MODEL_NAME=
export SERVER_URL=
export LLM_API_KEY="NONE"

Then run:

harbor run -d $BENCHMARK \
  -a openhands-sdk \
  -m hosted_vllm/$MODEL_NAME \
  --ae HOSTED_VLLM_API_BASE=$SERVER_URL/v1

Pi vLLM

Set the following variables in your environ:

export MODEL_NAME=

Create a models.json file with your vLLM server information:

export PI_MODELS_JSON='{ "providers": { "vllm": { "baseUrl": "<server-url>", "api": "openai-completions", "apiKey": "NONE", "models": [{ "id": "gemma4-26b", "name": "<model-name>", "contextWindow": 262000 }] } } }'
echo $PI_MODELS_JSON > models.json

Then run:

harbor run --agent pi -d $BENCHMARK \
    -m vllm/$MODEL_NAME \
    --ae PI_OFFLINE=1 \
    --ae PI_CODING_AGENT_DIR=/root/.pi/agent \
    --mounts-json '[ { "type": "bind", "source":"/path/to/models.json", "target": "/root/.pi/agent/models.json" } ]'

Qwen Code vLLM

Set the following variables in your environ:

export MODEL_NAME=
export SERVER_URL=
export OPENAI_BASE_URL=$SERVER_URL/v1
export OPENAI_API_KEY='NONE'

Then run:

harbor run --agent qwen-coder -d $BENCHMARK \
    -i $DATASET_PATTERN \
    -m $MODEL_NAME

Claude Code Anthropic

Copy .env.example to .env and set the following variables:

ANTHROPIC_API_KEY=

Then run:

set -a
source .env

harbor run --agent claude-code -d $BENCHMARK \
    -m claude-opus-4-8

Claude Code VertexAI

Set the following variables in your environ:

export CLOUD_ML_REGION=
export ANTHROPIC_VERTEX_PROJECT_ID=
export ANTHROPIC_MODEL=

Then run:

harbor run --agent claude-code -d $BENCHMARK \
    --ae CLAUDE_CODE_USE_VERTEX=1 \
    --ae CLOUD_ML_REGION=$CLOUD_ML_REGION \
    --ae ANTHROPIC_VERTEX_PROJECT_ID=$ANTHROPIC_VERTEX_PROJECT_ID \
    --ae ANTHROPIC_MODEL=$ANTHROPIC_MODEL \
    --ae GOOGLE_APPLICATION_CREDENTIALS='/app/.config/gcloud/application_default_credentials.json' \
    --mounts-json '[ { "type": "bind", "source":"~/.config/gcloud/application_default_credentials.json", "target": "/app/.config/gcloud/application_default_credentials.json" } ]'

Codex OpenAI

Copy .env.example to .env and set the following variables:

OPENAI_API_KEY=

Then run:

set -a
source .env

harbor run --agent codex -d $BENCHMARK \
    -m gpt-5.5

Deploy models with vLLM

Check out deploy/qwen-all-in-one.yml for a sample vLLM deployment of RedHatAI/Qwen3.6-35B-A3B-NVFP4.

Apply to your cluster by running:

oc apply -f deploy/qwen-all-in-one.yml

SWE-Bench Acceleration

Use accelerated images for SWE-bench-verified

  1. Download the SWE-Bench-Verified tasks
harbor download swe-bench/swe-bench-verified
  1. Replace images with the accelerated ones from Epoch AI
uv run scripts/replace_swe_bench_images.py <path-to-dataset>

Pre-pull base images

  1. Download the dataset
harbor download <dataset>
  1. Pull all the base images
uv run scripts/pull_images.py <path-to-dataset>

Run with Openshift

Run Tasks in Openshift (Orchestrate Locally)

Login to your cluster and select a project:

oc login --token=<token> --server=<server>
oc project <project>

Create ServiceAccounts and RoleBindings to run tasks:

oc apply -f deploy/harbor-task-sa.yml

Then in your harbor command, add the flag:

--env openshift

Run Tasks and Orchestrate in Openshift

Login to your cluster and select a project:

oc login --token=<token> --server=<server>
oc project <project>

Create ServiceAccounts and RoleBindings to run tasks and orchestrate:

oc apply -f deploy/harbor-task-sa.yml
oc apply -f deploy/harbor-orchestrator-sa.yml

Create a MinIO deployment to store your job results:

oc apply -f deploy/harbor-minio.yml

Using the CLI, start a job with the --remote flag enabled and set --environment openshift, e.g.:

uv run coding-agent-bench run \
    --agent claude-code \
    --dataset scale-ai/swe-bench-pro \
    --model-name my-model \
    --server-url http://my.server.url \
    --remote \
    --environment openshift

Skills used by a remote OpenShift Job must be public Git sources. Local paths are rejected because they are not available inside the orchestrator pod. The pod also needs outbound network access to the Git host.

uv run coding-agent-bench run \
    --agent claude-code \
    --dataset swe-bench/swe-bench-verified \
    --model-name my-model \
    --server-url http://my.server.url \
    --skill obra/superpowers@<ref> \
    --remote \
    --environment openshift

WIP

Run with Gemini and Gemini CLI

export GOOGLE_CLOUD_PROJECT="<your-project>"

harbor run --agent gemini-cli -d $BENCHMARK \
    -m $MODEL_NAME

Run with vLLM and Gemini CLI

harbor run --agent gemini-cli -d $BENCHMARK \
    --ae GOOGLE_GEMINI_BASE_URL=$SERVER_URL \
    --ae GEMINI_MODEL=$MODEL_NAME \
    -m $MODEL_NAME

About

Reproducible benchmarks for coding agents and models using Harbor

Resources

Readme
Activity
Custom properties

Stars

9 stars

Watchers

0 watching

Forks

6 forks
Report repository

Releases

Packages

Used by

Contributors

Languages