Add Copilot Studio SDK + Direct Line Opus 4.8 @ 256x leaderboard entries

[c-jatangithanuja]

Summary

Adds two Enterprise-Bench leaderboard entries for Copilot Studio + Claude Opus 4.8 @ 256x — same agent/model/prompt, two Harbor backends:

File	Backend	pass@1	pass@10	Harbor job
2026-08-05__copilot-studio-sdk__opus-4-8.yaml	SDK	32.1%	71.4%	https://hub.harborframework.com/jobs/77fbb373-4fcb-4e88-ac94-8c4cb419b1a3
2026-08-05__copilot-studio-direct-line__opus-4-8.yaml	Direct Line	37.9%	78.6%	https://hub.harborframework.com/jobs/d97b6f4e-1680-4850-83ea-e72c375a03f2

Both runs: 14 tasks × k=10 = 140 trials, 0 errors. Jobs are public on Harbor Hub.

Type of contribution

• Agent result submission
• Task addition or update
• Dataset addition or update
• Documentation
• Bug fix / setup improvement
• Other

Validation

• make validate passes (or run before opening PR — see Commands)
• I ran at least one affected task, or explained why not below
• Dataset changes are synthetic and safe to publish (N/A — no dataset changes)
• New/changed task criteria avoid answer leakage (N/A — no task changes)
• Documentation is updated (N/A — leaderboard entries only)

Commands run

# Harbor upload (from harbor/ dir, jobs already public)
harbor upload --public jobs/copilot-studio/256x/opus-4-8/2026-08-05__12-02-45
harbor upload --public jobs/copilot-studio/256x/direct-line-opus-4-8-minimal-k10/2026-08-05__14-05-10

# Reproduce (SDK)
cd harbor && source .venv/bin/activate
set -a && source configs/256x.env && set +a   # COPILOT_STUDIO_BACKEND=sdk
harbor run -p tasks/l1-l2-bench-v1 \
  -a agents.copilot_studio:CopilotStudioAgent \
  -m anthropic/claude-opus-4-8 \
  --env-file configs/256x.env --yes -k 10 -n 3 \
  --job-name "$(date +%Y-%m-%d__%H-%M-%S)__copilot-studio-sdk-opus-4-8__256x" \
  --jobs-dir jobs/copilot-studio/256x/opus-4-8

# Reproduce (Direct Line) — set COPILOT_STUDIO_BACKEND=direct_line in configs/256x.env
harbor run -p tasks/l1-l2-bench-v1 \
  -a agents.copilot_studio:CopilotStudioAgent \
  -m anthropic/claude-opus-4-8 \
  --env-file configs/256x.env --yes -k 10 -n 3 \
  --job-name "$(date +%Y-%m-%d__%H-%M-%S)__copilot-studio-direct-line-opus-4-8__256x" \
  --jobs-dir jobs/copilot-studio/256x/direct-line-opus-4-8-minimal-k10

# Validate enterprise-bench repo (before PR)
cd enterprise-bench && make validate

[nimit2801]

First-touch impression: leaderboard result submission adding two Copilot Studio + Claude Opus 4.8 @ 256x entries, one SDK backend and one Direct Line backend.

Checks: CI / validate passed. PR is mergeable and ready for review.

Risk: low-to-medium. This is metadata-only leaderboard data, but maintainers should verify the public Harbor sources match the claimed scores and that both runs are reproducible.

Maintainer verification points:

1. Confirm both Harbor jobs are public and accessible: SDK job 77fbb373-4fcb-4e88-ac94-8c4cb419b1a3 and Direct Line job d97b6f4e-1680-4850-83ea-e72c375a03f2.
2. Verify the YAML scores match the PR summary: SDK pass@1 32.1%, pass@10 71.4%; Direct Line pass@1 37.9%, pass@10 78.6%.
3. Spot-check that each entry references the correct backend/config and that both runs report 14 tasks x k=10 with 0 errors.

[c-jatangithanuja]

@nimit2801 — confirmed on the Opus 4.8 runs:

1. Public Harbor jobs

SDK: https://hub.harborframework.com/jobs/77fbb373-4fcb-4e88-ac94-8c4cb419b1a3
Direct Line: https://hub.harborframework.com/jobs/d97b6f4e-1680-4850-83ea-e72c375a03f2
2. YAML scores match

SDK: pass@1 32.1%, pass@10 71.4%
Direct Line: pass@1 37.9%, pass@10 78.6%
Checked against local result.json and Hub avg reward (0.32 / 0.38)
3. Backend / config spot-check

Both runs: 14 tasks × k=10 = 140 trials, 0 errors
SDK vs Direct Line backend metadata matches job names and traces