Run Enterprise-Bench L1–L2 with Grok Build + grok-4.5 (Harbor harness)

## Summary

Track an evaluation of **Grok Build** (`-a grok-build`) with model **`grok-4.5`** against **`Enterprise-Bench/l1-l2-bench`** on Harbor Hub.

**Harness:** Harbor (job runner, Docker trials, judge, jobs artifacts)  
**Agent under test:** Grok Build (xAI CLI, installed per trial container)  
**Dataset:** [Enterprise-Bench/l1-l2-bench](https://hub.harborframework.com/datasets/enterprise-bench/l1-l2-bench) (14 tasks)  
**Judge:** OpenAI GPT-5 via `OPENAI_API_KEY` (required for all agents)

## Why this run

- Add a **Grok Build + grok-4.5** data point next to existing Claude Code / DevRev Computer jobs in the README.
- Validate that Harbor's `grok-build` adapter works end-to-end with Enterprise-Bench MCP servers (crm / pm / file-server).
- Produce submit-ready results per [`docs/submit-results.md`](./docs/submit-results.md).

## Architecture

```text
Harbor Hub dataset (Enterprise-Bench/l1-l2-bench)
        ↓
Harbor job runner (Docker trials, scoring, jobs/)
        ↓  -a grok-build -m grok-4.5
Grok Build agent (CLI installed in each trial container)
        ↓  MCP (host.docker.internal:8011–8013)
Maple Payments synthetic data
        ↓  OPENAI_API_KEY
GPT-5 judge → reward 0.0 | 1.0
```

## Prerequisites

| Requirement | Notes |
|---|---|
| Harbor CLI **≥ 0.20** | `grok-build` is not in 0.18 factory map; upgrade via `uv tool upgrade harbor` |
| Docker Desktop | ~2 GB RAM + 2 CPUs per trial; on ~8 GiB Docker use **`-n 3`** max |
| Local base image | `make build-image` → `enterprise-bench/conversational-base:latest` (built locally, not pulled) |
| MCP servers | `make start-servers` (REST 9001–9003 + MCP 8011–8013) |
| `XAI_API_KEY` | Required by Harbor's Grok Build adapter inside Docker. **Host SuperGrok/browser OAuth is not used** for trials (fresh container; no `~/.grok/auth.json` mount). |
| `OPENAI_API_KEY` | LLM judge for every trial |

### Auth note (important for team)

- Interactive Grok Build on a laptop can use SuperGrok / OAuth (`grok login`).
- Harbor evals run **headless in Docker** and the upstream adapter **requires `XAI_API_KEY`**.
- Subscription alone is not sufficient for multi-trial container runs.

## Known gotchas (from field SKILL + prior local runs)

1. **`make install` before `make setup`** — otherwise setuptools flat-layout errors on `data/` / `images/`.
2. **Base image built locally** — `pull access denied` means image missing, not registry auth.
3. **MCP 421 Misdirected Request** — FastMCP host check rejects `Host: host.docker.internal`. Local fix: `allowed_hosts=["*"]` on each `mcp.run(...)`, then restart servers.
4. Ports 801x (MCP) vs 900x (REST) are both correct.
5. First trial is slower: Grok CLI installs inside the container (`curl https://x.ai/cli/install.sh`).
6. Grok Build disables web search by default (closed-book eval integrity). Keep that unless intentionally changing protocol.

## Execution plan

### Phase 0 — Tooling

```bash
uv tool upgrade harbor
harbor --version   # expect ≥ 0.20
harbor auth status
```

### Phase 1 — Dataset + environment

Prefer a clean Hub download or the maintained local tree.

```bash
harbor download enterprise-bench/l1-l2-bench -o ./enterprise-bench
cd ./enterprise-bench/l1-l2-bench

make install
make setup
# apply MCP allowed_hosts fix if Host-header probe returns 421
make build-image
make start-servers
```

Verify MCP from agent Host header:

```bash
curl -s -o /dev/null -w "%{http_code}\n" -X POST http://localhost:8012/mcp \
  -H "Host: host.docker.internal:8012" \
  -H "Accept: application/json, text/event-stream" \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{"protocolVersion":"2025-06-18","capabilities":{},"clientInfo":{"name":"t","version":"1"}}}'
# expect 200
```

### Phase 2 — Smoke (1 task, 1 attempt)

```bash
export XAI_API_KEY=...
export OPENAI_API_KEY=...

harbor run \
  -p eng-l1-a \
  -a grok-build \
  -m grok-4.5 \
  --mcp-config mcp.json \
  --ae XAI_API_KEY=$XAI_API_KEY \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  -k 1 -n 1 \
  --yes \
  --jobs-dir jobs/grok-build-grok-4.5-smoke
```

**Smoke pass criteria:** agent starts, MCP tools appear in trajectory, judge writes `reward.txt`, no auth failure in first minutes of setup.

### Phase 3 — Reliability pass (compare to prior local baseline)

Prior local claude-code / claude-opus-4-8 full job (`2026-07-09__09-26-22`): **70 trials**, mean reward **~0.74**, pass@5 **~0.93**.

```bash
# all 14 tasks × 5 attempts, 3 concurrent (safe on ~8 GiB Docker)
harbor run \
  -p . \
  -a grok-build \
  -m grok-4.5 \
  --mcp-config mcp.json \
  --ae XAI_API_KEY=$XAI_API_KEY \
  --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  -k 5 -n 3 \
  --yes \
  --jobs-dir jobs/grok-build-grok-4.5-k5
```

Run long jobs under `tmux`. Prefer explicit `harbor run` over Makefile defaults (Makefile still centers on claude-code).

### Phase 4 — Optional methodology-grade + submit

Paper protocol: **14 × 10 = 140** observations.

```bash
harbor run -p . -a grok-build -m grok-4.5 --mcp-config mcp.json \
  --ae XAI_API_KEY=$XAI_API_KEY --ae OPENAI_API_KEY=$OPENAI_API_KEY \
  -k 10 -n 3 --yes --jobs-dir jobs/grok-build-grok-4.5-k10
```

Submission package (see `docs/submit-results.md`):

- Agent name + version (Grok Build / CLI version)
- Model: `grok-4.5` (provider xAI)
- Harbor version
- Dataset ref: `enterprise-bench/l1-l2-bench` (or commit SHA)
- Full command, `-k`, `-n`
- OS, Docker memory/CPU, any agent kwargs
- Harbor Hub job link or archived `jobs/`
- Known failures / task exclusions

Do **not** submit modified tasks without disclosure, scrubbed failed trials, or logs containing secrets.

### Phase 5 — Teardown

```bash
make stop-servers
# avoid make clean unless intentionally wiping jobs + re-applying MCP fix
```

## Success metrics to report

- Mean reward / success rate
- pass@k (k=2,4,5 and/or 10)
- Per-task reliability (note: `eng-l1-a` criterion 6 has been hard across agents)
- Infra errors vs genuine task fails
- Rough wall-clock and cost notes (xAI agent + OpenAI judge)

## Checklist

- [ ] Harbor ≥ 0.20; `harbor run -h` lists `grok-build`
- [ ] `XAI_API_KEY` + `OPENAI_API_KEY` available to the runner
- [ ] Base image built; MCP servers healthy (Host-header probe 200)
- [ ] Smoke: `eng-l1-a` k=1 complete
- [ ] Reliability: 14 tasks × k=5 complete
- [ ] Results summarized vs Claude baseline
- [ ] Optional: k=10 methodology run
- [ ] Optional: Hub upload / result submission
- [ ] Servers stopped; keys rotated if pasted into shell history

## Time / cost rough order

| Scenario | Trials | Notes |
|---|---:|---|
| Smoke `eng-l1-a` | 1 | ~10–25 min (first CLI install in container) |
| Reliability k=5 | 70 | Several hours at n=3 |
| Methodology k=10 | 140 | Overnight / long day |

Task agent timeout in task.toml is typically **600s**; verifier **300s**.

## References

- Dataset: https://hub.harborframework.com/datasets/enterprise-bench/l1-l2-bench
- Harbor: https://github.com/harbor-framework/harbor
- Grok Build docs: https://docs.x.ai/build/overview
- Methodology: https://devrev.ai/enterprise-bench-methodology
- This repo: `README.md`, `SKILL.md`, `docs/submit-results.md`
- Existing public job links in README (Claude Code / DevRev Computer)

## Owner / next step

Execute Phase 0–2 as soon as API keys are available, then Phase 3 for a submit-ready Grok data point.

/cc @jeff-smith (adjust if handle differs)

Requirement	Notes
Harbor CLI ≥ 0.20	`grok-build` is not in 0.18 factory map; upgrade via `uv tool upgrade harbor`
Docker Desktop	~2 GB RAM + 2 CPUs per trial; on ~8 GiB Docker use `-n 3` max
Local base image	`make build-image` → `enterprise-bench/conversational-base:latest` (built locally, not pulled)
MCP servers	`make start-servers` (REST 9001–9003 + MCP 8011–8013)
`XAI_API_KEY`	Required by Harbor's Grok Build adapter inside Docker. Host SuperGrok/browser OAuth is not used for trials (fresh container; no `~/.grok/auth.json` mount).
`OPENAI_API_KEY`	LLM judge for every trial

Navigation Menu

Uh oh!

Run Enterprise-Bench L1–L2 with Grok Build + grok-4.5 (Harbor harness) #3

Summary

Why this run

Architecture

Prerequisites

Auth note (important for team)

Known gotchas (from field SKILL + prior local runs)

Execution plan

Phase 0 — Tooling

Phase 1 — Dataset + environment

Phase 2 — Smoke (1 task, 1 attempt)

Phase 3 — Reliability pass (compare to prior local baseline)

Phase 4 — Optional methodology-grade + submit

Phase 5 — Teardown

Success metrics to report

Checklist

Time / cost rough order

References

Owner / next step

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Scenario	Trials	Notes
Smoke `eng-l1-a`	1	~10–25 min (first CLI install in container)
Reliability k=5	70	Several hours at n=3
Methodology k=10	140	Overnight / long day

Uh oh!

Run Enterprise-Bench L1–L2 with Grok Build + grok-4.5 (Harbor harness) #3

Description

Summary

Why this run

Architecture

Prerequisites

Auth note (important for team)

Known gotchas (from field SKILL + prior local runs)

Execution plan

Phase 0 — Tooling

Phase 1 — Dataset + environment

Phase 2 — Smoke (1 task, 1 attempt)

Phase 3 — Reliability pass (compare to prior local baseline)

Phase 4 — Optional methodology-grade + submit

Phase 5 — Teardown

Success metrics to report

Checklist

Time / cost rough order

References

Owner / next step

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions