OpenEval — 1-Click Pinokio Launcher

A one-click Pinokio launcher for OpenEval, a local-first, harness-agnostic evaluation dashboard for agent CLIs.

OpenEval runs repeatable cases against agent harnesses (Claude Code, Codex, ncode, and any user descriptor under harnesses/), grades results with deterministic and rubric-based checks, persists run history to a local SQLite database, and turns everything into an operator-friendly Next.js dashboard. Your run history, transcripts, and stats stay on your own machine under app/data/.

What This Launcher Does

This launcher installs and runs OpenEval with a single click, so you don't have to clone the repo, install Node dependencies, or build and serve the app by hand.

• Install — clones the OpenEval repository into app/, runs npm install, and builds the production bundle (npm run build) so pages load fast at start time.
• Start — serves the pre-built dashboard (npm start on the next free port), waits for the local URL, and exposes an Open App button. All pages render per-request against your live local data — the production build doesn't make anything stale.
• Update — pulls the latest launcher and app code, reinstalls dependencies, and rebuilds the production bundle.
• Reset — deletes the app/ folder to revert to a clean, pre-install state (removes cloned code, dependencies, and local run history).

Requirements

• Node.js 20+ (bundled/managed by Pinokio's environment).
• OpenEval is local-first and runs entirely on your machine. Optional LLM-judge and cost-estimation features use OpenRouter or a local CLI and are only active if you configure them inside the app's Settings.

How To Use

1. Open this launcher in Pinokio.
2. Click Install and wait for the clone + dependency install to finish.
3. Click Start. When the server is ready, the sidebar shows an Open App button — click it to open the dashboard.
4. Use the dashboard:
   • / — summary: total runs, case count, latest pass rate, average tokens, recent runs.
   • /runs — historical run table with status, pass/fail, runner, cost, tokens, duration.
   • /runs/new — new-run wizard (harness, runner, parallelism, samples, filters, explicit cases).
   • /runs/[id] — run detail with per-case grader results, traces, cost, and usage.
   • /live — recent local CLI sessions with usage/cost/model provenance.
   • /collection — full-history usage rollups, top projects, and a permanent session archive.

To stop the server, use Pinokio's native stop control on the running Start script.

API

OpenEval exposes HTTP JSON endpoints from the running server. Replace http://localhost:3000 below with the exact URL shown by the Open App button (Pinokio auto-selects a free port, so it may differ).

Primary endpoints:

Method	Endpoint	Purpose
GET	/api/runs	List historical runs.
POST	/api/runs	Start a new run.
GET	/api/runs/{id}	Get a single run's detail.
GET	/api/cases	List available cases.
GET	/api/harnesses	List configured harnesses.
GET	/api/live	List recent local CLI sessions.
GET	/api/collection	Aggregated collection stats.

Start-a-run request body (all fields optional): harness, runner (headless | tmux), parallel (1–8), samples (1–8), caseIds (string[]), categories (string[]), tags (string[]), difficulty (string[]), model, name.

curl

# List runs
curl http://localhost:3000/api/runs

# Start a new run
curl -X POST http://localhost:3000/api/runs \
  -H "Content-Type: application/json" \
  -d '{"harness":"ncode","runner":"headless","parallel":2,"samples":1}'

# Inspect a run
curl http://localhost:3000/api/runs/RUN_ID

JavaScript

const BASE = "http://localhost:3000";

// List runs
const runs = await fetch(`${BASE}/api/runs`).then((r) => r.json());

// Start a new run
const started = await fetch(`${BASE}/api/runs`, {
  method: "POST",
  headers: { "Content-Type": "application/json" },
  body: JSON.stringify({ harness: "ncode", runner: "headless", parallel: 2, samples: 1 }),
}).then((r) => r.json());

console.log(runs, started);

Python

import requests

BASE = "http://localhost:3000"

# List runs
runs = requests.get(f"{BASE}/api/runs").json()

# Start a new run
started = requests.post(
    f"{BASE}/api/runs",
    json={"harness": "ncode", "runner": "headless", "parallel": 2, "samples": 1},
).json()

print(runs, started)

For deeper CLI usage (headless eval runs, accuracy audits, reports), see the upstream OpenEval documentation in app/README.md after installation.

Links

• OpenEval repository: https://github.com/RasputinKaiser/OpenEval
• Pinokio: https://pinokio.computer