Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
39 commits
Select commit Hold shift + click to select a range
ced2875
feat(harness): agent source registry and the understand-agent stage
KarthikAvinashFI Aug 16, 2026
42d3c08
feat(harness): database-backed generated worlds with a probe gate
KarthikAvinashFI Aug 16, 2026
320f271
fix(harness): isolate probes, surface tool results, make sequences co…
KarthikAvinashFI Aug 16, 2026
008766f
feat(harness): check the world's catalogue and argument reads, one co…
KarthikAvinashFI Aug 16, 2026
986a6a1
feat(harness): environment as world+simulator+sub-goals, proved scena…
KarthikAvinashFI Aug 16, 2026
419edb2
feat(harness): run stage dispatches on modality, live for hosted voic…
KarthikAvinashFI Aug 16, 2026
e954e76
fix(harness): slim the contract to consumed fields plus free-form not…
KarthikAvinashFI Aug 16, 2026
2a0ba45
feat(harness): stages hand a request to the stage that owns it, guard…
KarthikAvinashFI Aug 16, 2026
8adb1d7
fix(harness): make the contract tool schema self-describing and forgi…
KarthikAvinashFI Aug 17, 2026
45f8418
fix(harness): accept field synonyms, keep host tools out, report arti…
KarthikAvinashFI Aug 17, 2026
d61c567
feat(harness): scenarios own a folder, three gates, and one orchestra…
KarthikAvinashFI Aug 17, 2026
5c932f9
feat(harness-ui): scenarios show their gates and their files; contrac…
KarthikAvinashFI Aug 17, 2026
be0631f
feat(harness): one conversation is one folder, with its chat history …
KarthikAvinashFI Aug 17, 2026
eb941a9
chore(harness-ui): track the chat UI in the repo
KarthikAvinashFI Aug 17, 2026
deda850
fix(harness): reception can hand over in the turn that finds the agent
KarthikAvinashFI Aug 17, 2026
644cc25
docs(build-skill): document the three db methods a handler can call
KarthikAvinashFI Aug 17, 2026
cbd8df1
fix(build-tools): a crashed handler is told what a handler actually has
KarthikAvinashFI Aug 17, 2026
212d709
fix(harness): the turn that finds the agent can open the next stage
KarthikAvinashFI Aug 17, 2026
b439fac
fix(ui): keep paragraph breaks in a restored conversation
KarthikAvinashFI Aug 17, 2026
d1bd949
fix(scenarios): a short suite is told how to record the number asked for
KarthikAvinashFI Aug 17, 2026
6edb6b6
feat(prove): name a check that holds when nothing was done
KarthikAvinashFI Aug 17, 2026
1207efa
build(deps): declare the harness UI's fastapi and uvicorn
KarthikAvinashFI Aug 17, 2026
5207d20
docs(harness): setup and run instructions, sessions, and the third gate
KarthikAvinashFI Aug 17, 2026
1f428a2
feat(world): reuse the agent's own tools and stand up the store it al…
KarthikAvinashFI Aug 17, 2026
976d78e
fix(world): let an agent with no database of its own be built and run
KarthikAvinashFI Aug 18, 2026
93d0b96
fix(run): call the simulated user what it is, and keep it talking whi…
KarthikAvinashFI Aug 18, 2026
15609e0
feat(harness): run a suite through one simulation call and judge sub-…
KarthikAvinashFI Aug 18, 2026
0c3614a
fix(harness): correct modality after the fact, reach an agent inside …
KarthikAvinashFI Aug 18, 2026
6672501
fix(harness): give every simulated person the rules that decide wheth…
KarthikAvinashFI Aug 18, 2026
f423048
fix(ui): stop showing metrics that measured nothing as perfect scores…
KarthikAvinashFI Aug 18, 2026
8a6b18a
fix(world): answer a keyed collection's own key when a change names i…
KarthikAvinashFI Aug 18, 2026
96c16c7
fix(ui): render the markdown the stages actually write, and keep a to…
KarthikAvinashFI Aug 18, 2026
eb534a9
refactor(harness): split the misnamed environment module into catalog…
KarthikAvinashFI Aug 18, 2026
ce894df
fix(scenarios): let a suite grow, save what is proved, and show a who…
KarthikAvinashFI Aug 18, 2026
97c2f38
docs(skills): speak as the harness, guide the simulator prompt, and n…
KarthikAvinashFI Aug 18, 2026
7603134
fix(ui): read a world whose collections are mappings, not only lists
KarthikAvinashFI Aug 18, 2026
1005df0
feat(ui): let a front end on another origin reach this API
KarthikAvinashFI Aug 18, 2026
8d27fb8
feat(runs): report a run that is still going, so an hour-long suite i…
KarthikAvinashFI Aug 18, 2026
949e623
feat(harness): personas per scenario, suite-wide platform evals, and …
KarthikAvinashFI Aug 18, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,18 @@ Use it when you want one reproducible loop:
4. Promote the result into a replayable artifact.
5. Prove release readiness with local gates.

### The harness: point it at an agent and talk to it

`src/fi/alk/harness/` builds all of the above **for** an agent instead of asking you to write it.
Point it at an agent's source and it reads what that agent verifiably is, builds a real world its
tools act on, and writes test scenarios that are each proved before they are kept. It is driven
as a conversation, in a terminal or on a web page.

- **[Start here](src/fi/alk/harness/README.md)**: setup from nothing, then how to use it
- **[The web page](harness-ui/README.md)**: the same harness as a chat, on `localhost:8777`
- **[How it works](src/fi/alk/harness/HOW-IT-WORKS.md)** and
**[why it is shaped this way](src/fi/alk/harness/DESIGN.md)**

OpenEnv/Gymnasium shapes are compatibility inputs, not the product center.
Agent Learning Kit is the primary runtime and release contract, and the bar is
the executable `environment_10x_robustness` release gate.
Expand Down
34 changes: 34 additions & 0 deletions harness/.dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
# Credentials, first and deliberately. Nothing in the Dockerfile copies these today, but the
# whole context is handed to the daemon, and the day someone adds a broad COPY is the day live
# keys land in an image layer. Excluded here so that mistake cannot be made.
.env
.env.*
*.pem
*.key
*-credentials.json
service-account*.json

.git
.github
.venv
venv
__pycache__
*.pyc
.pytest_cache
.ruff_cache
.mypy_cache

# Session artifacts: contracts, worlds, transcripts, recordings. Written at run time into a
# mounted volume, and large enough to slow every build if they were sent as context.
artifacts
sessions

node_modules
dist
build
*.egg-info

docs
examples
typescript
oss
6 changes: 6 additions & 0 deletions harness/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
.venv/
.pytest_cache/
__pycache__/
*.py[cod]
artifacts/*
!artifacts/.gitkeep
52 changes: 52 additions & 0 deletions harness/Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,52 @@
# The harness in a container.
#
# Three runtimes have to be here, and each is here for a reason worth stating. Python runs the
# harness itself. Node runs the Claude Code CLI, which the Agent SDK drives as a subprocess --
# without it every stage fails at its first model call. The docker client is here because the
# harness starts containers of its own: a store for a world that needs a real engine, and an
# agent whose tools cannot be imported. It talks to a daemon over the socket proxy rather than
# running one of its own, so there is no Docker-in-Docker and no privileged mode.

FROM node:22-bookworm-slim

ENV PYTHONUNBUFFERED=1 \
PIP_NO_CACHE_DIR=1 \
PIP_DISABLE_PIP_VERSION_CHECK=1 \
PATH="/opt/venv/bin:${PATH}"

# Bookworm's python3 is 3.11, which satisfies requires-python. The venv is not isolation from
# anything here; it is what lets pip write at all, since a Debian-managed interpreter refuses.
RUN apt-get update \
&& apt-get install -y --no-install-recommends \
ca-certificates curl git python3 python3-venv \
&& rm -rf /var/lib/apt/lists/* \
&& python3 -m venv /opt/venv

# The client alone, not the daemon: the static tarball is a few megabytes where the distribution
# package would pull in an engine this container must never run.
ARG DOCKER_CLI_VERSION=27.3.1
RUN curl -fsSL "https://download.docker.com/linux/static/stable/$(uname -m)/docker-${DOCKER_CLI_VERSION}.tgz" \
| tar -xz -C /tmp docker/docker \
&& mv /tmp/docker/docker /usr/local/bin/docker \
&& rm -rf /tmp/docker

RUN npm install -g @anthropic-ai/claude-code

WORKDIR /app

# The harness has its own manifest but imports the parent checkout's public fi.simulate API.
# Build from the parent checkout: docker build -f harness/Dockerfile .
COPY pyproject.toml README.md ./
COPY src ./src
COPY harness/pyproject.toml harness/README.md ./harness/
COPY harness/src ./harness/src
RUN pip install -e . && pip install -e ./harness

COPY harness/ui ./harness/ui

# Reachable from outside the container, which loopback would not be.
ENV HARNESS_HOST=0.0.0.0 \
HARNESS_PORT=8777
EXPOSE 8777

CMD ["python", "harness/ui/server.py"]
Loading
Loading