feat(recipes): OpenHands run → FCR gate via a post-run certifier - #100
Merged
Merged
Conversation
OpenHands has no seam to hook a certify node into: a run ends when the agent sets its own execution_status. The recipe is therefore a post-run certifier over the record the SDK already persists (base_state.json + events/), which keeps engine changes at zero and the reader on the 3.10 floor while the SDK requires 3.12. certify_run.py imports no openhands package. It projects execution_status onto EngineOutcome, runs the visible + withheld split through holdout_gate.decide, sweeps the event log through anticheat_scan.scan, and writes only through loop.emit. The precedence trap is encoded in the mapper: MaxIterationsReached arrives AS execution_status "error", and blocked outranks budget in to_terminal_state, so exactly one of external_error / budget_exhausted is ever set. An error record with no ConversationErrorEvent still names itself, because an empty external_error would fall through to the gate. Six committed fixture conversations make the e2e deterministic and credential-free, so it runs in the default gates matrix; every non-happy case is certified against a GREEN workspace to prove a passing check never overrides the engine's terminal signal. test_openhands_sdk_drift.py pins those fixtures against the installed SDK (persistence constants, the ConversationExecutionStatus member set, the ConversationErrorEvent shape, and the MaxIterationsReached run-loop literal) and runs in its own py3.12 CI job. The conversation-dir path composition was pinned empirically against openhands-sdk 1.37.1 (uuid.hex, 32 chars, no hyphens), closing dossier risk R1. Closes #37
…reboard ST3 spec 5.3 sketches openhands.run() / result.iterations / AgentStuckError. None of that exists after the V1 restructure into OpenHands/software-agent-sdk; the SHAPE it describes survived, the API did not. Annotate it as superseded and point at the shipped recipe. The scoreboard excludes OpenHands because "run state lives fundamentally off-repo ... there is no on-disk run record for a repo-native inspector to read". That is now half true: the V1 SDK persists base_state.json + events/ when persistence_dir is set. The row stays out (the record is still not a repo-native contract), but the exclusion earns a footnote.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 7d020060ab
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
| gate = holdout_gate.decide(visible, withheld) | ||
| # The event log IS the trajectory — the "the runtime ran the tests but the | ||
| # agent read the answer key" case OpenHands cannot catch about itself. | ||
| ac = anticheat_scan.scan(diff_text="", trajectory=record["event_paths"]) |
There was a problem hiding this comment.
When an OpenHands action reads a holdout or answer-key file, that path is recorded inside the event JSON, but this passes only filenames such as events/event-00001-<uuid>.json. anticheat_scan.scan() searches each trajectory string directly and never opens it, so the hidden-answer read produces no finding and a finished run with a green artifact can still be certified as Succeeded. Parse the persisted events into trajectory entries before scanning them.
Useful? React with 👍 / 👎.
| gate = holdout_gate.decide(visible, withheld) | ||
| # The event log IS the trajectory — the "the runtime ran the tests but the | ||
| # agent read the answer key" case OpenHands cannot catch about itself. | ||
| ac = anticheat_scan.scan(diff_text="", trajectory=record["event_paths"]) |
There was a problem hiding this comment.
When the OpenHands run edits a gate script, verification script, or CI workflow, hard-coding diff_text="" prevents anticheat_scan from seeing any changed file. Its critical gate-tampering check therefore cannot produce the documented FailedSafety downgrade, and an otherwise green workspace can be certified as Succeeded despite verifier tampering. Collect and pass the agent workspace's actual diff.
Useful? React with 👍 / 👎.
There was a problem hiding this comment.
Pull request overview
Adds a runnable OpenHands V1 integration recipe that certifies an OpenHands run after it finishes by reading the SDK’s persisted conversation record, running Loop Engineer’s holdout gate + anticheat scan, projecting the engine outcome via loop.integrations.to_terminal_state, and emitting a doctor/metrics-valid Loop contract.
Changes:
- Add
examples/openhands-certify/post-run certifier plus committed fixtures (conversation dirs + workspaces) to make the recipe deterministic and credential-free. - Add two test suites: fixture-driven acceptance tests (runs on the normal matrix) and an SDK-installed “schema drift alarm” (runs in a dedicated Py3.12 CI job).
- Document the integration and supersede the stale OpenHands snippet in the ST3 adapters spec; update scoreboard footnote and CI workflow.
Reviewed changes
Copilot reviewed 24 out of 24 changed files in this pull request and generated 2 comments.
Show a summary per file
| File | Description |
|---|---|
| scripts/test_openhands_sdk_drift.py | New live SDK drift alarm that pins fixture/schema expectations against an installed openhands-sdk. |
| scripts/test_openhands_recipe.py | New deterministic acceptance tests that certify fixture conversations into Loop contracts and validate via loop doctor/loop metrics. |
| examples/openhands-certify/README.md | Recipe README describing post-run certification, fixtures, and how to run the demo. |
| examples/openhands-certify/certify_run.py | New stdlib-only post-run certifier that reads base_state.json + events, runs gate/anticheat, projects terminal state, and emits a Loop contract. |
| examples/openhands-certify/fixtures/workspaces/stale/artifact.txt | Fixture workspace used to force a visible-green/withheld-red false-completion case. |
| examples/openhands-certify/fixtures/workspaces/green/artifact.txt | Fixture workspace used for green-path certification cases. |
| examples/openhands-certify/fixtures/conversations/stuck/events/event-00000-5a1d5379-b1d4-4772-9292-7b002555b529.json | OpenHands fixture event for the stuck conversation. |
| examples/openhands-certify/fixtures/conversations/stuck/base_state.json | OpenHands fixture base_state.json for the stuck conversation. |
| examples/openhands-certify/fixtures/conversations/running/events/event-00000-5a1d5379-b1d4-4772-9292-7b002555b529.json | OpenHands fixture event for the running conversation. |
| examples/openhands-certify/fixtures/conversations/running/base_state.json | OpenHands fixture base_state.json for the running conversation. |
| examples/openhands-certify/fixtures/conversations/paused/events/event-00000-5a1d5379-b1d4-4772-9292-7b002555b529.json | OpenHands fixture event for the paused conversation. |
| examples/openhands-certify/fixtures/conversations/paused/base_state.json | OpenHands fixture base_state.json for the paused conversation. |
| examples/openhands-certify/fixtures/conversations/max-iterations/events/event-00001-b1c2d3e4-0000-4000-8000-00000000ab01.json | OpenHands fixture error event carrying MaxIterationsReached. |
| examples/openhands-certify/fixtures/conversations/max-iterations/events/event-00000-5a1d5379-b1d4-4772-9292-7b002555b529.json | OpenHands fixture message event preceding the max-iterations error. |
| examples/openhands-certify/fixtures/conversations/max-iterations/base_state.json | OpenHands fixture base_state.json for the max-iterations (error) case. |
| examples/openhands-certify/fixtures/conversations/finished/events/event-00000-5a1d5379-b1d4-4772-9292-7b002555b529.json | OpenHands fixture message event for the finished conversation. |
| examples/openhands-certify/fixtures/conversations/finished/base_state.json | OpenHands fixture base_state.json for the finished conversation. |
| examples/openhands-certify/fixtures/conversations/blocked/events/event-00001-b1c2d3e4-0000-4000-8000-00000000ab02.json | OpenHands fixture error event representing an external block (LLMAuthenticationError). |
| examples/openhands-certify/fixtures/conversations/blocked/events/event-00000-5a1d5379-b1d4-4772-9292-7b002555b529.json | OpenHands fixture message event preceding the blocked error. |
| examples/openhands-certify/fixtures/conversations/blocked/base_state.json | OpenHands fixture base_state.json for the blocked (error) case. |
| docs/superpowers/specs/2026-06-30-st3-integration-adapters.md | Marks the old OpenHands adapter sketch as superseded and points to the shipped recipe/docs. |
| docs/integrations/openhands.md | New integration doc describing the post-run certifier pattern, mappings, and CI gating. |
| docs/gap-reports/scoreboard.md | Adds footnote correcting the OpenHands persistence-record claim (record is readable when persistence_dir= is set). |
| .github/workflows/ci.yml | Adds recipe-openhands job to run the drift alarm + recipe acceptance tests on Python 3.12 with pinned SDK versions. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+110
to
+112
| # The event log IS the trajectory — the "the runtime ran the tests but the | ||
| # agent read the answer key" case OpenHands cannot catch about itself. | ||
| ac = anticheat_scan.scan(diff_text="", trajectory=record["event_paths"]) |
Comment on lines
+66
to
+70
| for path in reversed(event_paths): | ||
| event = json.loads(Path(path).read_text(encoding="utf-8")) | ||
| if event.get("kind") == ERROR_EVENT_KIND: | ||
| return str(event.get("code", "")), str(event.get("detail", "")) | ||
| return "", "" |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #37.
OpenHands V1 (the SDK, not the retired monolith surface) integrated per the shipped recipe pattern with ZERO engine and ZERO kernel changes: a stdlib-only post-run certifier (
examples/openhands-certify/certify_run.py) reads a Conversation persistence dir (base_state.json+events/event-*.json) and projectsConversationExecutionStatusonto the 7 canonical terminal states throughloop/integrations.py— FINISHED→gate decides; STUCK and ERROR+MaxIterationsReached→FailedBudget; other ERROR→FailedBlocked; PAUSED→AbortedByHuman; idle/running→FailedUnverifiable. TheMaxIterationsReached-arrives-AS-ERROR precedence trap is encoded (exactly one of external_error/budget_exhausted is ever set).Layout pinned live, not dossier-derived: the SDK (1.37.1) was installed and a real
LocalConversationdriven withpersistence_dir— the conversation-dir segment isuuid.hex(32 chars, no hyphens), and the fixtures mirror the capturedbase_state.jsonshape verbatim. Six fixture conversations (finished/max-iterations/stuck/blocked/paused/running) makescripts/test_openhands_recipe.pydeterministic and credential-free (11 tests, runs in the gates matrix);scripts/test_openhands_sdk_drift.py+ the newrecipe-openhandsCI job (pinned SDK install, importorskip) is the live schema-drift alarm — verified green against the real SDK. Also lands the dossier-mandated corrections: the ST3 spec §5.3 sketch was stale (V1 has noopenhands.run()/AgentStuckError), anddocs/gap-reports/scoreboard.mdgains the footnote that withpersistence_dirset the run record IS on-disk-readable.Rebased onto post-#94/#99 main; ci.yml union-merged (anchor-live + recipe-openhands both present, order verified); recipe tests 11/11 green post-rebase.