feat: add harness integration and lifecycle audit skill

[nikhilachale]

Code changes: +2519 -0
Tests: +1869 -0
Others: +1479 -0

Adds an AO contributor skill for assessing coding harnesses, implementing supported integration surfaces, and auditing native CLI/AO lifecycle behavior. The API-first runner preserves failed, blocked and not-run gates, including fresh instruction delivery, cancellation and exact restore.

Harness-specific audit reports, JSON receipts, screenshots and provenance must be published in comments and GitHub attachments on each harness's integration PR; new evidence files must not be committed. This PR contains reusable skill guidance, the runner, fixtures and CI wiring. Version-specific investigation narratives and the dated session index are excluded from its file diff; general integration and evidence requirements remain.

Historical index and per-harness evidence remain preserved at the pre-cleanup revision and through the respective integration PRs; existing evidence-file migration requires verified attachments before removal.

Operational guidance now includes a compact Prepare/Launch/Doctor/Drive/Evidence/Cleanup checklist and a journey map. It maps actions to proof prerequisites and observed results, adds read-only instance checks, identifies separate UI/Chat and independent-instruction coverage, and requires evidence to survive teardown.

The ACP/shared-Chat-UI guide maps the native binding, registry, typed contracts, daemon events and shared renderer. Its focused Chat checklist covers real UI creation, streaming, history/load, decisions, cancellation, reconnect, daemon recovery and exact restore. Chat Doctor uses controller/snapshot readiness rather than TUI composer cues; this adds guidance, not Chat automation to the runner.

Integration handoff now requires a completion matrix covering product registration/UI, model selection before launch, setup, OS/config readiness, branding, interface scope, permissions/instructions, lifecycle, evidence, and current CI/merge state. Each row records supported scope, verification status, and proof or an omission reason. Current-status receipts separate runtime-tested builds from the current CI head, preserve sanitized reproduction inputs and distinct attempts, and reconcile stale claims/counts before handoff. These are contributor review gates; the runner schema and automation are unchanged.

The skill now requires a per-harness, per-attempt screenshot index for every meaningful UI action and result: Agent Harness listing, install offer/progress/result, login/configuration or stored-credential state, Agent Harness models, New Task agent/model selection, running/output, follow-up and active-command cancellation, Kill/archive confirmation, archived entry, exact restore, and the same card back in Kanban with a fresh continuation. Supported Chat also requires real shared-UI streaming, decisions, cancellation, reconnect and recovery evidence. Each checkpoint links actual images and functional receipts with build, identity and capture provenance; failures retain original screenshots or explicit capture-unavailable reasons plus sanitized logs. Images are embedded in journey order with durable, verified links on the harness integration PR.

Already installed/connected does not prove a fresh install/login. The accepted saved-credential route can qualify after fresh verified authorization and a successful provider turn; its unexercised fresh-login alternative remains explicitly labelled without blocking that route. Manual-only plans retain an explicit unexercised in-page step and can qualify only agreed manual scope. A planned/absent Agent Harness model panel remains NOT_RUN / NOT_IMPLEMENTED; New Task models cannot substitute. Real credentials must not be deleted to stage captures. Later reproductions cannot replace originals. Screenshots prove visible states, while native identity, provider authorization, cancellation and restored continuation require independent functional proof. These are contributor gates; the API-first runner still captures failures only and does not automate successful-step screenshots. Detailed evidence belongs in PR comments/attachments; the description links to them. Existing committed evidence is removed only after its originals and published destinations are verified; source logos/licenses and synthetic fixtures remain source files. Isolated candidate registration enables real Chat qualification before production registration.

Validation: documentation checks and independent evidence-policy exercises passed; runner/tests/workflow are unchanged. Current-head CI is pending. Detailed verification and preserved failure receipts are in the PR comment. No live harness/UI qualification is claimed.

Historical run details and failures remain in the linked integration reports and earlier discussion. This contributor tooling does not add a desktop scheduler, automatic login or automatic PR publisher.

How it works

flowchart TD
  A["Read the skill<br/>SKILL.md and reference guides"]
  B["Prepare the agent integration<br/>and its audit configuration"]
  C["Run the audit<br/>audit_ao.mjs"]
  D["Check startup, instructions,<br/>cancellation and restore"]
  E["Save results and evidence<br/>audit_ao.mjs + audit_evidence.mjs"]
  F["Review passed, failed<br/>and blocked checks"]
  T["Tests + GitHub Actions<br/>Check that the audit tools work"]

  A --> B --> C --> D --> E --> F
  T -. validates .-> C
  T -. validates .-> E

Loading

Repository docs point contributors to the skill. The historical index above links to each agent's detailed evidence outside this PR's file diff. Tests validate the audit tools; a live audit checks the actual agent.

[github-actions]

🏆 Review leaderboard

Oct 3–10, 2026 · UTC

Rank	Reviewer	PRs reviewed	Review rounds	PR comments
🥇	@illegalcall	28	47	11
🥈	@ronishrohan	20	23	17
🥉	@Vaibhaav-Tiwari	16	20	11
4	@Prasad-D-Ware	10	27	1
5	@nikhilachale	8	20	3
6	@axisrow	8	10	48
7	@Rishet11	7	16	2
8	@mohakchakraborty2004	7	7	1
9	@Pritom14	5	5	3
10	@codebanditssss	4	4	2

Ranked by distinct external PRs reviewed, then review rounds, then PR comments. Self-activity and bot activity are excluded.

Show 6 more reviewers

Rank	Reviewer	PRs reviewed	Review rounds	PR comments
11	@Annieeeee11	3	6	1
12	@harshitsinghbhandari	2	4	0
13	@neversettle17-101	2	3	0
14	@whoisasx	2	2	0
15	@AgentWrapper	1	3	0
16	@ApexYash11	1	2	0

[nikhilachale]

Updated the screenshot evidence requirement in bf6a30d.

• Every harness/interface claimed as running inside AO now requires an embedded real AO desktop/web UI screenshot showing that session and its native output, with AO/native identity, native version, tested SHA/binary hash, UTC timestamp, capture method/path/hash, and a durable GitHub-renderable image URL.
• Failures keep explicit FAIL/BLOCKED/NOT_RUN labels and their actual relevant failure capture when available; unavailable screenshots retain the reason. Missing AO UI captures or attachments leave AO UI evidence: BLOCKED, even if the API runner passes. Functional logs/assertions remain necessary for cancellation and exact restore.
• Historical screenshots in this PR remain live terminal mirrors, not proof of execution inside the AO UI. They and the original failed attempts are unchanged. This documentation update makes no new live integration claim.

Validation on the VPS: git diff --check, all changed relative links/heading anchors, and the exact three-Markdown-file scope passed. No runner code changed; the previous 98-fixture result was not rerun or represented as a new result. No Mac compute or new provider run. Current-head GitHub CI is in progress; PR category counts were recalculated from GitHub per-file diffs (including 2 unchanged binary screenshot assets in the overall PR).

[i-trytoohard]

From the NotesBot call on 2026-10-11, this existing PR appears to be relevant.
Match confidence: 0.90 (author plus semantic: Nikhil's harness skill PR, review status given on the call).

Relevant discussion excerpts:

• [38:39] Dhruv Sharma: "With Nikhil, it's agent harness. He made a skill PR. So I'll have to review that and then harness page updates."
• [39:22] prateek: "Meraj will be working with Nikhil on the harness stuff right now."
• [39:41] Meraj: "He has given me one PR. I need to comment as well, Dhruv bhaiya, the PR."
• [39:41] prateek: "My expectation is by the end of the week you understand everything that's going on and have opinions on how to make our processes or our harness integrations better."

Source thread: https://discord.com/channels/1476302178913357958/1558894919202906274/1558894919202906274

Generated by AO Bot based on the transcript from NotesBot.

[nikhilachale]

Reusable guidance updated at def16c285aaa: independent standing-instruction controls. Ordinary tasks must not ask to read hidden instructions or reveal their paths/tokens; assisted runner gates are labeled separately. The guide also distinguishes user-authored sends from coordination, configured prompt presence from consumption/provider wire role, actual guarded multiline paste, missing-history validation and real Electron graphics fallback.

This commit contains guidance only. All harness JSON/reports/screenshots remain on their own integration PRs. Runner/fixtures are unchanged; current-head CI is pending, and earlier 150/150 success remains historical.

[i-trytoohard]

From the Oct 12 backend-pod call: Nikhil Achale said that in the last two days he made a skill for automatic harness integration and raised a couple of PRs with it, that the "vepl" harness is now working, and that harness sessions can already run in AO. This PR (harness integration and lifecycle audit skill) is directly that work product, so the call's updates and prateek's request for a screen recording apply here.

Related ask from the same call: Nikhil is to post a screen recording of the revamped harness page to the YouTube channel, and prateek's broader ask was that error states get native "report bug" and "trigger bug triage" actions.

Confidence: 0.66 (fuzzy semantic match, same workstream; Nikhil's skill-based harness integration PRs)

Excerpt (Nikhil Achale, 4:12): "...I was working on the automatic integration part of the harnesses, for that I also made a skill and in last two days raised a couple of PRs using that skill..."

Source: https://discord.com/channels/1476302178913357958/1559250521251127377

[nikhilachale]

Skill validation — comment-only evidence policy (396eb5d3b0f30b23cf83fd6f5a4ff3afc3bd4a50)

• Guidance now publishes audit reports, matrices, checkpoint images, logs and JSON receipts in the harness PR's comments and GitHub attachments. The source description links to the current evidence comment.
• New audit outputs stay outside the Git checkout. Product logos/licenses, source documentation and reusable synthetic fixtures remain source files.
• Historical committed evidence must be preserved, published/uploaded and verified before its removal from the current PR diff; no history rewriting or loss of original failures.
• Static validation: whitespace/diff checks and all 52 local documentation links/anchors passed. Independent baseline/forward exercises verified that committing raw-image/report URLs is rejected, failed uploads remain BLOCKED, migration preserves originals before removal, and product assets/fixtures are retained.
• This change edits four Markdown files only. Runner/tests/workflow bytes are unchanged. No local test/build/provider/live UI run was performed. New screenshot evidence is not claimed by this documentation review.
• Current-head CI: pending. Prior head 5af6c2306250c7b2ada2563af5247e2d2c7eddf0 completed all five checks. Its original fixture failure remains preserved: 149/150 passed and the existing 120 ms timeout fixture failed its timedOut assertion. The unchanged full rerun passed 150/150. Those are historical CI receipts, not live harness proof.