feat: add Tau terminal harness by nikhilachale · Pull Request #6493 · OrchestratorInc/agent-orchestrator

nikhilachale · 2026-10-09T23:29:40Z

Code changes: +757 -34
Tests: +1414 -5
Others: +5750 -3

Binary assets in the current PR diff: 14 (excluded from line counts).

Adds the pinned Tau terminal harness with native appended private instructions, preserved native/user/project rules, model selection and isolated lifecycle hooks. Chat, reviewers and cross-interface handoff are unsupported; explicit bypass permissions are required.

The actual AO audit exposed and fixed two defects: long multiline messages now reach Tau as one bracketed paste through the final admitted guard write, and exact restore now rejects missing/empty/corrupt/ambiguous/wrong-cwd native history instead of accepting metadata alone. Released-native CI exercises both boundaries. A fake-profile environment test and lint style issues found by CI are corrected for the new full run.

Draft: mixed live result. Current CI artifact e1ef49b12712b6c5fc17e595f931bc28296dd66f passed a GLM-4.7 ordinary fresh task with private/project/user actions, cwd, initial once and AGENTS preservation; executing-shell Escape, independent AO Kill/control isolation and missing-history rejection also passed. Exact native identity and new runtime generation restored, with NEW managed instructions confirmed by the supported read-only native SDK, but two ordinary restored tasks left the OLD private token: restored consumption FAIL. Current GLM-5.3-Flash partial fresh failure and earlier failures remain separate. The positive control changes task length/model; it does not identify the sole cause or erase failures.

Harness-owned audit report and real AO screenshots. Runtime source, native wheel/118-file fingerprint, serving PID/executable SHA, native-ID digest, generation and cleanup are recorded there. UI source is separately 3e80348c; later fixture/switch-style/evidence commits never relabel the old runtime binary. Controlled native/source payload role is system; authenticated third-party wire role was NOT_CAPTURED.

Validation is GitHub PR CI only; no routine VPS/Mac suites ran. All applicable current-head checks must finish before readiness. No per-harness report/JSON/PNG is added to PR6509.

github-actions · 2026-10-09T23:30:25Z

🏆 Review leaderboard

Oct 2–9, 2026 · UTC

Rank	Reviewer	PRs reviewed	Review rounds	PR comments
🥇	@illegalcall	27	45	10
🥈	@ronishrohan	19	22	16
🥉	@Vaibhaav-Tiwari	16	20	11
4	@Prasad-D-Ware	10	27	1
5	@axisrow	8	10	54
6	@Rishet11	7	17	2
7	@mohakchakraborty2004	7	7	1
8	@nikhilachale	6	14	3
9	@Pritom14	5	5	4
10	@codebanditssss	4	4	2

Ranked by distinct external PRs reviewed, then review rounds, then PR comments. Self-activity and bot activity are excluded.

Show 5 more reviewers

Rank	Reviewer	PRs reviewed	Review rounds
11	@neversettle17-101	3	4
12	@harshitsinghbhandari	2	4
13	@whoisasx	2	2
14	@AgentWrapper	1	3
15	@ApexYash11	1	2

nikhilachale · 2026-10-10T07:49:49Z

Evidence correction from the skill packaging review: the historical system_prompt_restore token was included in the initial standing instructions before the first turn. Its later appearance proves that token remained available after restore; it does not prove fresh instruction delivery or reinjection. The original 17/17 lifecycle result is a recorded runner result, not verified evidence for all 17 intended behaviors. Refreshed instructions remain unproven for this historical run. The reusable runner is being corrected to introduce a fresh token only after confirmed kill and before restore; fixture validation of that change is separate from real-provider validation, which has not been rerun. Original counts and artifacts below are retained for traceability.

Tau 0.4.7 — VPS audit report, 2026-10-10

Functional lifecycle: 17/17 PASS. CI on the audited commit: 24/24 PASS. Overall audit: BLOCKED. This is diagnostic lifecycle coverage; a full strict audit PASS is not claimed.

Audited commit: 189ca905dee3264ba5c682590ae04bcd8da88adf. Environment: Ubuntu 24.04 x86_64, the released native CLI, an isolated AO daemon/workspace/profile, and real Z.ai coding-plan calls using glm-5.3-flash. No tests, builds, lint, typechecks or provider execution ran on the user's Mac.

Gate totals: 25 PASS, 2 NOT_RUN, 1 BLOCKED. These are gate counts, separate from the 24 CI checks.

Local CLI

Gate	Result	Reason
local_binary	PASS	local executable resolved
local_version	PASS	local version command succeeded
local_integration	PASS	local integration probe succeeded
local_session_spawn	PASS	direct local CLI session produced the requested proof
local_models_list	NOT_RUN	local model-list command is not defined

AO / AOS11

Gate	Result	Reason
registered	PASS	agent is registered
installed	PASS	AO resolved the installed executable
fresh_probe	PASS	installation and authentication observations are fresh
authentication	BLOCKED	Tau has credentials configured, but AO could not verify them.
ao_models_api	PASS	AO model API listed 917 model(s)
tui_spawn	PASS	AO created the requested TUI session
proof_file_creation	PASS	AO_AUDIT_PROOF.json contains valid JSON
correct_working_directory	PASS	provider reported AO's session workspace
initial_prompt_exactly_once	PASS	unique initial token was handled once
hidden_ao_instructions	PASS	provider consumed hidden AO standing instructions
project_agents_md	PASS	provider consumed project AGENTS.md
activity_status	PASS	AO exposed ready status and an active→idle transition
second_message	PASS	second API message produced the requested provider-authored mutation
native_session_id	PASS	AO persisted a provider-native session identity
cancellation	PASS	AO terminal mux sent configured input to the observed active session and the same terminal settled
termination	PASS	kill was acknowledged and termination was confirmed
native_restore	PASS	AO restored in native mode with the exact provider-native ID
same_ao_session_workspace	PASS	restore retained the AO session ID and workspace
post_restore_terminal_ready	PASS	the restored native terminal displayed the configured ready cue on the same terminal generation
post_restore_message	PASS	post-restore message produced a provider-authored mutation
history_file_continuity	PASS	native history recall and original proof-file continuity were both observed
system_prompt_restore	PASS	restore-only hidden standing instruction was observed after restore

Cross-check

Gate	Result	Reason
model_catalog_consistency	NOT_RUN	both local and AO model catalogs are required for comparison

AO reports credentials as configured, not provider-verified. Real Z.ai calls succeeded, so this metadata gap is not evidence of an expired login. Tau's provider TSV must not be counted as one model ID per row; native model listing/exact catalog comparison remain NOT_RUN.

The live audit reproduced an ANSI-split composer marker causing a false startup timeout. The readiness fix strips ANSI before matching. VPS regression, session-manager/Tau race suites and pinned released-TUI conformance passed. Earlier renamed-binary hook-path failures and premature post-restore sends were test setup/runner failures; they are preserved separately. Final attempt 6 passed after using the canonical AO executable and waiting for the native composer.

The original strict skill remains unchanged. The separate diagnostic runner continues lifecycle testing only after successful direct native proof and AO installation/freshness checks; it retains the original non-passing auth/model gates. The runner revision used here passed 80/80 fixture tests; subsequent generic improvements passed 91/91.

Audited-head backend/aggregate CI · Pinned released-native conformance

Cleanup confirmed: the created session was terminated and the isolated daemon/native PTY processes stopped. A final audit-wide process scan found no remaining owned test processes. Credentials and raw native IDs are omitted here. Previous failed attempts and controlled comparisons remain preserved in the audit bundle. Real terminal text was captured; screenshots, Electron UI and native non-Linux harness coverage are not claimed.

Evidence identity

Final attempt: tau-diagnostic-attempt6
Source report: tau-diagnostic-attempt6/report.json
Original JSON report SHA-256: d4a17a7476d3f66c4db32fb731309695c2377d4cb93e479fa01f608ce611c17e
Audited AO binary SHA-256: ba1d3a51bd48bdf9d89d4e59dcdfa33449f86b72bf3bd0de6923888870187515
Diagnostic runner SHA-256: 09070c3086b73d8a5d54b28ab07697dda06b67b23bc3c1ddea4555f7144ec741
The retained export includes sanitized report copies, prior attempts, fixture logs, review findings and exact-head CI evidence; export copies hash native/opaque identities and retain original-file hashes.

nikhilachale · 2026-10-10T09:01:30Z

Evidence correction from the skill packaging review: the historical system_prompt_restore token was included in the initial standing instructions before the first turn. Its later appearance proves that token remained available after restore; it does not prove fresh instruction delivery or reinjection. The original 17/17 lifecycle result is a recorded runner result, not verified evidence for all 17 intended behaviors. Refreshed instructions remain unproven for this historical run. The reusable runner is being corrected to introduce a fresh token only after confirmed kill and before restore; fixture validation of that change is separate from real-provider validation, which has not been rerun. Original counts and artifacts below are retained for traceability.

Tau 0.4.7 — failed-attempt report, 2026-10-10

The original failures remain preserved even though the final diagnostic run passed all 17 lifecycle gates. The full strict audit remains BLOCKED: AO reports configured credentials, and native model discovery/comparison are NOT_RUN. See the final gate-by-gate report.

All times below are UTC on 2026-10-10. Testing ran on the Ubuntu VPS, with real Z.ai calls; no local Mac checks were run.

Attempt / start	Observed failure	Cause and resolution	Preserved evidence
Strict 1 / 05:44:26	Authentication BLOCKED; dependent lifecycle gates NOT_RUN.	AO's configured-only observation does not prove provider-verified auth. Successful direct calls allowed separately labeled diagnostic testing; the strict result was not rewritten.	`tau-strict-attempt1/report.json`
Diagnostic 1 / 05:53:03; 2 / 06:01:58	TUI spawn failed: HTTP 409 `SPAWN_TIMEOUT`. In attempt 2 the call took 20.978 s.	Integration bug: the visible `Ask Tau…` placeholder was split by cursor SGR styling. Raw-string readiness matching missed it. Fixed by stripping ANSI before matching in `189ca905`. The earlier Ring.Tail hypothesis was disproved.	`tau-diagnostic-attempt{1,2}/report.json`; attempt 2 `diagnosis.json`, `terminal-capture/003-plain.ansi`, `003-styled.ansi`, `raw-terminal.ansi`
Diagnostic 3 / 06:20:35; 4 / 06:24:54	Spawn remained pending; native startup hooks could not invoke `ao`. Downstream lifecycle gates were NOT_RUN.	Test launcher defect: the prebuilt executable had a noncanonical basename, so native children lacked `ao` on PATH. A symlink still resolved the renamed executable. A canonical executable/hardlink named `ao` fixed the fixture; this is not evidence of a production startup race.	`tau-diagnostic-attempt{3,4}/report.json`, `source.json`, `diagnosis.json`
Diagnostic 5 / 06:27:00	After exact-ID restore, send returned HTTP 409 `SESSION_STARTUP_PENDING`; post-restore execution, history and refreshed instructions failed to be proven.	Runner readiness defect: restore returned 06:28:08.015; send failed 06:28:08.043; native startup hook succeeded 06:28:09.508. Runner now waits for native readiness before sending.	`tau-diagnostic-attempt5/report.json`, `diagnosis.json`
Earlier diagnostic model comparison	Catalog comparison reported FAIL / no shared model IDs.	Invalid model-list fixture: `tau providers` TSV rows were interpreted as 29 model IDs. This was not a valid model-catalog comparison. Unsupported parsing was removed; final discovery/comparison remain NOT_RUN, not PASS.	Original diagnostic reports; `attempt-history.json`
Diagnostic 6 / 06:39:13–06:42:34	No functional lifecycle failures.	All 17 lifecycle gates passed with canonical binary and readiness-aware runner. Gate totals: 25 PASS, 1 BLOCKED, 2 NOT_RUN. All 24 CI checks passed on `189ca905dee3264ba5c682590ae04bcd8da88adf`.	`tau-diagnostic-attempt6/report.json`; `final-ci.json`

Representative original error:

POST /api/v1/sessions -> 409
code: SPAWN_TIMEOUT
message: Session spawn timed out before the agent could start
requestId: vps-linux/vbrD5uxUux-000010

The original native marker was A\x1b[0m\x1b[38;2;111;114;117;48;2;16;20;25msk Tau…. Of 25 plain samples, zero contained the literal marker and 23 contained it after ANSI stripping. This is preserved diagnostic evidence, not a screenshot.

Remaining gaps: configured credentials are authorized for this audit but still not classified as provider-verified by AO; this is not an expired-login finding. Local model listing and exact catalog comparison were not executed with a valid contract.

Screenshot coverage: the historical attempts have real terminal text/ANSI captures, but no PNG screenshots. No rendered log or recreated error image is claimed as an original screenshot.

Evidence paths above are relative to the preserved VPS report directory /home/azureuser/.ao/audits/harness-20261010/reports/. A sanitized report export is retained locally under ao-agent-harness/reports/vps-2026-10-10/, including attempt-history.json; raw terminal captures remain on the VPS. These paths are provenance references, not public downloadable GitHub attachments. Original failed/partial artifacts were not replaced by final passing results.

nikhilachale · 2026-10-10T09:56:15Z

The reusable integration/testing skill and consolidated session report are now in #6509.

The report includes two redacted, actual screenshots from the new Open Interpreter early-send reproduction; they are published repository evidence, not historical screenshots of Tau or Neovate. Original captures remain unchanged.

The historical instruction-refresh correction remains explicit: the old system_prompt_restore token was available before the first turn, so the recorded 17/17 result does not prove reinjected/refreshed instructions. The packaged runner now creates and installs a fresh token only after confirmed kill. Its exact commit passed 98/98 fixture tests on the Ubuntu VPS; the strengthened instruction gate has not been rerun against real providers. Auth/model gaps and the observed early-send failure remain unresolved in the historical integration results.

nikhilachale · 2026-10-10T18:35:52Z

Fresh current-runner follow-up, all computation and capture on the authorized VPS.

Tau 0.4.7 / Z.ai glm-5.3-flash / TUI / explicit bypass only: tau-002 finished at 2026-10-10T18:30:55.127Z with 25 PASS, 0 FAIL, 1 BLOCKED, 2 NOT_RUN. Functional lifecycle passes; strict overall BLOCKED / diagnostic-only. Auth remains configured, not verified. Native model listing and native/AO comparison remain NOT_RUN. No login, credential changes, or contract relaxation.

The fresh restore-only instruction was introduced through the project API after confirmed kill, read back, then consumed in the restored mutation. Same native ID/workspace, initialized empty composer, model/workspace cue, post-restore mutation and history continuity passed. Native digest a3f87013b77caef5c8860009aea4a55d567173d68e86562b9bed16afd4d8fdad; AO session audit-tau-0fb37380-1. Cancellation is observed active-turn Escape→idle (~0.25s), not proof of already-running subprocess reaping. The restored transcript says the 120-second wait completed; it did not establish cancellation of that shell.

Preserved tau-001: 21 PASS, 4 FAIL, 1 BLOCKED, 2 NOT_RUN. Provider-authored JSON acquired an extra closing brace on the second mutation; second_message, post_restore_message, history_file_continuity and system_prompt_restore failed. No file repair; fresh retry used identical source/pin/profile/contract. Original gate-time screenshot unavailable (API audit preceded GUI); the first image below is a later retained-session capture with its own timestamp. Historical failures remain unchanged. Historical 17/17 labels did not prove fresh instructions; native mirrors did not prove real AO UI.

Tested functional source 189ca905dee3264ba5c682590ae04bcd8da88adf; AO binary SHA-256 4c82435bec3bac6f08b1073d416c683b4035e20721b3d1b6d345a2a09216d4ef; UI source same. Runner SHA-256 8887aa2e7101b9e6d3950bfca08ffddf2e1e6c1c3f1d030c8dec2676443aecea. Evidence-only head b98a1bfb4ebad89a0e6501de204623ab69bb4bb2. Original exact source 24/24 CI successful. Final evidence-head checks tracked separately; no unnecessary full backend race rerun or frontend tsc/OOM retry.

Actual AO Electron, native bridge confirmed, exact target session selected, screenshots inspected for secrets/relevance. Supervisor reopened the same isolated data with the identical verified binary after stopping finalized runner; Tau-002 native digest and restored generation reconfirmed. Tau-001 records retained-session recovery and audit identity, but no explicit capture-time digest/generation recheck; it is failure-state UI evidence. PNGs are actual scrot/Xvfb captures, not terminal mirrors.

Failure capture UTC 2026-10-10T18:26:18.392600Z, PNG SHA-256 6ca864dc3d6fadb1e321f37245bc9d774f103c791826a4b1e8a4b5d4b05bfa56.

Success capture UTC 2026-10-10T18:32:59.570527Z, PNG SHA-256 1902fbcc696799f3b6cf36d80041e9b7248d35acb93d17cfc988833d7c841f69. Curated gate results, full provenance, native fingerprints and cleanup. Both owned sessions API-killed/termination confirmed; exact Electron/current run-file daemon executable verified before TERM and gone. Original reports/profiles/provenance binaries preserved; unrelated AO state untouched.

Final evidence-head CI verified at 2026-10-10T18:57:28.374670+00:00: 24/24 checks successful on b98a1bfb4ebad89a0e6501de204623ab69bb4bb2. Exact-head check example. Original source CI and final evidence CI are separate records; these are remote workflow results.

nikhilachale

[P2] Fence composer readiness to the current Tau launch

PromptReadinessHints uses the visible Ask Tau… marker, but waitForPromptReadiness obtains it from Runtime.GetOutput; the tmux implementation captures the last 80 lines with capture-pane -S, and Runtime.Restart explicitly preserves terminal history. When AO performs a fresh fallback/recovery on an existing Tau handle—for example, a live/exited pane whose native session ID was never captured—the prior Tau composer marker is still in scrollback. The very first poll can therefore report ready before the replacement Tau process has mounted its composer, and deliverAfterStartPrompt can paste the saved task into the initializing process, where it may be dropped.

Please fence readiness to the current runtime launch (for example, require the current-generation session_start signal promised by FirstSignalProvesInputReady in addition to the marker), or baseline/clear old pane output before matching. A focused regression test should return stale Ask Tau… output immediately after restart and prove delivery waits for current-launch readiness.

nikhilachale · 2026-10-12T16:52:14Z

Harness-specific reports, JSON receipts, screenshots and provenance now live on this integration PR branch, rather than the skill PR #6509.

Evidence index with embedded screenshots and JSON links.

Historical successes, failures and scope limitations remain as recorded. Moving these files does not rerun native/AO testing or upgrade any verdict.

This is evidence relocation only. Existing tested source/native hashes and failures are preserved; the documentation commit is not a fresh runtime pass. Applicable checks run in PR CI.

nikhilachale · 2026-10-12T18:07:26Z

Canonical main conflicts reconciled at 726047d23be0, preserving previous source/evidence ancestry and every existing harness report/screenshot byte. Reconciliation receipt.