feat(labs): humanish studies humanish, and fixes the two things it found (#495) - #501
Merged
Conversation
…und (#495) labs/first-contact.yaml — a real autonomous terminal agent lands in a fresh E2B shell, finds humanish from public surfaces only, and tries to get somewhere with it. The mission deliberately names no command: whether the participant discovers `init`, `doctor`, or `tui` IS the measurement, and a mission that listed them would answer the question by asking it. No E2B key is passed, so the participant cannot start a nested study — which is also the study condition, a developer on day one before any keys are set. Ran it live. It installed 0.55.0, scaffolded, ran a four-persona dry study, verified `share_ready`, served the Observer, drafted a GitHub issue, spent nothing, and reported back like a colleague: "promising enough for a controlled pilot, but I would not broadly adopt it until completing a capped live study." Two findings, both fixed here. - THE HANDOFF FAILS. `doctor` DID surface the TUI — discovery worked. The agent read "`humanish tui` is available in an interactive terminal", correctly concluded it was not in one, and dropped it: its report, written FOR A HUMAN WITH A TERMINAL, never mentioned the human surface at all. A capability described to a reader who cannot use it reads as "not for you" unless it is phrased as something to pass on. The row now knows who is reading, and terminalSurfaceMessage() is a pure function so both audiences are testable. - A REJECTED FLAG NOW NAMES ITS OWNER. The participant reached for `humanish run --no-open` by analogy with `lab run`, got a bare "unknown option", and filed it as a documentation mismatch. The flag really is absent (`run` opens nothing), but "unknown" answers the wrong question. Every command now points at the siblings that declare it, stays silent when none do, and reports truncation instead of quietly dropping owners. The refusal path this lab was built to test is still unmeasured: the agent never ran `humanish tui`, because doctor's old phrasing had already closed the question.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
feat(labs): humanish studies humanish, and fixes the two things it found (#495)
labs/first-contact.yaml — a real autonomous terminal agent lands in a fresh E2B
shell, finds humanish from public surfaces only, and tries to get somewhere with
it. The mission deliberately names no command: whether the participant discovers
init,doctor, ortuiIS the measurement, and a mission that listed themwould answer the question by asking it. No E2B key is passed, so the participant
cannot start a nested study — which is also the study condition, a developer on
day one before any keys are set.
Ran it live. It installed 0.55.0, scaffolded, ran a four-persona dry study,
verified
share_ready, served the Observer, drafted a GitHub issue, spentnothing, and reported back like a colleague: "promising enough for a controlled
pilot, but I would not broadly adopt it until completing a capped live study."
Two findings, both fixed here.
doctorDID surface the TUI — discovery worked. The agentread "
humanish tuiis available in an interactive terminal", correctlyconcluded it was not in one, and dropped it: its report, written FOR A HUMAN
WITH A TERMINAL, never mentioned the human surface at all. A capability
described to a reader who cannot use it reads as "not for you" unless it is
phrased as something to pass on. The row now knows who is reading, and
terminalSurfaceMessage() is a pure function so both audiences are testable.
humanish run --no-openby analogy withlab run, got a bare"unknown option", and filed it as a documentation mismatch. The flag really is
absent (
runopens nothing), but "unknown" answers the wrong question. Everycommand now points at the siblings that declare it, stays silent when none do,
and reports truncation instead of quietly dropping owners.
The refusal path this lab was built to test is still unmeasured: the agent never
ran
humanish tui, because doctor's old phrasing had already closed the question.