release: 0.55.0 — the effort a run used is now a decision, and part of the evidence - #499
Merged
Conversation
…f the evidence
Reasoning effort was reachable in the provider and unreachable from a lab, so
every computer-use run humanish had ever done was the provider default — by
omission, not by decision. A participant who abandons a flow at medium and
finishes it at high did not find a usability problem; the harness did.
- `actors[].reasoningEffort`, overridable per lane, validated against the
documented vocabulary. An unsupported level fails on the first turn with the
provider's own message rather than being silently downgraded into a trace
that would then misreport what ran.
- `ActorTrace.modelSettings` (humanish.model-settings.v1) records the effort
the request ACTUALLY carried, default included.
- The lab screen and the fan-out plan line show it, and the plan line is what
you read before spending money.
- labs/effort-contrast-demo.yaml — one persona, one mission, two efforts.
gpt-5.6-sol stays the default: Artificial Analysis, who ran OpenAI's pre-release
eval, reports Luna and Sol always on the Pareto frontier ahead of Terra.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
… effort (#497) A real run found the gap this closes: the trace recorded medium while the lab looked like it declared high. That one was my test setup, not the code — but the link it exercised is the one no unit test could see and only a paid run could check. Two lanes, one persona, one mission, different efforts, real orchestration on the fake substrate at $0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
release: 0.55.0 — the effort a run used is now a decision, and part of the evidence
Reasoning effort was reachable in the provider and unreachable from a lab, so
every computer-use run humanish had ever done was the provider default — by
omission, not by decision. A participant who abandons a flow at medium and
finishes it at high did not find a usability problem; the harness did.
actors[].reasoningEffort, overridable per lane, validated against thedocumented vocabulary. An unsupported level fails on the first turn with the
provider's own message rather than being silently downgraded into a trace
that would then misreport what ran.
ActorTrace.modelSettings(humanish.model-settings.v1) records the effortthe request ACTUALLY carried, default included.
you read before spending money.
gpt-5.6-sol stays the default: Artificial Analysis, who ran OpenAI's pre-release
eval, reports Luna and Sol always on the Pareto frontier ahead of Terra.