docs(fidelity): capability settings are recruiting decisions, not instrument settings - #500
Merged
Merged
Conversation
…trument settings I shipped 0.55.0 describing effort as a confound — "a participant who abandons at medium and finishes at high did not find a usability problem; the harness did." Daniel pushed back and he is right, so the text is corrected at the source rather than left to propagate. The error was applying two epistemic standards to two knobs of the same kind: persona treated as the participant, effort and model treated as the instrument. Both are things we set on a synthetic person before they touch the product, and per this document's own evidence the persona half is the more weakly grounded one (rule-file effects measured as largely content-independent). Calling effort "the harness" while calling persona "the participant" was special pleading. It also contradicted a frame this project already took: abandonment is a finding. So a lane that abandons at medium and completes at high produced two findings about two participants. The obligation that creates is to DECLARE and RECORD — which 0.55.0 does — not to hold the dial constant. Explicitly NOT claimed: that any (model, effort) pair corresponds to a real user population. Nothing here is calibrated to that, and a declared population says who was recruited, not whom they stand for. Corrects the same framing in reasoning-effort.ts, lab-config.ts, actor-contract.ts, schemas.md, and the effort-contrast-demo lab description.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
docs(fidelity): capability settings are recruiting decisions, not instrument settings
I shipped 0.55.0 describing effort as a confound — "a participant who abandons at
medium and finishes at high did not find a usability problem; the harness did."
Daniel pushed back and he is right, so the text is corrected at the source rather
than left to propagate.
The error was applying two epistemic standards to two knobs of the same kind:
persona treated as the participant, effort and model treated as the instrument.
Both are things we set on a synthetic person before they touch the product, and
per this document's own evidence the persona half is the more weakly grounded one
(rule-file effects measured as largely content-independent). Calling effort "the
harness" while calling persona "the participant" was special pleading. It also
contradicted a frame this project already took: abandonment is a finding.
So a lane that abandons at medium and completes at high produced two findings
about two participants. The obligation that creates is to DECLARE and RECORD —
which 0.55.0 does — not to hold the dial constant.
Explicitly NOT claimed: that any (model, effort) pair corresponds to a real user
population. Nothing here is calibrated to that, and a declared population says
who was recruited, not whom they stand for.
Corrects the same framing in reasoning-effort.ts, lab-config.ts, actor-contract.ts,
schemas.md, and the effort-contrast-demo lab description.