Every skill declares what success means, and nothing grades itself

[rookslog]

The gap

We test that skills install. We never test that they work (#21). Nothing
observes a skill after a user installs it, so every improvement to a shipped
skill is guesswork about its effect.

Success means something different for each skill. A conformance skill succeeds
when lint passes. A verbosity skill succeeds when signature density falls
AND comprehension holds. No single metric can say whether "the skills work",
and a single metric made into a target corrupts what it measures. Goodhart's
law states the first half. Campbell's law states the second: the more an
indicator drives decisions, the more it corrupts both itself and the process
it monitors.

The requirement structure

Three requirements, and the second and third are what keep the first honest.

1. Every skill declares. The contract is required, and absent is refused.

Each skill carries a measurement contract, one per skill, parallel to its
grounding matrix:

• the construct the skill claims to change
• the observable measures for it
• the counterweight measure the loop must not trade away
• gradeable-by: machine, human, or a declared pair
• the falsifier: what result would retire the skill

The honest starting state is declared, not measured: a new skill ships
"judgment-graded, no evidence yet". A skill that cannot state what success
would look like is a review finding, not a formality.

An absent contract reads as refused, never as "unmeasured". A soft default
turns the rule off for whoever forgot the file, which is the same reason an
absent quotation declaration reads as forbidden.

CI checks the shape: the contract exists, every measure names its
counterweight, gradeable-by is stated. The content is a person's decision,
like an audit date. The contract is an audit record for a person, so it lives
beside the grounding matrices, outside skills/. It must not ride into an
installed tree.

2. No skill collects on its own. One shared layer does.

Collection machinery is one component in the tool, parametrized by each
skill's contract. Per-skill machinery is N copies of the same code drifting
apart, the defect this repository already caught when two spawns carried two
allowlists.

Collection is opt-in and off by default. The data stays on the user's
machine. Whether anything is collected at all is the user's switch, never the
skill's. A user shares evidence by promoting a field report deliberately,
with provenance (#57). There is no background upload, and there never will
be.

3. The loop closes only through promoted studies.

Raw telemetry may propose a skill edit. Only a promoted study under the
measurement design may dispose of one. Outcomes are derived from retained
bytes, never asserted by the thing that produced them, and the owner's merge
is what ratifies (the ADR-0024 pattern).

The gradeable-by declaration decides the cadence. A machine-gradeable skill
can close its loop cheaply and often. A judgment-graded skill closes only
through studies with human reads, and that is the point: the declaration is
the skill stating on the record which grades an automated loop is not
licensed to give itself.

The Goodhart guard, in one place

• Measures come in pairs. A target with no counterweight is refused.
• The metric is never the actuator. An anomaly escalates to a human audit,
  never to silent recalibration.
• Audits are exogenous and unannounced. The loop must not know the calendar.

Parts already filed

#21 (efficacy), #43 (installed activation), #56 (status derived from
evidence), #57 (field reports carry provenance). This issue is the loop that
connects them.

Before any code

This decision touches the measurement design, consent, and data locality. It
needs a spec, as an amendment to
docs/specs/2026-08-04-measurement-design.md, and an ADR for the consent and
locality decision. The pull request that lands either records the ADR in the
same pass.

[rookslog]

Log-study input to this design (2026-08-13, unpromoted local study)

Operator transcript corpus, 3,887 files, ~92k API requests, priced at list rates. Findings that bear on this issue:

• User-facing prose is ~3% of list-priced spend. Context volume — cache reads (~half) and cache writes (~quarter) — dominates. Output tokens overall are ~a fifth, and within them user-visible text is ~14% (upper bound, estimated by character share).
• Consequence for the per-skill contracts here: for the verbosity skills (de-slop, compressed-deliberation, proportionate-execution), comprehension and user experience are the primary construct, and cost is the counterweight — not the reverse. The cost case for prose skills failed its own measurement.
• The dominant cost levers are delivery-side: what enters context (skill loading, tool results) and how long sessions run. skills: context-frugal skill discipline, and a lint that counts what a skill costs #104 (context-frugal skills) is the skill-shaped piece of that.

Observability gaps — what must be collected vs what is inferable

Inferable from existing transcripts, no new collection: per-request usage, per-skill invocation cost (join skill dispatches to cache-write spikes), session length distributions, model mix, subagent share, review output length.

Genuine gaps, not inferable beyond reasonable doubt:

1. Comprehension failures — nothing records that a reader had to re-read or re-ask. Needs a deliberate capture point (an on-request plain-language sidechat whose invocation rate is itself the metric, or manual signal capture). Anything else is self-grading.
2. Counterfactual triggering — "the skill should have fired and did not" appears in no log by construction. That is a trigger-eval experiment, not collection.
3. Behavioral attribution — whether a loaded skill changed the output needs A/B arms. The bench is that instrument; production logs cannot substitute for it.
4. Decision-time cost visibility — an agent cannot see a skill's size before invoking it. Computable offline as a size-annotated index; not visible in-session without harness support.

These gaps bound what the collection layer in this design should promise: it can carry 1 (as opt-in capture events) and the inferable joins, and must not claim to answer 2 or 3, which belong to the bench.