Keel impact score — a derived, evidence-first measure of Keel's per-session leverage#53
Merged
Merged
Conversation
An optional, evidence-first mechanism to score how much Keel actually shaped a session, turning the wrap-time promote/demote gut check into a tracked number plus the evidence behind it. - commands/keel-score.md: 0-100 five-axis rubric (convention adherence, retrieval hit-miss, rework avoided, guardrails fired, command leverage) under three honesty rules — no point without a citation, score the counterfactual delta, subtract a friction penalty (help - friction). - tools/keel-impact.sh: the deterministic half — date stamp, row append, rolling trend — over an append-only ledger. Model judges, script counts (same split as doctor). Interval-free awk regex for busybox parity. - docs/keel-impact.md: the ledger, seeded with its schema/header. - commands/wrap.md: optional step 7 wiring it into the session-wrap ritual. - tests/test_keel_impact.sh: 21 assertions (append, parse, rollup, range validation, table-safety escaping). - docs/loading-and-cost.md: honest command-size band bumped 1,070 -> 1,200 to match the new max; CHANGELOG entry. Portable by design (Markdown command + POSIX-ish Bash, not a Claude-only skill) so it works under any AI assistant, like the rest of Keel. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
Reground the mechanism on honesty: the model no longer picks a 0-100 number. It enumerates counted, cited EVENTS and the tool derives the score by a fixed formula, so the marketing number is a pure function of the evidence and cannot be inflated by vibe. - tools/keel-impact.sh: `add` now takes event counts (--guard/--fire/--hit/--miss/--friction/--silent), not --score, and computes score = round(100*HELP/(HELP+COST)), HELP=3*guard+2*fire+hit, COST=2*miss+2*friction. No events -> "—" (nothing to measure), never a fake 0. Records a confidence tier from the event count; keeps `silent` (unused always-loaded rules = demote candidates) out of the score. Rollup skips "—" from the mean and reports the honest cumulative signals (total guardrail fires, total retrieval misses). - commands/keel-score.md: rewritten around "you do not pick the score" — gather typed events each owing a concrete artifact; documents the only real counterfactual (an occasional A/B, Keel vs cold) as the ground truth these self-reported scores estimate. - docs/keel-impact.md: new derived-score schema + formula in the header. - commands/wrap.md: step 7 reworded (count events, tool derives). - tests/test_keel_impact.sh: 29 assertions pinning the derivation, confidence tiers, "—" handling, rollup aggregates, and validation. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
Push the most objective input to the score — a guardrail actually firing — out of the model's hands and into a deterministic shell event log, so it is portable (a git hook + a text file), deterministic (a line count), and free of token cost. - tools/secret-guard/secret-scan.sh: on a block, append a metadata-only event (guard/secret-guard/blocked — NEVER the matched secret) to $KEEL_IMPACT_LOG. Strictly opt-in: writes only when the env var is set, so default hook behaviour is byte-identical. - tools/keel-impact.sh: new `event TYPE [source] [detail]` producer entry point; `add` now auto-ingests any logged events into the counts, then truncates the log so nothing is double-counted (--no-ingest opts out). Log defaults to .keel/impact-events.log (gitignored), $KEEL_IMPACT_LOG overrides. So a secret-guard block reaches the derived score without the model counting it — and cannot be over- or under-stated by it. - commands/keel-score.md + docs/keel-impact.md: document that guard is collected for you; leave --guard at 0 unless a fire wasn't logged. - tests: test_keel_impact.sh (40) covers event/ingest/no-ingest/no-double- count and is now hermetic against an ambient log; test_secret_guard.sh (60) covers the opt-in, metadata-only, no-leak, off-by-default emission. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
Extend the deterministic, zero-token guardrail-fire capture from secret-guard to the other two guardrails, so every guardrail fire reaches the impact score without the model counting it. - tools/pre-pr-gate.sh: on a deny, append a metadata-only guard event (in deny(), covering both deny sites). Writes to the log file only, never stdout, so the hook's JSON decision payload stays intact. - tools/public-audit.sh: on a GAP (exit 1 — a real publication blocker caught), append a guard event. Not on a clean run or advisory WARNs. - Both are opt-in: they write only when $KEEL_IMPACT_LOG is set, so default behaviour is byte-identical (existing tests untouched). - Docs (keel-score.md, ledger header) now name all three guardrails as the producers; CHANGELOG generalized. - tests: test_pre_pr_gate.sh (+5) and test_public_audit.sh (+5) cover the opt-in, metadata-only, no-leak, off-by-default, stdout-integrity, and (public-audit) clean-run-records-nothing cases. - docs/loading-and-cost.md: command-size band ceiling 1,200 -> 1,250 for headroom (keel-score.md sits at ~1,201 after the guardrail-list edit). Verified end-to-end: all three guardrails write to one log and a single `add` auto-ingests all of them (guard col = 3, model passed --guard 0). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
Make the guardrail-fire loop turn-key: enable it once per repo and it works in any commit context (git hook, IDE, CI) with no exported variable — the objective signal reaches the score with zero config. - Guardrails now resolve where to record a fire by: $KEEL_IMPACT_LOG override, else a repo-local .keel/ marker (found from the git top level). With neither, nothing is written (behaviour unchanged). This replaces the env-only gate, which couldn't survive GUI/IDE commits or be scoped per project. Applied to secret-guard, pre-pr-gate (via the hook's cwd), and public-audit (via the audited dir). - tools/keel-impact.sh: new `enable [dir]` opts a repo in — creates the gitignored .keel/ marker the hooks look for. Idempotent. - tools/init-project.sh: new projects are born tracked (.keel/ created + gitignored); --no-impact opts out. - install.sh: Done message points to `keel-impact.sh enable` for existing repos. - tests/lib.sh: pin KEEL_IMPACT_LOG into the sandbox so a suite run never records into a maintainer's real .keel/. Every guardrail test now covers all three enable paths (override / marker-only / neither), plus enable-subcommand and init-project born-tracked / --no-impact cases. - Marker resolution written as explicit `if` (not an && chain) so it is unambiguously set -e safe in the security-adjacent hooks. - Docs (keel-score.md, ledger header, CHANGELOG) updated for the marker. Verified end-to-end with all env unset: enable a repo -> a secret-guard block records into .keel/ -> `add` auto-ingests it -> score derived. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
Two additions at the right altitude: the cross-project "usefulness of
Keel" view lives in the impact tool, and doctor gains only a hygiene
check — it does not become a feature-status dashboard.
- tools/keel-impact.sh:
- `rollup --registry FILE` sweeps every project in an INSTANCE.md
Projects table (same parser as doctor --registry) and reports each
one's mean score from its own .keel/ledger.md, a grand total, and the
cumulative guardrail-fire / retrieval-miss signals.
- Resolve the ledger (and event log) from the tracked repo's .keel/
marker, mirroring the guardrails — so a project's sessions score into
.keel/ledger.md with no env, and the registry sweep has a canonical
per-project path. Falls back to Keel's docs/keel-impact.md (ledger)
when a repo has no marker, preserving Keel's own dogfooding default.
- tools/doctor.sh: WARN (not GAP) when a .keel/ marker exists but isn't
gitignored — its event log could leak into history. Mirrors the
existing unignored-private-context check; it never nags a project to
enable tracking (optional feature != baseline drift).
- tests/lib.sh: also pin KEEL_IMPACT_LEDGER into the sandbox so a stray
`add` in a test never writes into Keel's real docs/keel-impact.md.
- tests: registry sweep (scored / zero / unscored projects + grand total
+ missing-registry error) in test_keel_impact.sh; the .keel/ hygiene
WARN (and its absence once ignored) in test_doctor.sh.
Verified end-to-end with all env unset: two enabled repos score into
their .keel/ledger.md and `rollup --registry` reports both plus a total.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
The ledger is the durable deliverable (per-session score history) — the whole point of "know Keel's usefulness over time". Ignoring all of .keel/ made that history local-only, lost on a fresh clone / CI and unshareable. Split the two: ignore only the ephemeral event log; let .keel/ledger.md be committed. - .gitignore / enable / init-project: ignore /.keel/impact-events.log (not the whole /.keel/). The ledger beside it stays trackable — a repo can `git add .keel/ledger.md` for a shareable, cross-clone record. - doctor: the hygiene WARN now targets the log specifically (check-ignore .keel/impact-events.log), so it never flags the committable ledger. Still a WARN, never a GAP, never nags to enable. - tests: assert the split directly (log ignored, ledger trackable via git check-ignore) and update the doctor / init-project expectations. Verified: after enable+score, `git add -A -n` stages .keel/ledger.md but not .keel/impact-events.log. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
An optional way to answer "is Keel earning its keep?" — quantify how much Keel actually shaped a session, then aggregate it across projects. Built around one rule that keeps it honest: the score is derived, not asserted.
The design deliberately avoids the trap of a self-congratulatory number: the model that used Keel does not pick its own score. It enumerates counted, cited events; a Bash tool computes the number by a fixed formula. So the "marketing" score is a pure function of the evidence and can't be inflated by a good mood.
How it works
1. Derived score (
/keel-score→commands/keel-score.md+tools/keel-impact.sh). The model reports typed events, each owing a concrete artifact from the session (a rule id, a diff line, a guardrail block). The tool derives:Guardrail fires (objective blocks) weigh most; retrieval misses and friction pull it down. Each row also carries a
conftier (a score behind one event is visibly weak) and asilent-rules count (demote candidates), kept out of the score.2. The objective signal is collected in the shell, at zero token cost. A guardrail actually firing is the most objective input, so it's pushed out of the model's hands:
secret-guard,pre-pr-gate, andpublic-auditeach record a metadata-only event (never the matched secret) to a per-repo event log thataddauto-ingests. Deterministic, portable (a git hook + a text file), and free of tokens — the model can't over- or under-state it.3. Works out of the box, per project, no env. Recording is gated on a repo-local
.keel/marker (resolved from the git top level), so the loop needs no exported variable and works in any commit context.init-projectenables it by default (--no-impactto skip);keel-impact.sh enable [dir]opts an existing repo in;$KEEL_IMPACT_LOG/$KEEL_IMPACT_LEDGERremain explicit overrides.4. Cross-project view.
keel-impact.sh rollup --registry <INSTANCE.md>sweeps every project in the Projects table (same parser asdoctor --registry) and reports each one's mean score from its own.keel/ledger.md, plus a grand total and the cumulative guardrail-fire / retrieval-miss signals.5. One narrow
doctorcheck.doctorgains a WARN (not a GAP) when a.keel/marker exists but isn't gitignored — mirroring its existing unignored-private-context check so an enabled event log can't leak into history. It deliberately does not gain an "impact tracking on/off" line: that would be false drift for an optional feature, anddoctorstays a baseline audit.Design choices worth flagging
awkuses no interval expressions (busybox/Alpine parity).pre-pr-gatewrites to the log file only, never stdout, so its JSON decision stays intact.Tests
Full suite green — 337 assertions across 11 files, including new coverage in
test_keel_impact.sh(derivation, confidence tiers,—handling, event/ingest/no-double-count,enable, registry rollup), and every guardrail test now exercises all three enable paths (override / marker-only / neither → nothing written) plus no-secret-leak and stdout-integrity. The test harness pins the impact log/ledger into its sandbox so a suite run never records into a real.keel/.Files
commands/keel-score.md,tools/keel-impact.sh,docs/keel-impact.md,tests/test_keel_impact.shtools/secret-guard/secret-scan.sh,tools/pre-pr-gate.sh,tools/public-audit.shtools/init-project.sh,tools/doctor.sh,install.sh,.gitignore,commands/wrap.md(optional step 7)docs/loading-and-cost.md,CHANGELOG.md,tests/lib.sh, and the four instrumented tools' tests🤖 Generated with Claude Code
https://claude.ai/code/session_01Qghzk5C4mWTAcLCvXofYdN
Generated by Claude Code