Make the impact score an auditable trail; add hold + retro#54
Merged
Conversation
Three refinements to the (still-unreleased) Keel impact score: - Auditable counts (thread A): the score is no longer built from bare integer flags. `keel-score` passes one cited event per flag (repeatable --fire/--hit/--miss/--friction "..."); the count is the number of citations, so "no citation -> no count" is mechanical. Every citation (including each auto-ingested guardrail fire, cited by its source|detail) is archived to a durable, trackable .keel/evidence.md next to the ledger. The row's evidence cell is auto-filled with the strongest citation. Breaking for the unreleased `add` interface: --guard N / --evidence are replaced by repeatable citation flags (--silent N stays a bare count). - `hold` (thread B): a new scored event type for when Keel restrained the agent from weakening/bypassing a rule or guardrail (vs guard, which blocks bad content). Weighs highest: HELP = 4*hold + 3*guard + 2*fire + hit. Own ledger column (after guard) and cumulative signal. - Retrospective scoring, quarantined (thread C): `add --retro [--asof DATE]` records a score reconstructed from a past session without contaminating the live signal — never touches the live log, conf-tagged `-retro` and dropped one tier, excluded from the live rollup (`rollup --retro` shows only these). Transcript extraction stays a manual, opt-in step. A/B calibration remains documented-only (the sole true counterfactual is not zero-cost, so it is deliberately not automated). Also fixes found in review: the auto-ingest loop now processes a final log line with no trailing newline (was dropped, then lost on truncate); an empty/missing citation is rejected with a clean exit 2; shared `_flatten` for the one-event-one-line invariant; column-position legends at all the positional awk/printf sites. commands/*.md token band raised 1,250 -> 1,450 (keel-score.md grew). Covered by tests/test_keel_impact.sh (378 tests green). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #53 (impact score). Extends the still-unreleased feature along three threads, plus review fixes.
What changed
/keel-scorepasses one cited event per flag (repeatable--fire/--hit/--miss/--friction "…"); the count is the number of citations, so no citation → no count, mechanically — the same "derived, not asserted" honesty, pushed down to the counts. Every citation (including each auto-ingested guardrail fire, cited by itssource | detail) is archived to a durable, trackable.keel/evidence.mdnext to the ledger, so a score is a checkable record. The row'sevidencecell is auto-filled with the strongest citation. Breaking for the unreleasedaddinterface:--guard N …/--evidenceare replaced by repeatable citation flags (--silent Nstays a bare count).hold— Keel's highest function, scored above guard (thread B). A new event type for when Keel restrained the agent from weakening/bypassing a rule or guardrail (vsguard, which blocks bad content).HELP = 4·hold + 3·guard + 2·fire + hit. Own ledger column (afterguard) and cumulative signal. Schema change: the ledger gains aholdcolumn.add --retro [--asof YYYY-MM-DD]records a score reconstructed from a past session without contaminating the live signal: never touches the live event log, conf-tagged-retroand dropped one tier, excluded from the liverollup(rollup --retroshows only these). Transcript→events extraction stays a manual, opt-in step.Review fixes (from
/code-review high)${2:?}exit 1)._flattenfor the one-event-one-line invariant; column-position legends at all positional awk/printf sites.commands/*.mdtoken band raised 1,250 → 1,450 (keel-score.md grew; band ⊇ actual).Test plan
bash tests/run.sh→ 378 passed, 0 failed (was 341).shellcheck --severity=warningclean on the changed scripts.rollup --retroincludes), unterminated-line ingest, empty-citation rejection.🤖 Generated with Claude Code