Skip to content

Make the impact score an auditable trail; add hold + retro#54

Merged
dbudnikau-personal merged 1 commit into
mainfrom
claude/ecstatic-khorana-dc43ea
Jul 8, 2026
Merged

Make the impact score an auditable trail; add hold + retro#54
dbudnikau-personal merged 1 commit into
mainfrom
claude/ecstatic-khorana-dc43ea

Conversation

@dbudnikau-personal

Copy link
Copy Markdown
Collaborator

Follow-up to #53 (impact score). Extends the still-unreleased feature along three threads, plus review fixes.

What changed

  • Auditable counts (thread A). The score is no longer built from bare integer flags. /keel-score passes one cited event per flag (repeatable --fire/--hit/--miss/--friction "…"); the count is the number of citations, so no citation → no count, mechanically — the same "derived, not asserted" honesty, pushed down to the counts. Every citation (including each auto-ingested guardrail fire, cited by its source | detail) is archived to a durable, trackable .keel/evidence.md next to the ledger, so a score is a checkable record. The row's evidence cell is auto-filled with the strongest citation. Breaking for the unreleased add interface: --guard N … / --evidence are replaced by repeatable citation flags (--silent N stays a bare count).
  • hold — Keel's highest function, scored above guard (thread B). A new event type for when Keel restrained the agent from weakening/bypassing a rule or guardrail (vs guard, which blocks bad content). HELP = 4·hold + 3·guard + 2·fire + hit. Own ledger column (after guard) and cumulative signal. Schema change: the ledger gains a hold column.
  • Retrospective scoring, quarantined (thread C). add --retro [--asof YYYY-MM-DD] records a score reconstructed from a past session without contaminating the live signal: never touches the live event log, conf-tagged -retro and dropped one tier, excluded from the live rollup (rollup --retro shows only these). Transcript→events extraction stays a manual, opt-in step.
  • A/B calibration (thread D): kept documented-only — the sole true counterfactual is not zero-cost, so it is deliberately not automated.

Review fixes (from /code-review high)

  • Auto-ingest now processes a final log line with no trailing newline (was dropped, then lost on truncate — a real data-loss regression vs the old awk counter).
  • An empty/missing citation is rejected with a clean exit 2 (was bash's cryptic ${2:?} exit 1).
  • Shared _flatten for the one-event-one-line invariant; column-position legends at all positional awk/printf sites.
  • commands/*.md token band raised 1,250 → 1,450 (keel-score.md grew; band ⊇ actual).

Test plan

  • bash tests/run.sh378 passed, 0 failed (was 341).
  • shellcheck --severity=warning clean on the changed scripts.
  • Manual end-to-end: hold weight-4 scoring, retro quarantine (live rollup excludes, rollup --retro includes), unterminated-line ingest, empty-citation rejection.

🤖 Generated with Claude Code

Three refinements to the (still-unreleased) Keel impact score:

- Auditable counts (thread A): the score is no longer built from bare
  integer flags. `keel-score` passes one cited event per flag (repeatable
  --fire/--hit/--miss/--friction "..."); the count is the number of
  citations, so "no citation -> no count" is mechanical. Every citation
  (including each auto-ingested guardrail fire, cited by its source|detail)
  is archived to a durable, trackable .keel/evidence.md next to the ledger.
  The row's evidence cell is auto-filled with the strongest citation.
  Breaking for the unreleased `add` interface: --guard N / --evidence are
  replaced by repeatable citation flags (--silent N stays a bare count).

- `hold` (thread B): a new scored event type for when Keel restrained the
  agent from weakening/bypassing a rule or guardrail (vs guard, which blocks
  bad content). Weighs highest: HELP = 4*hold + 3*guard + 2*fire + hit. Own
  ledger column (after guard) and cumulative signal.

- Retrospective scoring, quarantined (thread C): `add --retro [--asof DATE]`
  records a score reconstructed from a past session without contaminating
  the live signal — never touches the live log, conf-tagged `-retro` and
  dropped one tier, excluded from the live rollup (`rollup --retro` shows
  only these). Transcript extraction stays a manual, opt-in step.

A/B calibration remains documented-only (the sole true counterfactual is
not zero-cost, so it is deliberately not automated).

Also fixes found in review: the auto-ingest loop now processes a final log
line with no trailing newline (was dropped, then lost on truncate); an
empty/missing citation is rejected with a clean exit 2; shared `_flatten`
for the one-event-one-line invariant; column-position legends at all the
positional awk/printf sites. commands/*.md token band raised 1,250 -> 1,450
(keel-score.md grew). Covered by tests/test_keel_impact.sh (378 tests green).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@dbudnikau-personal dbudnikau-personal merged commit 147f742 into main Jul 8, 2026
4 checks passed
@dbudnikau-personal dbudnikau-personal deleted the claude/ecstatic-khorana-dc43ea branch July 8, 2026 21:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant