Skip to content

effective-prompt: raw/baseline snapshots are pinned to a kimaki version, so every kimaki release fails the test on every managed install #606

Description

@chubes4

The raw and baseline snapshots are rendered from whatever kimaki is installed on the host running the test. The committed ones were taken on 2026-08-21 against kimaki 0.27.x. kimaki has since shipped 0.29.0, so upgrade.sh now reports a failure on every managed install, every night, until someone runs --update:

[wp-coding-agents]   effective-prompt test FAILED — dm-context-filter may be leaking banned phrases
    baseline : 18,853 chars (stripped 21,285, ~5321 tokens)
    filtered : 726 chars (stripped 39,412, ~9853 tokens)
    baseline leaks: 50
    filtered leaks: 0
      FAIL:
        - raw snapshot drift — run with --update to refresh
        - baseline snapshot drift — run with --update to refresh
    FAIL — 2 scenario(s) failed

The filter is fine. filtered leaks: 0, and the filtered snapshot still matches byte-for-byte — dm-context-filter is replacing the whole upstream prompt as designed. Only raw and baseline drifted, and they drifted because upstream added prompt sections:

+## sleeping the session      (kimaki_sleep)
+## updating the session title
+### who edited a file        (kimaki session editors)
+kimaki session list --active / wait-loop guidance

The structural problem

raw and baseline are snapshots of someone else's artifact. They change whenever remorses/kimaki ships, which is often, and they are asserted on production hosts during unattended maintenance. So the test is guaranteed to go red on a schedule that has nothing to do with this repo, and the failure it prints — "dm-context-filter may be leaking banned phrases" — describes a security regression that did not happen.

That is the wrong default. The warning has now fired on the h44 nightly for weeks — the stale .actual files on chubes.net date to 2026-09-03 — and it trains you to skim past it while looking for the real failure, which is exactly what a boy-who-cried-wolf assertion does.

The invariants worth asserting on a managed install are the ones about our behavior: filtered_leaks === 0, no leak regression vs baseline, non-Kimaki blocks preserved, and the filtered snapshot itself. Those hold across upstream prompt changes. Byte-equality against an upstream prompt is useful as a local review aid, not as an unattended production assertion.

Suggested split:

  • Keep the filtered snapshot and the three leak invariants as hard assertions everywhere.
  • Demote raw/baseline byte-equality to informational when the installed kimaki version differs from the one recorded with the snapshots, and record that version alongside them.
  • If they stay as hard assertions, make the failure text say "upstream kimaki prompt changed" rather than implying the filter leaked.

Independent of the above, the snapshots need a refresh for 0.29.0.

Context

Found while diagnosing a red h44 nightly: https://github.com/chubes4/h44-lacrosse/actions/runs/35323714007

This was not the cause of that failure — bridges/kimaki.sh correctly treats it as a warning. The real cause was Extra-Chill/data-machine#3512. It did cost diagnosis time, though, because it is the loudest thing in the log.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions