The raw and baseline snapshots are rendered from whatever kimaki is installed on the host running the test. The committed ones were taken on 2026-08-21 against kimaki 0.27.x. kimaki has since shipped 0.29.0, so upgrade.sh now reports a failure on every managed install, every night, until someone runs --update:
[wp-coding-agents] effective-prompt test FAILED — dm-context-filter may be leaking banned phrases
baseline : 18,853 chars (stripped 21,285, ~5321 tokens)
filtered : 726 chars (stripped 39,412, ~9853 tokens)
baseline leaks: 50
filtered leaks: 0
FAIL:
- raw snapshot drift — run with --update to refresh
- baseline snapshot drift — run with --update to refresh
FAIL — 2 scenario(s) failed
The filter is fine. filtered leaks: 0, and the filtered snapshot still matches byte-for-byte — dm-context-filter is replacing the whole upstream prompt as designed. Only raw and baseline drifted, and they drifted because upstream added prompt sections:
+## sleeping the session (kimaki_sleep)
+## updating the session title
+### who edited a file (kimaki session editors)
+kimaki session list --active / wait-loop guidance
The structural problem
raw and baseline are snapshots of someone else's artifact. They change whenever remorses/kimaki ships, which is often, and they are asserted on production hosts during unattended maintenance. So the test is guaranteed to go red on a schedule that has nothing to do with this repo, and the failure it prints — "dm-context-filter may be leaking banned phrases" — describes a security regression that did not happen.
That is the wrong default. The warning has now fired on the h44 nightly for weeks — the stale .actual files on chubes.net date to 2026-09-03 — and it trains you to skim past it while looking for the real failure, which is exactly what a boy-who-cried-wolf assertion does.
The invariants worth asserting on a managed install are the ones about our behavior: filtered_leaks === 0, no leak regression vs baseline, non-Kimaki blocks preserved, and the filtered snapshot itself. Those hold across upstream prompt changes. Byte-equality against an upstream prompt is useful as a local review aid, not as an unattended production assertion.
Suggested split:
- Keep the
filtered snapshot and the three leak invariants as hard assertions everywhere.
- Demote
raw/baseline byte-equality to informational when the installed kimaki version differs from the one recorded with the snapshots, and record that version alongside them.
- If they stay as hard assertions, make the failure text say "upstream kimaki prompt changed" rather than implying the filter leaked.
Independent of the above, the snapshots need a refresh for 0.29.0.
Context
Found while diagnosing a red h44 nightly: https://github.com/chubes4/h44-lacrosse/actions/runs/35323714007
This was not the cause of that failure — bridges/kimaki.sh correctly treats it as a warning. The real cause was Extra-Chill/data-machine#3512. It did cost diagnosis time, though, because it is the loudest thing in the log.
The
rawandbaselinesnapshots are rendered from whatever kimaki is installed on the host running the test. The committed ones were taken on 2026-08-21 against kimaki 0.27.x. kimaki has since shipped 0.29.0, soupgrade.shnow reports a failure on every managed install, every night, until someone runs--update:The filter is fine.
filtered leaks: 0, and thefilteredsnapshot still matches byte-for-byte —dm-context-filteris replacing the whole upstream prompt as designed. Onlyrawandbaselinedrifted, and they drifted because upstream added prompt sections:The structural problem
rawandbaselineare snapshots of someone else's artifact. They change whenever remorses/kimaki ships, which is often, and they are asserted on production hosts during unattended maintenance. So the test is guaranteed to go red on a schedule that has nothing to do with this repo, and the failure it prints — "dm-context-filter may be leaking banned phrases" — describes a security regression that did not happen.That is the wrong default. The warning has now fired on the h44 nightly for weeks — the stale
.actualfiles on chubes.net date to 2026-09-03 — and it trains you to skim past it while looking for the real failure, which is exactly what a boy-who-cried-wolf assertion does.The invariants worth asserting on a managed install are the ones about our behavior:
filtered_leaks === 0, no leak regression vs baseline, non-Kimaki blocks preserved, and thefilteredsnapshot itself. Those hold across upstream prompt changes. Byte-equality against an upstream prompt is useful as a local review aid, not as an unattended production assertion.Suggested split:
filteredsnapshot and the three leak invariants as hard assertions everywhere.raw/baselinebyte-equality to informational when the installed kimaki version differs from the one recorded with the snapshots, and record that version alongside them.Independent of the above, the snapshots need a refresh for 0.29.0.
Context
Found while diagnosing a red h44 nightly: https://github.com/chubes4/h44-lacrosse/actions/runs/35323714007
This was not the cause of that failure —
bridges/kimaki.shcorrectly treats it as a warning. The real cause was Extra-Chill/data-machine#3512. It did cost diagnosis time, though, because it is the loudest thing in the log.