test(e2e): cover GrubSetKargs with story 138 - #447
Conversation
Story 138 drives a one-step high-risk GrubSetKargs plan (append quiet, delete splash), closing the last Ubuntu-only action gap. The suite is now 138 stories (54 atomic + 84 Ubuntu) with 0 Ubuntu-only uncovered actions; committed live-VM evidence stays at 79/79 per LTS, with the five additional stories (134-138) not yet in a recorded run. The public-claims contract test is adapted to the derived-zero gap: stale nonzero gap prose is rejected against derived zero, and the universal claim is rejected against a fixture gap reopened by neutralising the story quote.
vladimirrott
left a comment
There was a problem hiding this comment.
Reviewed at 5082f3f3ad3f47428d16d8d7e5bc6dea7264526e. Approved, and I am merging it.
The assertion I want to point at is APPEND == ["quiet"]. prompt.rs line 928 hands the model the example {"append":["quiet","nomodeset"],"delete":["splash"]}, so a story that accepted any non-empty append would pass on a model that copied the example back verbatim and learned nothing. Pinning the exact single-element list is what makes story 138 a test of comprehension rather than of recall. Stories 134 to 137 set that bar and you met it.
Rewriting public-claims.test.sh is the part I read twice. A contract test adapted to a new tree state is the easiest place in this repository to destroy an assertion while every check stays green, and the house rule is that a failing contract gets the prose fixed, never the assertion widened. So I went looking for that, and it is not what you did. Evidence below.
maintainer screen 447 returns DO NOT EXECUTE, so everything ran in rootless podman, --network=none, uid 1000, against an extracted copy of your head.
Clean run first
$ cd /home/entropia/.local/state/sysknife-maint/checkout && rm -rf /tmp/wt447 && mkdir -p /tmp/wt447 && git archive p447 | tar -x -C /tmp/wt447 && echo "extracted head: $(git rev-parse p447)" && chmod -R a+rX /tmp/wt447 && rm -rf /tmp/tmp447 && mkdir -p /tmp/tmp447 && chmod 777 /tmp/tmp447 && out="$(podman run --rm --network=none --user 1000:1000 -v /tmp/wt447:/repo:z -v /tmp/tmp447:/sktmp:z -w /repo -e TMPDIR=/sktmp -e HOME=/sktmp localhost/sk-yamlgate:1 bash -c 'python3 -V; bash tests/release/public-claims.test.sh; echo "TEST rc=$?"' 2>&1)"; rc=$?; printf '%s\n' "$out" | grep -v 'level=warning' | tail -20; echo "podman rc=$rc"
extracted head: 5082f3f3ad3f47428d16d8d7e5bc6dea7264526e
Python 3.12.14
Published figures match the evidence artifacts.
Public claims are internally consistent.
Public claims contract passed.
TEST rc=0
podman rc=0
Story metadata and provider parity agree with the counts you published:
$ rm -rf /tmp/tmp447b && mkdir -p /tmp/tmp447b && chmod 777 /tmp/tmp447b && out="$(podman run --rm --network=none --user 1000:1000 -v /tmp/wt447:/repo:z -v /tmp/tmp447b:/sktmp:z -w /repo -e TMPDIR=/sktmp -e HOME=/sktmp localhost/sk-yamlgate:1 bash -c 'bash tests/e2e/story-metadata.test.sh; echo "story-metadata rc=$?"; bash tests/e2e/provider-parity.test.sh; echo "provider-parity rc=$?"' 2>&1)"; rc=$?; printf '%s\n' "$out" | grep -v 'level=warning' | tail -12; echo "podman rc=$rc"; echo "=== shellcheck story-138 (CI flags) ==="; out2="$(shellcheck --severity=warning /tmp/wt447/tests/e2e/stories/story-138.sh /tmp/wt447/tests/release/public-claims.test.sh 2>&1)"; rc2=$?; printf '%s\n' "$out2"; echo "shellcheck rc=$rc2"
Story metadata derived cleanly: 138 stories (84 ubuntu, 54 atomic).
story-metadata rc=0
Provider parity passed: 7 providers reachable from 5 entry points.
provider-parity rc=0
podman rc=0
=== shellcheck story-138 (CI flags) ===
shellcheck rc=0
The contract test still bites, in four directions
Two mutations to scripts/check_evidence_claims.py, which your PR does not touch, and two to the tree the claims describe:
$ set -e
for m in cm1 cm2 tm1 tm2; do rm -rf /tmp/wt447$m; cp -a /tmp/wt447 /tmp/wt447$m; done
python3 - <<'PY'
# CM1: make the universal "every Debian-only action has a story" claim unmatchable
p='/tmp/wt447cm1/scripts/check_evidence_claims.py'; s=open(p).read()
old=' r"\\bEvery\\s+Debian-only\\s+action\\s+(?:now\\s+)?has\\s+(?:one|a\\s+story)\\b",'
assert s.count(old)==1, ('cm1', s.count(old))
open(p,'w').write(s.replace(old,' r"\\bZZ_NEVER_MATCHES_ZZ\\b",'))
# CM2: stop reporting a published/derived gap-count mismatch in the CONTRIBUTING rule
p='/tmp/wt447cm2/scripts/check_evidence_claims.py'; s=open(p).read()
old=''' if gap_claim:
if _claim_count(gap_claim.group("count")) != uncovered["Ubuntu"]:'''
assert s.count(old)==1, ('cm2', s.count(old))
open(p,'w').write(s.replace(old,''' if gap_claim:
if False:'''))
print("checker mutations applied")
PY
rm -f /tmp/wt447tm1/tests/e2e/stories/story-138.sh
python3 - <<'PY'
p='/tmp/wt447tm2/docs/introduction.md'; s=open(p).read()
old='The current suite is 84 Ubuntu stories.'
assert s.count(old)==1
open(p,'w').write(s.replace(old,'The current suite is 83 Ubuntu stories.'))
PY
echo "--- cm1 ---"; grep -n 'ZZ_NEVER_MATCHES_ZZ' /tmp/wt447cm1/scripts/check_evidence_claims.py
echo "--- cm2 ---"; grep -n -A1 'if gap_claim:' /tmp/wt447cm2/scripts/check_evidence_claims.py | head -4
echo "--- tm1 ---"; ls /tmp/wt447tm1/tests/e2e/stories/story-138.sh 2>&1 || echo "story-138.sh removed"
echo "--- tm2 ---"; grep -n 'current suite is 8[0-9] Ubuntu stories' /tmp/wt447tm2/docs/introduction.md
checker mutations applied
--- cm1 ---
439: r"\bZZ_NEVER_MATCHES_ZZ\b",
--- cm2 ---
525: if gap_claim:
526- if False:
--- tm1 ---
ls: cannot access '/tmp/wt447tm1/tests/e2e/stories/story-138.sh': No such file or directory
story-138.sh removed
--- tm2 ---
107:> The current suite is 83 Ubuntu stories. All three Ubuntu LTS releases have a
$ for m in cm1 cm2 tm1 tm2; do echo "=== M:$m ==="; t=/tmp/tmp447$m; rm -rf $t; mkdir -p $t; chmod 777 $t; chmod -R a+rX /tmp/wt447$m; out="$(podman run --rm --network=none --user 1000:1000 -v /tmp/wt447$m:/repo:z -v $t:/sktmp:z -w /repo -e TMPDIR=/sktmp -e HOME=/sktmp localhost/sk-yamlgate:1 bash -c 'bash tests/release/public-claims.test.sh; echo "TEST rc=$?"' 2>&1)"; rc=$?; printf '%s\n' "$out" | grep -v 'level=warning' | tail -8; echo "podman rc=$rc"; echo; done
=== M:cm1 ===
Claims that do not derive from evidence:
- CONTRIBUTING.md: could not derive the Debian-only coverage prose; expected every Debian-only action to have a story
Invalid public claim: a published figure does not derive from evidence
TEST rc=1
podman rc=0
=== M:cm2 ===
Published figures match the evidence artifacts.
Public claims are internally consistent.
FAIL: checker accepted stale claim: stale nonzero gap prose against a derived zero gap
TEST rc=1
podman rc=0
=== M:tm1 ===
- CONTRIBUTING.md: says every Debian-only action has a story, but the tree leaves 1 Ubuntu-only actions uncovered
- docs/introduction.md: says every Debian-only action has a story, but the tree leaves 1 Ubuntu-only actions uncovered
- CONTRIBUTING.md: claims a 138-story suite, which matches neither the stories on disk nor any recorded run (known: 54, 79, 83, 137). Quote a figure from tests/evidence/story-runs/ or drop it.
- CONTRIBUTING.md: Ubuntu family story count has published 84, derived 83
- CONTRIBUTING.md: Ubuntu family story count has published 84, derived 83
- docs/introduction.md: Ubuntu family story count has published 84, derived 83
Invalid public claim: a published figure does not derive from evidence
TEST rc=1
podman rc=0
=== M:tm2 ===
Claims that do not derive from evidence:
- docs/introduction.md: Ubuntu family story count has published 84, derived 83
Invalid public claim: a published figure does not derive from evidence
TEST rc=1
podman rc=0
M:cm2 is the one that answers my worry: it is your rewritten stale-prose block that catches a checker which stopped comparing published against derived, and it catches it by name. M:tm1 is the other half, and it settles the coverage arithmetic without my having to trust a count: delete story 138 and the derived Ubuntu-only gap goes from zero to one, in both claim files. The story is load-bearing on the sentence it licensed.
Your new blocks all open with a derived != 0 precondition. I inverted one to prove it is not decoration:
$ set -e
rm -rf /tmp/wt447cm3 && cp -a /tmp/wt447 /tmp/wt447cm3
python3 - <<'PY'
p='/tmp/wt447cm3/tests/release/public-claims.test.sh'; s=open(p).read()
old='''if derived != 0:
raise SystemExit(
f"fixture gap is not zero before the stale-prose mutation: {derived}"
)'''
assert s.count(old)==1, s.count(old)
open(p,'w').write(s.replace(old, old.replace('if derived != 0:','if derived == 0:')))
PY
grep -n -A1 'if derived == 0:' /tmp/wt447cm3/tests/release/public-claims.test.sh | head -4
t=/tmp/tmp447cm3; rm -rf $t; mkdir -p $t; chmod 777 $t; chmod -R a+rX /tmp/wt447cm3
out="$(podman run --rm --network=none --user 1000:1000 -v /tmp/wt447cm3:/repo:z -v $t:/sktmp:z -w /repo -e TMPDIR=/sktmp -e HOME=/sktmp localhost/sk-yamlgate:1 bash -c 'bash tests/release/public-claims.test.sh; echo "TEST rc=$?"' 2>&1)"; rc=$?; printf '%s\n' "$out" | grep -v 'level=warning' | tail -6; echo "podman rc=$rc"
278:if derived == 0:
279- raise SystemExit(
Published figures match the evidence artifacts.
Public claims are internally consistent.
fixture gap is not zero before the stale-prose mutation: 0
TEST rc=1
podman rc=0
The SystemExit reaches the [[ -z ... ]] guard and the run stops, naming the reason. No path through those blocks passes on an empty read.
The live-VM caveat, and why I am merging anyway
You asked for a VM story run before merge, which was the right thing to ask. I cannot do one from here. I am merging without it on the same basis #401 landed two days ago: stories 134 to 137 went in from that PR and are still outside the committed evidence, and both files you edited now say so in the prose a reader sees. CONTRIBUTING.md says five additional stories are not in a committed live-VM run, introduction.md says the same, and the committed figure stays 79/79 per LTS. Nobody is being told the story has been executed. When the next VM run happens, 134 to 138 go in together.
One thing I checked because the claim depends on it: GrubSetKargs is grounded for the Debian path, not only present in the catalogue. prompt.rs:817 lists it, :843 documents append and delete as its params, :880 sets it HIGH, and action_family.rs:113 puts it in DEBIAN_ONLY_ACTIONS. Your story asks for the action the Debian prompt is built to propose.
Optional, nothing blocking
The old gap mutations derived their target action from docs/action-reference.md at run time. Yours name GrubSetKargs and the two prose sentences as literals. That reads as a step backwards, and I decided it is not one: every literal you introduced is guarded by an explicit uniqueness check that raises rather than matching nothing in silence, so a rename or a reword stops the suite with a sentence saying which literal went missing. It costs a maintainer one confusing failure instead of a silent vacuous pass, which is the right side to fail on. Worth a comment above the first block saying that, so the next person to touch it knows the literals were a choice.
About the two you are holding
This one refs #233, which is yours, along with #219. Both have been quiet for nine days and there is no deadline on either. If either has gone cold or turned out bigger than it looked, say so and I will put it back in the pool with no hard feelings at all; if you are still on them, ignore me. Either answer is a good answer, and this PR is the kind of work #233 was asking for.
You mentioned working from Windows. If you have an Ubuntu VM or a WSL instance around for the story work, SysKnife's read-only side runs there and takes nothing away: sysknife --dry-run "show me what wants a reboot" plans and prints and executes nothing. You have now read more of the story assertions than most people ever will, so if the real thing disagrees with one of them on a live box, you would be the first to notice.
Summary
Adds
tests/e2e/stories/story-138.shforGrubSetKargs, following the stronger assertion style of stories 134-137: exactly one plan step, actionGrubSetKargs, riskhigh, and exact params (append["quiet"],delete["splash"]) from the benign deterministic intent "in GRUB, append quiet to the kernel command line and delete splash".prompt.rsalready grounds the action, so it is untouched.Derived story-coverage claims updated for the new tree (138 stories = 54 atomic + 84 Ubuntu, 0 Ubuntu-only uncovered actions):
CONTRIBUTING.mdanddocs/introduction.md. Committed live-VM evidence stays at 79/79 per LTS; the five additional stories (134-138) are not yet in a recorded run.The
public-claimscontract test assumed a nonzero Debian-only gap, so its four gap-mutation blocks are adapted to the derived-zero world: stale nonzero gap prose is rejected against derived zero, and the universal claim is rejected against a fixture gap reopened by neutralising the story quote.Related Issue
Refs #233
Validation
story-138.sh;story-metadata,story-runner-verdicts,public-claimscontract all pass host-side)CONTRIBUTING.md,docs/introduction.md)Notes for Reviewers
Story 138 has not been run against a live VM from this host (Windows); the script is syntax/ShellCheck/metadata verified only. Suggest a VM story run before merge. No fail2ban cassette/VM work and no #229 guard changes included.