Skip to content

Commit ed217e6

Browse files
claude[bot]claude
andauthored
tooling(scripts): row batteries for four table-driven self-tests (#13799 batch 7b) (#15286)
* tooling(scripts): row batteries for check-skill-id-lint's self-test `failed === 0` was this self-test's only success condition, so "every case held" and "the cases never ran" printed the same line (#13489). Recipe A, as the batch 7 pilot (PR #15271) settled it for the table-driven class: each of the 14 rows of the literal `cases` table is a declared battery under its own label verbatim, floor 1, and `registerCase(name)` is the first statement of the driving loop body. The roster is a LITERAL the table is checked against — never derived from it — and its own size is pinned at 14, which is also half of the duplicate-label refusal. No assertion condition touched, no control flow rewritten, no comment promoted to a section head, verdict handshake unchanged. The red-run line becomes `N failure(s) (cases and floor)` because floor breaches report into the same sink; the green line is byte-identical to main's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zGPuVVX3deAx9LdjK8jCk * tooling(scripts): row batteries for check-agent-model-declared's self-test `failed === 0` was this self-test's only success condition, so "every case held" and "the cases never ran" printed the same line (#13489). Recipe A, as the batch 7 pilot (PR #15271) settled it: each of the 18 rows of the literal `cases` table is a declared battery under its own `label` verbatim, floor 1, roster size pinned at 18. `registerCase(c.label)` is the FIRST statement of the driving loop body — ahead of the `try` whose `catch` ends in `continue` and ahead of the two guards that do the same, because the floor asserts REACH and a registration behind a guard reintroduces the inversion. Scope stated in place: the four discovery-block assertions below the loop are not rows of the table (the verdict has always printed 18 rows, not the 22 assertions that run), so they are left exactly as they are. No assertion condition touched, no control flow rewritten, no comment promoted to a section head, verdict handshake unchanged. Red-run line becomes `N failure(s) (cases and floor)`; the green line is byte-identical to main's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zGPuVVX3deAx9LdjK8jCk * tooling(scripts): row batteries for check-startup-registry-verdict's self-test `failures === 0` was this self-test's only success condition, so "every case held" and "the cases never ran" printed the same line (#13489). Recipe A, as the batch 7 pilot (PR #15271) settled it: each of the 18 rows of the literal `cases` table is a declared battery under its own `name` verbatim, floor 1, roster size pinned at 18, and `registerCase(c.name)` is the first statement of the driving loop body. Scope stated in place: the eight `expectRoot(...)` assertions of the dead-root reverse proof below the loop are not rows of the table — the verdict line has always counted the 18 analysis rows and named that proof separately — so they are left exactly as they are. No assertion condition touched, no control flow rewritten, no comment promoted to a section head, verdict handshake unchanged. Red-run line becomes `N failure(s) (cases and floor)`; the green line is byte-identical to main's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zGPuVVX3deAx9LdjK8jCk * tooling(scripts): row batteries for js-comment-mask's self-test `failed === 0` was this self-test's only success condition, so "every case held" and "the cases never ran" printed the same line (#13489). Recipe A, as the batch 7 pilot (PR #15271) settled it: each of the 23 rows of the literal mask/strip `cases` table is a declared battery under its own label verbatim, floor 1, roster size pinned at 23, and `registerCase(name)` is the first statement of the driving loop body. Scope stated in place: the twelve `interpolation`-view assertions below the corpus loop are NOT a second literal table — their rows exist only because an `x(...)` call pushed them onto `extra` at runtime, so a roster taken from that loop would be DERIVED and a deleted call would delete its own floor. They are left exactly as they are and the verdict keeps counting them separately. This module is a LIBRARY the gates import, so the change is proved from the consumer side too: six importers' `--self-test` runs (comment-mask-corpus, comment-mask-adoption, error-code-casing, entry-guard, declaration-mirrors, watch-hint-literal) are byte-identical to the base tree's on both streams and green in normal mode on the edited tree. No assertion condition touched, no control flow rewritten, no comment promoted to a section head, verdict handshake unchanged. Red-run line becomes `N failure(s) (cases and floor)`; the green line is byte-identical to main's. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_012zGPuVVX3deAx9LdjK8jCk --------- Co-authored-by: Claude <noreply@anthropic.com>
1 parent 9a3dfac commit ed217e6

4 files changed

Lines changed: 506 additions & 4 deletions

File tree

‎scripts/check-agent-model-declared.mjs‎

Lines changed: 130 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -334,6 +334,72 @@ function report(problems) {
334334
// handshake is a flag rather than a returned sentinel.
335335
let selfTestReachedVerdict = false;
336336

337+
// ── The self-test's own battery roster and floor (#13489) ──────────────────
338+
//
339+
// `failed === 0` used to be this self-test's ONLY success condition, so "every
340+
// case held" and "the cases never ran" printed the same line. Closed the PR
341+
// #13487 way: what is pinned is the registered NAMES, not a number.
342+
//
343+
// This self-test is TABLE-DRIVEN — one literal `cases` table, one loop over it,
344+
// and a sink (`failed += 1`) that writes only when a case FAILS. Routing THAT
345+
// sink through `registerCase()` would register a case only when it fails: a
346+
// fully green run would register 0 and every battery would read DID NOT RUN,
347+
// the floor inverted rather than installed. So the roster is the table's own
348+
// rows. Each row LABEL is a declared battery, verbatim, with a floor of 1, and
349+
// `registerCase(c.label)` is the FIRST statement of the driving loop body —
350+
// before the `try` whose `catch` ends in `continue`, and before the two guards
351+
// that also end in `continue`. The floor asserts REACH, so a registration
352+
// placed after a guard would reintroduce the very inversion it exists to avoid:
353+
// a row whose guard always skips would read DID NOT RUN, indistinguishable from
354+
// a deleted row. There is no `battery()` opener: for a table-driven self-test
355+
// the ROW is the battery, so attribution is the loop variable rather than a
356+
// most-recently-opened section.
357+
//
358+
// ⛔ A pinned TOTAL is not the repair, and neither is a roster DERIVED from the
359+
// table: `cases.length` moves with the table, so a deleted row would delete its
360+
// own floor. The roster below is a LITERAL the table is checked against, which
361+
// is what lets a deleted or renamed row name ITSELF in the refusal.
362+
//
363+
// The counts are a FLOOR, not an equality — a row that grows into several
364+
// registrations must not red. 1 is the honest floor for a table row: the loop
365+
// reaches it exactly once per run.
366+
//
367+
// SCOPE, stated so the next reader does not mistake the number: the four
368+
// discovery-block assertions below the loop (`*.md` only, a missing
369+
// `.claude/agents/`, a directory holding no `.md`, and real-tree discovery) are
370+
// NOT rows of this table — they are written inline, one at a time, and the
371+
// verdict line has always counted the 18 ROWS rather than the 22 assertions
372+
// that run. This roster floors the 18 rows; those four assertions are left
373+
// exactly as they are.
374+
const SELF_TEST_BATTERIES = Object.freeze({
375+
'a definition pinning a tier alias → GREEN': 1,
376+
'a definition pinning a fully-qualified model id → GREEN': 1,
377+
'a quoted value → GREEN (both YAML spellings are legal)': 1,
378+
'a value with a trailing comment → GREEN': 1,
379+
'R1 — frontmatter with name+description and NO model: → RED': 1,
380+
'R2 — `model:` present but empty → RED': 1,
381+
'R3 — no frontmatter at all → RED': 1,
382+
'R4 — unterminated frontmatter → RED': 1,
383+
'R5 — a typo of a tier alias → RED (declared-looking, undeclared in effect)': 1,
384+
'R5b — an alias with a version suffix is not an alias → RED': 1,
385+
'R6 — `model: inherit` with nothing justifying it → RED': 1,
386+
'R6b — `model: inherit` WITH a justification entry whose reason is in the file → GREEN': 1,
387+
'R7 — a justified inherit whose written reason was edited away → RED (self-invalidating)': 1,
388+
'R8 — a justification entry naming an unscanned file → RED (anti-dormancy)': 1,
389+
'R9 — a justification entry on a file that now pins a tier → RED (dead config)': 1,
390+
'R10 — an empty scan → RED, never a green skip (#4690, the whole point)': 1,
391+
'one bad definition among several → RED, naming the bad one only': 1,
392+
'an indented `model:` inside description prose does NOT satisfy the rule → RED': 1,
393+
});
394+
395+
// DELETING an entry silences that battery's floor exactly as effectively as
396+
// zeroing it, so the roster's own size is pinned too. This pin is also half of
397+
// the duplicate-label refusal: two rows sharing a label collapse to ONE key in
398+
// the literal above, so the roster falls below this number; the table
399+
// cross-check in the floor block is the other half, and names WHICH label
400+
// collided.
401+
const SELF_TEST_BATTERY_FLOOR = 18;
402+
337403
function selfTest() {
338404
const fm = (modelLine) =>
339405
`---\nname: demo\ndescription: >\n A demo agent.\n${modelLine ? `${modelLine}\n` : ''}---\n\nBody.\n`;
@@ -466,8 +532,15 @@ function selfTest() {
466532
},
467533
];
468534

535+
// The ledger this self-test's floor is evaluated against (#13489).
536+
const batterySeen = new Map();
537+
const registerCase = (name) => {
538+
batterySeen.set(name, (batterySeen.get(name) ?? 0) + 1);
539+
};
540+
469541
let failed = 0;
470542
for (const c of cases) {
543+
registerCase(c.label);
471544
let problems;
472545
try {
473546
({ problems } = runAllChecks(c.files, c.justified ?? []));
@@ -542,8 +615,64 @@ function selfTest() {
542615
console.log(` ✓ real-tree discovery: ${disc.files.length} agent definition(s), ${disc.problems.length} layout problem(s)`);
543616
}
544617

618+
// ── The floor: every declared row RAN, and ran its case (#13489) ───────
619+
//
620+
// Evaluated after every row has had its chance and BEFORE the verdict, so the
621+
// success line below can only be printed by a run in which the set of rows
622+
// that registered EQUALS the set declared. A set difference names WHICH row
623+
// stopped; a count says only that something did.
624+
const floorFailure = (message) => {
625+
console.error(`✗ self-test floor: ${message}`);
626+
failed += 1;
627+
};
628+
const declaredBatteries = Object.keys(SELF_TEST_BATTERIES);
629+
let floorBreached = false;
630+
if (declaredBatteries.length < SELF_TEST_BATTERY_FLOOR) {
631+
floorBreached = true;
632+
floorFailure(
633+
`SELF_TEST_BATTERIES declares ${declaredBatteries.length} batteries, below the pinned ` +
634+
`${SELF_TEST_BATTERY_FLOOR} — a battery deleted from the roster takes its own floor with it.`,
635+
);
636+
}
637+
const rowLabels = cases.map((c) => c.label);
638+
const duplicated = [...new Set(rowLabels.filter((name, i) => rowLabels.indexOf(name) !== i))];
639+
if (duplicated.length > 0) {
640+
floorBreached = true;
641+
floorFailure(
642+
`the cases table uses ${duplicated.map((n) => JSON.stringify(n)).join(', ')} as a row label more than once — ` +
643+
'two rows sharing a label are ONE battery, so the second can stop running while the first keeps the floor met.',
644+
);
645+
}
646+
for (const [name, count] of batterySeen) {
647+
if (declaredBatteries.includes(name)) continue;
648+
floorBreached = true;
649+
floorFailure(
650+
`self-test battery "${name}" registered ${count} case(s) but is not declared in ` +
651+
'SELF_TEST_BATTERIES — a case attributed to no declared battery is one nothing floors.',
652+
);
653+
}
654+
for (const name of declaredBatteries) {
655+
const count = batterySeen.get(name) ?? 0;
656+
if (count >= SELF_TEST_BATTERIES[name]) continue;
657+
floorBreached = true;
658+
floorFailure(
659+
count === 0
660+
? `self-test battery "${name}" DID NOT RUN — 0 cases registered, ${SELF_TEST_BATTERIES[name]} pinned. ` +
661+
'The verdict below would have claimed that case holds.'
662+
: `self-test battery "${name}" registered ${count} case(s), below its pinned floor of ` +
663+
`${SELF_TEST_BATTERIES[name]} — cases that used to run no longer do.`,
664+
);
665+
}
666+
if (floorBreached) {
667+
floorFailure(
668+
'A battery at or below its floor means cases STOPPED RUNNING — the battery is the bug, not the ' +
669+
'number. Find what stopped registering (a deleted row, a renamed label, a loop that no longer ' +
670+
'reaches it) and restore it.',
671+
);
672+
}
673+
545674
if (failed > 0) {
546-
console.error(`\n✗ check-agent-model-declared self-test failed (${failed} case(s)).`);
675+
console.error(`\n✗ check-agent-model-declared self-test: ${failed} failure(s) (cases and floor).`);
547676
process.exit(1);
548677
}
549678
console.log(`\n✓ check-agent-model-declared self-test: ${cases.length} cases pass.`);

‎scripts/check-startup-registry-verdict.mjs‎

Lines changed: 126 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1059,6 +1059,68 @@ function run({ list = false, packagesDir } = {}) {
10591059
// handshake is a flag rather than a returned sentinel.
10601060
let selfTestReachedVerdict = false;
10611061

1062+
// ── The self-test's own battery roster and floor (#13489) ──────────────────
1063+
//
1064+
// `failures === 0` used to be this self-test's ONLY success condition, so
1065+
// "every case held" and "the cases never ran" printed the same line. Closed the
1066+
// PR #13487 way: what is pinned is the registered NAMES, not a number.
1067+
//
1068+
// This self-test is TABLE-DRIVEN — one literal `cases` table, one loop over it,
1069+
// and a sink (`failures++`) that writes only when a case FAILS. Routing THAT
1070+
// sink through `registerCase()` would register a case only when it fails: a
1071+
// fully green run would register 0 and every battery would read DID NOT RUN,
1072+
// the floor inverted rather than installed. So the roster is the table's own
1073+
// rows. Each row `name` is a declared battery, verbatim, with a floor of 1, and
1074+
// `registerCase(c.name)` is the first statement of the driving loop body — so
1075+
// the case is attributed to the row actually being run, whatever that row
1076+
// asserts afterwards. There is no `battery()` opener: for a table-driven
1077+
// self-test the ROW is the battery, so attribution is the loop variable rather
1078+
// than a most-recently-opened section.
1079+
//
1080+
// ⛔ A pinned TOTAL is not the repair, and neither is a roster DERIVED from the
1081+
// table: `cases.length` moves with the table, so a deleted row would delete its
1082+
// own floor. The roster below is a LITERAL the table is checked against, which
1083+
// is what lets a deleted or renamed row name ITSELF in the refusal.
1084+
//
1085+
// The counts are a FLOOR, not an equality — a row that grows into several
1086+
// registrations must not red. 1 is the honest floor for a table row: the loop
1087+
// reaches it exactly once per run.
1088+
//
1089+
// SCOPE, stated so the next reader does not mistake the number: the dead-root
1090+
// reverse proof below the loop drives `expectRoot(...)` eight times over a real
1091+
// temporary tree. Those are NOT rows of this table — they are written inline,
1092+
// one at a time — and the verdict line has always counted the 18 analysis rows
1093+
// and named the dead-root proof separately. This roster floors the 18 rows;
1094+
// the eight `expectRoot` assertions are left exactly as they are.
1095+
const SELF_TEST_BATTERIES = Object.freeze({
1096+
'#4777/A flags: init() probes a service and ANNOUNCES the absence': 1,
1097+
'#4777/A flags: init() probes a service and CACHES the absence on the instance': 1,
1098+
'#4777/A flags: init() probe whose absence is PERSISTED': 1,
1099+
'#4777/A passes: probe DEFERRED into a lazily-resolved closure': 1,
1100+
'#4777/A passes: probe deferred into a kernel:ready hook': 1,
1101+
'#4777/A passes: read-only probe — used, never recorded': 1,
1102+
'#4777/A passes: the same shape in start() — every init() has completed by then': 1,
1103+
'#4777/A passes: non-literal service name is skipped, never guessed': 1,
1104+
'#4777/A flags: the probe hidden behind a same-file helper init() calls': 1,
1105+
'#4777/A passes: the probe is licensed by optionalDependencies (the kernel orders the provider ahead)': 1,
1106+
'#4777/A passes: the probe is licensed by requiresServices': 1,
1107+
'#4777/A passes: a plugin probing the service it provides itself': 1,
1108+
'#4777/B flags: open node-type registry judged and announced before any seal': 1,
1109+
'#4777/B passes: the verdict is drawn at the declared seal': 1,
1110+
'#4777/B passes: read-only audit — returns state, never announces': 1,
1111+
'#4777/B passes: keyed lookup at registration warning about a duplicate': 1,
1112+
'#4777/B passes: keyed descriptor lookup on a runtime path': 1,
1113+
'#4777/B flags: the read hidden behind one same-file indirection': 1,
1114+
});
1115+
1116+
// DELETING an entry silences that battery's floor exactly as effectively as
1117+
// zeroing it, so the roster's own size is pinned too. This pin is also half of
1118+
// the duplicate-label refusal: two rows sharing a label collapse to ONE key in
1119+
// the literal above, so the roster falls below this number; the table
1120+
// cross-check in the floor block is the other half, and names WHICH label
1121+
// collided.
1122+
const SELF_TEST_BATTERY_FLOOR = 18;
1123+
10621124
function selfTest() {
10631125
// Every fixture is analysed on its own, so the provider index is built from
10641126
// the fixture itself — which means each one must declare the provider it is
@@ -1355,8 +1417,15 @@ function selfTest() {
13551417
},
13561418
];
13571419

1420+
// The ledger this self-test's floor is evaluated against (#13489).
1421+
const batterySeen = new Map();
1422+
const registerCase = (name) => {
1423+
batterySeen.set(name, (batterySeen.get(name) ?? 0) + 1);
1424+
};
1425+
13581426
let failures = 0;
13591427
for (const c of cases) {
1428+
registerCase(c.name);
13601429
const sf = parseSourceFile('t.ts', `${PROVIDERS}\n${c.code}`, ts.ScriptKind.TS);
13611430
const findings = [];
13621431
const seams = [];
@@ -1437,8 +1506,64 @@ function selfTest() {
14371506
rmSync(dir, { recursive: true, force: true });
14381507
}
14391508

1509+
// ── The floor: every declared row RAN, and ran its case (#13489) ───────
1510+
//
1511+
// Evaluated after every row has had its chance and BEFORE the verdict, so the
1512+
// success line below can only be printed by a run in which the set of rows
1513+
// that registered EQUALS the set declared. A set difference names WHICH row
1514+
// stopped; a count says only that something did.
1515+
const floorFailure = (message) => {
1516+
console.error(`✗ self-test floor: ${message}`);
1517+
failures++;
1518+
};
1519+
const declaredBatteries = Object.keys(SELF_TEST_BATTERIES);
1520+
let floorBreached = false;
1521+
if (declaredBatteries.length < SELF_TEST_BATTERY_FLOOR) {
1522+
floorBreached = true;
1523+
floorFailure(
1524+
`SELF_TEST_BATTERIES declares ${declaredBatteries.length} batteries, below the pinned ` +
1525+
`${SELF_TEST_BATTERY_FLOOR} — a battery deleted from the roster takes its own floor with it.`,
1526+
);
1527+
}
1528+
const rowLabels = cases.map((c) => c.name);
1529+
const duplicated = [...new Set(rowLabels.filter((name, i) => rowLabels.indexOf(name) !== i))];
1530+
if (duplicated.length > 0) {
1531+
floorBreached = true;
1532+
floorFailure(
1533+
`the cases table uses ${duplicated.map((n) => JSON.stringify(n)).join(', ')} as a row label more than once — ` +
1534+
'two rows sharing a label are ONE battery, so the second can stop running while the first keeps the floor met.',
1535+
);
1536+
}
1537+
for (const [name, count] of batterySeen) {
1538+
if (declaredBatteries.includes(name)) continue;
1539+
floorBreached = true;
1540+
floorFailure(
1541+
`self-test battery "${name}" registered ${count} case(s) but is not declared in ` +
1542+
'SELF_TEST_BATTERIES — a case attributed to no declared battery is one nothing floors.',
1543+
);
1544+
}
1545+
for (const name of declaredBatteries) {
1546+
const count = batterySeen.get(name) ?? 0;
1547+
if (count >= SELF_TEST_BATTERIES[name]) continue;
1548+
floorBreached = true;
1549+
floorFailure(
1550+
count === 0
1551+
? `self-test battery "${name}" DID NOT RUN — 0 cases registered, ${SELF_TEST_BATTERIES[name]} pinned. ` +
1552+
'The verdict below would have claimed that case holds.'
1553+
: `self-test battery "${name}" registered ${count} case(s), below its pinned floor of ` +
1554+
`${SELF_TEST_BATTERIES[name]} — cases that used to run no longer do.`,
1555+
);
1556+
}
1557+
if (floorBreached) {
1558+
floorFailure(
1559+
'A battery at or below its floor means cases STOPPED RUNNING — the battery is the bug, not the ' +
1560+
'number. Find what stopped registering (a deleted row, a renamed label, a loop that no longer ' +
1561+
'reaches it) and restore it.',
1562+
);
1563+
}
1564+
14401565
if (failures > 0) {
1441-
console.error(`\n✗ self-test: ${failures} case(s) failed\n`);
1566+
console.error(`\n✗ self-test: ${failures} failure(s) (cases and floor)\n`);
14421567
return 1;
14431568
}
14441569
console.log(`\n✓ self-test: ${cases.length} analysis case(s) + the dead-root hard error (red when the scan root is renamed, green when restored) all passed\n`);

0 commit comments

Comments
 (0)