feat(scripts): report-only skill-example schema probe — ruling B step one readings

[objectstack-fleet]

Part of #22059
Clause-②: no

Step ① of ruling B (comment 6050857613): the probe and its readings, not the gate. It adds a report-only mode, node scripts/check-corpus-claim-drift.mjs --probe-examples [--json FILE], backed by a new module scripts/skill-example-probe.mjs. The mode exits 0 on any reading. It is wired into no check:* script, no workflow and no baseline. The drift script's --self-test gains four Probe … batteries (65 cases) that pin the probe's pure parts over synthetic schemas, so neither the self-test nor the ratchet needs a build.

Headline

reading	value
hit rate, cards filed before #22059 with a landed published-catalog repair	2 / 19 (10.5%)
hit rate, every card the premise listing matched at filing	2 / 20 (10.0%)
hit rate, every card the listing matches today	2 / 22 (9.1%)
the ruling's bar	one half
ceiling with a perfect transform (cards whose repair changed example code)	6 / 19 (31.6%)

Every denominator is below one half, and so is the ceiling. That ceiling does not depend on how good the transform is: 9 of the 19 repairs touched no example fence at all, and 4 more changed only comments or string contents inside one.

How the probe works (the module header is the authority)

• Population. Every *.md under skills/objectstack-* with evals/ excluded: 49 files. A fence is in scope when the first token of its opener is ts, typescript, tsx, json, json5, jsonc or yaml/yml. jsonc is outside the dispatched list; it is admitted as JSON-with-comments and reported on its own column. Every other fence is counted as other.
• Transform. TS and JSON fences are parsed through scripts/ts-parse.mjs and never executed. Literals are evaluated statically. The supported shapes are defineX({…}), X.create({…}) / XSchema.parse({…}), typed literals (const a: T =, satisfies T, as T), key: value fragments, bare literals and YAML documents. A fence that does not read as a whole is split at blank lines. Same-fence const bindings are followed, and Field.kind(…) runs on the spec's own pure helper. Anything else is opaque: issues at, under or above an opaque node are suppressed and counted, never guessed.
• Mapping. Mapping is tried in this order. (1) The code's own claim: the factory or constructor it calls, or the spec type it is annotated with. (2) An os:check-yaml marker's declaration. (3) Key ownership. Tier 1 is the one registry metadata type (or the stack) that declares every key, judged partially. Tier 2 is used only when tier 1 finds no owner: the one named schema the fragment is a complete instance of. A stack's map-format collections are normalised as defineStack normalises them. A component's properties is judged by its ComponentPropsMap row, the dispatch check-yaml-examples already performs. Schemas come from this tree's built packages/spec/dist, rebuilt first. The probe refuses a dist older than src.
• Verdicts kept apart.
  • Structural red: a Zod issue, reported per code.
  • Counter-example: code the text marks as wrong, either under a ❌ or "Incorrect" heading or with a leading ❌ comment. These are never a red and never a replay signature.
  • Coverage miss: a fence that is not read or not mapped, with its reason.

Census: population fences by skill and language

skill	typescript	ts	tsx	json	jsonc	other fences	judged whole	partly	coverage miss
objectstack-ai	2	1	0	0	0	2	2	0	1
objectstack-api	3	1	0	0	0	4	3	0	1
objectstack-automation	10	4	0	0	0	2	8	1	5
objectstack-data	103	1	0	0	1	6	66	0	39
objectstack-formula	0	9	0	0	0	1	0	0	9
objectstack-i18n	6	0	0	1	0	5	4	0	3
objectstack-platform	64	0	0	1	0	11	6	1	58
objectstack-query	41	0	0	0	0	1	1	0	40
objectstack-ui	32	0	1	0	0	5	25	0	8
objectstack-upgrade	1	0	0	0	1	12	2	0	0
total	262	16	1	2	2	49	117	2	164

There are no YAML fences in the published catalog: the YAML half of the ruling has an empty population. The other fences are 27 bash, 13 with no language, 3 js, 2 md, 1 each of jsx, markdown, sql and text.

Transform and mapping coverage

• 279 TypeScript fences. The transform produced a value for 203 (72.8%). It missed 76: 67 no-candidate (runtime code such as hook bodies and engine calls, with no metadata literal) and 9 syntax (ellipsis placeholders and prose inside the fence).
• Mapping. Of the 203, 117 were mapped whole, 2 partly, and 86 not mapped: 46 ambiguous-owner and 40 unmapped (nested sub-schema fragments such as where: and node config:).
• Whole. 115 of the 279 TS fences (41.2%) were read and mapped whole; 117 of all 283 population fences (41.3%).
• Mapped items by shape. factory-call 45 · create-call 25 · property-list 39 · bare-literal 33 · typed-literal 6. By tier: 76 by the code's own claim, 60 by tier-1 ownership, 12 by tier-2 inference.
• os:check blocks. 33 skills fences carry the os:check marker. The probe judges 32 of them whole.

Replay: the cards premise 1 counts

The set. The premise's own listing (domain:skills, state=all, since=2026-09-23T00:00Z, title regex as quoted) returns 116 issues across pages today, and 22 titles match. Two of them, #22120 and #22123, were created after #22059 was filed (2026-10-07T05:02Z), so 20 matched at filing, not 21. The command as written reads page 1 only; today that page holds 21 matches, the two post-filing cards included and #18964 (page 2) excluded. All 12 ids the premise names are in the set.

Method. Each card's landing PR is the merged PR whose merge instant is the card's closed timeline event, located on main by its squash subject. Both sides are read from the local object store and judged by this tree's schemas, which is a stated bias in both directions. A fence is touched when its body is not found verbatim on the other side. A HIT is a structural signature on a touched BEFORE fence that no touched AFTER fence carries.

card	landing PR	squash	class	touched fences (before/after)	note
#18964	#19792	98d83361d9	not-in-corpus	0/0	edited .claude/skills/pm-dispatch/SKILL.md (a seat process finding)
#20090	#20118	180ef90fb7	comment-or-string	1/1	the error was a // comment in the fence
#20173	#20257	8af914a30d	HIT	1/1	page:header properties.actions held an Action object; ComponentPropsMap refuses it (expected string); os:check block
#20196	#20256	6704717188	comment-or-string	2/2	comment text only
#20275	#20298	29720975b6	comment-or-string	1/1	comment text only; rest is prose
#20385	#20410	789b2ae54f	prose	0/0
#20500	#20776	80dc9c0231	prose	0/0	table row
#20569	#20796	eac538c96d	prose	0/0	table row
#20657	#20778	7a09eee1b1	neither	1/1	an http node carrying a webhook url is schema-valid (a lint advisory, not a parse)
#20782	#20811	1d2024538c	transform	5/1	where: fragments ambiguous across 4 query schemas; see note 3
#20888	#20902	4957ee5ef0	transform	3/2	same
#21211	#21283	9fb9b25362	prose	0/0
#21288	#21302	125ce9f718	prose	0/0
#21392	#21402	a5139404d8	HIT	1/1	select with no options; FieldSchema refinement; os:check block
#21415	#21652	bff5aa2d98	prose	0/0
#21537	#21651	fea67065a3	neither	2/2	formViews.default is a schema-valid named form; the error is semantic
#21567	#21650	1a230548cf	prose	0/0
#21583	#21656	506fb6dac3	prose	0/0	table row
#21588	#21677	eed2dee481	prose	0/0
#21627	#21667	8ad9694fb7	comment-or-string	1/1	the error sat inside a source: string
#22120 (after filing)	#22122	033e5c536d	prose	0/0	5 bash fences touched
#22123 (after filing)	none		no-fix		still open

Misses by class over the 22: prose 10 · comment-or-string 4 · neither 2 · transform 2 · not-in-corpus 1 · no-fix 1.

Notes:

1. Both HITs sit in blocks carrying the os:check marker. check:skill-examples type-checks those blocks, and it cannot see either error: a Zod refinement and an open properties record both type-check.
2. skills(objectstack-ui): pages.md and its eval teach full Action objects in page:header.properties.actions; the contract is action ids (ruling B on #11592) and os validate flags the taught shape #20173's AFTER side is still red on today's schemas, for page:header breadcrumb, which was retired in spec 17 after that repair. The signature the repair removed is what makes it a HIT.
3. Counterfactual for the two transform misses. Judged by the where member of any of the four candidate schemas, both sides are accepted: that member accepts { status: { $foo: 1 } } too. They would read neither with a perfect mapping.

Control: the current catalog on this tree

verdict	fences
green	110
red	3
counter-example	4
partly	2
coverage miss	164

There are 14 issues suppressed as opaque or open. Counter-examples: 8 are refused, as their text says (indexing.md ×2, lifecycle.md ×5 rules, relationships.md:382). 2 are accepted by the schema; both are semantic ❌ (relationships.md:351 lookup vs master_detail, pagination.md:75 keyset field).

The three reds:

site	verdict	reading
skills/objectstack-ui/rules/actions.md:77 (os:check block)	custom at body	a teaching error on main: type: 'modal' with a body. The built defineAction throws on it ("body only runs for type: 'script' …").
skills/objectstack-ui/rules/actions.md:176	custom at target	a teaching error on main: type: 'url' with no target. defineAction throws "Action 'target' is required when type is 'url', …".
skills/objectstack-upgrade/references/examples-upgrade.md:7	invalid_type / invalid_value	a probe gap, not an error: the deliberate protocol-16 FROM shape the upgrade skill migrates

The two actions.md sites are reported as findings for the seat to file. ⛔ This PR does not edit them. A control check confirms the existing gate misses the first one: pnpm --filter @objectstack/spec run check:skill-examples is green on this head ("262 prose examples type-check across 3 surface(s)") with the actions.md:77 block in it.

Acceptance notes

• Premise corrections ([Decision] published skills/** teaching errors are found by hand (21 of 109 lane cards in two weeks): run the skill evals or a schema check over skill examples in CI (a new gate), or keep maintainer-scored evals #22059's Dedupe paragraph and premise 1):
  • Four existing gates already read example blocks, opt-in: check:skill-examples (os:check, TypeScript), check:yaml-examples (os:check-yaml), check:doc-formula-expressions and check:doc-security-posture.
  • The listing counts 20 cards at filing, not 21.
• Probe gaps a gate built on this probe would have to close:
  • an expect-red convention for the upgrade skill's FROM shapes;
  • the 86 ownership misses (nested fragments; ambiguous query fragments);
  • the Plugin type, held out by TYPE_NOT_A_SPEC_SHAPE.
• Stated bias: the replay is judged by today's schemas, not the schemas of each repair's own day.
• [finding] The teaching corpus has no lexical anti-drift pin, so a filter operator's prose can contradict its driver forever — the reachable half of "no gate can catch it" #13582, Published-skills factual sweep: verify every behavioral claim in skills/** against the implementation — program anchor #13658, Skills catalog optimization program (maintainer-directed): audit every skill on both roots, then optimize — program anchor #14292 and Published skills: the four SKILL.md files over 500 lines become routing + rules with examples and the platform ops tail moved under references/; TOCs on the 7 reference files over 300 lines; data's description trimmed under its cap (maintainer direct dispatch, batch #213 item 1) #19715 remain as is. The 22 replayed cards remain as is.

Verification (head e6d00e1421)

• pnpm check:corpus-claim-drift: exit 0. The self-test prints OK with the four Probe … batteries at 14/18/21/12 cases. The ratchet prints "OK, no new claim sites", over 252 files.
• node scripts/check-corpus-claim-drift.mjs --probe-examples --json …: exit 0, with the readings above. Apart from its head line, the output is byte-identical on every run from before the first commit to this head.
• One-off ablations, each through scripts/ablation-replace.mjs, each restored with blob equal to HEAD and an empty git diff HEAD:
  • opaque suppression disabled: self-test exit 1 (2 cases red);
  • the HIT rule disabled: exit 1 (1 case red);
  • the probe registration call disabled: exit 1 (all four probe batteries "DID NOT RUN").
• Gate families derived by node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack on this head (30 commands): all exit 0, check:pm-dispatch-gates included.
• Roster families keyed under scripts/:
  • check-published-list-mirrors (both), check:dts-closure, check:engine-double-contract, check:i18n-stale-fill and check-dts-references --self-test: exit 0.
  • NOT MEASURED: check:published-readme-exports, PREREQUISITE NOT MET (exit 3, 45 packages unbuilt). check:console-injection skipped (no console dist).
• eslint, narrowed to the 2 touched files:
  • eslint --no-inline-config --format json reports 2 files, 0 errors, 0 warnings.
  • The population comes from eslint.config.mjs (files: ['**/*.{ts,tsx,mts,cts,js,jsx,mjs,cjs}']).
  • No parserOptions.project or typed rules are configured, so this diff cannot move a verdict on an untouched file.
• pnpm check:nul-bytes: exit 0. The control-byte self-scan of both files: none.

Reads and writes

• REST GETs (reads): the card and its comments (2), the lane listing (3 pages) and 22 timelines.
• Writes: git push, this draft, one label-write, one os-dev-report comment. No MCP calls.

---

Generated by Claude Code

[objectstack-fleet]

Closing without merging, by the ruling on #22059 (comment 6050857613, letter B, maintainer 「批 #287 同意」): the probe's hit rate is 2 / 19 (10.5%), under the bar of one half on every denominator the body reports, and its ceiling with a perfect transform is 6 / 19, so the ruling's own text falls to C: no gate, the finding process as today, and the probe's readings stay on the card. Skills seat 1, session_01CXydFDyiQwNbGFkmwrcRQq, 2026-10-08T04:00Z.

What this PR delivered is accepted as step ① of the ruling: the readings (this body, and the report 6051826214 on #22059) are the record. What it also carries, a 1,711-line probe module and four self-test batteries that no gate would run, is the shape the ruling declined as D ("an offline report nobody reads"), so it does not land. The branch claude/issue-22059-skill-example-schema-probe stays at e6d00e1421 as the probe's source of record; reopening this PR is one act if the maintainer ever names the tool as wanted.

The two teaching errors the catalog control surfaced (skills/objectstack-ui/rules/actions.md:77 and :176) are filed as #22176; this PR did not edit them, as its body says.

---

Generated by Claude Code