Verifiable gates: a mechanical checker suite + CI, and the fixes it caught - #59
Open
richm-spp wants to merge 8 commits into
Open
Verifiable gates: a mechanical checker suite + CI, and the fixes it caught#59richm-spp wants to merge 8 commits into
richm-spp wants to merge 8 commits into
Conversation
The skill's runtime behaviour is 'read the index, load only the picks', so a link that doesn't resolve is a failed file read mid-build. Nothing in the repo checked this. Reports 31 broken instances / 28 unique file->target pairs on the current tree, all in docs/ and site/_tests/. The runtime graph inside skills/ is clean at 274/274, which --runtime asserts separately. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Derives the theme list and the gate list from the artifacts, then asserts the prose agrees. Catches the class of drift that produced the gate 57 defect: grid was added by PR Nutlope#57 and the gate enumerating the catalog was never updated, so the gate silently stopped covering the case it exists for. Finds 3 real disagreements on the current tree: README claims 57 gates twice (the artifact has 58 - 1-57 plus sub-gate 38a), and gate 57's allowlist omits grid. Each assertion was falsified before being trusted: version drift, a 22nd theme landing, and a gate deleted without renumbering all turn it red. An allowlist is detected as a comma-separated run of 5+ theme names, so the example mentions in gates 38a and 55 ('Studio / Garden / Sport') no longer false-positive. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve of the 58 gates are pure pattern-matching over emitted CSS/HTML. This runs them for real: a small dependency-free CSS parser (rules carry their at-rule context, so a declaration inside @media (prefers-reduced-motion) is distinguishable from one outside — several gates hinge on exactly that), one-level var() resolution, and inline <style> extraction. Hard tier fails the run; advisory tier (needs-judgment gates: prose measure, off-scale spacing, token bypass, 1fr-with-image) reports without failing, and --strict promotes it. Against the 30 committed examples: 61 hard failures, 246 advisories across 18 pages — while the pages' own CSS stamps declare 'gates: all-pass'. The findings are the gates' own letter: <em> inside h1/h2 (banned by discipline 6), body { overflow-x: hidden } (gate 34 mandates clip), uppercase display at line-height 0.82-0.92 (gate 55 floor is 1.0), paper tokens at chroma 0.003 (gate 22 minimum is 0.005). Self-scoring said pass; the machine disagrees. Checker was falsified against its own false positives before being trusted: black box-shadows are compositing not palette (gate 22 now skips shadow/mask props and alpha'd values), display headlines are not prose (gate 25 excludes heading selectors and is advisory), .masthead-meta is not in gate 38a's enumeration (selector tightened to the gate's own list). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
links and counts gate hard (they will go green in the fix commits that follow). gates runs as a regression baseline: the historical examples carry 61 known hard failures, so the job fails only if that number grows. Ratchet the baseline down as examples are fixed; never up. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every relative link in docs/ was broken (wrong depth prefix), the verbs fixtures were one level short of the repo root, two links also pointed at study-examples.md in a directory it never lived in, and one link targeted the never-committed refine verb fixture (unlinked; the prose contrast stands). The runtime graph in skills/ needed nothing - it was already 274/274. links.mjs: 31 broken -> 0. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Grid landed via PR Nutlope#57 and neither the README's gate count nor gate 57's own catalog enumeration was updated - so the gate written to catch studied-DNA drift could not fire for the newest theme, and the README undercounted by one (1-57 plus sub-gate 38a). Both now guarded by tools/verify/counts.mjs, which derives the real numbers from the artifacts. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The minimal wordmark nav was addressed as both N1 and N1a across seven files
('N1 (N1a)' in the cookbook index), while the load rule maps a code to
components/<code>-<slug>.md - so the N1a spelling pointed at a file that does
not exist (n1a-*.md). Prose now converges on N1, matching
n1-wordmark-2-links.md on disk; N1b keeps its own file and code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
python3 -m http.server is not available on a standard Windows install, and Windows users are half the audience of a Claude Code / Cursor skill. npx serve works wherever node does - which the skill already requires. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@richm-spp is attempting to deploy a commit to the Together AI Team on Vercel. A member of the Team first needs to authorize it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Hallmark's 58 gates are all self-scored by the model that produced the output. This PR makes the mechanically checkable subset actually able to fail, and fixes what the checkers found on the current tree.
Three zero-dependency checkers (
tools/verify/, Node ≥ 18, no packages)links.mjsdocs/andsite/_tests/; the runtime graph insideskills/was already clean (274/274)counts.mjspackage.jsonvs SKILL.md version) match the artifacts they describegates.mjs1frimage tracks, uppercase +line-height < 1, double stickytop:0, missing reduced-motion, …) over any HTML/CSS, with a small CSS parser that respects@mediacontextEach checker was watched to fail against the tree before anything was fixed, then the fixes landed as separate commits: the 28 link repairs, README 57→58, gate 57 + Grid, and folding the
N1/N1adouble-naming into one code per archetype (SKILL.md used both for the same file; an ambiguous code is a failed file load at build time).CI
.github/workflows/verify.ymlruns all three on push/PR (runs are ~10–20 s). Green on the fork: https://github.com/richm-spp/hallmark/actions — this repo has no CI today, so nothing runs here until the workflow lands.Also:
npm run serveusedpython3, which isn't on PATH for most Windows users — nownpx serve site.Happy to split any of this out or adjust the workflow triggers to taste.