feat(rhodibot): gate criteria on declared capabilities, and classify what a file check found - #545
Merged
Merged
Conversation
…capability 26 of the canon's 74 criteria are gated. Asked as universal questions they demand files a repository has no reason to carry -- an FFI seam from a docs site, a container from a library -- which is the same mistake as reading `template_ref` as a requirement, one level up. `src/canon/profile.rs` reads `.machine_readable/rsr-profile.a2ml` and answers whether a criterion applies: - effective set = (direct list | preset expansion) + add - remove, per the gate table's own model - applicability is Universal, Declared(capability) or NotDeclared(capability); `na` leaves the denominator rather than counting against the repository - a misspelt capability, an unknown preset, a misspelt key and removing something never declared are all refused: `capabilities = ["russt"]` quietly collapsing to an empty set would switch a whole gate of checks off and report the repository as clean The gate table is vendored and pinned like the criteria -- same release, same rule -- and `scripts/check-canon-drift.sh` now checks both artefacts against the canon and against `canon.lock`. `GateTable::unknown_gates` fails if a criterion gates on a capability the vocabulary does not define, because such a criterion can never become applicable: that is how `deno` came to be listed. Against the five pilot repositories, which declare nothing: of the 31 criteria whose description names files, 22 are scored and 9 are `na` (6 governance-tier, 2 web-ui, 1 docs-site). The pilot's 11 universal criteria are all in the scored set, so the earlier pilot numbers still hold.
Contributor
|
Warning Review limit reachedNext included review available in 15 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (6)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
hyperpolymath
changed the title
hyperpolymath
added a commit
that referenced
this pull request
Sep 19, 2026
Third of the four steps that turn the canon's criteria into checks (steps one and two landed in #543 and #545). A criterion that names files has four possible answers, and they are not interchangeable: | verdict | meaning | worth | |---|---|---| | present at the path | the file is where the canon records it | nothing to do | | present under a retired location | the file exists, in a place the canon has retired | move it | | present elsewhere | the file exists, in neither place | check it | | absent | the file is not there | add it | ## The rule this holds **The description decides what is required; `template_ref` only says where the canon's own template keeps it.** The advisory pilot read `template_ref` as the requirement and failed repositories that satisfied the criterion elsewhere -- both of its false positives were criterion 1.2.2, satisfied by a root `.pre-commit-config.yaml` even though the template keeps its copy under `ci/`. A bare filename -- "STATE.a2ml", "Justfile" -- is accepted in the two places that are unambiguously the repository's own: **the canon's recorded location, or the repository root**. Everywhere else is `Elsewhere`, a finding, with the canon's location named as a note. Both halves of that rule were learned by running this against real trees: - **"anywhere"** passed `1.2.4` (`.tool-versions`) on a copy at `robot-repo-automaton/templates/skeleton/.tool-versions` -- another project's template skeleton; - **"root only"** would have failed gitbot-fleet's `.machine_readable/descriptiles/*.a2ml`, which satisfy 3.1.2-3.1.7 exactly where the canon keeps them. A description that gives a path (`.well-known/security.txt`) still means that path: a copy elsewhere is `Elsewhere`. ## Retirements are read, not hardcoded The canon states `.machine_readable/6a2/` is retired -- in prose, inside criterion 3.1.1's description: "NOT 6a2/, which is deprecated 2026-06-30". There is no machine-readable deprecation table, so the sentence is parsed, resolved against the criterion's own `template_ref`, and asserted by a lockstep test. A canon that stops saying it fails the test rather than quietly reclassifying every retired path as ordinary. ## Details the real trees forced - a repository holding the file in **both** a live place and a retired one is `relocated` with the leftover named, not `deprecated`: the move is under way, and "deprecated location" would overstate it (nesy-solver, 3.1.2-3.1.4); - an absent file names what is sitting there instead -- gitbot-fleet has `CODE_OF_CONDUCT.adoc` where 2.1.4 asks for `.md`, and a bare "absent" would send a reader looking for a file that is right there under another name; - severities are ordered `Missing > Deprecated > Relocated > Satisfied`. ## Against real repositories Trees read on 2026-09-19: | repository | satisfied | relocated | deprecated | missing | |---|---|---|---|---| | nesy-solver | 12 | 4 | 3 | 3 | | gitbot-fleet | 17 | 3 | 0 | 2 | The pilot's own record for nesy-solver -- three findings, all descriptive files left under the retired path -- is asserted in the lockstep suite, as is 1.2.2 staying fixed. ## Verification - `cargo test --locked` -- 117 lib + 10 lockstep + 50 integration, all pass - `cargo clippy --locked --all-targets -- -D warnings` -- clean - `scripts/check-canon-drift.sh` -- exit 0 - `rustfmt --edition 2024 --check` clean on every file touched Advisory only: rhodibot reports against canon criterion ids; hypatia remains the canon's single normative checker. Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
hyperpolymath
added a commit
that referenced
this pull request
Sep 19, 2026
… line (#549) Fourth and last step of the canon-sourced rules (#543, #545, #547). The three pieces before this one read the rules from the canon, decide which criteria apply, and classify what was found. Nothing ran them. This adds `rhodibot canon`: ```sh rhodibot canon --owner hyperpolymath --repo ubicity rhodibot canon --path /path/to/checkout --format json rhodibot canon --owner O --repo R --fail-on missing ``` **Advisory by default** — exits 0 whatever it finds. The canon designates hypatia's `rsr-conformance` as the single normative checker, so a gate here would be a second opinion claiming an authority it does not have. `--fail-on` is opt-in. ## Two details that carry their weight - `GitHubClient::tree_paths` asks once for the whole file list and **refuses a truncated response**. A partial tree is missing files, a missing file reads as an absent one, and the report would manufacture findings against a repository that has the file. - `get_file_content_if_present` distinguishes 404 from everything else: an absent `rsr-profile.a2ml` means "declares no capabilities", a 500 means "could not read the declaration". Treating the second as the first would silently shrink the check to the universal criteria and report a cleaner scorecard than the truth. `canon/report.rs` assembles the report (counts, findings worst-first, retired locations quoted) and serialises to JSON. `canon/local.rs` reads a checkout, preferring `git ls-files` over a walk — a walked list includes build output, and a `CODE_OF_CONDUCT.md` inside `target/` is not the repository's code of conduct. ## The pilot re-run through the tool Unauthenticated, against the five repositories of the advisory pilot: | repository | at path | elsewhere | deprecated | absent | |---|---|---|---|---| | gitbot-fleet | 17 | 3 | 0 | 2 | | ubicity | 7 | 2 | 6 | 7 | | awesome-nickel | 8 | 1 | 7 | 6 | | julia-professional-registry | 8 | 1 | 7 | 6 | | nesy-solver | 12 | 4 | 3 | 3 | **110 questions, 52 satisfied, 58 findings** (11 relocated, 23 at the retired `6a2/`, 24 absent). 22 of the canon's 74 criteria apply to every one of them: 26 are gated on a capability none declares, 26 more ask content questions rather than naming files. The 23 files under `.machine_readable/6a2/` are still 23, re-derived from criterion 3.1.1's own sentence about the retirement rather than from a hardcoded path, with the same per-repository breakdown as the hand run. Both of the hand run's false positives (criterion 1.2.2) now come back satisfied. ## Two findings the earlier scope could not see - `2.1.3` asks for `SECURITY.md`; `ubicity` has `SECURITY.adoc`. The verdict is `absent` and the report now says what is there instead. Same for `CODE_OF_CONDUCT.md`/`CODE_OF_CONDUCT.adoc` at `2.1.4`. If `.adoc` is the estate's convention, the canon is where that belongs. - `2.2.1` is `relocated` for all five: the `.well-known/` files live inside a `www/` publication bundle, not at the repository root the criterion names. The origin-versus-repository question is worth an answer. ## Verification - `cargo test --locked` — 135 lib + 10 lockstep + 50 integration, all pass - `cargo clippy --locked --all-targets -- -D warnings` — clean - new wiremock tests cover the tree listing (directories excluded), the truncated tree refusal, and 404-versus-500 on the optional file read - `scripts/check-canon-drift.sh` — exit 0 against upstream `main` Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two steps of the four that turn the canon's criteria into checks. (The first, reading requirements from
descrather thantemplate_ref, landed in #543.)1. Capability gates --
4d7eea426 of the canon's 74 criteria are gated on a capability. Asked as universal questions they demand files a repository has no reason to carry -- an FFI seam from a docs site, a container from a library.
src/canon/profile.rsreads.machine_readable/rsr-profile.a2ml:(direct list | preset expansion) + add - remove, the gate table's own model;ApplicabilityisUniversal,Declared(capability)orNotDeclared(capability); a gated criterion leaves the denominator asnarather than counting against the repository;capabilities = ["russt"]quietly collapsing to an empty set would switch a whole gate of checks off and report the repository as clean.The gate table is vendored and pinned like the criteria -- same release, same rule -- under
[gates]incanon/pin.toml(b65ce754..., matchingcanon.lock'sgatesslot and the spine profile'sgates_sha256).scripts/check-canon-drift.shnow checks both artefacts; both failure paths are exercised.Against the five pilot repositories, which declare nothing: of the 31 criteria whose description names files, 22 are scored and 9 are
na(6 governance-tier, 2 web-ui, 1 docs-site).2. Verdict classification --
96c70e3The four answers a file check can give are not interchangeable: present where the canon records it, present under a location the canon retired, present elsewhere, absent.
src/canon/verdict.rsholds the rule the pilot established: the description decides what is required;template_refonly says where the canon's own template keeps it. Both of the pilot's false positives were criterion 1.2.2 -- satisfied by a root.pre-commit-config.yamleven though the template keeps its copy underci/.A bare filename is accepted in the two places that are unambiguously the repository's own: the canon's recorded location, or the repository root. Both halves of that rule were learned by running it against real trees:
1.2.4(.tool-versions) on a copy atrobot-repo-automaton/templates/skeleton/.tool-versions-- another project's template skeleton;.machine_readable/descriptiles/*.a2ml, which satisfy 3.1.2-3.1.7 exactly where the canon keeps them.Deprecated locations are read from the canon, not hardcoded: the canon states
.machine_readable/6a2/is retired in prose, inside criterion 3.1.1's description, and there is no machine-readable table. The sentence is parsed, resolved against the criterion's owntemplate_ref, and asserted by a lockstep test -- a canon that stops saying it fails the test rather than quietly reclassifying every retired path as ordinary.A repository holding the file in both a live place and a retired one is reported as relocated with the leftover named, not as living in the retired path. An absent file names what is sitting there instead (
CODE_OF_CONDUCT.adocwhere 2.1.4 asks for.md).Against two real trees, read on 2026-09-19:
The pilot's own record for nesy-solver -- three findings, all descriptive files left under the retired path -- is asserted in the lockstep suite, as is 1.2.2 staying fixed.
Verification
cargo test --locked-- 117 lib + 10 lockstep + 50 integration, all passcargo clippy --locked --all-targets -- -D warnings-- cleanscripts/check-canon-drift.sh-- exit 0 againstmain; exit 1 when either pin is forgedrustfmt --edition 2024 --checkclean on every file touchedAdvisory only: rhodibot reports against canon criterion ids; hypatia remains the canon's single normative checker.