feat(rhodibot): classify what a file check found - #547
Merged
Merged
Conversation
…lot's mistake
A criterion that names files has four possible answers -- present where the
canon records it, present under a location the canon retired, present
somewhere else, absent -- and they are not interchangeable. This adds the
classifier that decides which.
The rule it holds: the description decides what is required; `template_ref`
only says where the canon's own template keeps it. The pilot read
`template_ref` as the requirement and failed every repository that satisfied
the criterion elsewhere -- both of its false positives were criterion 1.2.2,
which wants "a .pre-commit-config.yaml with real hooks" and is satisfied by one
at the repository root even though the template keeps its copy under `ci/`.
A bare filename -- "STATE.a2ml", "Justfile" -- is therefore accepted in the two
places that are unambiguously the repository's own: the canon's recorded
location, or the repository root. Everywhere else is `Elsewhere`, a finding,
with the canon's location named as a note. Both halves of that rule were
learned by running this against real trees:
- "anywhere" passed 1.2.4 (`.tool-versions`) on a copy at
`robot-repo-automaton/templates/skeleton/.tool-versions`, another project's
template skeleton;
- "root only" would have failed gitbot-fleet's
`.machine_readable/descriptiles/*.a2ml`, which satisfy 3.1.2-3.1.7 exactly
where the canon keeps them.
Deprecated locations are read from the canon rather than hardcoded. The canon
states that `.machine_readable/6a2/` is retired in prose, inside criterion
3.1.1's description -- "NOT 6a2/, which is deprecated 2026-06-30" -- and there
is no machine-readable table, so the sentence is parsed, resolved against the
criterion's own `template_ref`, and asserted by a lockstep test. A canon that
stops saying it fails the test instead of quietly reclassifying every retired
path as ordinary.
A repository carrying the file in both a live place and a retired one is
reported as relocated with the leftover named, not as living in the retired
path: the move is under way, and saying "deprecated location" would overstate
it. An absent file also names what is sitting there in its place by extension
-- gitbot-fleet has CODE_OF_CONDUCT.adoc where 2.1.4 asks for .md.
Against two real repositories, from their trees on 2026-09-19:
nesy-solver 12 satisfied, 4 relocated, 3 deprecated, 3 missing
gitbot-fleet 17 satisfied, 3 relocated, 0 deprecated, 2 missing
The pilot's own record for nesy-solver -- three findings, all descriptive files
left under the retired path -- is asserted in the lockstep suite, as is the
1.2.2 false positive staying fixed.
Advisory only: rhodibot reports against canon criterion ids.
Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
Contributor
|
Note Currently processing new changes in this PR. This may take a few minutes, please wait... ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Advanced Run ID: 📒 Files selected for processing (3)
✨ Finishing Touches📝 Generate docstrings
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| @@ -0,0 +1,1044 @@ | |||
| // SPDX-License-Identifier: MPL-2.0 | |||
hyperpolymath
added a commit
that referenced
this pull request
Sep 19, 2026
… line (#549) Fourth and last step of the canon-sourced rules (#543, #545, #547). The three pieces before this one read the rules from the canon, decide which criteria apply, and classify what was found. Nothing ran them. This adds `rhodibot canon`: ```sh rhodibot canon --owner hyperpolymath --repo ubicity rhodibot canon --path /path/to/checkout --format json rhodibot canon --owner O --repo R --fail-on missing ``` **Advisory by default** — exits 0 whatever it finds. The canon designates hypatia's `rsr-conformance` as the single normative checker, so a gate here would be a second opinion claiming an authority it does not have. `--fail-on` is opt-in. ## Two details that carry their weight - `GitHubClient::tree_paths` asks once for the whole file list and **refuses a truncated response**. A partial tree is missing files, a missing file reads as an absent one, and the report would manufacture findings against a repository that has the file. - `get_file_content_if_present` distinguishes 404 from everything else: an absent `rsr-profile.a2ml` means "declares no capabilities", a 500 means "could not read the declaration". Treating the second as the first would silently shrink the check to the universal criteria and report a cleaner scorecard than the truth. `canon/report.rs` assembles the report (counts, findings worst-first, retired locations quoted) and serialises to JSON. `canon/local.rs` reads a checkout, preferring `git ls-files` over a walk — a walked list includes build output, and a `CODE_OF_CONDUCT.md` inside `target/` is not the repository's code of conduct. ## The pilot re-run through the tool Unauthenticated, against the five repositories of the advisory pilot: | repository | at path | elsewhere | deprecated | absent | |---|---|---|---|---| | gitbot-fleet | 17 | 3 | 0 | 2 | | ubicity | 7 | 2 | 6 | 7 | | awesome-nickel | 8 | 1 | 7 | 6 | | julia-professional-registry | 8 | 1 | 7 | 6 | | nesy-solver | 12 | 4 | 3 | 3 | **110 questions, 52 satisfied, 58 findings** (11 relocated, 23 at the retired `6a2/`, 24 absent). 22 of the canon's 74 criteria apply to every one of them: 26 are gated on a capability none declares, 26 more ask content questions rather than naming files. The 23 files under `.machine_readable/6a2/` are still 23, re-derived from criterion 3.1.1's own sentence about the retirement rather than from a hardcoded path, with the same per-repository breakdown as the hand run. Both of the hand run's false positives (criterion 1.2.2) now come back satisfied. ## Two findings the earlier scope could not see - `2.1.3` asks for `SECURITY.md`; `ubicity` has `SECURITY.adoc`. The verdict is `absent` and the report now says what is there instead. Same for `CODE_OF_CONDUCT.md`/`CODE_OF_CONDUCT.adoc` at `2.1.4`. If `.adoc` is the estate's convention, the canon is where that belongs. - `2.2.1` is `relocated` for all five: the `.well-known/` files live inside a `www/` publication bundle, not at the repository root the criterion names. The origin-versus-repository question is worth an answer. ## Verification - `cargo test --locked` — 135 lib + 10 lockstep + 50 integration, all pass - `cargo clippy --locked --all-targets -- -D warnings` — clean - new wiremock tests cover the tree listing (directories excluded), the truncated tree refusal, and 404-versus-500 on the optional file read - `scripts/check-canon-drift.sh` — exit 0 against upstream `main` Signed-off-by: Jonathan D.A. Jewell <6759885+hyperpolymath@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Third of the four steps that turn the canon's criteria into checks (steps one and two landed in #543 and #545).
A criterion that names files has four possible answers, and they are not interchangeable:
The rule this holds
The description decides what is required;
template_refonly says where the canon's own template keeps it. The advisory pilot readtemplate_refas the requirement and failed repositories that satisfied the criterion elsewhere -- both of its false positives were criterion 1.2.2, satisfied by a root.pre-commit-config.yamleven though the template keeps its copy underci/.A bare filename -- "STATE.a2ml", "Justfile" -- is accepted in the two places that are unambiguously the repository's own: the canon's recorded location, or the repository root. Everywhere else is
Elsewhere, a finding, with the canon's location named as a note. Both halves of that rule were learned by running this against real trees:1.2.4(.tool-versions) on a copy atrobot-repo-automaton/templates/skeleton/.tool-versions-- another project's template skeleton;.machine_readable/descriptiles/*.a2ml, which satisfy 3.1.2-3.1.7 exactly where the canon keeps them.A description that gives a path (
.well-known/security.txt) still means that path: a copy elsewhere isElsewhere.Retirements are read, not hardcoded
The canon states
.machine_readable/6a2/is retired -- in prose, inside criterion 3.1.1's description: "NOT 6a2/, which is deprecated 2026-06-30". There is no machine-readable deprecation table, so the sentence is parsed, resolved against the criterion's owntemplate_ref, and asserted by a lockstep test. A canon that stops saying it fails the test rather than quietly reclassifying every retired path as ordinary.Details the real trees forced
relocatedwith the leftover named, notdeprecated: the move is under way, and "deprecated location" would overstate it (nesy-solver, 3.1.2-3.1.4);CODE_OF_CONDUCT.adocwhere 2.1.4 asks for.md, and a bare "absent" would send a reader looking for a file that is right there under another name;Missing > Deprecated > Relocated > Satisfied.Against real repositories
Trees read on 2026-09-19:
The pilot's own record for nesy-solver -- three findings, all descriptive files left under the retired path -- is asserted in the lockstep suite, as is 1.2.2 staying fixed.
Verification
cargo test --locked-- 117 lib + 10 lockstep + 50 integration, all passcargo clippy --locked --all-targets -- -D warnings-- cleanscripts/check-canon-drift.sh-- exit 0rustfmt --edition 2024 --checkclean on every file touchedAdvisory only: rhodibot reports against canon criterion ids; hypatia remains the canon's single normative checker.