chore(skills): routing lines, routing and behavior evals, and fixes found by running them - #75
Merged
Merged
Conversation
…ound by running them Descriptions say what each skill does, when to use it, and what it is not for, including where a general PDF, PPTX or inbox catch-up skill takes over. make evals runs a must-fire case per skill and must-not-fire collision cases with rival stand-ins loaded, plus behavior cases that grade deslop and docsync against fixture repos (evals-bash for cases that need Bash in the run). Fixes: sisters check 1 matched no "- uses:" pin, and its README check exempted team repos without reading kind; checks move to a reference file and non-Python sisters are skipped cleanly. theoros ships its own up/down/status script and no longer calls a repo's make theoros, which can start a second autonomous Claude. auto-commit fingerprints untracked files and excludes COMMITS.md, and sets PR assignee and label. paper builds through techne:latex. elenchus recommends ultra instead of launching it. deslop runs small scopes inline.
… evals prove the right skill fired theoros.sh addresses panes by id, so base-index 1 no longer breaks the split; reads prerequisites in either key order; unquotes YAML values; normalizes session names the way tmux does, so a dotted repo slug no longer orphans a session; fails when the REPL exits at once; drops a stale state file on status; escapes its JSON; and aborts every command on a missing session_name. The driving instructions use those pane ids. fingerprint.sh hashes the staged diff separately from the unstaged one, handles an untracked nested repo and a repo with no commits, answers the same from any directory, and prints nothing when git fails. Each collision eval now also requires the expected skill to fire. The tmux tests run on a private server and HOME. Em-dashes are gone from the lines this branch added, and two doc claims now match the code.
Owner
Author
|
Findings (all fixed in 80e0e74; each has a regression test, and the new tests fail against 0135420)
Refuted
Verified clean
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Routing lines on every description that competes with a general skill, the first eval suite, and the fixes that running things turned up.
Measured
make evals, 23 cases × 3 runs, general PDF / PPTX / inbox stand-ins loaded in every case): 69 of 69 runs pass. Every skill fires on natural phrasing. On merging or filling a PDF, a.texbuild, editing a slide's title, an email catch-up, and launching an app with no spectator, the expected skill fires (the general stand-in,techne:latex, or the built-inrun) and the techne skill stays quiet, 18 of 18 runs.main's descriptions: all 14 skills reached fired 3/3 (sisters 2/2); the run stopped on an account spend limit beforeslides,theorosand the collision cases, so the before/after on collisions is unmeasured.deslopflagged the planted slop and kept the load-bearing why-comment, 2/2;docsynccaught a nonexistentmake buildand--verbosewithout flagging the realmake test/--quiet, 2/2.auto-commitwrote a correct two-group plan with a current fingerprint in 2 of 2 headless runs; its eval case needs Bash in the run, which the eval sandbox refuses on a machine whose~/.dockerholds Docker Desktop's WSL links, so it sits undermake evals-bash.Fixed
sisterscheck 1 matched^\s*uses:and so missed every- uses:step: 8 of 34 pins seen in one repo, 6 of 17 in another. Its README check claimed a team exemption it never read fromtechne.toml. Checks move toreferences/checks.md; all twelve were run against the live sisters.theoroscalled a repo'smake theoros; a repo whose target launches its own autonomous driver would start a second, permission-bypassing Claude beside the one already driving. The skill now shipsscripts/theoros.sh(up / status / down, prerequisites, ops pane), covered by tmux tests.auto-commit's staleness hash (git diff HEAD | git hash-object) ignored untracked files, so a plan made before a new file appeared still read as current.scripts/fingerprint.shcovers them and excludesCOMMITS.md; PRs get assignee and label at creation.paperbuilds throughtechne:latex(latexmk) instead of tectonic; a fresh scaffold's only finding is itsTODOmarkers.elenchusrecommendsultrarather than trying to launch it;deslopruns small scopes inline.make validategreen (180 tests). Shellcheck now covers skill-shipped scripts.