Skip to content

chore(skills): routing lines, routing and behavior evals, and fixes found by running them - #75

Merged
ajbarea merged 2 commits into
mainfrom
chore/skill-routing-and-prose
Sep 23, 2026
Merged

ajbarea merged 2 commits into
mainfrom
chore/skill-routing-and-prose

Conversation

@ajbarea

@ajbarea ajbarea commented Sep 23, 2026

Copy link
Copy Markdown
Owner

Routing lines on every description that competes with a general skill, the first eval suite, and the fixes that running things turned up.

Measured

  • Routing, this branch (make evals, 23 cases × 3 runs, general PDF / PPTX / inbox stand-ins loaded in every case): 69 of 69 runs pass. Every skill fires on natural phrasing. On merging or filling a PDF, a .tex build, editing a slide's title, an email catch-up, and launching an app with no spectator, the expected skill fires (the general stand-in, techne:latex, or the built-in run) and the techne skill stays quiet, 18 of 18 runs.
  • Routing, main's descriptions: all 14 skills reached fired 3/3 (sisters 2/2); the run stopped on an account spend limit before slides, theoros and the collision cases, so the before/after on collisions is unmeasured.
  • Behavior: deslop flagged the planted slop and kept the load-bearing why-comment, 2/2; docsync caught a nonexistent make build and --verbose without flagging the real make test / --quiet, 2/2. auto-commit wrote a correct two-group plan with a current fingerprint in 2 of 2 headless runs; its eval case needs Bash in the run, which the eval sandbox refuses on a machine whose ~/.docker holds Docker Desktop's WSL links, so it sits under make evals-bash.

Fixed

  • sisters check 1 matched ^\s*uses: and so missed every - uses: step: 8 of 34 pins seen in one repo, 6 of 17 in another. Its README check claimed a team exemption it never read from techne.toml. Checks move to references/checks.md; all twelve were run against the live sisters.
  • theoros called a repo's make theoros; a repo whose target launches its own autonomous driver would start a second, permission-bypassing Claude beside the one already driving. The skill now ships scripts/theoros.sh (up / status / down, prerequisites, ops pane), covered by tmux tests.
  • auto-commit's staleness hash (git diff HEAD | git hash-object) ignored untracked files, so a plan made before a new file appeared still read as current. scripts/fingerprint.sh covers them and excludes COMMITS.md; PRs get assignee and label at creation.
  • paper builds through techne:latex (latexmk) instead of tectonic; a fresh scaffold's only finding is its TODO markers.
  • elenchus recommends ultra rather than trying to launch it; deslop runs small scopes inline.

make validate green (180 tests). Shellcheck now covers skill-shipped scripts.

…ound by running them

Descriptions say what each skill does, when to use it, and what it is
not for, including where a general PDF, PPTX or inbox catch-up skill
takes over. make evals runs a must-fire case per skill and must-not-fire
collision cases with rival stand-ins loaded, plus behavior cases that
grade deslop and docsync against fixture repos (evals-bash for cases
that need Bash in the run).

Fixes: sisters check 1 matched no "- uses:" pin, and its README check
exempted team repos without reading kind; checks move to a reference
file and non-Python sisters are skipped cleanly. theoros ships its own
up/down/status script and no longer calls a repo's make theoros, which
can start a second autonomous Claude. auto-commit fingerprints untracked
files and excludes COMMITS.md, and sets PR assignee and label. paper
builds through techne:latex. elenchus recommends ultra instead of
launching it. deslop runs small scopes inline.
@ajbarea ajbarea added the enhancement New feature or request label Sep 23, 2026
@ajbarea ajbarea self-assigned this Sep 23, 2026
… evals prove the right skill fired

theoros.sh addresses panes by id, so base-index 1 no longer breaks the
split; reads prerequisites in either key order; unquotes YAML values;
normalizes session names the way tmux does, so a dotted repo slug no
longer orphans a session; fails when the REPL exits at once; drops a
stale state file on status; escapes its JSON; and aborts every command
on a missing session_name. The driving instructions use those pane ids.

fingerprint.sh hashes the staged diff separately from the unstaged one,
handles an untracked nested repo and a repo with no commits, answers the
same from any directory, and prints nothing when git fails.

Each collision eval now also requires the expected skill to fire. The
tmux tests run on a private server and HOME. Em-dashes are gone from the
lines this branch added, and two doc claims now match the code.
@ajbarea

ajbarea commented Sep 23, 2026

Copy link
Copy Markdown
Owner Author

Findings (all fixed in 80e0e74; each has a regression test, and the new tests fail against 0135420)

  • theoros.sh: panes were addressed as :0.0 / :0.1, so a tmux.conf with base-index 1 broke the split and left a live session with no state file. I reproduced it; panes are now addressed by id (%N), and the driving instructions use those ids.
  • theoros.sh: a prerequisite written message: before command: was silently skipped, and up reported ready. Reproduced; either key order now aborts on a failing check.
  • theoros.sh: quoted YAML values kept their quotes ("cat -u" ran as one word); a REPL that died on start still printed "ready"; status printed a dead session's state; a missing session_name fell through to exit 0 in status. All reproduced and fixed.
  • theoros.sh: tmux rewrites . in a session name to _, so a dotted slug like demo.site-theoros reported a false "REPL exited" and orphaned a session status and down could never find. Reproduced; names are normalized the way tmux stores them.
  • theoros.sh: the state file's JSON was unescaped. A repo path containing " and \ now round-trips through json.loads.
  • fingerprint.sh: an untracked nested repo exited 123 while still printing a partial hash; a repo with no commits printed the empty-input hash; outside a repo it printed a hash and exited 0; untracked paths printed relative to the cwd, so a subdirectory gave a different answer. All reproduced; it now handles each, and prints nothing when git fails.
  • fingerprint.sh: two different partial stagings of one file hashed the same. The staged diff is now hashed separately from the unstaged one.
  • Collision evals only asserted that techne stayed quiet, so a run where nothing fired passed. Each now also requires the expected skill: the general PDF / PPTX / inbox stand-in, techne:latex, or the built-in run. 18 of 18 runs pass.
  • The tmux tests ran against the default server and personal tmux.conf, and teardown called tmux unconditionally. They now use a private server and HOME.
  • The Makefile comment and IMPL.md overstated and understated the eval scope respectively; both now match. Em-dashes are gone from the lines this branch added.

Refuted

  • none

Verified clean

scripts/eval-plugin.sh's rm -rf targets a path fixed from the script's own location, unreachable from any argument; theoros removes only $STATE_DIR/<session>.state; routing suite 69 of 69 on this branch's descriptions; behavior cases for deslop and docsync 2 of 2 each; make validate green (189 tests); shellcheck clean on every skill-shipped script.

@ajbarea
ajbarea merged commit ae5fc74 into main Sep 23, 2026
3 checks passed
@ajbarea
ajbarea deleted the chore/skill-routing-and-prose branch September 23, 2026 01:45
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant