diff --git a/.github/workflows/validate.yml b/.github/workflows/validate.yml index 4c5e8d1..cc294bd 100644 --- a/.github/workflows/validate.yml +++ b/.github/workflows/validate.yml @@ -41,6 +41,9 @@ jobs: # cmarker from Typst Universe. Declining that keeps the pipeline # hermetic; the markdown front end is covered here unconditionally. TECHNE_NO_TYPST: "1" + # techne:theoros's lifecycle tests drive real tmux sessions. Where the runner + # has tmux they run regardless; the declaration only permits its absence. + TECHNE_NO_TMUX: "1" - name: Build docs site (same command docs.yml deploys with) diff --git a/.gitignore b/.gitignore index 92a793b..62176de 100644 --- a/.gitignore +++ b/.gitignore @@ -20,6 +20,10 @@ site/ # scripts/dev-runner.sh archives logs/ +# Routing evals: the assembled eval-only plugin and per-run results +plugins/techne/eval-fixtures/with-rivals/ +plugins/techne/evals/results/ + # Claude Code session worktrees .claude/worktrees/ diff --git a/IMPL.md b/IMPL.md index 494d1f6..a9bcd16 100644 --- a/IMPL.md +++ b/IMPL.md @@ -11,19 +11,24 @@ has crept in — extract it back to ROADMAP. ## In flight -**`techne:slides`** (branch `feat/slides-skill`). - -- **Why:** a talk deck rebuilt for a student audience surfaced the same checks by hand - every time: real slide titles, contrast, fonts that survive Google Slides, numbers the - room does not need, and rendering through the app that will present it. -- **Decisions:** Typst + Touying for decks presented as PDF, pptxgenjs when a `.pptx` is - required. The checker parses OOXML with the stdlib and needs no Office install; render - prefers PowerPoint over COM (reached from WSL) because LibreOffice substitutes fonts. - `# research(2026-09)`: WCAG 1.4.6 thresholds (7:1, large 4.5:1 at 18pt / 14pt bold); - pptxgenjs table margins are inches since v3.8.0 (read in `pptxgen.cjs.js`, docs say points). -- **Scope:** `SKILL.md`, `scripts/slides.py` (`check`, `render`), unit tests, every listing. -- **Out of scope:** a Touying template; the pptx API itself (the Anthropic `pptx` skill). -- **Done when:** `make validate` green, independent review clean, CI green, merged. +**Skill routing and prose pass** (branch `chore/skill-routing-and-prose`). + +- **Why:** general document skills (PDF, PPTX, inbox catch-up) now share sessions with techne, + and their descriptions claim any PDF or `.pptx` work. Nothing tested which skill fires. +- **Decisions:** `# research(2026-09)`: platform.claude.com skill-authoring best practices + (what + when in the description, under 1,024 characters, SKILL.md under 500 lines, + references one level deep, scripts for deterministic work) and code.claude.com plugin-evals + (`tool_used: Skill` graders; a case's `plugins` list replaces the plugin under test, so + techne and the rival stand-ins are assembled into one eval-only plugin). +- **Scope:** routing and scope lines in descriptions; `make evals` routing suite with + must-fire and must-not-fire cases; `paper` builds through `techne:latex`; `sisters` split + into a reference file with its pin-regex, team-exemption and non-Python fixes; behavior + cases for deslop, docsync and auto-commit against fixture repos; theoros ships + its lifecycle script and stops calling a repo's `make theoros`; auto-commit fingerprint + covers untracked files; elenchus recommends `ultra` instead of launching it. +- **Out of scope:** behavioral evals for the prose-only skills beyond deslop, docsync and + auto-commit. +- **Done when:** `make validate` green, routing suite passing, reviewed, CI green, merged. ## Skill collection state diff --git a/Makefile b/Makefile index 5e96f53..a0fa00f 100644 --- a/Makefile +++ b/Makefile @@ -4,7 +4,7 @@ ## that techne itself documents at docs/conventions.md. ## -.PHONY: help check-env setup manifests frontmatter fix lint shellcheck guards test-unit zizmor test validate build ci clean docs +.PHONY: help check-env setup manifests frontmatter fix lint shellcheck guards test-unit zizmor test validate build ci clean docs evals evals-bash .DEFAULT_GOAL := help check-env: ## Verify required tools are on PATH @@ -33,8 +33,8 @@ lint: ## ruff check + format check + ty on scripts/ and skill- @uv run ruff format --check scripts/ plugins/ tests/ @uv run ty check scripts/ plugins/ tests/ -shellcheck: ## shellcheck on scripts/*.sh (via shellcheck-py PyPI binary) - @uv run shellcheck --severity=warning scripts/*.sh +shellcheck: ## shellcheck on repo and skill-shipped shell scripts (shellcheck-py binary) + @uv run shellcheck --severity=warning scripts/*.sh plugins/techne/skills/*/scripts/*.sh # Skill names are derived from the directory listing, so a new skill is guarded # the day it lands rather than when someone remembers to extend the pattern. @@ -74,6 +74,24 @@ build: ## Build docs site (strict; mirrors docs.yml deploy) ci: setup validate ## Mirror CI end-to-end (validate.yml, which includes the docs build) +# Routing evals run real Claude sessions on your own credential, so they cost money and +# stay out of validate. Every routing case loads stand-ins for the general document and +# catch-up skills techne shares a session with, so a collision shows up as a failed case. +# Behavior cases build a fixture repo with a scaffold script and grade what the skill produced. +evals: ## Routing + behavior evals (claude plugin eval; runs on your credential) + @bash scripts/eval-plugin.sh + @cd plugins/techne && claude plugin eval . --tag routing --ablation none --trust-plugin \ + --no-publish -j 2 --threshold 0.9 + @cd plugins/techne && claude plugin eval . --tag behavior --ablation none --trust-plugin \ + --no-publish -j 2 --threshold 0.9 --scaffold + +# Cases that need Bash inside the run. The eval sandbox refuses to grant Bash on a machine +# whose Docker credential store holds a symlink (Docker Desktop's WSL integration does). +evals-bash: ## Behavior evals that grant Bash (needs a symlink-free ~/.docker) + @cd plugins/techne && claude plugin eval . --tag behavior-bash --ablation none \ + --trust-plugin --no-publish -j 2 --threshold 0.9 --scaffold \ + --allow-tools Write 'Bash(git *)' 'Bash(bash *)' + clean: ## Remove ruff + build caches @rm -rf .ruff_cache .pytest_cache site/ diff --git a/ROADMAP.md b/ROADMAP.md index 04f62ae..778e02b 100644 --- a/ROADMAP.md +++ b/ROADMAP.md @@ -156,6 +156,14 @@ that drift class recurs. Detail lives in git history (`git log`) and the live skill code. This log is pruned once work is durably shipped. +- 2026-09-22 — **Routing and prose pass, and the first evals.** Descriptions now say what each + skill is not for and where a general PDF or PPTX skill takes over. `make evals` runs a routing + suite (a must-fire case per skill, must-not-fire collision cases, rival stand-ins loaded in + every case) and behavior cases against fixture repos. Fixes found by running things: + `sisters` check 1 missed every `- uses:` pin, and its team exemption never read `kind`; + theoros called a repo's `make theoros`, which can start a second, permission-bypassing Claude, + so it now ships its own lifecycle script; auto-commit's staleness hash ignored untracked files; + `paper` builds through `techne:latex` instead of tectonic. - 2026-09-22 — **`techne:slides`.** A stdlib OOXML checker gates a talk deck on real title placeholders, contrast resolved through the surface actually behind the text, alt text, portable fonts, and figures or walls of text on talk slides; a renderer exports through diff --git a/docs/architecture.md b/docs/architecture.md index d39cdb1..3767ee5 100644 --- a/docs/architecture.md +++ b/docs/architecture.md @@ -42,6 +42,14 @@ The maintenance invariant: whenever a Claude Code release, MCP spec revision, Gi This invariant runs philosophically; there is no automated check, it's a stance the maintainer holds when reading release notes. The sibling sister repos apply the same audit to their own code (see `phalanx-fl`, `velocity-fl`, `kourai-khryseai` ROADMAPs); techne's variant is meta, it audits the audit tools themselves. +## How the skills are tested + +Two layers, for two kinds of failure. + +**Scripts** (`make test-unit`). Every script a skill ships has pytest coverage: the LaTeX log gates, the markdown-to-Typst renderer, the catch-up sweep, the slide-deck checker, auto-commit's staleness fingerprint, and the theoros session lifecycle, which drives real tmux sessions. These run in CI. + +**Routing** (`make evals`). A skill that never fires does nothing, and a skill that fires on someone else's request does harm. `claude plugin eval` sends natural-language prompts to fresh sessions and records which skill Claude chose. Each skill has a case that must fire it, and collision cases must not: merging a PDF is not `techne:pdf`, an email catch-up is not `techne:catchup`, editing a slide's title is not `techne:slides`. Every case also loads stand-ins for the general PDF, PPTX, and inbox catch-up skills that share a session with techne, so an overlapping description fails a case instead of shipping. Evals run on your own credential and are not part of CI. + ## See also - [Conventions](conventions.md): the standard file locations and adoption path diff --git a/docs/examples.md b/docs/examples.md index d026c4d..67aafca 100644 --- a/docs/examples.md +++ b/docs/examples.md @@ -42,7 +42,7 @@ A collaborator wants to watch you refactor a complex module in real time without 2. Run `/techne:theoros`. It starts a detached tmux session named per the config. 3. Share the session name with collaborators. They attach read-only: `tmux attach -r -t `. 4. You drive the work through Claude; collaborators see live output. The split-window layout is optional; add an `ops_command` to the skill-context if you want a tailing logs pane underneath the driver pane. -5. Tear down with `make theoros-down` (if you've adopted the tier-2 Makefile targets) or `tmux kill-session -t `. +5. Tell Claude you are done; it runs the skill's `theoros.sh down`, which kills the session and removes its state file. ## CI noise cleanup diff --git a/docs/skills/elenchus.md b/docs/skills/elenchus.md index e9117e8..8aff38b 100644 --- a/docs/skills/elenchus.md +++ b/docs/skills/elenchus.md @@ -20,7 +20,7 @@ Adversarial pre-merge code review. Drives `/code-review`, then runs the three pa The model that writes a change is a poor judge of it: the author reviews what they intended, under ship-it momentum, against a mental model that hides what the code actually does. The fix is not a smarter model but an independent reader running a fixed protocol against the whole repo, reproducing as it goes. -Independence is restored mechanically, not by willpower. `/code-review` at `high` spawns independent local agents; `ultra` runs a multi-agent cloud review. Never hand-read your own diff and call it reviewed. +Independence is restored mechanically, not by willpower. `/code-review` at `high` spawns independent local agents; `ultra` runs a billed multi-agent cloud review that you launch yourself, and the skill recommends it rather than starting it. Never hand-read your own diff and call it reviewed. ## The three passes diff --git a/docs/skills/paper.md b/docs/skills/paper.md index 60901d6..fcf27b4 100644 --- a/docs/skills/paper.md +++ b/docs/skills/paper.md @@ -15,8 +15,9 @@ on day one and you only write prose. /techne:paper [--from ] [--venue ] ``` -Reads the `## paper` section of `.claude/skill-context.md` (author, bib path, LaTeX engine, -portfolio file), scaffolds `papers//`, build-verifies with the configured engine, and +Reads the `## paper` section of `.claude/skill-context.md` (author, bib path, portfolio +file), scaffolds `papers//`, build-verifies through the `techne:latex` runner (a fresh +scaffold's only finding is the template's `TODO` draft markers), and adds a row to the portfolio file. Then write prose into the `% HARVEST:` blocks; run `python harvest.py` to regenerate the evaluation table from the source repo's corpus. diff --git a/docs/skills/theoros.md b/docs/skills/theoros.md index 6000449..6ce27ea 100644 --- a/docs/skills/theoros.md +++ b/docs/skills/theoros.md @@ -22,17 +22,15 @@ The skill checks for an existing tmux session, starts one if absent, prints the tmux attach -t -r ``` -When done, the session tears down via `make theoros-down` (tier 2) or `tmux kill-session` (tier 1). +When done, the skill runs its bundled `theoros.sh down`, which kills the session and removes the state file. ## Prerequisites The `## theoros` section of `.claude/skill-context.md` must supply at minimum `repl_command` and `session_name`. Without it the skill aborts and explains what to add. -Two tiers: -- **Tier 1**: two-field YAML block in skill-context, no other files needed. -- **Tier 2**: `scripts/theoros.sh` + Makefile targets (`theoros`, `theoros-down`, `theoros-status`) for prerequisite gating and extended lifecycle. +Optional fields add a split log pane (`ops_command`) and checks that must pass before the session starts (`prerequisites`). The lifecycle script (`up`, `down`, `status`) ships with the skill, so the repo needs no script or Makefile target of its own. -The skill auto-detects tier 2 by checking for a `theoros:` target in the `Makefile`. +A repo's own `make theoros`, if it has one, is for a human starting a session outside Claude Code. The skill does not call it, because such a target may launch its own autonomous driver. See [Conventions](../conventions.md) for the scaffolding walkthrough. diff --git a/plugins/techne/eval-fixtures/rival-skills/skills/inbox-catch-up/SKILL.md b/plugins/techne/eval-fixtures/rival-skills/skills/inbox-catch-up/SKILL.md new file mode 100644 index 0000000..214ebdb --- /dev/null +++ b/plugins/techne/eval-fixtures/rival-skills/skills/inbox-catch-up/SKILL.md @@ -0,0 +1,8 @@ +--- +name: inbox-catch-up +description: Catch-up sweep across the user's email inboxes, calendar, and course site, reporting only what changed since the last run. Use when the user asks to be caught up on messages, mail, or course announcements. +--- + +# inbox-catch-up + +Eval stand-in. Say which file or source you would work on, then stop. diff --git a/plugins/techne/eval-fixtures/rival-skills/skills/pdf-toolkit/SKILL.md b/plugins/techne/eval-fixtures/rival-skills/skills/pdf-toolkit/SKILL.md new file mode 100644 index 0000000..f0a6638 --- /dev/null +++ b/plugins/techne/eval-fixtures/rival-skills/skills/pdf-toolkit/SKILL.md @@ -0,0 +1,8 @@ +--- +name: pdf-toolkit +description: Use this skill for any work with PDF files: reading or extracting text and tables, merging, splitting, rotating, watermarking, creating new PDFs, filling PDF forms, encrypting, extracting images, and OCR. Whenever the user mentions a .pdf file or asks to produce one, use this skill. +--- + +# pdf-toolkit + +Eval stand-in. Say which file or source you would work on, then stop. diff --git a/plugins/techne/eval-fixtures/rival-skills/skills/pptx-toolkit/SKILL.md b/plugins/techne/eval-fixtures/rival-skills/skills/pptx-toolkit/SKILL.md new file mode 100644 index 0000000..9672d47 --- /dev/null +++ b/plugins/techne/eval-fixtures/rival-skills/skills/pptx-toolkit/SKILL.md @@ -0,0 +1,8 @@ +--- +name: pptx-toolkit +description: Use this skill any time a .pptx file is involved: creating slide decks or presentations, reading or extracting their text, editing existing decks, templates, layouts, speaker notes, or comments. Trigger whenever the user asks for a PowerPoint or references a .pptx filename. +--- + +# pptx-toolkit + +Eval stand-in. Say which file or source you would work on, then stop. diff --git a/plugins/techne/evals/behavior-auto-commit/case.yaml b/plugins/techne/evals/behavior-auto-commit/case.yaml new file mode 100644 index 0000000..91373e0 --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: behavior-auto-commit +tags: [behavior-bash] +context: + scaffold_script: scaffold.sh diff --git a/plugins/techne/evals/behavior-auto-commit/graders/fired.md b/plugins/techne/evals/behavior-auto-commit/graders/fired.md new file mode 100644 index 0000000..1d704bd --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/graders/fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:auto-commit"' +--- diff --git a/plugins/techne/evals/behavior-auto-commit/graders/header.md b/plugins/techne/evals/behavior-auto-commit/graders/header.md new file mode 100644 index 0000000..7a96aba --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/graders/header.md @@ -0,0 +1,5 @@ +--- +type: regex +pattern: 'techne:auto-commit[\s\S]*tree-hash:\s+[0-9a-f]{12}' +target: { source: file, path: COMMITS.md } +--- diff --git a/plugins/techne/evals/behavior-auto-commit/graders/not-self-listed.md b/plugins/techne/evals/behavior-auto-commit/graders/not-self-listed.md new file mode 100644 index 0000000..d1aca75 --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/graders/not-self-listed.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '^Files:.*COMMITS\.md' +flags: m +match: not_contains +target: { source: file, path: COMMITS.md } +--- diff --git a/plugins/techne/evals/behavior-auto-commit/graders/plan-written.md b/plugins/techne/evals/behavior-auto-commit/graders/plan-written.md new file mode 100644 index 0000000..3bc33a7 --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/graders/plan-written.md @@ -0,0 +1,4 @@ +--- +type: file_exists +path: COMMITS.md +--- diff --git a/plugins/techne/evals/behavior-auto-commit/graders/two-groups.md b/plugins/techne/evals/behavior-auto-commit/graders/two-groups.md new file mode 100644 index 0000000..368e3d0 --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/graders/two-groups.md @@ -0,0 +1,7 @@ +--- +type: regex +pattern: '^(feat|fix|docs|chore|refactor)\([^)]+\): ' +flags: m +match: "count:2" +target: { source: file, path: COMMITS.md } +--- diff --git a/plugins/techne/evals/behavior-auto-commit/prompt.md b/plugins/techne/evals/behavior-auto-commit/prompt.md new file mode 100644 index 0000000..79a0a77 --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/prompt.md @@ -0,0 +1,8 @@ +--- +max_turns: 15 +timeout_seconds: 300 +runs: 2 +allowed_tools: [Read, Glob, Grep, Skill, Write, "Bash(git *)", "Bash(bash *)"] +--- + +Group my pending changes into sensible commits. Just write the plan; don't commit anything. diff --git a/plugins/techne/evals/behavior-auto-commit/scaffold.sh b/plugins/techne/evals/behavior-auto-commit/scaffold.sh new file mode 100755 index 0000000..0c73810 --- /dev/null +++ b/plugins/techne/evals/behavior-auto-commit/scaffold.sh @@ -0,0 +1,9 @@ +#!/bin/bash +set -euo pipefail +git init -q . && git config user.email t@t && git config user.name t +mkdir -p src +printf '# Tool\n\nTeh tool does things.\n' > README.md +printf 'def add(a, b):\n return a + b\n' > src/calc.py +git add -A && git commit -qm "feat(calc): add" +printf '# Tool\n\nThe tool does things.\n' > README.md +printf 'def mul(a, b):\n return a * b\n' > src/mul.py diff --git a/plugins/techne/evals/behavior-deslop/case.yaml b/plugins/techne/evals/behavior-deslop/case.yaml new file mode 100644 index 0000000..6845081 --- /dev/null +++ b/plugins/techne/evals/behavior-deslop/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: behavior-deslop +tags: [behavior] +context: + scaffold_script: scaffold.sh diff --git a/plugins/techne/evals/behavior-deslop/graders/fired.md b/plugins/techne/evals/behavior-deslop/graders/fired.md new file mode 100644 index 0000000..b9474bf --- /dev/null +++ b/plugins/techne/evals/behavior-deslop/graders/fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:deslop"' +--- diff --git a/plugins/techne/evals/behavior-deslop/graders/verdicts.md b/plugins/techne/evals/behavior-deslop/graders/verdicts.md new file mode 100644 index 0000000..f7e4083 --- /dev/null +++ b/plugins/techne/evals/behavior-deslop/graders/verdicts.md @@ -0,0 +1,6 @@ +--- +type: llm +--- + +PASS if the response proposes deleting or rewriting the "Seamlessly and robustly loads the configuration (April 2026 best practice)" comment AND proposes deleting at least one of the narration comments ("Now we open the file", "Return the result") AND does not propose deleting the comment explaining why the handle outlives the call (atexit / context manager). +FAIL if it leaves the "Seamlessly and robustly" comment alone, or proposes removing the atexit / context-manager comment. diff --git a/plugins/techne/evals/behavior-deslop/prompt.md b/plugins/techne/evals/behavior-deslop/prompt.md new file mode 100644 index 0000000..39d0c53 --- /dev/null +++ b/plugins/techne/evals/behavior-deslop/prompt.md @@ -0,0 +1,8 @@ +--- +max_turns: 12 +timeout_seconds: 300 +runs: 2 +allowed_tools: [Read, Glob, Grep, Skill] +--- + +Copilot filled src/app.py with fluffy AI comments. Clean the slop out of the comments. diff --git a/plugins/techne/evals/behavior-deslop/scaffold.sh b/plugins/techne/evals/behavior-deslop/scaffold.sh new file mode 100755 index 0000000..0fabab6 --- /dev/null +++ b/plugins/techne/evals/behavior-deslop/scaffold.sh @@ -0,0 +1,17 @@ +#!/bin/bash +set -euo pipefail +git init -q . && git config user.email t@t && git config user.name t +mkdir -p src +cat > src/app.py <<'PY' +# Seamlessly and robustly loads the configuration (April 2026 best practice). +def load_config(path): + # Now we open the file + with open(path) as f: + # Return the result + return f.read() + + +# The handle outlives this call on purpose: atexit closes it, so a context manager does not fit. +_LOG = open("/dev/null", "w") +PY +git add -A && git commit -qm init diff --git a/plugins/techne/evals/behavior-docsync/case.yaml b/plugins/techne/evals/behavior-docsync/case.yaml new file mode 100644 index 0000000..8a1bff5 --- /dev/null +++ b/plugins/techne/evals/behavior-docsync/case.yaml @@ -0,0 +1,5 @@ +schema_version: "1.1" +name: behavior-docsync +tags: [behavior] +context: + scaffold_script: scaffold.sh diff --git a/plugins/techne/evals/behavior-docsync/graders/findings.md b/plugins/techne/evals/behavior-docsync/graders/findings.md new file mode 100644 index 0000000..fb3b835 --- /dev/null +++ b/plugins/techne/evals/behavior-docsync/graders/findings.md @@ -0,0 +1,6 @@ +--- +type: llm +--- + +PASS if the response reports BOTH that `make build` has no matching Makefile target AND that `--verbose` is not a real flag of `python -m app`, and does not claim that `make test` or `--quiet` is wrong. +FAIL if it misses either real drift, or reports `make test` or `--quiet` as drift. diff --git a/plugins/techne/evals/behavior-docsync/graders/fired.md b/plugins/techne/evals/behavior-docsync/graders/fired.md new file mode 100644 index 0000000..d3bb90f --- /dev/null +++ b/plugins/techne/evals/behavior-docsync/graders/fired.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:docsync"' +--- diff --git a/plugins/techne/evals/behavior-docsync/prompt.md b/plugins/techne/evals/behavior-docsync/prompt.md new file mode 100644 index 0000000..d868ba5 --- /dev/null +++ b/plugins/techne/evals/behavior-docsync/prompt.md @@ -0,0 +1,8 @@ +--- +max_turns: 15 +timeout_seconds: 300 +runs: 2 +allowed_tools: [Read, Glob, Grep, Skill] +--- + +Check README.md for drift against the code. Report what is wrong; do not edit anything yet. diff --git a/plugins/techne/evals/behavior-docsync/scaffold.sh b/plugins/techne/evals/behavior-docsync/scaffold.sh new file mode 100755 index 0000000..85a3533 --- /dev/null +++ b/plugins/techne/evals/behavior-docsync/scaffold.sh @@ -0,0 +1,20 @@ +#!/bin/bash +set -euo pipefail +git init -q . && git config user.email t@t && git config user.name t +mkdir -p app +printf 'test:\n\tpytest\n\nlint:\n\truff check .\n' > Makefile +cat > app/__main__.py <<'PY' +import argparse + +parser = argparse.ArgumentParser() +parser.add_argument("--quiet", action="store_true") +args = parser.parse_args() +PY +cat > README.md <<'MD' +# app + +Build it with `make build`, then run the suite with `make test`. + +Run it with `python -m app --verbose` for detailed output, or `--quiet` to silence it. +MD +git add -A && git commit -qm init diff --git a/plugins/techne/evals/fires-audit/graders/fires-audit.md b/plugins/techne/evals/fires-audit/graders/fires-audit.md new file mode 100644 index 0000000..511b88b --- /dev/null +++ b/plugins/techne/evals/fires-audit/graders/fires-audit.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:audit"' +--- diff --git a/plugins/techne/evals/fires-audit/prompt.md b/plugins/techne/evals/fires-audit/prompt.md new file mode 100644 index 0000000..e37315c --- /dev/null +++ b/plugins/techne/evals/fires-audit/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Is my local build clean? Run the full audit of the make targets before I push. diff --git a/plugins/techne/evals/fires-auto-commit/graders/fires-auto-commit.md b/plugins/techne/evals/fires-auto-commit/graders/fires-auto-commit.md new file mode 100644 index 0000000..1d704bd --- /dev/null +++ b/plugins/techne/evals/fires-auto-commit/graders/fires-auto-commit.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:auto-commit"' +--- diff --git a/plugins/techne/evals/fires-auto-commit/prompt.md b/plugins/techne/evals/fires-auto-commit/prompt.md new file mode 100644 index 0000000..a3d406f --- /dev/null +++ b/plugins/techne/evals/fires-auto-commit/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +My working tree is a mess. Group these changes into sensible commits and draft the messages. diff --git a/plugins/techne/evals/fires-catchup/graders/fires-catchup.md b/plugins/techne/evals/fires-catchup/graders/fires-catchup.md new file mode 100644 index 0000000..2247d38 --- /dev/null +++ b/plugins/techne/evals/fires-catchup/graders/fires-catchup.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:catchup"' +--- diff --git a/plugins/techne/evals/fires-catchup/prompt.md b/plugins/techne/evals/fires-catchup/prompt.md new file mode 100644 index 0000000..dd4e70d --- /dev/null +++ b/plugins/techne/evals/fires-catchup/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Catch me up on the ajbarea/techne repo. Did anyone reply to my PRs while I was away? diff --git a/plugins/techne/evals/fires-ci-audit/graders/fires-ci-audit.md b/plugins/techne/evals/fires-ci-audit/graders/fires-ci-audit.md new file mode 100644 index 0000000..915f9d0 --- /dev/null +++ b/plugins/techne/evals/fires-ci-audit/graders/fires-ci-audit.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:ci-audit"' +--- diff --git a/plugins/techne/evals/fires-ci-audit/prompt.md b/plugins/techne/evals/fires-ci-audit/prompt.md new file mode 100644 index 0000000..7b627fe --- /dev/null +++ b/plugins/techne/evals/fires-ci-audit/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +CI passed on my PR but the GitHub Actions logs are full of warnings. Why, and can you fix them? diff --git a/plugins/techne/evals/fires-deslop/graders/fires-deslop.md b/plugins/techne/evals/fires-deslop/graders/fires-deslop.md new file mode 100644 index 0000000..b9474bf --- /dev/null +++ b/plugins/techne/evals/fires-deslop/graders/fires-deslop.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:deslop"' +--- diff --git a/plugins/techne/evals/fires-deslop/prompt.md b/plugins/techne/evals/fires-deslop/prompt.md new file mode 100644 index 0000000..3812eec --- /dev/null +++ b/plugins/techne/evals/fires-deslop/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Copilot left fluffy AI comments all over this codebase. Clean the slop out of the comments. diff --git a/plugins/techne/evals/fires-docs-site/graders/fires-docs-site.md b/plugins/techne/evals/fires-docs-site/graders/fires-docs-site.md new file mode 100644 index 0000000..df99285 --- /dev/null +++ b/plugins/techne/evals/fires-docs-site/graders/fires-docs-site.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:docs-site"' +--- diff --git a/plugins/techne/evals/fires-docs-site/prompt.md b/plugins/techne/evals/fires-docs-site/prompt.md new file mode 100644 index 0000000..efdb10f --- /dev/null +++ b/plugins/techne/evals/fires-docs-site/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Audit my Zensical docs site: the nav, the Pages deploy workflow, and broken anchors. diff --git a/plugins/techne/evals/fires-docsync/graders/fires-docsync.md b/plugins/techne/evals/fires-docsync/graders/fires-docsync.md new file mode 100644 index 0000000..d3bb90f --- /dev/null +++ b/plugins/techne/evals/fires-docsync/graders/fires-docsync.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:docsync"' +--- diff --git a/plugins/techne/evals/fires-docsync/prompt.md b/plugins/techne/evals/fires-docsync/prompt.md new file mode 100644 index 0000000..d3d3fa5 --- /dev/null +++ b/plugins/techne/evals/fires-docsync/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Check whether README.md still matches the code. I think the documented commands drifted. diff --git a/plugins/techne/evals/fires-elenchus/graders/fires-elenchus.md b/plugins/techne/evals/fires-elenchus/graders/fires-elenchus.md new file mode 100644 index 0000000..0da5b41 --- /dev/null +++ b/plugins/techne/evals/fires-elenchus/graders/fires-elenchus.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:elenchus"' +--- diff --git a/plugins/techne/evals/fires-elenchus/prompt.md b/plugins/techne/evals/fires-elenchus/prompt.md new file mode 100644 index 0000000..28d71b6 --- /dev/null +++ b/plugins/techne/evals/fires-elenchus/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Review this branch like a hostile reviewer and try to break it before I merge. diff --git a/plugins/techne/evals/fires-latex/graders/fires-latex.md b/plugins/techne/evals/fires-latex/graders/fires-latex.md new file mode 100644 index 0000000..ba2bbc8 --- /dev/null +++ b/plugins/techne/evals/fires-latex/graders/fires-latex.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:latex"' +--- diff --git a/plugins/techne/evals/fires-latex/graders/not-pdf.md b/plugins/techne/evals/fires-latex/graders/not-pdf.md new file mode 100644 index 0000000..d710a09 --- /dev/null +++ b/plugins/techne/evals/fires-latex/graders/not-pdf.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pdf"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/fires-latex/prompt.md b/plugins/techne/evals/fires-latex/prompt.md new file mode 100644 index 0000000..446cb38 --- /dev/null +++ b/plugins/techne/evals/fires-latex/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Compile thesis.tex and tell me whether the PDF is actually ready to submit. diff --git a/plugins/techne/evals/fires-paper-review/graders/fires-paper-review.md b/plugins/techne/evals/fires-paper-review/graders/fires-paper-review.md new file mode 100644 index 0000000..5ab6f43 --- /dev/null +++ b/plugins/techne/evals/fires-paper-review/graders/fires-paper-review.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:paper-review"' +--- diff --git a/plugins/techne/evals/fires-paper-review/prompt.md b/plugins/techne/evals/fires-paper-review/prompt.md new file mode 100644 index 0000000..0681c73 --- /dev/null +++ b/plugins/techne/evals/fires-paper-review/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Before I submit, check whether my paper's contributions are actually novel and what related work I'm missing. diff --git a/plugins/techne/evals/fires-paper/graders/fires-paper.md b/plugins/techne/evals/fires-paper/graders/fires-paper.md new file mode 100644 index 0000000..23b6ef3 --- /dev/null +++ b/plugins/techne/evals/fires-paper/graders/fires-paper.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:paper"' +--- diff --git a/plugins/techne/evals/fires-paper/prompt.md b/plugins/techne/evals/fires-paper/prompt.md new file mode 100644 index 0000000..6160f70 --- /dev/null +++ b/plugins/techne/evals/fires-paper/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Start a new paper called fedsim in my papers monorepo. diff --git a/plugins/techne/evals/fires-pdf/graders/fires-pdf.md b/plugins/techne/evals/fires-pdf/graders/fires-pdf.md new file mode 100644 index 0000000..eb0cb1b --- /dev/null +++ b/plugins/techne/evals/fires-pdf/graders/fires-pdf.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pdf"' +--- diff --git a/plugins/techne/evals/fires-pdf/graders/not-latex.md b/plugins/techne/evals/fires-pdf/graders/not-latex.md new file mode 100644 index 0000000..9339ed5 --- /dev/null +++ b/plugins/techne/evals/fires-pdf/graders/not-latex.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:latex"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/fires-pdf/prompt.md b/plugins/techne/evals/fires-pdf/prompt.md new file mode 100644 index 0000000..24d1c60 --- /dev/null +++ b/plugins/techne/evals/fires-pdf/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Turn notes.md into a nicely typeset PDF I can print. diff --git a/plugins/techne/evals/fires-research-grounded/graders/fires-research-grounded.md b/plugins/techne/evals/fires-research-grounded/graders/fires-research-grounded.md new file mode 100644 index 0000000..514e12d --- /dev/null +++ b/plugins/techne/evals/fires-research-grounded/graders/fires-research-grounded.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:research-grounded"' +--- diff --git a/plugins/techne/evals/fires-research-grounded/prompt.md b/plugins/techne/evals/fires-research-grounded/prompt.md new file mode 100644 index 0000000..0e2cf57 --- /dev/null +++ b/plugins/techne/evals/fires-research-grounded/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Check the library and architecture choices in my ROADMAP.md against current best practice. diff --git a/plugins/techne/evals/fires-reslop/graders/fires-reslop.md b/plugins/techne/evals/fires-reslop/graders/fires-reslop.md new file mode 100644 index 0000000..adad07a --- /dev/null +++ b/plugins/techne/evals/fires-reslop/graders/fires-reslop.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:reslop"' +--- diff --git a/plugins/techne/evals/fires-reslop/prompt.md b/plugins/techne/evals/fires-reslop/prompt.md new file mode 100644 index 0000000..8d34228 --- /dev/null +++ b/plugins/techne/evals/fires-reslop/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +These docstrings are hallucinated marketing. Rewrite them from what the code actually does. diff --git a/plugins/techne/evals/fires-sisters/graders/fires-sisters.md b/plugins/techne/evals/fires-sisters/graders/fires-sisters.md new file mode 100644 index 0000000..27ebeea --- /dev/null +++ b/plugins/techne/evals/fires-sisters/graders/fires-sisters.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:sisters"' +--- diff --git a/plugins/techne/evals/fires-sisters/prompt.md b/plugins/techne/evals/fires-sisters/prompt.md new file mode 100644 index 0000000..4eb4018 --- /dev/null +++ b/plugins/techne/evals/fires-sisters/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Are my sister repos in sync? Check for cross-repo drift. diff --git a/plugins/techne/evals/fires-slides/graders/fires-slides.md b/plugins/techne/evals/fires-slides/graders/fires-slides.md new file mode 100644 index 0000000..5014084 --- /dev/null +++ b/plugins/techne/evals/fires-slides/graders/fires-slides.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:slides"' +--- diff --git a/plugins/techne/evals/fires-slides/prompt.md b/plugins/techne/evals/fires-slides/prompt.md new file mode 100644 index 0000000..8f663bb --- /dev/null +++ b/plugins/techne/evals/fires-slides/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +I present talk.pptx to a room of students tomorrow. Is it ready? diff --git a/plugins/techne/evals/fires-theoros/graders/fires-theoros.md b/plugins/techne/evals/fires-theoros/graders/fires-theoros.md new file mode 100644 index 0000000..a21e50f --- /dev/null +++ b/plugins/techne/evals/fires-theoros/graders/fires-theoros.md @@ -0,0 +1,5 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:theoros"' +--- diff --git a/plugins/techne/evals/fires-theoros/prompt.md b/plugins/techne/evals/fires-theoros/prompt.md new file mode 100644 index 0000000..b0623f1 --- /dev/null +++ b/plugins/techne/evals/fires-theoros/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing] +--- + +Let's do a live smoke of the REPL: you drive it in tmux and I'll spectate. diff --git a/plugins/techne/evals/never-app-check-no-spectator/graders/fires-expected.md b/plugins/techne/evals/never-app-check-no-spectator/graders/fires-expected.md new file mode 100644 index 0000000..8a60bdf --- /dev/null +++ b/plugins/techne/evals/never-app-check-no-spectator/graders/fires-expected.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"(?:[\w-]+:)?run"' +arm: both +--- diff --git a/plugins/techne/evals/never-app-check-no-spectator/graders/not-theoros.md b/plugins/techne/evals/never-app-check-no-spectator/graders/not-theoros.md new file mode 100644 index 0000000..97e04c7 --- /dev/null +++ b/plugins/techne/evals/never-app-check-no-spectator/graders/not-theoros.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:theoros"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/never-app-check-no-spectator/prompt.md b/plugins/techne/evals/never-app-check-no-spectator/prompt.md new file mode 100644 index 0000000..758cfd9 --- /dev/null +++ b/plugins/techne/evals/never-app-check-no-spectator/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing, collision] +--- + +Start the dev server and confirm the login page renders. diff --git a/plugins/techne/evals/never-edit-pptx-title/graders/fires-expected.md b/plugins/techne/evals/never-edit-pptx-title/graders/fires-expected.md new file mode 100644 index 0000000..39edc2d --- /dev/null +++ b/plugins/techne/evals/never-edit-pptx-title/graders/fires-expected.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pptx-toolkit"' +arm: both +--- diff --git a/plugins/techne/evals/never-edit-pptx-title/graders/not-slides.md b/plugins/techne/evals/never-edit-pptx-title/graders/not-slides.md new file mode 100644 index 0000000..b83feb8 --- /dev/null +++ b/plugins/techne/evals/never-edit-pptx-title/graders/not-slides.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:slides"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/never-edit-pptx-title/prompt.md b/plugins/techne/evals/never-edit-pptx-title/prompt.md new file mode 100644 index 0000000..24393a3 --- /dev/null +++ b/plugins/techne/evals/never-edit-pptx-title/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing, collision] +--- + +In deck.pptx, change the title on slide 3 to "Results". diff --git a/plugins/techne/evals/never-email-catchup/graders/fires-expected.md b/plugins/techne/evals/never-email-catchup/graders/fires-expected.md new file mode 100644 index 0000000..a2139ac --- /dev/null +++ b/plugins/techne/evals/never-email-catchup/graders/fires-expected.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:inbox-catch-up"' +arm: both +--- diff --git a/plugins/techne/evals/never-email-catchup/graders/not-catchup.md b/plugins/techne/evals/never-email-catchup/graders/not-catchup.md new file mode 100644 index 0000000..1c5f4d9 --- /dev/null +++ b/plugins/techne/evals/never-email-catchup/graders/not-catchup.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:catchup"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/never-email-catchup/prompt.md b/plugins/techne/evals/never-email-catchup/prompt.md new file mode 100644 index 0000000..5199331 --- /dev/null +++ b/plugins/techne/evals/never-email-catchup/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing, collision] +--- + +Catch me up on my email and course announcements since Friday. diff --git a/plugins/techne/evals/never-fill-pdf-form/graders/fires-expected.md b/plugins/techne/evals/never-fill-pdf-form/graders/fires-expected.md new file mode 100644 index 0000000..bf00a34 --- /dev/null +++ b/plugins/techne/evals/never-fill-pdf-form/graders/fires-expected.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pdf-toolkit"' +arm: both +--- diff --git a/plugins/techne/evals/never-fill-pdf-form/graders/not-pdf.md b/plugins/techne/evals/never-fill-pdf-form/graders/not-pdf.md new file mode 100644 index 0000000..d710a09 --- /dev/null +++ b/plugins/techne/evals/never-fill-pdf-form/graders/not-pdf.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pdf"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/never-fill-pdf-form/prompt.md b/plugins/techne/evals/never-fill-pdf-form/prompt.md new file mode 100644 index 0000000..1825696 --- /dev/null +++ b/plugins/techne/evals/never-fill-pdf-form/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing, collision] +--- + +Fill in the name and address fields of application.pdf. diff --git a/plugins/techne/evals/never-merge-pdfs/graders/fires-expected.md b/plugins/techne/evals/never-merge-pdfs/graders/fires-expected.md new file mode 100644 index 0000000..bf00a34 --- /dev/null +++ b/plugins/techne/evals/never-merge-pdfs/graders/fires-expected.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pdf-toolkit"' +arm: both +--- diff --git a/plugins/techne/evals/never-merge-pdfs/graders/not-latex.md b/plugins/techne/evals/never-merge-pdfs/graders/not-latex.md new file mode 100644 index 0000000..9339ed5 --- /dev/null +++ b/plugins/techne/evals/never-merge-pdfs/graders/not-latex.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:latex"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/never-merge-pdfs/graders/not-pdf.md b/plugins/techne/evals/never-merge-pdfs/graders/not-pdf.md new file mode 100644 index 0000000..d710a09 --- /dev/null +++ b/plugins/techne/evals/never-merge-pdfs/graders/not-pdf.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pdf"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/never-merge-pdfs/prompt.md b/plugins/techne/evals/never-merge-pdfs/prompt.md new file mode 100644 index 0000000..f8b6bdb --- /dev/null +++ b/plugins/techne/evals/never-merge-pdfs/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing, collision] +--- + +Merge report1.pdf and report2.pdf into one file. diff --git a/plugins/techne/evals/never-tex-is-not-markdown/graders/fires-expected.md b/plugins/techne/evals/never-tex-is-not-markdown/graders/fires-expected.md new file mode 100644 index 0000000..23d5b3b --- /dev/null +++ b/plugins/techne/evals/never-tex-is-not-markdown/graders/fires-expected.md @@ -0,0 +1,6 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:latex"' +arm: both +--- diff --git a/plugins/techne/evals/never-tex-is-not-markdown/graders/not-pdf.md b/plugins/techne/evals/never-tex-is-not-markdown/graders/not-pdf.md new file mode 100644 index 0000000..d710a09 --- /dev/null +++ b/plugins/techne/evals/never-tex-is-not-markdown/graders/not-pdf.md @@ -0,0 +1,8 @@ +--- +type: tool_used +tool: Skill +input_match: '"skill"\s*:\s*"techne:pdf"' +min: 0 +max: 0 +arm: both +--- diff --git a/plugins/techne/evals/never-tex-is-not-markdown/prompt.md b/plugins/techne/evals/never-tex-is-not-markdown/prompt.md new file mode 100644 index 0000000..4795e2f --- /dev/null +++ b/plugins/techne/evals/never-tex-is-not-markdown/prompt.md @@ -0,0 +1,10 @@ +--- +max_turns: 6 +timeout_seconds: 240 +runs: 3 +allowed_tools: [Read, Glob, Grep, Skill] +plugins: ["../../eval-fixtures/with-rivals"] +tags: [routing, collision] +--- + +Build main.tex into a PDF. diff --git a/plugins/techne/skills/auto-commit/SKILL.md b/plugins/techne/skills/auto-commit/SKILL.md index e13e697..8b8ac70 100644 --- a/plugins/techne/skills/auto-commit/SKILL.md +++ b/plugins/techne/skills/auto-commit/SKILL.md @@ -2,7 +2,7 @@ name: auto-commit description: Analyze pending git changes and write a structured, conventional-commit plan to COMMITS.md so the user can review and stage commits in batches before committing. Use whenever the user wants to group working-tree or staged changes into sensible commits, draft commit messages for a dirty working tree, or prepare a commit plan from a diff. disable-model-invocation: false -allowed-tools: Bash(git status) Bash(git diff *) Bash(git log *) Bash(git rev-parse *) Bash(git hash-object *) Read Write +allowed-tools: Bash(git status) Bash(git diff *) Bash(git log *) Bash(git rev-parse *) Bash(bash *scripts/fingerprint.sh) Read Write --- # Auto-Commit Generator @@ -23,9 +23,9 @@ Analyze pending git changes and group them into a structured commit plan written - `git diff --cached` (staged changes — don't skip this; staged work must be included in the plan) - `git log -20 --oneline` (to match the repo's existing commit style, e.g. scope names already in use) - `git rev-parse HEAD` (for the staleness header) - - `git diff HEAD | git hash-object --stdin` (fingerprint of the working tree for the staleness header) + - `bash ${CLAUDE_SKILL_DIR}/scripts/fingerprint.sh` (fingerprint of the working tree for the staleness header) -3. **Trust `.gitignore`, with one exception: always exclude `COMMITS.md` itself.** Do not hardcode directory filters (no "skip `.claude/`", "skip `designs/`"). `git status` already respects `.gitignore`; if a file shows up, the user wants it tracked. The sole baked-in filter is `COMMITS.md` at the repo root — it is the regeneratable output of this skill, and including it in its own plan is circular (and leads to committing it to `main`, which the user has explicitly regretted). Drop it from every group; if `COMMITS.md` is the *only* pending change, treat the tree as clean and don't rewrite the file. Beyond that, if `git status` reports nothing worth committing, say so in one sentence and don't write the file. +3. **Trust `.gitignore`, with one exception: always exclude `COMMITS.md` itself.** Do not hardcode directory filters (no "skip `.claude/`", "skip `designs/`"). `git status` already respects `.gitignore`; if a file shows up, the user wants it tracked. The sole baked-in filter is `COMMITS.md` at the repo root; it is the regeneratable output of this skill, and including it in its own plan is circular (and ends with the plan committed to `main`). Drop it from every group; if `COMMITS.md` is the *only* pending change, treat the tree as clean and don't rewrite the file. Beyond that, if `git status` reports nothing worth committing, say so in one sentence and don't write the file. 4. **Group.** Decide the grouping that best reflects the actual work. See [Grouping](#grouping). @@ -85,7 +85,7 @@ branch: feat/dev-toolchain-upgrade --> ``` -`tree-hash` is the short (12-char) output of `git diff HEAD | git hash-object --stdin`. `scanned-at` is UTC ISO-8601. +`tree-hash` is the output of `scripts/fingerprint.sh`: tracked edits, index state, and untracked file contents, excluding `COMMITS.md` itself. A `git diff HEAD` hash alone misses a new untracked file, so a plan made before that file appeared would still read as current. `scanned-at` is UTC ISO-8601. ## Output format @@ -169,7 +169,7 @@ If either check fails, stop and report. Don't execute a stale plan — the group 4. **Open the PR.** `gh pr create` with the plan as source material. Build the body from the commit headlines and bullets — drop the header comment and the `Suggested branch:` line (those were scaffolding). Format: ``` - gh pr create --title "" --body "$(cat <<'EOF' + gh pr create --title "" --assignee @me [--label