Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 3 additions & 15 deletions .claude-plugin/marketplace.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,25 +4,13 @@
"name": "QuantEcon"
},
"description": "QuantEcon's shared Claude Code skills and tools",
"version": "0.2.0",
"version": "0.3.0",
"plugins": [
{
"name": "qe",
"source": "./qe",
"version": "0.6.0",
"description": "QuantEcon's author-facing base skills — working through a PR's review feedback, turning audit reports into tracked work projects, and creating, reading and carrying work-plan state across agent sessions"
},
{
"name": "benchmark",
"source": "./benchmark",
"version": "0.4.0",
"description": "Benchmarking and acceleration-evaluation tools for QuantEcon lecture code"
},
{
"name": "audit",
"source": "./audit",
"version": "0.2.0",
"description": "Bulk, read-only audits of a QuantEcon repository — issue triage, PR review, technical debt, translation parity — each producing an evidence-cited report bundle"
"version": "0.7.0",
"description": "QuantEcon's shared agent skills under one namespace — working through a PR's review feedback, advising on and scoring lecture acceleration (benchmark), bulk read-only repository audits, turning audit reports into tracked work projects, and creating, reading and carrying work-plan state across agent sessions"
}
]
}
14 changes: 7 additions & 7 deletions .github/workflows/validate.yml
Original file line number Diff line number Diff line change
Expand Up @@ -45,21 +45,21 @@ jobs:
BASE_REF: ${{ github.base_ref }}
run: python scripts/check-version-bump.py --base "origin/$BASE_REF"

# The benchmark plugin's claim is that no score is ever written by hand:
# The benchmark skill's claim is that no score is ever written by hand:
# every scorecard is a deterministic function of its evidence.json. That
# only stays true if it is checked. A non-empty diff here means either a
# scorecard was hand-edited, or a rubric change moved a published verdict
# without the baselines being regenerated — in the second case the fix is
# to re-run these two commands and commit, so the verdict move lands in
# the diff where a reviewer sees it. Stdlib only; no install step needed.
- name: Scorecards reproduce from evidence
working-directory: benchmark
working-directory: qe
run: |
python scripts/scoring/score.py references/examples/ge_arrow
python scripts/scoring/score.py references/examples/markov_asset
python scripts/scoring/score.py references/fixtures/rubric_v2
git diff --exit-code -- 'references/examples/*/results/scorecard.json' \
'references/fixtures/*/results/scorecard.json'
python scripts/benchmark/scoring/score.py references/benchmark/examples/ge_arrow
python scripts/benchmark/scoring/score.py references/benchmark/examples/markov_asset
python scripts/benchmark/scoring/score.py references/benchmark/fixtures/rubric_v2
git diff --exit-code -- 'references/benchmark/examples/*/results/scorecard.json' \
'references/benchmark/fixtures/*/results/scorecard.json'

# A separate job on purpose: this one installs an ~85 MB npm toolchain, and
# running it beside `validate` rather than inside it keeps that job four fast
Expand Down
12 changes: 6 additions & 6 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ Guidance for AI coding agents and human contributors working in `QuantEcon/skill

## What this repository is

A [Claude Code plugin marketplace](https://code.claude.com/docs/en/plugin-marketplaces) holding QuantEcon's shared agent skills and the deterministic scripts they drive. The goal is to share institutional knowledge — the checks, rubrics and procedures experienced maintainers already apply by hand — so the same work produces consistent results wherever it runs. Orientation is in [README.md](README.md); what has actually shipped is in [CATALOG.md](CATALOG.md); work in flight is in the per-plugin tracking issues.
A [Claude Code plugin marketplace](https://code.claude.com/docs/en/plugin-marketplaces) holding QuantEcon's shared agent skills and the deterministic scripts they drive. The goal is to share institutional knowledge — the checks, rubrics and procedures experienced maintainers already apply by hand — so the same work produces consistent results wherever it runs. Orientation is in [README.md](README.md); what has actually shipped is in [CATALOG.md](CATALOG.md); work in flight is in the per-family tracking issues.

**The repo is early, and its conventions are deliberately loose.** Where a doc describes a report shape, a phase division, a naming form or a directory layout, read it as what an existing skill does rather than as a contract a new one must satisfy — see [CATALOG.md § Principles](CATALOG.md#principles). The few things that genuinely must hold are stated plainly, with their reasons, and they are all about keeping a skill's output checkable by someone who will not re-run it.

Expand All @@ -20,7 +20,7 @@ What this means in practice:

- **Skills point outward.** When a skill needs a rule, a convention, or a procedure that already exists in the [QuantEcon manual](https://manual.quantecon.org) or in `QuantEcon/style-guide`, it cites and links to it. A `SKILL.md` carries only what the skill itself adds: the procedure it runs, the judgement it applies, the output it produces.
- **Rule text is authored upstream only.** Style-rule text lives in `QuantEcon/style-guide`; when the style skills land ([#3](https://github.com/QuantEcon/skills/issues/3)) this repo will carry a rendered, drift-checked snapshot of it — never hand-authored here.
- **Numbers drift fastest** — weights, thresholds, verdict bands, versions. The rubric's weights are stated in [`benchmark/references/EVALUATION_FRAMEWORK.md`](benchmark/references/EVALUATION_FRAMEWORK.md) and implemented once in `benchmark/scripts/scoring/rubric.py`; anywhere else they come up, quote with a pointer rather than re-tabulating.
- **Numbers drift fastest** — weights, thresholds, verdict bands, versions. The rubric's weights are stated in [`qe/references/benchmark/EVALUATION_FRAMEWORK.md`](qe/references/benchmark/EVALUATION_FRAMEWORK.md) and implemented once in `qe/scripts/benchmark/scoring/rubric.py`; anywhere else they come up, quote with a pointer rather than re-tabulating.
- **Every topic has an owning doc** (see the map below). Before adding a section, work out which file owns the topic, put it there, and link from wherever else it comes up.
- **Across boundaries, link — don't copy.** An installed plugin ships only its own directory, so a reference to a repo-level file or another plugin is an absolute GitHub URL, never a duplicated paragraph ([developing-skills § Conventions](docs/developing-skills.md#conventions)).
- **The one deliberate exception**: a `SKILL.md` frontmatter `description` must stand alone, because it is what natural-language invocation matches against. Restate what the skill does in that one sentence; the details stay behind the link.
Expand All @@ -34,15 +34,15 @@ Before adding a paragraph, check whether it already exists. If it does, link to
| What the marketplace is, installation (local, lecture repos, CI) | [README.md](README.md) |
| Using the skills: setup, invocation, what to expect | [docs/using-skills.md](docs/using-skills.md) |
| Contributing: layout, conventions, dev loop, local testing, versioning, releases and tagging, PR flow | [docs/developing-skills.md](docs/developing-skills.md) |
| What changed in a plugin, release by release | `<plugin>/CHANGELOG.md`one per plugin, since the plugin is the released artifact and ships only its own directory |
| What changed, release by release | [qe/CHANGELOG.md](qe/CHANGELOG.md)inside the plugin, since the plugin is the released artifact and ships only its own directory; the retired benchmark/audit streams are historical sections there |
| Running an evaluation by hand, end to end | [docs/tutorial-run-an-evaluation.md](docs/tutorial-run-an-evaluation.md) |
| Running a whole-tracker audit, and reviewing what it produces | [docs/tutorial-run-an-audit.md](docs/tutorial-run-an-audit.md) |
| The benchmark skill: modes, report format, manual pipeline | [benchmark/README.md](benchmark/README.md) |
| Rubric: dimensions, weights, anchors, verdict bands | [benchmark/references/EVALUATION_FRAMEWORK.md](benchmark/references/EVALUATION_FRAMEWORK.md) |
| The benchmark skill: modes, report format, manual pipeline | [qe/references/benchmark/README.md](qe/references/benchmark/README.md) |
| Rubric: dimensions, weights, anchors, verdict bands | [qe/references/benchmark/EVALUATION_FRAMEWORK.md](qe/references/benchmark/EVALUATION_FRAMEWORK.md) |
| Style rule text and schema | `QuantEcon/style-guide` (upstream — never authored in this repo) |
| What has shipped, and the principles behind it | [CATALOG.md](CATALOG.md) |
| Parked ideas, not committed to | [Low-priority enhancement issues](https://github.com/QuantEcon/skills/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement+label%3Alow-priority) in the tracker, one per idea with its merit assessment |
| Work in flight, per plugin | issues [#3](https://github.com/QuantEcon/skills/issues/3) (`qe`), [#4](https://github.com/QuantEcon/skills/issues/4) (`benchmark`), [#12](https://github.com/QuantEcon/skills/issues/12) (`audit`) |
| Work in flight, per skill family | issues [#3](https://github.com/QuantEcon/skills/issues/3) (style), [#4](https://github.com/QuantEcon/skills/issues/4) (benchmark), [#12](https://github.com/QuantEcon/skills/issues/12) (audit) |

## Working in this repo

Expand Down
18 changes: 10 additions & 8 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,14 +1,16 @@
# Skill catalog

What this marketplace installs today, filtered to what is **operational**: a skill appears here once it is merged *and* runs to its stated purpose, not merely once its files are on `main`. Listing is not a claim about how far a skill has been validated — the State column answers that for each one, and says so plainly when the answer is "not much". Anything not listed does not exist yet as a usable skill, however firmly it has been discussed. Work in flight lives in the tracking issue for its plugin; ideas nobody has committed to live in the tracker as [low-priority enhancement issues](https://github.com/QuantEcon/skills/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement+label%3Alow-priority).
What this marketplace installs today, filtered to what is **operational**: a skill appears here once it is merged *and* runs to its stated purpose, not merely once its files are on `main`. Listing is not a claim about how far a skill has been validated — the State column answers that for each one, and says so plainly when the answer is "not much". Anything not listed does not exist yet as a usable skill, however firmly it has been discussed. Work in flight lives in the tracking issue for its skill family; ideas nobody has committed to live in the tracker as [low-priority enhancement issues](https://github.com/QuantEcon/skills/issues?q=is%3Aissue+is%3Aopen+label%3Aenhancement+label%3Alow-priority).

Scaffolding is deliberately absent — and as of `qe` 0.6.0 it no longer ships at all: an unbuilt skill lives only as the plan in its plugin's tracking issue, so installing a plugin never puts a menu entry in front of you that does nothing. (Through 0.5.0 the style skills shipped as stubs reporting "not yet operational"; that practice is retired.)
Scaffolding is deliberately absent — and as of `qe` 0.6.0 it no longer ships at all: an unbuilt skill lives only as the plan in its family's tracking issue, so installing the plugin never puts a menu entry in front of you that does nothing. (Through 0.5.0 the style skills shipped as stubs reporting "not yet operational"; that practice is retired.)

| Plugin | Skills | State | Tracking |
Everything ships in the single `qe` plugin ([#43](https://github.com/QuantEcon/skills/issues/43), qe 0.7.0); the families keep their own tracking issues.

| Family | Skills | State | Tracking |
|---|---|---|---|
| **`qe`** — author-facing skills, from drafting a lecture to merging its PR | `/qe:copilot-review` | Operational, and validated from an installed plugin on 2026-08-03: plugin-root path resolution, cross-repo mode, and running from outside a working tree. The style skills (`check-style` and six per-category siblings) shipped as scaffolding through 0.5.0 and were removed in 0.6.0; their plan lives in [#3](https://github.com/QuantEcon/skills/issues/3), and they return once the rule snapshot and deterministic preflight land. The `workplan-*` family — `/qe:workplan-project` (0.3.0 as `/qe:workplan`, renamed 0.4.0; report bundle → tracking issue with sub-issues) and `/qe:workplan` (0.5.0, consolidating 0.4.0's `workplan-issue` and `workplan-update` and adding a read verb; the work-plan issue's whole lifecycle — create, read/validate/recommend, resume, update, close-and-succeed) — is merged as complete procedures with no validated runs yet, so it is likewise not listed until it has one. Note the name reuse: `/qe:workplan` meant the report-to-project skill for one morning in 0.3.0; from 0.5.0 it is the lifecycle skill. | [#3](https://github.com/QuantEcon/skills/issues/3) |
| **`benchmark`** — evaluating accelerated lecture implementations | `/benchmark:review-acceleration` | Operational for workspace runs: rubric v2, a deterministic scoring engine, and two complete worked evaluations as regression baselines. | [#4](https://github.com/QuantEcon/skills/issues/4) |
| **`audit`** — bulk, read-only repository audits | `/audit:issues` | Run once **as a skill** — a 230-item tracker on 2026-07-28, 22 minutes, seven plugin defects found and recorded ([record](https://github.com/QuantEcon/skills/blob/main/reviews/audit-run-action-translation-2026-07-28.md)). Both severity-1 defects are fixed as of 0.2.0 ([#34](https://github.com/QuantEcon/skills/pull/34)): a `[verified]` citation must now resolve on the ref the audit named, and phase 2 checkpoints its closed pass as well as its open one. That validates the method as far as one run goes and no further: its central claim, resumability, is still untested, because the run was never interrupted. Further runs: [#16](https://github.com/QuantEcon/skills/issues/16). | [#12](https://github.com/QuantEcon/skills/issues/12), [#16](https://github.com/QuantEcon/skills/issues/16) |
| **Author-facing** — from drafting a lecture to merging its PR | `/qe:copilot-review` | Operational, and validated from an installed plugin on 2026-08-03: plugin-root path resolution, cross-repo mode, and running from outside a working tree. The style skills (`check-style` and six per-category siblings) shipped as scaffolding through 0.5.0 and were removed in 0.6.0; their plan lives in [#3](https://github.com/QuantEcon/skills/issues/3), and they return once the rule snapshot and deterministic preflight land. The `workplan-*` family — `/qe:workplan-project` (0.3.0 as `/qe:workplan`, renamed 0.4.0; report bundle → tracking issue with sub-issues) and `/qe:workplan` (0.5.0, consolidating 0.4.0's `workplan-issue` and `workplan-update` and adding a read verb; the work-plan issue's whole lifecycle — create, read/validate/recommend, resume, update, close-and-succeed) — is merged as complete procedures with no validated runs yet, so it is likewise not listed until it has one. Note the name reuse: `/qe:workplan` meant the report-to-project skill for one morning in 0.3.0; from 0.5.0 it is the lifecycle skill. | [#3](https://github.com/QuantEcon/skills/issues/3) |
| **Benchmark** — evaluating accelerated lecture implementations | `/qe:benchmark` (until qe 0.7.0: `/benchmark:review-acceleration`) | Operational for workspace runs: rubric v2, a deterministic scoring engine, and two complete worked evaluations as regression baselines. | [#4](https://github.com/QuantEcon/skills/issues/4) |
| **Audit** — bulk, read-only repository audits | `/qe:audit-issues` (until qe 0.7.0: `/audit:issues`) | Run once **as a skill** — a 230-item tracker on 2026-07-28, 22 minutes, seven plugin defects found and recorded ([record](https://github.com/QuantEcon/skills/blob/main/reviews/audit-run-action-translation-2026-07-28.md)). Both severity-1 defects are fixed as of audit 0.2.0 ([#34](https://github.com/QuantEcon/skills/pull/34)): a `[verified]` citation must now resolve on the ref the audit named, and phase 2 checkpoints its closed pass as well as its open one. That validates the method as far as one run goes and no further: its central claim, resumability, is still untested, because the run was never interrupted. Further runs: [#16](https://github.com/QuantEcon/skills/issues/16). | [#12](https://github.com/QuantEcon/skills/issues/12), [#16](https://github.com/QuantEcon/skills/issues/16) |

Installation and setup are in [README.md](README.md); what it is like to run one is in [docs/using-skills.md](docs/using-skills.md).

Expand All @@ -18,7 +20,7 @@ The point of the marketplace is to **share institutional knowledge** — the che

- **Few, high-frequency skills** over many niche ones, each validated against actual PR history. The 2026-07-21 analysis of ~630 merged PRs across the four main lecture repos is the evidence base: style was the largest recurring theme by a wide margin, which is why it is the flagship. A skill justified by breadth rather than frequency, as the audit family is, should say so.
- **Report first, fix on request.** Skills produce a structured report and offer fixes; they never silently edit. Safe to run in CI, and authors stay in control.
- **Cited claims; computed scores.** Every finding carries a citation — a rule ID plus `file:line`, or a number plus its source. Skills whose output is a findings list need nothing more. Skills that aggregate judgements into a scored verdict use the evidence-file pattern from the benchmark plugin: judgement recorded as cited answers, every score computed by a deterministic engine, never typed by hand (see [docs/developing-skills.md](docs/developing-skills.md)).
- **Scaffolding is advice, not instruction.** Report shapes, phase divisions, naming forms and directory conventions are described as what an existing skill does, not as contracts a new one has to satisfy. Three plugins is not enough to know which of them generalise, and a rule invented from one worked example mostly succeeds at forcing the next skill into the first one's shape. A skill can be a single `SKILL.md`. Where something genuinely must hold — read-only boundaries, cited claims, a stated coverage of what was and was not checked — say so plainly and give the reason; everything else can converge later, once there is something to generalise from.
- **Cited claims; computed scores.** Every finding carries a citation — a rule ID plus `file:line`, or a number plus its source. Skills whose output is a findings list need nothing more. Skills that aggregate judgements into a scored verdict use the evidence-file pattern from the benchmark skill: judgement recorded as cited answers, every score computed by a deterministic engine, never typed by hand (see [docs/developing-skills.md](docs/developing-skills.md)).
- **Scaffolding is advice, not instruction.** Report shapes, phase divisions, naming forms and directory conventions are described as what an existing skill does, not as contracts a new one has to satisfy. Three skill families is not enough to know which of them generalise, and a rule invented from one worked example mostly succeeds at forcing the next skill into the first one's shape. A skill can be a single `SKILL.md`. Where something genuinely must hold — read-only boundaries, cited claims, a stated coverage of what was and was not checked — say so plainly and give the reason; everything else can converge later, once there is something to generalise from.

Note what the last two have in common: the rules stated firmly are the ones that keep output *checkable by someone who will not re-run it*. That is the test worth applying before writing any new rule down.
Loading
Loading