Skip to content

feat(skills): vendor the ponytail skill pack across all three host trees - #36

Merged
jasoncham-itmax merged 1 commit into
masterfrom
feat/ponytail-skill-pack
Sep 25, 2026
Merged

jasoncham-itmax merged 1 commit into
masterfrom
feat/ponytail-skill-pack

Conversation

@limcheehow

Copy link
Copy Markdown
Contributor

What

Vendors the ponytail skill pack — six MIT-licensed skills from
dietrichgebert/ponytail, pinned at commit
e3ba2aa (v4.10.0) — into all three host trees, and wires them into every declaration point.

ponytail forces the laziest solution that actually works: YAGNI first, standard library before
custom code, native platform features before dependencies, one line before fifty. It is
auto-triggering on coding tasks, so once loaded it shapes how code gets written for every
coding task, not only on request.

Why

We A/B'd it on five real repos before adopting. Same tickets, same model, same repo SHAs,
per-cell clones; the arms differ only by preloading the skill (30 headless sessions, n=3 per cell,
deepseek-v4.1-flash):

Metric ponytail vs baseline
Input tokens −40.3%
Output tokens −50.4%
Total tokens (incl. cache reads) −53.7%
API calls −38.4%
Cost −50.4%
Source LOC written −38.4%

Every repo used less; no repo regressed. Correctness was 100% (15/15) baseline vs
93% (14/15) ponytail
. The one ponytail miss was a malformed generation — a tool-call fragment
leaked as final text with zero edits — not a quality regression. Importantly, neither arm ever
dropped a validation guard or broke an existing suite
, which is the risk you'd expect a "write
less code" rule to introduce.

Honest caveats: n=3 per cell with up to 3.6× run-to-run variance, so read the means as
±10pp. One repo (city-nexus) overlaps between arms and its per-task number is not reliable; the
aggregate across all five is what holds. Five Python repos, one ticket each — this is not a
production-readiness claim.

How

Rule text is byte-for-byte upstream in all three trees. Only two things were adapted:

  1. Frontmatter — license: MIT, upstream, upstream-commit added everywhere; Hermes/Codex
    host keys (version, author, platforms, metadata.hermes.*) on the ports.
  2. An appended host-notes section — how each host triggers/preloads the skill.

Deliberate deviation from convention: descriptions are not given the usual
(Hermes port of ...) suffix. The description is the trigger, and rewriting it would change
when the skill fires — the one thing the A/B was measuring.

⚠️ Reviewer decision: vendoring vs the externalSkills precedent

This repo's existing pattern for third-party skills is to document them (externalSkills in
plugin.json, e.g. code-refactor) rather than vendor them. This PR knowingly vendors
instead, because that gives the team a one-command install and puts the rule text in-tree where it
can be reviewed and diffed against upstream. If you'd rather it be documentation-only, say so and
I'll rework it — the vendored copies are self-contained, so the revert is clean.

Attribution is explicit so nothing is misrepresented: skills/ponytail/LICENSE retains the upstream
MIT licence, every skill pins the upstream repo + commit, and README/CHANGELOG state plainly that
these six are third-party and do not carry the Tapway/Karpathy framing the other 24 do.

Testing

New tests/test_ponytail_pack.py — 109 checks, written RED first, and proven to
discriminate
:

  • removing one vendored file → 5 checks fail
  • making a declared count stale → 1 check fails

It asserts that every canonical rule line survives in each host port (so the skill that ships
is the skill that was measured), upstream provenance, that declared counts agree with the
filesystem, the Hermes guard-trigger scrub, and that ponytail's description still carries its
coding-task triggers (a pack that never fires saves nothing).

  • tests/test_hermes_install.py — EXPECTED and blocked grow by six, so the
    community-install guard scrub is enforced, not just documented
  • tests/check_docs_consistency.py — hardcoded expectations updated
  • Full suite green: 10 test files, 811 checks, 0 failures; docs consistency ALL PASS

Rollout

  • counts synced everywhere: 30 canonical / 31 Hermes / 30 + $tapway umbrella Codex
  • version 2.3.0 → 2.4.0 (plugin.json, marketplace.json)
  • drive-by fix: marketplace.json had pre-existing stale counts (19 skills / 4 agents) that didn't
    match the filesystem — corrected to 30 skills / 5 agents

Notes

Vendors six MIT-licensed skills verbatim from dietrichgebert/ponytail
(pinned at commit e3ba2aa, v4.10.0) into skills/, hermes/skills/ and
codex/skills/, so the whole team gets the token-saving rule set from one
install.

The rule text is byte-for-byte upstream in every tree; only the YAML
frontmatter and an appended host-notes section were adapted. Descriptions
are deliberately not rewritten with the usual "(Hermes port of ...)"
suffix, because the description is the trigger and changing it would
change when the skill fires — the one thing the A/B measured.

Attribution:
- upstream MIT licence retained at skills/ponytail/LICENSE
- upstream repo + commit pinned in every vendored skill's frontmatter
- credited in README and CHANGELOG, explicitly as third-party (these
  skills are not Tapway-authored and do not carry the Karpathy framing)

Rollout:
- register all six in plugin.json, hermes/install.sh and codex/install.sh
- bump version 2.3.0 -> 2.4.0 (plugin.json, marketplace.json)
- sync counts: 30 canonical / 31 Hermes / 30 + $tapway umbrella (Codex)
  across README.md, hermes/README.md, codex/README.md, install comments
  and tests/check_docs_consistency.py
- correct pre-existing stale marketplace counts (19 skills / 4 agents ->
  30 skills / 5 agents)

Tests (RED -> GREEN):
- tests/test_ponytail_pack.py (109 checks): every canonical rule line
  survives in each host port, upstream provenance, declared counts agree
  with the filesystem, Hermes guard-trigger scrub, and that ponytail's
  description still carries its coding-task triggers
- proven to discriminate: removing a vendored file fails 5 checks,
  making a declared count stale fails 1
- test_hermes_install.py: EXPECTED and blocked grow by six, so the
  community-install guard scrub is enforced, not merely documented
- full suite green (10 test files, 811 checks) + docs consistency

Measured before adopting (A/B, 5 Tapway repos, 30 headless sessions, n=3,
deepseek-v4.1-flash): -40.3% input tokens, -50.4% output, -53.7% total,
-38.4% API calls, -50.4% cost, -38.4% source LOC; correctness 100%
baseline vs 93% ponytail, where the single miss was a malformed
generation with zero edits rather than a quality regression. Caveat:
n=3 with up to 3.6x run-to-run variance.

Plan: docs/plans/2026-09-24-ponytail-skill-pack-adoption.md
@github-actions

Copy link
Copy Markdown
Contributor

✅ Workflow audit passed — conventional commits ✓, plan ✓, no secrets ✓.

@jasoncham-itmax
jasoncham-itmax merged commit a8dfed9 into master Sep 25, 2026
1 check passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants