diff --git a/docs/README.md b/docs/README.md new file mode 100644 index 0000000..27e07aa --- /dev/null +++ b/docs/README.md @@ -0,0 +1,5 @@ +# Playground docs + +Public write-ups from experiments run against this repo. + +- [ADR 0091 skill activation matrix](./adr-0091-skill-activation-matrix.html) — three-arm triage proof for always-on harness skills (fullsend [#6378](https://github.com/fullsend-ai/fullsend/pull/6378) / [#6380](https://github.com/fullsend-ai/fullsend/issues/6380)). diff --git a/docs/adr-0091-skill-activation-matrix.html b/docs/adr-0091-skill-activation-matrix.html new file mode 100644 index 0000000..487bcaf --- /dev/null +++ b/docs/adr-0091-skill-activation-matrix.html @@ -0,0 +1,199 @@ + + + + + + ADR 0091 — How we tested always-on skills + + + + + + +
+
+
ADR 0091 · playground proof · 2026-08-21
+

Three ways to load a team skill — only one worked every time

+

+ We ran the same triage job nine times on this repo (three setups × three repeats). + Each run used a fresh bug issue so triage had a real reason to label. + The goal: find a reliable way for a team add-on skill to actually get opened — + not just uploaded into the skill list. +

+
+
+
1 · Baseline
+ No extra team skill + normal labeling 3/3 + no team skill to open +
+
+
2 · Listed only
+ Skill on harness, nothing else + normal labeling 3/3 + team skill opened 0/3 +
+
+
3 · Always-on
+ metadata.apply: always + name in prompt + team skill opened 3/3 + skill changed the comment 3/3 +
+
+

Result: putting a skill on the harness list is not enough. Marking it always-on and naming it in the agent prompt (so the model opens it with the Skill tool) worked on every run.

+
+ +
+

What each setup did

+
    +
  1. Baseline — stock triage harness. No smoke team skill. Checks that triage still works.
    2. +
  2. Listed only — same harness, plus a smoke skill listed under harness skills:. No always-on flag. This is the “just put it on the list” approach.
    3. +
  3. Always-on — same smoke skill with metadata.apply: always, and a placeholder in the agent file that bootstrap fills with that skill’s name before the run. The model is told to open those skills with the Skill tool. Skill bodies are not pasted into the agent file.
    4. +
+

We scored “did it work?” two ways: (1) did the transcript show a Skill open for the smoke skill, and (2) did the triage comment start with the skill’s marker (CAVEMAN:).

+
+ +
+

Run-by-run results

+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
SetupRepIssueActions runSmoke skill opened?Marker in comment?
Always-on1#2832516211708yesyes
Always-on2#3232518488943yesyes
Always-on3#3332518844288yesyes
Listed only1#3432519258690nono
Listed only2#3532519914084nono
Listed only3#3632520306596nono
Baseline1#3732520676476n/an/a
Baseline2#3832521092300n/an/a
Baseline3#3932521477717n/an/a
+
+
+ +
+

What this means for ADR 0091

+

+ The ADR should land on the always-on pattern we proved here: + opt in with metadata.apply: always, fill skill names into the agent prompt before the run, + and let the model open each skill with the Skill tool. Do not paste SKILL.md bodies into the agent file. +

+

Working pieces we used

+ +
+ + +
+ + diff --git a/docs/adr-0091-skill-activation-matrix.json b/docs/adr-0091-skill-activation-matrix.json new file mode 100644 index 0000000..c6b350f --- /dev/null +++ b/docs/adr-0091-skill-activation-matrix.json @@ -0,0 +1,161 @@ +[ + { + "arm": "always-on", + "rep": 1, + "issue": 28, + "run_id": "32516211708", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32516211708", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/28", + "opened_additive": true, + "additive_name": "smoke-always-on", + "caveman_comment": true, + "skill_opens": [ + "smoke-always-on", + "github-forge", + "github", + "smoke-always-on", + "github-forge", + "github" + ] + }, + { + "arm": "always-on", + "rep": 2, + "issue": 32, + "run_id": "32518488943", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32518488943", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/32", + "opened_additive": true, + "additive_name": "smoke-always-on", + "caveman_comment": true, + "skill_opens": [ + "smoke-always-on", + "github-forge", + "github", + "smoke-always-on", + "github-forge", + "github" + ] + }, + { + "arm": "always-on", + "rep": 3, + "issue": 33, + "run_id": "32518844288", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32518844288", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/33", + "opened_additive": true, + "additive_name": "smoke-always-on", + "caveman_comment": true, + "skill_opens": [ + "smoke-always-on", + "github-forge", + "github", + "smoke-always-on", + "github-forge", + "github" + ] + }, + { + "arm": "listed", + "rep": 1, + "issue": 34, + "run_id": "32519258690", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32519258690", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/34", + "opened_additive": false, + "additive_name": "smoke-listed", + "caveman_comment": false, + "skill_opens": [ + "github-forge", + "github", + "github-forge", + "github" + ] + }, + { + "arm": "listed", + "rep": 2, + "issue": 35, + "run_id": "32519914084", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32519914084", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/35", + "opened_additive": false, + "additive_name": "smoke-listed", + "caveman_comment": false, + "skill_opens": [ + "github-forge", + "github", + "github-forge", + "github" + ] + }, + { + "arm": "listed", + "rep": 3, + "issue": 36, + "run_id": "32520306596", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32520306596", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/36", + "opened_additive": false, + "additive_name": "smoke-listed", + "caveman_comment": false, + "skill_opens": [ + "github-forge", + "github", + "github-forge", + "github" + ] + }, + { + "arm": "baseline", + "rep": 1, + "issue": 37, + "run_id": "32520676476", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32520676476", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/37", + "opened_additive": false, + "additive_name": null, + "caveman_comment": false, + "skill_opens": [ + "github-forge", + "github", + "github-forge", + "github" + ] + }, + { + "arm": "baseline", + "rep": 2, + "issue": 38, + "run_id": "32521092300", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32521092300", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/38", + "opened_additive": false, + "additive_name": null, + "caveman_comment": false, + "skill_opens": [ + "github-forge", + "github", + "github-forge", + "github" + ] + }, + { + "arm": "baseline", + "rep": 3, + "issue": 39, + "run_id": "32521477717", + "run_url": "https://github.com/fullsend-playground/python-app/actions/runs/32521477717", + "issue_url": "https://github.com/fullsend-playground/python-app/issues/39", + "opened_additive": false, + "additive_name": null, + "caveman_comment": false, + "skill_opens": [ + "github-forge", + "github", + "github-forge", + "github" + ] + } +]