Revamp adctoolbox-user-guide skill with progressive disclosure#13
Merged
Revamp adctoolbox-user-guide skill with progressive disclosure#13
Conversation
Progressive Disclosure split: basic tier (spectrum + calibration) stays in SKILL.md; advanced debug (dashboards, phase-plane, bit-level, error decomp) moves to references/advanced-debug.md. Rewrite is driven by skill-creator evals with fixed 10-prompt suite and locked success criteria so every commit in the branch is measured, not taste-driven. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Task-by-task plan driving the spec's progressive-disclosure rewrite through a locked 10-prompt skill-creator eval. Each commit is gated by measured pass/fail against basic recall, advanced escalation, and response length criteria. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
10-prompt run against the pre-change SKILL.md (commit 687dbf1). Method: 10 fresh subagents in parallel, each given the full SKILL.md content + one prompt + read access to references/ files. (Adapted from skill-creator's full benchmark loop, which requires interactive HTML review.) Result vs locked criteria: - Basic 5 / 5 PASS (criterion >= 4/5) - Advanced escalation 0 / 5 EXPECTED FAIL (criterion >= 3/5; references/ advanced-debug.md does not yet exist and SKILL.md does not point to it) - Avg length 200.7 words The advanced-escalation gap is exactly what the next commits are designed to close. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Adds a task-keyword-organized advanced reference file covering dashboards, phase-plane / error decomposition, bit-level analysis, static nonlinearity, and cap-to-weight conversion. SKILL.md does not yet point to this file -- that comes in the next commit. All 20 named public APIs verified present in the package's __init__.py exports before commit. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Replaces the bundled adctoolbox-user-guide SKILL.md with a basic tier
focused on spectrum + calibration + critical conventions. Drops the
mid-flow "Installing AI Skills" CLI section (operator-facing, belongs
in the project README). Adds explicit "NOT for" disambiguation vs the
analog-agents skills and adctoolbox-contributor-guide. Section 6 now
escalates to references/advanced-debug.md for dashboards / phase-plane
/ bit-level / error-decomposition / static nonlinearity / cap-to-weight.
iter-1 eval (10-prompt suite, same protocol as baseline):
- Basic correct 4 / 5 PASS (>= 4/5; B5 dropped: ADC_Signal_Generator
no longer named in the basic tier ->
subagent hallucinated `generate_sine_dout`)
- Advanced escalate 4 / 5 PASS (baseline 0/5 -> +4; A1, A2, A3, A4
all escalated; A5 answered inline)
- Avg length 151.6 PASS (baseline 200.7, -24%)
All three locked criteria pass -> Task 7 (description / handoff tuning)
is not required.
Note vs plan target line count (90 +/- 15): SKILL.md is 123 lines,
+18 above the upper bound. Content is verbatim from the spec/plan;
trimming further would deviate from the reviewed spec content. The
real length criterion (avg response length non-increasing) is met
with significant margin.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Restructure api-quickref.md so all entries sit under either ## Basic or ## Advanced. No API names removed; verified by diffing the identifier set old vs new (zero drops). Basic tier mirrors SKILL.md basic workflow: analyze_spectrum, analyze_spectrum_polar, compute_spectrum, find_coherent_frequency, fit_sine_4param, calibrate_weight_sine, calibrate_weight_sine_lite, ADC_Signal_Generator, validate_aout_data, validate_dout_data. Everything else (error decomposition, bit-level, dashboards, phase-plane, static nonlinearity, cap-to-weight, FOM/limits, unit conversions, NTF) is Advanced. CLI and Key Conventions stay as their own ## sections. Deviation vs plan Step 5.2: net +38 lines, plan budget +4. The +4 budget assumed two header inserts would suffice, but the file already had 5 top-level ## sections (Flat Imports / Submodule Imports / CLI / Key Conventions / Default Entry Points), and the basic / advanced split needed re-grouping inside each — done with minimal added prose (one short orientation line per section). Substantive intent of Step 5 is met. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Append ## Advanced examples section listing packaged scripts that correspond to the categories in references/advanced-debug.md: dashboards, phase-plane, bit-level / overflow / ENOB sweep, error decomposition, static nonlinearity. Existing entries unchanged. cap-to-weight has no standalone example; the advanced-debug.md inline snippet is the canonical reference for that one. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Re-runs the locked 10-prompt suite against the post-Task-6 HEAD
(SKILL.md + advanced-debug.md + partitioned api-quickref + example-map
with Advanced examples section).
Result vs locked criteria:
- Basic correct 4 / 5 PASS (>=4 -- same B5 hallucination as iter-1)
- Advanced escalate 5 / 5 PASS (iter-1 was 4/5; A5 now escalates,
likely from the cleaner basic/advanced
boundary in api-quickref.md)
- Avg length 148.5 PASS (baseline 200.7, -26%)
Known follow-up (not blocking PR):
The basic-tier SKILL.md no longer names ADC_Signal_Generator for the
"generate synthetic ADC stimulus" use case (B5), and subagents fall
back to a hallucinated `generate_sine_dout`. Adding one bullet to
SKILL.md section 4 ("synthetic stimulus -> from adctoolbox.siggen
import ADC_Signal_Generator") would close this without changing the
overall basic-tier scope.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
adctoolbox-user-guideskill into a basic tier (SKILL.md) and an advanced tier (references/advanced-debug.md).SKILL.mdto focus on spectrum + calibration + critical conventions; dropped the mid-flow "Installing AI Skills" section (operator-facing CLI, belongs in the project README).analog-design,analog-verify,analog-audit) andadctoolbox-contributor-guide.references/advanced-debug.mdfor dashboards / phase-plane / bit-level / error decomposition / static nonlinearity / cap-to-weight.references/api-quickref.mdandreferences/example-map.mdinto Basic / Advanced sections (no entries removed).Eval results (locked 10-prompt suite, locked criteria)
Locked thresholds: Basic ≥ 4/5, Advanced escalation ≥ 3/5, avg length non-increasing vs baseline. Final run passes all three.
Method: 10 fresh subagents per run, each given the SKILL.md + read access to the references/ directory + one of the locked prompts. JSON artifacts under
docs/superpowers/specs/eval/.Known follow-up (not in this PR)
ADC_Signal_Generatorexplicitly, and subagents fall back to a hallucinatedgenerate_sine_doutfor the "generate synthetic ADC output" prompt. One bullet in SKILL.md §4 (e.g.from adctoolbox.siggen import ADC_Signal_Generator) would close this. Deferred to keep the basic tier scope as reviewed.adctoolbox-install-skillCLI docs from the skill into the project README.Spec / plan
docs/superpowers/specs/2026-04-22-user-guide-skill-revamp-design.mddocs/superpowers/plans/2026-04-22-user-guide-skill-revamp.mddocs/superpowers/specs/eval/prompts.mdDeviations from the plan (documented in commits)
SKILL.mdis 123 lines vs plan target 90 ± 15 — content is verbatim from the reviewed spec; the avg-response-length criterion still passes with margin.api-quickref.mdnet diff +38 lines vs plan budget +4 — the original file had 5 top-level sections; clean basic/advanced partitioning required restructuring rather than two header inserts. All API names preserved (verified by identifier diff).🤖 Generated with Claude Code