skill-vet

One-command, three-verdict (APPROVE / REVIEW / REJECT) wrapper over Cisco skill-scanner. Hides the flags, the env-var setup, and the deterministic-vs-LLM analyzer math so a reviewer can decide in seconds.

$ skill-vet /path/to/skill/

[REJECT] /path/to/skill
  HIGH     PIPELINE_TAINT_FLOW              SKILL.md:1
  HIGH     PIPELINE_TAINT_FLOW              SKILL.md:2

  Decision driver: deterministic HIGH+ findings.
  Cite the rule_ids + file:line above in your decision note.
  Advisory layer (5 LLM findings) corroborates but did not drive the decision.

Exit code: 0 APPROVE · 1 REJECT · 2 REVIEW.

---

Why this exists

The stock Cisco scanner emits a rich JSON report. Operationally that puts a lot of weight on the reviewer: read findings[], group by analyzer, ignore the top-level is_safe, apply the runbook's decision matrix. We did that for a few months and learned the policy is stable and well-defined — the reviewer just shouldn't have to apply it by hand every time.

skill-vet is a ~215-line Python wrapper that runs the canonical full-stack invocation and applies the verdict policy from the Cisco config wiki page for you. The wrapper doesn't add new analyzers, doesn't change Cisco's decisions, and doesn't ship its own intelligence — it's a translation layer from "rich JSON" to "one verdict

• rationale".

Internal context: Agent Skill Review Runbook.

---

Quick start

1. Clone and set up

git clone https://git.lab.smartsheet.com/application-security/security-reviews/skill-vet.git
cd skill-vet

# The wrapper imports from the Cisco scanner package. Either:
#   (a) Have skill-scanner installed in the same venv (recommended), or
#   (b) Clone it as a sibling and point PYTHONPATH at it.
git clone https://github.com/cisco-ai-defense/skill-scanner.git ../skill-scanner
cd ../skill-scanner
uv venv .venv --python 3.12
uv sync
uv pip install --python .venv/bin/python boto3   # not in default sync
cd ../skill-vet

2. Configure Bedrock

The wrapper defaults to Opus 4.8 on Bedrock. Authenticate with AWS:

export AWS_PROFILE=prod-eng-ai-sandbox
export AWS_REGION=us-west-2
# Verify
aws sts get-caller-identity

Or set your own backend by exporting any of:

SKILL_SCANNER_LLM_PROVIDER=aws-bedrock   # default
SKILL_SCANNER_LLM_MODEL=...               # default: Opus 4.8 inference-profile
SKILL_SCANNER_LLM_TEMPERATURE=none        # required for Claude 4.x

3. Run

../skill-scanner/.venv/bin/python skill_vet.py /path/to/skill/

# Optional: shell alias
alias skill-vet='/full/path/to/skill-scanner/.venv/bin/python /full/path/to/skill-vet/skill_vet.py'

---

What you get

APPROVE — green banner, exit 0

[APPROVE] /path/to/skill
  Deterministic findings: 1 (max severity INFO)
  Advisory layer: 0 LLM finding(s), all below MEDIUM — not blocking.

  Next step: install with the runbook pinning checklist
  (pin exact version, disable auto-update, re-scan on change, restrict connectors).

What to do: Install the skill, pin the exact version, disable auto-update, restrict connectors per the runbook. Re-scan on any version change.

REVIEW — yellow banner, exit 2

[REVIEW] /path/to/skill
  INFO     MANIFEST_MISSING_LICENSE         SKILL.md
  HIGH     LLM_PROMPT_INJECTION             (no file)  (advisory)

  Why REVIEW: no deterministic HIGH+, but advisory layer flagged ≥ MEDIUM
  or deterministic MEDIUM findings need manual judgment.

  Next step: read the runbook's manual checklist; re-run once if the
  LLM stage produced the only HIGH/CRITICAL (LLM findings flap).

What to do: Apply the manual checklist in the runbook. The wrapper has deferred to you because the deterministic core didn't auto-reject and the LLM layer raised a concern. Re-run once if the only HIGH came from the LLM stage (those flap run-to-run).

REJECT — red banner, exit 1

[REJECT] /path/to/skill
  HIGH     PIPELINE_TAINT_FLOW              SKILL.md:1
  CRITICAL YARA_prompt_injection_generic    SKILL.md:14

  Decision driver: deterministic HIGH+ findings.
  Cite the rule_ids + file:line above in your decision note.

What to do: Reject. Cite rule_id + file:line in your decision note — those are stable across runs. Tell the requester what specifically failed.

---

Flags

Flag	Effect
--show-findings	Print every finding (with descriptions) after the verdict. Useful for the manual-review case.
--no-llm	Skip the LLM stage entirely. Faster, no Bedrock cost, but only deterministic findings — see "caveats" below.

---

When to use --no-llm

The deterministic core (static / pipeline / behavioral / bytecode) is stable across runs. The LLM stage adds the semantic catches (prompt injection in plain English, role-play hijack, intent reading). The eval data shows:

• --no-llm never APPROVED an expected-REJECT skill in our corpus — safe as a fast CI pre-filter or air-gapped gate.
• --no-llm does collapse 26 expected-REVIEW skills to APPROVE that full-stack would have caught — not safe as a sole reviewer tool for marginal cases.

Rule of thumb: use --no-llm for CI gating where speed matters and you only care about the auto-reject signal. Use full-stack (the default) when you're the human deciding.

---

Caveats

1. The wrapper inherits every Cisco scanner blind-spot. Time bombs, runtime-only payloads, image-embedded instructions, composition attacks with MCP servers, etc. See the Scanner Comparison wiki page §5 for the full list. The wrapper makes Cisco easier to use, not better at detection.
2. LLM findings flap; deterministic findings don't. A skill can produce a different LLM-CRITICAL count run-to-run. The wrapper's verdict is stable because we drive from the deterministic layer.
3. The 13.7% "REVIEW on safe" rate in the eval corpus is partly a labeling artifact — many of those skills should be REVIEW. Pair- rating in progress.
4. Skills without SKILL.md cannot be scanned. The wrapper returns REVIEW + a clear "scanner failed" message so you don't silently approve.

---

Repository contents

Path	What's there
skill_vet.py	The wrapper. ~215 lines. Imports the Cisco scanner Python API.
eval/	Test harness, metrics, corpus manifest. Reproducible evidence.
corpus/	Adversarial fixtures + real-world sampling notes.
runs/	Eval results (results.jsonl) and the auto-rendered report.md.
upstream/	Draft GitHub issue + PR for cisco-ai-defense/skill-scanner.
study/	Reviewer-experience study kit (3 reviewers × 3 skills × 2 conditions).
runs/INTERNAL-SUMMARY.md	Plain-language write-up of what the eval shows.

---

Reproducing the eval

bash eval/repro.sh           # full eval (~1h wall, ~$50 Bedrock)
bash eval/repro.sh --smoke   # 5 skills × 1 run, ~5min, ~$5

See eval/INTERNAL-SUMMARY.md for the latest numbers.

---

Owner / questions

Rohan Isawe (Information Security). File issues against this repo or ping in the security Slack channel. For upstream Cisco discussion, see upstream/issue-draft.md.