Quickstart · How it works · Commands · Eval harness · Contributing
When a CVE drops, detection engineers spend hours reading advisory text and patch diffs to hand-write a detection rule. SigCraft does the mechanical first pass and hands a reviewer something to correct rather than a blank file.
Warning
Every emitted rule is an unreviewed candidate. Nothing here is production-ready out of the box. The validation layer tells you which parts to trust; a human still signs off before deployment.
git clone https://github.com/tal7aouy/sigcraft.git
cd sigcraft
uv pip install -e ".[dev]"
# No API key needed — run the deterministic half against a sample patch
sigcraft inspect --diff examples/path-traversal.diff local-diff: path-traversal
2 diff hunk(s) across 1 file(s)
6 signal(s): added_guard=2, http_param=1, function=1, file_path=1, binary_string=1
╭──────────────────────────────── signals ─────────────────────────────────╮
│ http_param: │
│ - file # name := req.URL.Query().Get("file") │
│ file_path: │
│ - internal/handler/download.go │
│ function: │
│ - registerRoutes # func registerRoutes(r *mux.Router) { │
│ added_guard: │
│ - path-normalisation # target := filepath.Clean(filepath.Join(...)) │
│ - traversal-check # if !strings.HasPrefix(target, storageRoot...) │
│ binary_string: │
│ - read failed: # present in vulnerable version, removed by patch │
╰──────────────────────────────────────────────────────────────────────────╯
nuclei (nuclei 2.7 vs yara 0.5)
- 1 HTTP-shaped signal(s) extracted
- 1 literal string(s) recovered from the diff
- patch touches mostly web-stack code (2 web vs 0 native files)
Every indicator carries the line it came from. That provenance is what the validator later audits the generated rule against.
An LLM writing YAML is not the product. The validation layer is the product.
Anyone can feed an advisory to a model and get a plausible-looking template. What makes the output usable is knowing which parts to trust. So:
- A deterministic extractor mines indicators — paths, params, versions, functions, strings — from the advisory and diff before the model is called.
- The model is instructed it may only use indicators traceable to those inputs.
- The validator cross-checks the generated rule against the extracted signals. If a template probes a path the extractor never saw, that's flagged as possibly invented.
That cross-check (signal-anchored in both validators) is the differentiator. The extractor is the ground truth the output is audited against, and it is never weakened to make the model's job easier.
ingest → extract → route → generate (LLM) → validate → emit
| Stage | What it does | Network | API key |
|---|---|---|---|
| ingest | NVD, GitHub commits/PRs, GHSA, local diffs | yes¹ | — |
| extract | Deterministic signal mining | — | — |
| route | Scores nuclei vs yara | — | — |
| generate | Anthropic structured-output call | yes | yes |
| validate | 3 layers: structure / semantics / engine | — | — |
| emit | Provenance header + file write | — | — |
| eval | Corpus scoring | — | — |
¹ Not needed with --diff.
Only generate needs a key. inspect, check, eval, and gen --dry-run run the entire deterministic half without one — that's how you develop and demo this without spending tokens.
Both validators run cheapest-first, and each layer is independently callable:
| Layer | Catches |
|---|---|
| structure | Doesn't parse, missing required fields, bad severity |
| semantics | Status-code-only matchers, generic words (error, admin), missing matchers-condition: and, YARA strings under 6 bytes, any of them with many strings, missing filesize/magic guard, invented indicators |
| engine | Real nuclei -validate / yara compile |
Findings are pass / warn / fail. A missing binary is always a warn, never a fail — absent tooling says nothing about a rule's quality.
Requires Python 3.11+. uv recommended.
uv pip install -e ".[dev]"Optional extras:
uv pip install -e ".[yara]" # yara-python, enables YARA compile-checkExternal validators are auto-detected and optional:
nuclei— engine validation for templatesyaraCLI oryara-python— compile-check for rules
Run sigcraft version to see what's available.
# Deterministic half — no API key needed
sigcraft inspect --cve CVE-2023-4966
sigcraft inspect --diff examples/path-traversal.diff
sigcraft inspect --commit https://github.com/o/r/commit/SHA --show-prompt
# Generate a candidate rule (needs ANTHROPIC_API_KEY)
sigcraft gen --cve CVE-2024-1234 --target auto -o out/
sigcraft gen --cve CVE-2024-1234 --dry-run # everything except the model call
# Batch triage a whole list of CVEs (Patch Tuesday)
sigcraft gen --from-file cves.txt --dry-run # no key needed
sigcraft gen --from-file cves.txt --concurrency 4 -o out/
# Score the pipeline against known-good rules
sigcraft eval
sigcraft eval --generated out/ --verbose
sigcraft eval --json
# Validate any existing rule file
sigcraft check out/cve-2024-1234.yaml
sigcraft versionExit codes: 0 ok · 1 error · 2 validation failed (the rule is still written so you can inspect it). eval exits 1 if routing accuracy falls below --min-routing or a known-good reference rule fails validator alignment.
sigcraft eval scores the deterministic pipeline against a corpus of CVEs with known-good published rules — no model call, no key. It turns extractor/router/prompt tuning from vibes into measurement.
╭──────────────────────────── eval: 5 case(s) ─────────────────────────────╮
│ case target route recall align sim │
│ cve-2023-4966-citrix-bleed nuclei ok 0.33 ok skip │
│ cve-2024-3389-use-after-free yara ok 1.00 ok skip │
│ cve-2024-5561-path-traversal nuclei ok 0.67 ok skip │
│ cve-2024-7780-heap-overflow yara ok 1.00 ok skip │
│ cve-2024-9020-reflected-xss nuclei ok 0.50 ok skip │
╰──────────────────────────────────────────────────────────────────────────╯
routing accuracy: 1.00 mean recall: 0.70 alignment failures: 0
Three scores, each independently meaningful:
- routing accuracy — did the router pick the engine the reference rule is written for?
- extraction recall — of the indicators the reference rule actually uses, how many did the extractor surface? Misses are reported as missing rather than collapsed to zero, because an indicator may simply not be present in the inputs — that's not an extractor gap. Read the missing list to tell them apart.
- validator alignment — does our validator accept the known-good rule? A hard fail is either a validator bug or a low-quality reference; both are worth surfacing.
generation-similarity is only computable once gen has run against a real key, so it reports skipped unless --generated is passed — a keyless run can't be misread as "0% similar".
Add a case by dropping a directory in eval/corpus/ containing case.yaml, a reference.yaml/reference.yar, and an optional *.diff. No registration needed. Because the corpus is offline and deterministic, a score regression is a code regression — not a flaky API.
sigcraft/
├── models.py # Pydantic domain model, shared by every stage
├── cli.py # gen / inspect / check / eval / version
├── route.py # nuclei-vs-yara scoring
├── emit.py # provenance header + file writing
├── ingest/ # NVD · GitHub · GHSA · unified-diff parser
├── extract/signals.py # all deterministic signal mining
├── llm/client.py # Anthropic call, structured output
└── validate/ # nuclei_check.py · yara_check.py
eval/ # corpus loading, metrics, runner
prompts/ # nuclei.md, yara.md — tune without touching code
examples/ # fixture diffs (web + native) for testing routing
Detection guidance lives in prompts/*.md and is loaded at runtime — tune it there rather than inlining prompt text into Python.
| Variable | Required | Purpose |
|---|---|---|
ANTHROPIC_API_KEY |
for gen only |
Model access |
NVD_API_KEY |
optional | Raises NVD rate limit |
GITHUB_TOKEN |
optional | Raises GitHub rate limit |
Contributions are welcome. Start with CONTRIBUTING.md — it covers the dev setup, the invariants that must not be broken, and how to add extraction patterns or eval cases safely.
By participating you agree to the Code of Conduct. To report a vulnerability, see SECURITY.md.