diff --git a/skills-templates/assemble-lib-rules/main.md.j2 b/skills-templates/assemble-lib-rules/main.md.j2 index 501e57dd3..7f3d526f5 100644 --- a/skills-templates/assemble-lib-rules/main.md.j2 +++ b/skills-templates/assemble-lib-rules/main.md.j2 @@ -1,6 +1,6 @@ --- name: assemble-lib-rules -description: Write the per-vuln-class security join rules that merge the created source/sink lib rules with the built-ins. Use to wire lib rules into project-level joins +description: Expand rule-family tags and write only the per-vulnerability security joins still needed to connect created source/sink library rules with the built-ins. Use to wire lib rules into project-level joins license: Apache-2.0 metadata: author: opentaint diff --git a/skills-templates/assemble-lib-rules/sections/gotchas.md b/skills-templates/assemble-lib-rules/sections/gotchas.md index 7d59c4466..310d3d954 100644 --- a/skills-templates/assemble-lib-rules/sections/gotchas.md +++ b/skills-templates/assemble-lib-rules/sections/gotchas.md @@ -1,2 +1,3 @@ - Ref the existing lib rules (built-in + created), never re-declare a source or sink +- A tag is an open family, not shorthand for one rule — use it only when every active same-language member should fan out through the join - Keep produced joins comment-free diff --git a/skills-templates/assemble-lib-rules/sections/output.md b/skills-templates/assemble-lib-rules/sections/output.md index 0c33a7501..fd56832d3 100644 --- a/skills-templates/assemble-lib-rules/sections/output.md +++ b/skills-templates/assemble-lib-rules/sections/output.md @@ -1,8 +1,8 @@ ### Artifacts -- one join file per (vuln class, sink rule) under `.opentaint/rules//security/--lib-ext.yaml`, each refing all relevant sources + its one sink -- `.opentaint/tracking/rules/joins/.yaml` — one per vuln class, recording every join produced (per Tracking) +- zero or more extension join files under `.opentaint/rules//security/`, only for concrete source-to-sink combinations not already covered through an existing rule/tag join +- `.opentaint/tracking/rules/joins/.yaml` — one per vuln class, recording the concrete components covered by existing and created joins (per Tracking) ### Summary -- one line per join: class, sink, source count, and which ends are new +- one line per vulnerability class: concrete source/sink counts, reused tag-expanded joins, and any extension join created diff --git a/skills-templates/assemble-lib-rules/sections/preamble.md b/skills-templates/assemble-lib-rules/sections/preamble.md index 240f0d32a..37670ba1a 100644 --- a/skills-templates/assemble-lib-rules/sections/preamble.md +++ b/skills-templates/assemble-lib-rules/sections/preamble.md @@ -1,3 +1,3 @@ # Skill: Assemble Lib Rules -Source and sink library rules are authored per package but never paired across them. Write the security joins that pair them — one per vuln class, each merging the created source/sink rules with the built-ins, mirroring the built-in security rules. The joins carry no test project, the main scan verifies them. +Source and sink library rules expose open role families through tags. Related custom sources may share a project-specific tag, and a custom join may consume that family with one `tag:` ref. Expand the existing built-in and custom joins first: a created rule that reused a consumed family tag may already be wired without another file. Add only the security joins needed for uncovered source-to-sink combinations, using tags for deliberate family expansion and exact rule refs for isolated components. The joins carry no test project; the main scan verifies them. diff --git a/skills-templates/assemble-lib-rules/sections/tracking.md b/skills-templates/assemble-lib-rules/sections/tracking.md index c4e388693..ccc7d3465 100644 --- a/skills-templates/assemble-lib-rules/sections/tracking.md +++ b/skills-templates/assemble-lib-rules/sections/tracking.md @@ -1,3 +1,3 @@ -This skill writes the joins tracking, one file per vuln class, setting each file's `stages.written: done`. The main scan verifies the joins, don't touch `verified`. +This skill writes the joins tracking, one file per vuln class, setting each file's `stages.written: done`. Record concrete source/sink refs after expanding tags so deterministic status checks don't need to reimplement rule loading. The main scan verifies the joins; don't touch `verified`. {% include "shared/tracking/joins.md" %} diff --git a/skills-templates/assemble-lib-rules/sections/workflow.md b/skills-templates/assemble-lib-rules/sections/workflow.md index cc1b1749e..19b051763 100644 --- a/skills-templates/assemble-lib-rules/sections/workflow.md +++ b/skills-templates/assemble-lib-rules/sections/workflow.md @@ -1,32 +1,29 @@ -### 1. Read the created lib rules and the built-ins +### 1. Inventory components and expand existing joins -Read every source unit under `.opentaint/tracking/rules/sources/` and sink unit under `.opentaint/tracking/rules/sinks/` — the `rule_id`s already recorded, the sinks carrying their `vuln_class` — and the built-in source/sink lib rules: +Read every source unit under `.opentaint/tracking/rules/sources/` and sink unit under `.opentaint/tracking/rules/sinks/` — the concrete `rule_id`s already recorded and each sink's `vuln_class` — plus every created library/security rule. Locate and read the built-in library/security rules through: ```bash opentaint health --rules ``` -Collect every source rule (built-in + created) and every sink rule grouped by vuln class. A rule is built-in when its ref resolves in the loaded built-in ruleset, created otherwise — that membership, not a stored tag, tells the two apart. +For each source and sink component, record its concrete ref, tags, exposed metavariable, language, and relevant vulnerability class. A `tag:` ref is a language-scoped union of every active rule carrying that exact tag, whether the rules are built-in or custom; an explicit `rule:` ref selects only one rule. Several custom sources may therefore share one project-specific tag and be consumed together by a custom join. Expand both forms in every existing security join and enumerate the concrete source-to-sink pairs its `on` clauses cover. A created rule that reused a tag already consumed by a built-in or custom join is wired by that join automatically. -When join files already exist from a prior run, reuse them as the baseline: re-assemble to fold in any new lib rule — a new source widens the `on` of every join for its vuln class, a new sink adds a join — and leave a join whose wiring no new rule touches as-is. Don't rewrite joins that already hold. +When custom join files already exist, use them as part of this baseline. Don't rewrite a join whose expansion already covers the intended pairs. -### 2. Write one security join per (vuln class, sink rule) +### 2. Add only uncovered combinations -A join references exactly ONE sink rule — several sinks can't merge into one join. So a vuln class with more than one relevant sink becomes several joins: one per sink rule, each refing all the relevant sources on the left. Sources are many, the sink is always one. +For each vulnerability class, compare all relevant source × sink combinations with the expanded pairs from step 1. Write a new `mode: join` rule only for combinations still uncovered: -For each vuln class, and within it each sink rule that needs new wiring, write a join under `.opentaint/rules//security/--lib-ext.yaml` with `mode: join`, refing the relevant sources + that one sink, wiring only the new-end combinations in `on`: +- Prefer `tag:` when every active member of that role family should participate. Tagged source and sink sides expand as a Cartesian product, so one join can deliberately cover several components. +- Use an exact `rule:` ref when tag expansion would pull in an unrelated component or duplicate a pair already covered elsewhere. +- A ref contains exactly one of `tag` or `rule`, every `as` alias is unique, and a join cannot reference another join. +- Read the component before writing `on`: custom components expose `$UNTRUSTED`, while a built-in may expose another name. -- a created sink ← from every relevant source (built-in + created) -- a built-in sink ← from created sources only (a built-in source → built-in sink pair is already covered by the built-in join) - -Two rules here: - -- Unique id — use `id: --lib-ext`, never the bare class name; a custom join named `ssrf`/`xxe`/`path-traversal` collides silently with the built-in join of that id and is dropped with no error (only the scan's rule statistics reveal it) -- Right metavariable each side — the source side is always `$UNTRUSTED`. The sink side is `$UNTRUSTED` for a custom rules, but a built-in sink may bind another name — read it from how that sink is wired in the built-in security rules and use it: `source.$UNTRUSTED -> sink.$` +Use a unique extension id such as `-lib-ext`; never reuse the built-in class id because duplicate ids are dropped. Preserve the built-in rule's user-facing metadata for that vulnerability class. ```yaml rules: - - id: ssrf-webclient-ssrf-sink-lib-ext + - id: ssrf-lib-ext severity: ERROR message: Untrusted data reaches an SSRF sink metadata: @@ -36,21 +33,18 @@ rules: mode: join join: refs: - - rule: java/lib/generic/servlet-untrusted-data-source.yaml#java-servlet-untrusted-data-source - as: servlet-source - - rule: java/lib/spring/webflux-request-source.yaml#webflux-request-source - as: webflux-source - - rule: java/lib/spring/webclient-ssrf-sink.yaml#webclient-ssrf-sink + - tag: orders-untrusted-data-source + as: orders-source + - tag: ssrf-sink as: sink on: - - 'servlet-source.$UNTRUSTED -> sink.$UNTRUSTED' - - 'webflux-source.$UNTRUSTED -> sink.$UNTRUSTED' + - 'orders-source.$UNTRUSTED -> sink.$UNTRUSTED' ``` -The same class's built-in sink is a second file (e.g. `ssrf-java-ssrf-sink-lib-ext.yaml`), refing only the created sources → that built-in sink. +This example intentionally adds every custom source whose `tags` list contains `orders-untrusted-data-source` to every active Java SSRF sink carrying `ssrf-sink`. If only one source should connect, replace its `tag:` ref with that component's exact `rule:` ref. -### 3. Verify every rule is wired, then stop +### 3. Record concrete coverage and verify -With the joins written, confirm no orphan before returning — a source or sink not merged into a join. Cross-check the `rule_id`s in the source and sink units against the joins: every created source rule must appear as a source end in the `on` of at least one join, and every created sink rule must be the sink of a join, and each created sink's join must ref all its relevant sources (built-in + created) on the left. Add a join (per step 2) for anything still unwired. +Update each class's joins tracking with concrete rule refs, even when the covering join uses tags: list every created source covered under `sources`, and one `joins` entry per concrete sink with the `rule_id` of the existing or newly created security join that covers it. Several sink entries may name the same tag-expanded join. Don't create a redundant join file merely to produce tracking state. -Set `stages.written: done` (per Tracking) and return per Output. +Expand all joins again and confirm there is no orphan or missing pair: every created source reaches every relevant sink class, every created sink is reachable from all relevant sources, and no pair is represented twice. Set `stages.written: done` (per Tracking) and return per Output. diff --git a/skills-templates/create-rule/references/java.md.j2 b/skills-templates/create-rule/references/java.md.j2 index c5db61cfb..b32dff195 100644 --- a/skills-templates/create-rule/references/java.md.j2 +++ b/skills-templates/create-rule/references/java.md.j2 @@ -10,6 +10,10 @@ Built-in source/sink lib rules live under `java/lib/generic/` (framework-neutral ### 2. Author the library rules +Every custom library source or sink declares its role family under top-level `tags`. Tags are exact, language-scoped strings used only to expand `join.refs`; they don't change matching or activate a library rule. Follow the built-in vocabulary: source families use `-untrusted-data-source`, and sink families use the security rule's exact `-sink` tag (for example `servlet-untrusted-data-source`, `spring-untrusted-data-source`, `sqli-sink`, `ssrf-sink`, or `path-traversal-sink`). Read the built-in library and security rules before choosing. + +A tag is an open extension point: adding `sqli-sink` opts the rule into every Java join that references that tag. Reuse it only when that fan-out is correct. Otherwise use a stable project-specific family such as `-untrusted-data-source` or `--sink`; the assembly stage will add the required join. Multiple custom sources can declare the same project-specific source tag, and a custom join can select the whole family with one `tag:` ref. Use an explicit `rule:` ref when only one component should be selected. + Reference a built-in: ```yaml @@ -27,6 +31,8 @@ rules: - id: my-custom-source options: lib: true + tags: + - servlet-untrusted-data-source severity: NOTE message: Custom untrusted data source languages: [java] @@ -42,6 +48,28 @@ rules: - pattern: doPost ``` +To make several custom sources a reusable family, repeat the same exact list-form tag on every source rule: + +```yaml +tags: + - orders-untrusted-data-source +``` + +A custom security join can then consume all active Java source rules in that family: + +```yaml +join: + refs: + - tag: orders-untrusted-data-source + as: orders-source + - tag: sqli-sink + as: sink + on: + - 'orders-source.$UNTRUSTED -> sink.$UNTRUSTED' +``` + +Keep the shared tag project-specific unless these sources are intentionally members of a built-in family. Tag expansion is language-scoped and includes every active matching rule, including other custom rules. + Custom sink library rule (`.opentaint/rules/java/lib/generic/my-sink.yaml`): ```yaml @@ -49,6 +77,8 @@ rules: - id: my-custom-sink options: lib: true + tags: + - sqli-sink severity: NOTE message: Custom dangerous operation languages: [java] @@ -90,7 +120,7 @@ rules: - 'src.$UNTRUSTED -> sink.$UNTRUSTED' ``` -The marker rules resolve from the sub-project's `test-rules` root, your lib rules from `.opentaint/rules` — `test rule run` is passed both. Metavariable names must match across `refs` and `on`. +The marker rules resolve from the sub-project's `test-rules` root, your lib rules from `.opentaint/rules` — `test rule run` is passed both. Metavariable names must match across `refs` and `on`. Keep these test refs explicit: replacing `rule:` with `tag:` would pull every active family member into the test and stop it isolating the new component. ### 4. Test until success @@ -103,6 +133,7 @@ The concrete operators per verdict: ## Constraints - Custom library rules go under `java/lib/generic/` or `java/lib/spring/`, mirroring the built-in layout — never directly under `java/lib/` +- Tags are exact strings, not display labels. Match the built-in security join's spelling, including distinctions such as `sqli-sink` versus the vulnerability class `sql-injection` - For a simple structural pattern (no dataflow), omit `mode` — it uses the default mode - Don't unpack or grep the analyzer JAR diff --git a/skills-templates/create-rule/sections/constraints.md b/skills-templates/create-rule/sections/constraints.md index c256d8fef..04924e3ac 100644 --- a/skills-templates/create-rule/sections/constraints.md +++ b/skills-templates/create-rule/sections/constraints.md @@ -1,6 +1,7 @@ {% include "shared/engine/facts.md" %} - Library rules MUST have `options.lib: true` and `severity: NOTE` +- Every custom source/sink library rule MUST declare at least one deliberate role-family `tags` value. Reuse an existing tag only when open expansion into all of that tag's consumers is intended - The test joins (`mode: join`) MUST have `metadata.cwe` and `metadata.short-description` - Metavariable names must match across `refs` and `on` clauses, or the join won't connect. Bind the tainted value to one consistent metavariable in every lib source/sink rule - The `rule` path in `refs` is relative to its ruleset root — a marker ref resolves under the test project's marker rules, a lib ref under the scanned rules tree diff --git a/skills-templates/create-rule/sections/output.md b/skills-templates/create-rule/sections/output.md index 3b47321de..c31ab4da6 100644 --- a/skills-templates/create-rule/sections/output.md +++ b/skills-templates/create-rule/sections/output.md @@ -5,7 +5,7 @@ ### Summary -- the lib rule ids (created or referenced), a one-line test summary for the side, and the exact `test rule run` command used +- the lib rule ids (created or referenced), each rule's family tags, a one-line test summary for the side, and the exact `test rule run` command used - if blocked at step 5: `stages.tests_passing` left pending, and the cause — - a dropped library method on the failing sample's flow → the methods that need a model, to be approximated before a re-dispatch - nothing dropped and no clear rule cause → non-convergence, for the orchestrator to intervene diff --git a/skills-templates/create-rule/sections/workflow.md b/skills-templates/create-rule/sections/workflow.md index 61858ed2e..17f78715f 100644 --- a/skills-templates/create-rule/sections/workflow.md +++ b/skills-templates/create-rule/sections/workflow.md @@ -10,11 +10,13 @@ In fix mode, don't author from the unit: go straight to that one flagged rule, a ### 2. Author the library rules -Derive each rule's pattern from the unit's fully-qualified names, recorded signatures, and annotations. Bind the tainted value to one consistent metavariable in every rule so the security joins assembled later reference one name. The rule forms — a built-in `refs`, a custom source rule, a custom sink rule, and where custom rules go — are in the language reference. +Derive each rule's pattern from the unit's fully-qualified names, recorded signatures, and annotations. Bind the tainted value to one consistent metavariable in every rule so the security joins assembled later reference one name. + +Give every custom library rule its source/sink family `tags`. Reuse a built-in role tag only when this rule should intentionally extend every join that consumes that family; otherwise create a stable project-specific role tag. Give related custom sources the same project-specific tag when custom joins should consume them as one family. Tags don't alter matching, and a fix must preserve them unless the rule's family membership is itself wrong. The rule forms, tag selection, and where custom rules go are in the language reference. ### 3. Write the test joins -A library rule emits nothing on its own — to exercise it, wire it to the generic taint marker in a throwaway test join. Write one join for the side into the test project's marker rules, referencing the generic marker on one end and each new lib rule on the other, so a positive sample's tainted value flows marker-to-rule (a sink side) or rule-to-marker (a source side). These joins live only in the test project, never in the scanned rules tree, so the main scan never loads them. The join form, its naming, and where it goes are in the language reference. +A library rule emits nothing on its own — to exercise it, wire it to the generic taint marker in a throwaway test join. Write one join for the side into the test project's marker rules, referencing the generic marker on one end and each new lib rule on the other, so a positive sample's tainted value flows marker-to-rule (a sink side) or rule-to-marker (a source side). Use exact `rule:` refs here, not the library rule's open family tag, so the test isolates only the component under test. These joins live only in the test project, never in the scanned rules tree, so the main scan never loads them. The join form, its naming, and where it goes are in the language reference. ### 4. Test until success diff --git a/skills-templates/orchestrate-stage/references/sinks.md b/skills-templates/orchestrate-stage/references/sinks.md index c3182b7d3..338becd36 100644 --- a/skills-templates/orchestrate-stage/references/sinks.md +++ b/skills-templates/orchestrate-stage/references/sinks.md @@ -23,7 +23,7 @@ Inputs: - `unit` - `fix-target` (optional) — only the scan-flagged rule correction explicitly assigned by the task -Expect back — each sink's `rule_id` set and `stages.tests_passing: done`. +Expect back — each sink's `rule_id` set, every custom rule carrying its deliberate sink-family tag, and `stages.tests_passing: done`. ## Assemble the full joins @@ -32,7 +32,7 @@ Once no sink unit is pending, status names unwired sink rules. Dispatch assemble Inputs: - `language` -Expect back — the created-sink joins added to the joins tally, each refing all relevant sources. Then delete the sink units' `test-compiled/` models. +Expect back — existing tag-expanded joins reused where they already cover the created sinks, extension joins added only for uncovered combinations, and the concrete coverage recorded in the joins tally. Then delete the sink units' `test-compiled/` models. ## Stage gate diff --git a/skills-templates/orchestrate-stage/references/sources.md b/skills-templates/orchestrate-stage/references/sources.md index 71eff8168..961bb7003 100644 --- a/skills-templates/orchestrate-stage/references/sources.md +++ b/skills-templates/orchestrate-stage/references/sources.md @@ -53,7 +53,7 @@ Inputs: - `unit` - `fix-target` (optional) — only the scan-flagged rule correction explicitly assigned by the task -Expect back — each source's `rule_id` set and `stages.tests_passing: done`. +Expect back — each source's `rule_id` set, every custom rule carrying its deliberate source-family tag, and `stages.tests_passing: done`. ## Assemble source joins @@ -62,7 +62,7 @@ Once no source unit is pending, status names unwired created sources. Dispatch a Inputs: - `language` -Expect back — the joins tally written: one join per built-in sink, each refing all created sources for that vulnerability class. Then delete the source units' `test-compiled/` models. +Expect back — existing tag-expanded joins reused where they already cover the created sources, extension joins added only for uncovered combinations, and the concrete coverage recorded in the joins tally. Then delete the source units' `test-compiled/` models. ## Stage gate diff --git a/skills-templates/shared/tracking/joins.md b/skills-templates/shared/tracking/joins.md index ff4df4ac4..3d6daa200 100644 --- a/skills-templates/shared/tracking/joins.md +++ b/skills-templates/shared/tracking/joins.md @@ -1,13 +1,13 @@ -`.opentaint/tracking/rules/joins/.yaml` — one file per vuln class (class = filename), each listing the joins written for it (one per sink rule), verified later by the main scan. `sink` is a plain ref; built-in-vs-created is derived by ruleset membership, so no tag is stored, and each join's artifact path is derivable from its `rule_id`. Keep it clear from comments +`.opentaint/tracking/rules/joins/.yaml` — one file per vuln class (class = filename), listing concrete source/sink refs and the existing or created security joins that cover them, verified later by the main scan. Tags are expanded before writing this state: `sources` and each `sink` remain concrete refs so status checks stay deterministic; several sinks may share one tag-expanded join `rule_id`. Built-in-vs-created is derived by ruleset membership. Keep it clear from comments ```yaml sources: - java/lib/generic/servlet-untrusted-data-source.yaml#java-servlet-untrusted-data-source - java/lib/spring/webflux-request-source.yaml#webflux-request-source joins: - - rule_id: java/security/ssrf-webclient-ssrf-sink-lib-ext.yaml:ssrf-webclient-ssrf-sink-lib-ext + - rule_id: java/security/ssrf.yaml:ssrf sink: java/lib/spring/webclient-ssrf-sink.yaml#webclient-ssrf-sink - - rule_id: java/security/ssrf-java-ssrf-sink-lib-ext.yaml:ssrf-java-ssrf-sink-lib-ext + - rule_id: java/security/ssrf.yaml:ssrf sink: java/lib/generic/ssrf-sinks.yaml#java-ssrf-sink stages: written: done diff --git a/skills/assemble-lib-rules/SKILL.md b/skills/assemble-lib-rules/SKILL.md index d5efb8961..f83b7b5e2 100644 --- a/skills/assemble-lib-rules/SKILL.md +++ b/skills/assemble-lib-rules/SKILL.md @@ -1,6 +1,6 @@ --- name: assemble-lib-rules -description: Write the per-vuln-class security join rules that merge the created source/sink lib rules with the built-ins. Use to wire lib rules into project-level joins +description: Expand rule-family tags and write only the per-vulnerability security joins still needed to connect created source/sink library rules with the built-ins. Use to wire lib rules into project-level joins license: Apache-2.0 metadata: author: opentaint @@ -9,7 +9,7 @@ metadata: # Skill: Assemble Lib Rules -Source and sink library rules are authored per package but never paired across them. Write the security joins that pair them — one per vuln class, each merging the created source/sink rules with the built-ins, mirroring the built-in security rules. The joins carry no test project, the main scan verifies them. +Source and sink library rules expose open role families through tags. Related custom sources may share a project-specific tag, and a custom join may consume that family with one `tag:` ref. Expand the existing built-in and custom joins first: a created rule that reused a consumed family tag may already be wired without another file. Add only the security joins needed for uncovered source-to-sink combinations, using tags for deliberate family expansion and exact rule refs for isolated components. The joins carry no test project; the main scan verifies them. ## Inputs @@ -20,35 +20,32 @@ Provided by the caller, fall back to the default value when omitted. Ask back on ## Workflow -### 1. Read the created lib rules and the built-ins +### 1. Inventory components and expand existing joins -Read every source unit under `.opentaint/tracking/rules/sources/` and sink unit under `.opentaint/tracking/rules/sinks/` — the `rule_id`s already recorded, the sinks carrying their `vuln_class` — and the built-in source/sink lib rules: +Read every source unit under `.opentaint/tracking/rules/sources/` and sink unit under `.opentaint/tracking/rules/sinks/` — the concrete `rule_id`s already recorded and each sink's `vuln_class` — plus every created library/security rule. Locate and read the built-in library/security rules through: ```bash opentaint health --rules ``` -Collect every source rule (built-in + created) and every sink rule grouped by vuln class. A rule is built-in when its ref resolves in the loaded built-in ruleset, created otherwise — that membership, not a stored tag, tells the two apart. +For each source and sink component, record its concrete ref, tags, exposed metavariable, language, and relevant vulnerability class. A `tag:` ref is a language-scoped union of every active rule carrying that exact tag, whether the rules are built-in or custom; an explicit `rule:` ref selects only one rule. Several custom sources may therefore share one project-specific tag and be consumed together by a custom join. Expand both forms in every existing security join and enumerate the concrete source-to-sink pairs its `on` clauses cover. A created rule that reused a tag already consumed by a built-in or custom join is wired by that join automatically. -When join files already exist from a prior run, reuse them as the baseline: re-assemble to fold in any new lib rule — a new source widens the `on` of every join for its vuln class, a new sink adds a join — and leave a join whose wiring no new rule touches as-is. Don't rewrite joins that already hold. +When custom join files already exist, use them as part of this baseline. Don't rewrite a join whose expansion already covers the intended pairs. -### 2. Write one security join per (vuln class, sink rule) +### 2. Add only uncovered combinations -A join references exactly ONE sink rule — several sinks can't merge into one join. So a vuln class with more than one relevant sink becomes several joins: one per sink rule, each refing all the relevant sources on the left. Sources are many, the sink is always one. +For each vulnerability class, compare all relevant source × sink combinations with the expanded pairs from step 1. Write a new `mode: join` rule only for combinations still uncovered: -For each vuln class, and within it each sink rule that needs new wiring, write a join under `.opentaint/rules//security/--lib-ext.yaml` with `mode: join`, refing the relevant sources + that one sink, wiring only the new-end combinations in `on`: +- Prefer `tag:` when every active member of that role family should participate. Tagged source and sink sides expand as a Cartesian product, so one join can deliberately cover several components. +- Use an exact `rule:` ref when tag expansion would pull in an unrelated component or duplicate a pair already covered elsewhere. +- A ref contains exactly one of `tag` or `rule`, every `as` alias is unique, and a join cannot reference another join. +- Read the component before writing `on`: custom components expose `$UNTRUSTED`, while a built-in may expose another name. -- a created sink ← from every relevant source (built-in + created) -- a built-in sink ← from created sources only (a built-in source → built-in sink pair is already covered by the built-in join) - -Two rules here: - -- Unique id — use `id: --lib-ext`, never the bare class name; a custom join named `ssrf`/`xxe`/`path-traversal` collides silently with the built-in join of that id and is dropped with no error (only the scan's rule statistics reveal it) -- Right metavariable each side — the source side is always `$UNTRUSTED`. The sink side is `$UNTRUSTED` for a custom rules, but a built-in sink may bind another name — read it from how that sink is wired in the built-in security rules and use it: `source.$UNTRUSTED -> sink.$` +Use a unique extension id such as `-lib-ext`; never reuse the built-in class id because duplicate ids are dropped. Preserve the built-in rule's user-facing metadata for that vulnerability class. ```yaml rules: - - id: ssrf-webclient-ssrf-sink-lib-ext + - id: ssrf-lib-ext severity: ERROR message: Untrusted data reaches an SSRF sink metadata: @@ -58,50 +55,47 @@ rules: mode: join join: refs: - - rule: java/lib/generic/servlet-untrusted-data-source.yaml#java-servlet-untrusted-data-source - as: servlet-source - - rule: java/lib/spring/webflux-request-source.yaml#webflux-request-source - as: webflux-source - - rule: java/lib/spring/webclient-ssrf-sink.yaml#webclient-ssrf-sink + - tag: orders-untrusted-data-source + as: orders-source + - tag: ssrf-sink as: sink on: - - 'servlet-source.$UNTRUSTED -> sink.$UNTRUSTED' - - 'webflux-source.$UNTRUSTED -> sink.$UNTRUSTED' + - 'orders-source.$UNTRUSTED -> sink.$UNTRUSTED' ``` -The same class's built-in sink is a second file (e.g. `ssrf-java-ssrf-sink-lib-ext.yaml`), refing only the created sources → that built-in sink. +This example intentionally adds every custom source whose `tags` list contains `orders-untrusted-data-source` to every active Java SSRF sink carrying `ssrf-sink`. If only one source should connect, replace its `tag:` ref with that component's exact `rule:` ref. -### 3. Verify every rule is wired, then stop +### 3. Record concrete coverage and verify -With the joins written, confirm no orphan before returning — a source or sink not merged into a join. Cross-check the `rule_id`s in the source and sink units against the joins: every created source rule must appear as a source end in the `on` of at least one join, and every created sink rule must be the sink of a join, and each created sink's join must ref all its relevant sources (built-in + created) on the left. Add a join (per step 2) for anything still unwired. +Update each class's joins tracking with concrete rule refs, even when the covering join uses tags: list every created source covered under `sources`, and one `joins` entry per concrete sink with the `rule_id` of the existing or newly created security join that covers it. Several sink entries may name the same tag-expanded join. Don't create a redundant join file merely to produce tracking state. -Set `stages.written: done` (per Tracking) and return per Output. +Expand all joins again and confirm there is no orphan or missing pair: every created source reaches every relevant sink class, every created sink is reachable from all relevant sources, and no pair is represented twice. Set `stages.written: done` (per Tracking) and return per Output. ## Output ### Artifacts -- one join file per (vuln class, sink rule) under `.opentaint/rules//security/--lib-ext.yaml`, each refing all relevant sources + its one sink -- `.opentaint/tracking/rules/joins/.yaml` — one per vuln class, recording every join produced (per Tracking) +- zero or more extension join files under `.opentaint/rules//security/`, only for concrete source-to-sink combinations not already covered through an existing rule/tag join +- `.opentaint/tracking/rules/joins/.yaml` — one per vuln class, recording the concrete components covered by existing and created joins (per Tracking) ### Summary -- one line per join: class, sink, source count, and which ends are new +- one line per vulnerability class: concrete source/sink counts, reused tag-expanded joins, and any extension join created ## Tracking -This skill writes the joins tracking, one file per vuln class, setting each file's `stages.written: done`. The main scan verifies the joins, don't touch `verified`. +This skill writes the joins tracking, one file per vuln class, setting each file's `stages.written: done`. Record concrete source/sink refs after expanding tags so deterministic status checks don't need to reimplement rule loading. The main scan verifies the joins; don't touch `verified`. -`.opentaint/tracking/rules/joins/.yaml` — one file per vuln class (class = filename), each listing the joins written for it (one per sink rule), verified later by the main scan. `sink` is a plain ref; built-in-vs-created is derived by ruleset membership, so no tag is stored, and each join's artifact path is derivable from its `rule_id`. Keep it clear from comments +`.opentaint/tracking/rules/joins/.yaml` — one file per vuln class (class = filename), listing concrete source/sink refs and the existing or created security joins that cover them, verified later by the main scan. Tags are expanded before writing this state: `sources` and each `sink` remain concrete refs so status checks stay deterministic; several sinks may share one tag-expanded join `rule_id`. Built-in-vs-created is derived by ruleset membership. Keep it clear from comments ```yaml sources: - java/lib/generic/servlet-untrusted-data-source.yaml#java-servlet-untrusted-data-source - java/lib/spring/webflux-request-source.yaml#webflux-request-source joins: - - rule_id: java/security/ssrf-webclient-ssrf-sink-lib-ext.yaml:ssrf-webclient-ssrf-sink-lib-ext + - rule_id: java/security/ssrf.yaml:ssrf sink: java/lib/spring/webclient-ssrf-sink.yaml#webclient-ssrf-sink - - rule_id: java/security/ssrf-java-ssrf-sink-lib-ext.yaml:ssrf-java-ssrf-sink-lib-ext + - rule_id: java/security/ssrf.yaml:ssrf sink: java/lib/generic/ssrf-sinks.yaml#java-ssrf-sink stages: written: done @@ -111,4 +105,5 @@ stages: ## Gotchas - Ref the existing lib rules (built-in + created), never re-declare a source or sink +- A tag is an open family, not shorthand for one rule — use it only when every active same-language member should fan out through the join - Keep produced joins comment-free diff --git a/skills/create-rule/SKILL.md b/skills/create-rule/SKILL.md index 323c89356..b80ab358c 100644 --- a/skills/create-rule/SKILL.md +++ b/skills/create-rule/SKILL.md @@ -35,11 +35,13 @@ In fix mode, don't author from the unit: go straight to that one flagged rule, a ### 2. Author the library rules -Derive each rule's pattern from the unit's fully-qualified names, recorded signatures, and annotations. Bind the tainted value to one consistent metavariable in every rule so the security joins assembled later reference one name. The rule forms — a built-in `refs`, a custom source rule, a custom sink rule, and where custom rules go — are in the language reference. +Derive each rule's pattern from the unit's fully-qualified names, recorded signatures, and annotations. Bind the tainted value to one consistent metavariable in every rule so the security joins assembled later reference one name. + +Give every custom library rule its source/sink family `tags`. Reuse a built-in role tag only when this rule should intentionally extend every join that consumes that family; otherwise create a stable project-specific role tag. Give related custom sources the same project-specific tag when custom joins should consume them as one family. Tags don't alter matching, and a fix must preserve them unless the rule's family membership is itself wrong. The rule forms, tag selection, and where custom rules go are in the language reference. ### 3. Write the test joins -A library rule emits nothing on its own — to exercise it, wire it to the generic taint marker in a throwaway test join. Write one join for the side into the test project's marker rules, referencing the generic marker on one end and each new lib rule on the other, so a positive sample's tainted value flows marker-to-rule (a sink side) or rule-to-marker (a source side). These joins live only in the test project, never in the scanned rules tree, so the main scan never loads them. The join form, its naming, and where it goes are in the language reference. +A library rule emits nothing on its own — to exercise it, wire it to the generic taint marker in a throwaway test join. Write one join for the side into the test project's marker rules, referencing the generic marker on one end and each new lib rule on the other, so a positive sample's tainted value flows marker-to-rule (a sink side) or rule-to-marker (a source side). Use exact `rule:` refs here, not the library rule's open family tag, so the test isolates only the component under test. These joins live only in the test project, never in the scanned rules tree, so the main scan never loads them. The join form, its naming, and where it goes are in the language reference. ### 4. Test until success @@ -79,7 +81,7 @@ A positive that won't pass after ~3 rule fixes may have a cause no rule edit can ### Summary -- the lib rule ids (created or referenced), a one-line test summary for the side, and the exact `test rule run` command used +- the lib rule ids (created or referenced), each rule's family tags, a one-line test summary for the side, and the exact `test rule run` command used - if blocked at step 5: `stages.tests_passing` left pending, and the cause — - a dropped library method on the failing sample's flow → the methods that need a model, to be approximated before a re-dispatch - nothing dropped and no clear rule cause → non-convergence, for the orchestrator to intervene @@ -117,6 +119,7 @@ stages: OpenTaint is a whole-program, interprocedural, field-sensitive alias analysis engine. It already propagates through visible application code, calls, aliases, and individual fields; custom rules and approximations model only the assigned source, sink, or opaque-method boundary. Compile-time constants and literals carry no taint, so a source or carrier whose output is only a constant introduces nothing. - Library rules MUST have `options.lib: true` and `severity: NOTE` +- Every custom source/sink library rule MUST declare at least one deliberate role-family `tags` value. Reuse an existing tag only when open expansion into all of that tag's consumers is intended - The test joins (`mode: join`) MUST have `metadata.cwe` and `metadata.short-description` - Metavariable names must match across `refs` and `on` clauses, or the join won't connect. Bind the tainted value to one consistent metavariable in every lib source/sink rule - The `rule` path in `refs` is relative to its ruleset root — a marker ref resolves under the test project's marker rules, a lib ref under the scanned rules tree diff --git a/skills/create-rule/references/java.md b/skills/create-rule/references/java.md index c5db61cfb..b32dff195 100644 --- a/skills/create-rule/references/java.md +++ b/skills/create-rule/references/java.md @@ -10,6 +10,10 @@ Built-in source/sink lib rules live under `java/lib/generic/` (framework-neutral ### 2. Author the library rules +Every custom library source or sink declares its role family under top-level `tags`. Tags are exact, language-scoped strings used only to expand `join.refs`; they don't change matching or activate a library rule. Follow the built-in vocabulary: source families use `-untrusted-data-source`, and sink families use the security rule's exact `-sink` tag (for example `servlet-untrusted-data-source`, `spring-untrusted-data-source`, `sqli-sink`, `ssrf-sink`, or `path-traversal-sink`). Read the built-in library and security rules before choosing. + +A tag is an open extension point: adding `sqli-sink` opts the rule into every Java join that references that tag. Reuse it only when that fan-out is correct. Otherwise use a stable project-specific family such as `-untrusted-data-source` or `--sink`; the assembly stage will add the required join. Multiple custom sources can declare the same project-specific source tag, and a custom join can select the whole family with one `tag:` ref. Use an explicit `rule:` ref when only one component should be selected. + Reference a built-in: ```yaml @@ -27,6 +31,8 @@ rules: - id: my-custom-source options: lib: true + tags: + - servlet-untrusted-data-source severity: NOTE message: Custom untrusted data source languages: [java] @@ -42,6 +48,28 @@ rules: - pattern: doPost ``` +To make several custom sources a reusable family, repeat the same exact list-form tag on every source rule: + +```yaml +tags: + - orders-untrusted-data-source +``` + +A custom security join can then consume all active Java source rules in that family: + +```yaml +join: + refs: + - tag: orders-untrusted-data-source + as: orders-source + - tag: sqli-sink + as: sink + on: + - 'orders-source.$UNTRUSTED -> sink.$UNTRUSTED' +``` + +Keep the shared tag project-specific unless these sources are intentionally members of a built-in family. Tag expansion is language-scoped and includes every active matching rule, including other custom rules. + Custom sink library rule (`.opentaint/rules/java/lib/generic/my-sink.yaml`): ```yaml @@ -49,6 +77,8 @@ rules: - id: my-custom-sink options: lib: true + tags: + - sqli-sink severity: NOTE message: Custom dangerous operation languages: [java] @@ -90,7 +120,7 @@ rules: - 'src.$UNTRUSTED -> sink.$UNTRUSTED' ``` -The marker rules resolve from the sub-project's `test-rules` root, your lib rules from `.opentaint/rules` — `test rule run` is passed both. Metavariable names must match across `refs` and `on`. +The marker rules resolve from the sub-project's `test-rules` root, your lib rules from `.opentaint/rules` — `test rule run` is passed both. Metavariable names must match across `refs` and `on`. Keep these test refs explicit: replacing `rule:` with `tag:` would pull every active family member into the test and stop it isolating the new component. ### 4. Test until success @@ -103,6 +133,7 @@ The concrete operators per verdict: ## Constraints - Custom library rules go under `java/lib/generic/` or `java/lib/spring/`, mirroring the built-in layout — never directly under `java/lib/` +- Tags are exact strings, not display labels. Match the built-in security join's spelling, including distinctions such as `sqli-sink` versus the vulnerability class `sql-injection` - For a simple structural pattern (no dataflow), omit `mode` — it uses the default mode - Don't unpack or grep the analyzer JAR diff --git a/skills/orchestrate-stage/references/sinks.md b/skills/orchestrate-stage/references/sinks.md index c3182b7d3..338becd36 100644 --- a/skills/orchestrate-stage/references/sinks.md +++ b/skills/orchestrate-stage/references/sinks.md @@ -23,7 +23,7 @@ Inputs: - `unit` - `fix-target` (optional) — only the scan-flagged rule correction explicitly assigned by the task -Expect back — each sink's `rule_id` set and `stages.tests_passing: done`. +Expect back — each sink's `rule_id` set, every custom rule carrying its deliberate sink-family tag, and `stages.tests_passing: done`. ## Assemble the full joins @@ -32,7 +32,7 @@ Once no sink unit is pending, status names unwired sink rules. Dispatch assemble Inputs: - `language` -Expect back — the created-sink joins added to the joins tally, each refing all relevant sources. Then delete the sink units' `test-compiled/` models. +Expect back — existing tag-expanded joins reused where they already cover the created sinks, extension joins added only for uncovered combinations, and the concrete coverage recorded in the joins tally. Then delete the sink units' `test-compiled/` models. ## Stage gate diff --git a/skills/orchestrate-stage/references/sources.md b/skills/orchestrate-stage/references/sources.md index 71eff8168..961bb7003 100644 --- a/skills/orchestrate-stage/references/sources.md +++ b/skills/orchestrate-stage/references/sources.md @@ -53,7 +53,7 @@ Inputs: - `unit` - `fix-target` (optional) — only the scan-flagged rule correction explicitly assigned by the task -Expect back — each source's `rule_id` set and `stages.tests_passing: done`. +Expect back — each source's `rule_id` set, every custom rule carrying its deliberate source-family tag, and `stages.tests_passing: done`. ## Assemble source joins @@ -62,7 +62,7 @@ Once no source unit is pending, status names unwired created sources. Dispatch a Inputs: - `language` -Expect back — the joins tally written: one join per built-in sink, each refing all created sources for that vulnerability class. Then delete the source units' `test-compiled/` models. +Expect back — existing tag-expanded joins reused where they already cover the created sources, extension joins added only for uncovered combinations, and the concrete coverage recorded in the joins tally. Then delete the source units' `test-compiled/` models. ## Stage gate