From 467016052671d65e314f175311e99c42c0f2fbb0 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Tue, 25 Aug 2026 11:32:01 -0700 Subject: [PATCH 01/13] docs: document AI configuration and usage AI-assisted assessments have shipped for three requirements, but nothing told users how to turn them on, which providers work, what the settings mean, or how to read an [AI-Assisted] verdict. Add a concise "AI-Assisted Checks" section to the README covering the six ai_* keys and their PVTR_AI_* environment variables, a minimal configuration, and how to interpret results. Add docs/ai-assisted-checks.md for the detail: provider examples for openai and anthropic, configuration precedence, verdict and confidence semantics, failure behavior, the evidence record, per- requirement coverage, size limits, and cost considerations. Add a commented AI block to example-config.yml, split into required and optional keys. The keys, defaults, verdict schema, and failure paths documented here were verified against privateer-sdk v1.32.2 and the step implementations. Dry-run is deliberately not documented. It briefly existed in the SDK (privateerproj/privateer-sdk#227) and was removed when the ai package was restructured (privateerproj/privateer-sdk#252); this scanner never implemented it. The docs instead explain how to validate a configuration without provider spend, relying on the SDK validating provider, model, and credential locally before any network call. Closes #319 Signed-off-by: vinayada1 --- README.md | 72 ++++++++ docs/ai-assisted-checks.md | 335 +++++++++++++++++++++++++++++++++++++ example-config.yml | 17 +- 3 files changed, 423 insertions(+), 1 deletion(-) create mode 100644 docs/ai-assisted-checks.md diff --git a/README.md b/README.md index 88257a40..29daeeec 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,78 @@ If the binaries and the config files are in different directories specify the co You may have to adjust the plugin name in the config.yaml file to match them. +## AI-Assisted Checks + +A few OSPS Baseline requirements ask whether a project *documents* something, or +whether a setting is *appropriate for what it is used for*. Those questions +cannot be answered by pattern matching, so the scanner can optionally ask a +large language model and record its answer as evidence. + +AI is **opt-in**. With no `ai_*` settings the scanner behaves exactly as it +always has, contacts no provider, and sends nothing anywhere. + +### Configuration + + + +| Key | Environment variable | Required | Default | Purpose | +| --- | --- | :---: | --- | --- | +| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Backend adapter: `openai` or `anthropic`. | +| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Provider-specific model identifier. | +| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Provider credential. | +| `ai_base_url` | `PVTR_AI_BASE_URL` | no | adapter default | Alternate endpoint: proxy, gateway, or self-hosted deployment. | +| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Per-call timeout, as a Go duration string. | +| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Cap on the model's response length. | + + + +Declare the keys at the top level to have every service inherit them, or inside +a single service's `vars` block to scope them to that service: + +```yaml +ai_provider: openai +ai_model: gpt-4o-mini +# Supply the credential via PVTR_AI_API_KEY rather than writing it here. + +services: + my-scan: + plugin: github-repo + vars: + owner: + repo: + token: +``` + +### Reading The Results + +A requirement answered with AI help is prefixed with `[AI-Assisted]`: + +```text +[AI-Assisted] CONTRIBUTING.md tells contributors to run `go test ./...` before opening a pull request. +``` + +The model's verdict maps to `Passed`, `Failed`, or `Needs Review`, with a +`low` / `medium` / `high` confidence. Anything unexpected — a malformed +response, a provider error, a timeout, a missing credential — becomes +`Needs Review` at low confidence and the scan continues. **An AI-assisted check +never silently passes a requirement.** + +The model's full reasoning, the exact prompt, the material it was shown, and the +model used are all recorded as evidence in the evaluation log, so a reviewer can +audit or dispute the answer. + +Three requirements use AI today: `OSPS-QA-06.02`, `OSPS-QA-06.03`, and +`OSPS-AC-04.02`. At most one provider call is made per applicable requirement +per scan, and only the specific documentation or workflow files a question needs +are sent — never the whole repository. + +Note that repository content is sent to whichever provider you configure, and +that provider usage is billed to you. + +For provider examples, configuration precedence, how to validate a +configuration without provider spend, size limits, and the full evidence +format, see [docs/ai-assisted-checks.md](docs/ai-assisted-checks.md). + ## Docker Usage ```sh diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md new file mode 100644 index 00000000..56737ab4 --- /dev/null +++ b/docs/ai-assisted-checks.md @@ -0,0 +1,335 @@ +# AI-Assisted Checks + +This guide explains how to enable AI-assisted assessments in the Privateer +GitHub repository scanner, how to configure a provider, and how to read the +results. + +The behavior described here reflects +[`privateer-sdk`](https://github.com/privateerproj/privateer-sdk) **v1.32.2**, +the version this scanner currently builds against. The SDK owns the AI client, +the verdict schema, and the `ai_*` configuration keys, so upgrading the SDK can +change this surface. + +For a short introduction, see the +[AI-Assisted Checks](../README.md#ai-assisted-checks) section of the README. + +## What AI Assistance Does + +A handful of OSPS Baseline requirements ask whether a project *documents* +something, or whether a configuration is *appropriate for what a job actually +does*. These questions cannot be answered by pattern matching alone. For those +requirements, the scanner can send a bounded slice of repository content to a +large language model and ask for a structured verdict. + +AI never replaces a deterministic check. It is only consulted where a +deterministic check does not exist or cannot reach a conclusion, and its answer +is recorded as evidence alongside every other observation in the evaluation log. + +## AI Is Opt-In + +When none of the `ai_*` keys are set, AI is off. The scanner makes no requests +to any provider, and every AI-capable step keeps the same non-AI verdict it +produced before AI support existed. + +Setting *any* `ai_*` key while a required one is missing is treated as a +misconfiguration rather than as "disabled" — including setting only an optional +key such as `ai_base_url`. The affected step reports `Needs Review` and logs a +warning; the scan itself still completes. + +## Configuration Keys + + + +| Key | Environment variable | Required | Default | Purpose | +| --- | --- | :---: | --- | --- | +| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Backend adapter: `openai` or `anthropic`. | +| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Provider-specific model identifier. | +| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Provider credential. | +| `ai_base_url` | `PVTR_AI_BASE_URL` | no | adapter default | Alternate endpoint: a proxy, gateway, or self-hosted deployment. | +| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Per-call timeout, as a Go duration string. | +| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Cap on the length of the model's response. | + + + +The default endpoints are `https://api.openai.com/v1` for `openai` and +`https://api.anthropic.com/v1` for `anthropic`. + +`ai_max_tokens` must be an integer; the remaining keys must be strings. An +`ai_timeout` value that is not a valid Go duration is a hard error rather than a +silent fallback to the default. + +Lowering `ai_max_tokens` much below the default is not recommended. The response +has to carry a short message, a long-form explanation, and any citations, so an +aggressive cap can truncate the answer and cause the step to fall back to manual +review. + +### Where To Put The Keys + +AI settings may be declared at the top level of the config file, in which case +every service inherits them, or inside a single service's `vars` block. For a +given key, the scanner resolves the first of these that is present: + +1. `services..vars.` +2. top-level `vars.` +3. a top-level `` entry +4. the corresponding `PVTR_*` environment variable + +A per-service value therefore overrides an inherited one, which is how a single +service can use a different model or endpoint from the rest. See +[Running Only Some Services With AI](#running-only-some-services-with-ai) before +trying to use an override to disable AI for one service. + +### Keeping The Credential Out Of The Config File + +Prefer supplying the API key through `PVTR_AI_API_KEY`, or from whatever secret +store your CI already uses, rather than writing it into `config.yml`: + +```sh +export PVTR_AI_API_KEY='' +./pvtr run --binaries-path . +``` + +## Provider Examples + +### OpenAI + +```yaml +ai_provider: openai +ai_model: gpt-4o-mini +# Supply the credential via PVTR_AI_API_KEY rather than this file. + +services: + my-scan: + plugin: github-repo + vars: + owner: + repo: + token: +``` + +### Anthropic + +```yaml +ai_provider: anthropic +ai_model: # check the provider's current model list +# Supply the credential via PVTR_AI_API_KEY rather than this file. + +services: + my-scan: + plugin: github-repo + vars: + owner: + repo: + token: +``` + +### A Gateway Or Self-Hosted Endpoint + +Use `ai_base_url` when calls must travel through a corporate gateway, an +observability proxy, or a self-hosted deployment that speaks the selected +provider's wire protocol. Keep `ai_provider` set to the protocol the endpoint +implements. + +```yaml +ai_provider: openai +ai_model: +ai_base_url: https://ai-gateway.internal.example.com/v1 +``` + +### Per-Service Overrides + +A service's own `vars` win over anything inherited, so one service can use a +different model from the rest: + +```yaml +ai_provider: openai +ai_model: gpt-4o-mini + +services: + routine-scan: + plugin: github-repo + vars: + owner: + repo: + token: + + careful-scan: + plugin: github-repo + vars: + owner: + repo: + token: + ai_model: +``` + +### Running Only Some Services With AI + +If only a subset of your services should use AI, configure AI **inside those +services** rather than at the top level. Services that say nothing about AI then +run with it disabled. + +Opting a single service out of inherited settings is possible but easy to get +wrong. An explicitly empty value does override an inherited one, but AI counts +as "not configured" only when *every* `ai_*` key resolves to empty. Blanking +`ai_provider` and `ai_model` while an `ai_api_key` is still reachable — from a +top-level entry or from `PVTR_AI_API_KEY` — leaves the service configured but +invalid, and its AI-assisted requirements report `Needs Review`. Scoping AI to +the services that need it avoids the problem entirely. + +## Validating Your Configuration Without Provider Spend + +The scanner has no dry-run mode. AI dry-run existed briefly in the SDK +([privateer-sdk#227](https://github.com/privateerproj/privateer-sdk/pull/227)) +and was removed when the AI packages were restructured in +[privateer-sdk#252](https://github.com/privateerproj/privateer-sdk/pull/252). +This scanner never implemented dry-run itself, so there is nothing to enable on +the current SDK. + +You can still check most of a configuration cheaply: + +- **Typos cost nothing.** The provider name, model, and credential are + validated locally before any network call is made. An unsupported + `ai_provider`, an empty `ai_model`, an `ai_api_key` left unset while other AI + keys are present, or an unparseable `ai_timeout` all fail without contacting a + provider. +- **Watch the logs.** Abandoned AI assessments are logged at `warn` level with + the requirement id and the reason, so keep `loglevel` at `info` or lower (the + default in `example-config.yml`) — at `error` these warnings are suppressed. + A run that produces no such warning and still reports manual review for the + requirements below means AI was never picked up at all, which usually points + at the keys being in the wrong place. +- **Point at a local endpoint.** Set `ai_base_url` to a local + OpenAI-compatible mock server to exercise the full AI code path, including + evidence gathering and response validation, without any provider usage. +- **Scope the run.** Use `--service=` to run a single service + while you are iterating on the configuration. + +## Reading `[AI-Assisted]` Results + +An assessment answered with AI help carries an `[AI-Assisted]` prefix on its +message: + +```text +[AI-Assisted] CONTRIBUTING.md tells contributors to run `go test ./...` before opening a pull request. +``` + +When the model returns a verdict but no usable message, the prefix is followed +by the verdict itself: + +```text +[AI-Assisted] verdict: needs_review (medium confidence) +``` + +The message is always a single line and is capped at 160 characters, so it reads +like every other assessment message. The model's longer reasoning is not +discarded; it is kept in the evidence record described below. + +### Verdicts + +The model answers with `pass`, `fail`, or `needs_review`, which map to `Passed`, +`Failed`, and `Needs Review`. Anything else the model might return — an +unexpected value, a missing field, a malformed payload — maps to `Needs Review`. + +An AI-assisted check therefore **never silently passes a requirement**. The +worst case is that a human is asked to look at it. + +### Confidence + +Confidence is a `low`, `medium`, or `high` enum reported by the model, not a +numeric score. Treat it as a triage aid: a `Passed` result at `low` confidence +deserves a spot check before you rely on it. + +For `OSPS-AC-04.02` the scanner applies an extra guard of its own. Unless the +model returns a definite verdict at `high` confidence, the result is recorded as +`Needs Review` at `low` confidence, with the model's summary appended to the +deterministic finding. + +### When AI Cannot Answer + +Every failure path degrades to `Needs Review` at `low` confidence, logs a +warning naming the requirement, records no AI evidence, and lets the scan +continue. This covers: + +- AI is configured incorrectly and the client cannot be built. +- The repository content needed for the question could not be retrieved. +- The content exceeds the size limits described below. +- The provider returned an error, timed out, rate-limited the request, or + rejected the credential. +- The response did not conform to the expected verdict schema. + +A failed AI call never turns into a `Failed` requirement. + +## Evidence And Auditing + +When the model answers, the scanner records a `gemara` evidence entry of type +`ai-assessment` in the normal evaluation log. There is no separate evidence file +or directory to collect. + +The entry contains: + +- the verdict, confidence, short message, long-form explanation, and any + citations the model supplied; +- the exact prompt and the exact material the model was shown; +- provenance: the provider, the model actually used, and the provider's request + identifier; +- a description naming the files the assessment was based on, as a permalink to + each file at the scanned commit where one can be constructed, and otherwise as + a repository-absolute path such as `/README.md`. + +That is enough for a reviewer to judge, reproduce, or dispute the answer without +access to provider-side logs. + +> **Do not point AI-assisted checks at content you would not publish.** The +> prompt and the material are written verbatim into the results file, and no +> redaction is performed on them. Anything that should not appear in a results +> file should not be sent to an AI provider in the first place. + +## Which Checks Use AI + + + +| Requirement | Question asked | When AI is consulted | +| --- | --- | --- | +| `OSPS-QA-06.02` | Does project documentation explain when and how tests are run? | Whenever AI is configured. Without AI the requirement reports `Needs Review`. | +| `OSPS-QA-06.03` | Does project documentation state a policy for maintaining tests? | Whenever AI is configured. Without AI the requirement reports `Needs Review`. | +| `OSPS-AC-04.02` | Are the permissions a CI/CD job grants itself the minimum it needs? | Only when the deterministic check is inconclusive. | + + + +`OSPS-AC-04.02` is evaluated deterministically first. A `write-all` grant fails +outright, permissions set to `none` or left empty pass, and a workflow with no +explicit `permissions:` block is not applicable — none of those consult a model. +AI is only asked about the remaining case, where a job holds a specific grant +whose necessity depends on what the job actually does. + +### Size Limits + +The scanner bounds what it will send: + +- README and CONTRIBUTING material for the `OSPS-QA-06` requirements is capped + at 64 KiB combined. +- Workflow material for `OSPS-AC-04.02` is capped at 50 workflow files and + 64 KiB. Both caps apply only to the workflows that actually need semantic + review, not to every workflow in the repository, so a repository with many + workflows can still be assessed as long as few of them are ambiguous. + +Exceeding a limit defers to manual review instead of truncating the input. +Truncation could drop the very passage the verdict depends on and produce a +confidently wrong answer, so the scanner refuses to guess. + +## Cost And Operational Notes + +- At most one provider call is made per applicable requirement per scan. With + the checks listed above, a single scan makes at most three calls, and fewer + when a deterministic check already answered. +- Request size is bounded by the caps above, and response size by + `ai_max_tokens`. +- Only the specific files a question needs are sent — documentation and + workflow definitions — never the whole repository or its source code. +- Usage charges are yours. A small, inexpensive model is generally sufficient + for these questions. +- If a provider is unreachable or over quota, scans still complete; the affected + requirements report `Needs Review`. +- The repository content is sent to whichever provider you configure. Confirm + that is acceptable for the repositories you scan, particularly for private + ones, before enabling AI. diff --git a/example-config.yml b/example-config.yml index 32eb0c7c..5157fc68 100644 --- a/example-config.yml +++ b/example-config.yml @@ -19,9 +19,24 @@ services: # - Maturity Level 2 # - Maturity Level 3 - # All variables are required to run the evaluation + # The owner, repo, and token variables are required to run the evaluation vars: owner: repo: token: + # AI-assisted checks are opt-in and disabled while these are unset. + # They may also be declared at the top level of this file, in which case + # every service inherits them, or supplied as PVTR_AI_* environment + # variables. See docs/ai-assisted-checks.md. + + # --- Required to enable AI --- + # ai_provider: openai # openai or anthropic + # ai_model: gpt-4o-mini # provider-specific model id + # ai_api_key: # prefer the PVTR_AI_API_KEY env var instead + + # --- Optional (defaults shown) --- + # ai_base_url: https://api.openai.com/v1 # proxy, gateway, or self-hosted + # ai_timeout: 30s # per-call timeout + # ai_max_tokens: 1024 # response length cap + From 61943bab6ad31b28294e310ef21372cd9ec9f132 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Tue, 25 Aug 2026 11:45:13 -0700 Subject: [PATCH 02/13] docs: clarify that provider selection is SDK-driven The scanner has no provider-specific code; it calls ai.NewClient and the SDK picks an adapter from ai_provider. Say so explicitly, and note that each adapter has its own SDK test suite so readers can weigh the examples fairly. Signed-off-by: vinayada1 --- docs/ai-assisted-checks.md | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 56737ab4..d6930c78 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -91,6 +91,12 @@ export PVTR_AI_API_KEY='' ## Provider Examples +The scanner contains no provider-specific code. It asks the SDK for a client and +the SDK selects an adapter from `ai_provider`, so the available providers are +whichever ones the pinned SDK registers — currently `openai` and `anthropic`. +Each adapter is covered by its own test suite in the SDK, so neither is a +second-class path. + ### OpenAI ```yaml From 2c278761ddc02ecb026a1250764e0d3327a67300 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Tue, 25 Aug 2026 12:00:44 -0700 Subject: [PATCH 03/13] docs: simplify wording in AI documentation Replace jargon with plain language throughout: 'deterministic check' becomes 'regular check', 'provenance' becomes 'where the answer came from', 'adapter', 'schema', 'enum' and 'wire protocol' are dropped in favour of ordinary terms. Use 'settings' consistently instead of alternating with 'keys', and describe evidence as landing in the results file rather than alternating between that and 'evaluation log'. Also make two claims exact: the short message is truncated with an ellipsis at 160 characters rather than simply capped, and the retained explanation is bounded at 1500 characters. Signed-off-by: vinayada1 --- README.md | 56 +++---- docs/ai-assisted-checks.md | 317 ++++++++++++++++++------------------- example-config.yml | 15 +- 3 files changed, 192 insertions(+), 196 deletions(-) diff --git a/README.md b/README.md index 29daeeec..9e88e190 100644 --- a/README.md +++ b/README.md @@ -30,9 +30,9 @@ You may have to adjust the plugin name in the config.yaml file to match them. ## AI-Assisted Checks A few OSPS Baseline requirements ask whether a project *documents* something, or -whether a setting is *appropriate for what it is used for*. Those questions -cannot be answered by pattern matching, so the scanner can optionally ask a -large language model and record its answer as evidence. +whether a setting is *appropriate for what it is used for*. Searching for +keywords cannot answer those questions, so the scanner can optionally ask an AI +model and record its answer as evidence. AI is **opt-in**. With no `ai_*` settings the scanner behaves exactly as it always has, contacts no provider, and sends nothing anywhere. @@ -41,24 +41,24 @@ always has, contacts no provider, and sends nothing anywhere. -| Key | Environment variable | Required | Default | Purpose | +| Setting | Environment variable | Required | Default | Purpose | | --- | --- | :---: | --- | --- | -| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Backend adapter: `openai` or `anthropic`. | -| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Provider-specific model identifier. | -| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Provider credential. | -| `ai_base_url` | `PVTR_AI_BASE_URL` | no | adapter default | Alternate endpoint: proxy, gateway, or self-hosted deployment. | -| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Per-call timeout, as a Go duration string. | -| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Cap on the model's response length. | +| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Which AI service to use: `openai` or `anthropic`. | +| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name, spelled the way your provider spells it. | +| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Your API key for that provider. | +| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | A different endpoint, such as a proxy or gateway. | +| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | How long to wait for one answer, for example `30s` or `2m`. | +| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Longest answer to allow back from the model. | -Declare the keys at the top level to have every service inherit them, or inside -a single service's `vars` block to scope them to that service: +Put the settings at the top level so every service picks them up, or inside a +single service's `vars` block to apply them to just that service: ```yaml ai_provider: openai ai_model: gpt-4o-mini -# Supply the credential via PVTR_AI_API_KEY rather than writing it here. +# Pass the API key via PVTR_AI_API_KEY rather than writing it here. services: my-scan: @@ -77,27 +77,27 @@ A requirement answered with AI help is prefixed with `[AI-Assisted]`: [AI-Assisted] CONTRIBUTING.md tells contributors to run `go test ./...` before opening a pull request. ``` -The model's verdict maps to `Passed`, `Failed`, or `Needs Review`, with a -`low` / `medium` / `high` confidence. Anything unexpected — a malformed -response, a provider error, a timeout, a missing credential — becomes -`Needs Review` at low confidence and the scan continues. **An AI-assisted check -never silently passes a requirement.** +The model answers `Passed`, `Failed`, or `Needs Review`, with a confidence of +`low`, `medium`, or `high`. Anything unexpected — a malformed answer, a provider +error, a timeout, a missing API key — becomes `Needs Review` at low confidence +and the scan continues. **An AI-assisted check never silently passes a +requirement.** -The model's full reasoning, the exact prompt, the material it was shown, and the -model used are all recorded as evidence in the evaluation log, so a reviewer can -audit or dispute the answer. +The model's full reasoning, the exact question it was asked, the content it was +shown, and the model used are all saved as evidence in the results file, so a +reviewer can check or dispute the answer. Three requirements use AI today: `OSPS-QA-06.02`, `OSPS-QA-06.03`, and -`OSPS-AC-04.02`. At most one provider call is made per applicable requirement -per scan, and only the specific documentation or workflow files a question needs -are sent — never the whole repository. +`OSPS-AC-04.02`. Each one makes at most one call to the provider per scan, and +only the specific documentation or workflow files a question needs are sent — +never the whole repository. Note that repository content is sent to whichever provider you configure, and -that provider usage is billed to you. +that provider charges you for the calls. -For provider examples, configuration precedence, how to validate a -configuration without provider spend, size limits, and the full evidence -format, see [docs/ai-assisted-checks.md](docs/ai-assisted-checks.md). +For provider examples, where to put the settings, how to check a configuration +without paying for provider calls, size limits, and the full evidence format, +see [docs/ai-assisted-checks.md](docs/ai-assisted-checks.md). ## Docker Usage diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index d6930c78..278136fd 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -1,14 +1,13 @@ # AI-Assisted Checks -This guide explains how to enable AI-assisted assessments in the Privateer -GitHub repository scanner, how to configure a provider, and how to read the -results. +This guide explains how to turn on AI-assisted checks in the Privateer GitHub +repository scanner, how to set up a provider, and how to read the results. -The behavior described here reflects +What is described here matches [`privateer-sdk`](https://github.com/privateerproj/privateer-sdk) **v1.32.2**, the version this scanner currently builds against. The SDK owns the AI client, -the verdict schema, and the `ai_*` configuration keys, so upgrading the SDK can -change this surface. +the answer format, and the `ai_*` settings, so a later SDK version can change +any of this. For a short introduction, see the [AI-Assisted Checks](../README.md#ai-assisted-checks) section of the README. @@ -16,72 +15,70 @@ For a short introduction, see the ## What AI Assistance Does A handful of OSPS Baseline requirements ask whether a project *documents* -something, or whether a configuration is *appropriate for what a job actually -does*. These questions cannot be answered by pattern matching alone. For those -requirements, the scanner can send a bounded slice of repository content to a -large language model and ask for a structured verdict. +something, or whether a setting is *appropriate for what a job actually does*. +Searching for keywords cannot answer those questions. For those requirements, +the scanner can send a limited amount of repository content to an AI model and +ask it to answer in a fixed format. -AI never replaces a deterministic check. It is only consulted where a -deterministic check does not exist or cannot reach a conclusion, and its answer -is recorded as evidence alongside every other observation in the evaluation log. +AI never replaces a regular check. It is only used where a regular check does +not exist or cannot reach a conclusion, and its answer is saved as evidence +alongside every other observation the scan records. ## AI Is Opt-In -When none of the `ai_*` keys are set, AI is off. The scanner makes no requests -to any provider, and every AI-capable step keeps the same non-AI verdict it -produced before AI support existed. +When none of the `ai_*` settings are set, AI is off. The scanner makes no +requests to any provider, and every AI-capable check gives the same answer it +gave before AI support existed. -Setting *any* `ai_*` key while a required one is missing is treated as a -misconfiguration rather than as "disabled" — including setting only an optional -key such as `ai_base_url`. The affected step reports `Needs Review` and logs a -warning; the scan itself still completes. +Setting *any* `ai_*` setting while a required one is missing counts as a mistake +in the configuration rather than as "turned off" — including setting only an +optional one such as `ai_base_url`. The affected check reports `Needs Review` +and logs a warning; the scan itself still finishes. -## Configuration Keys +## Configuration Settings -| Key | Environment variable | Required | Default | Purpose | +| Setting | Environment variable | Required | Default | Purpose | | --- | --- | :---: | --- | --- | -| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Backend adapter: `openai` or `anthropic`. | -| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Provider-specific model identifier. | -| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Provider credential. | -| `ai_base_url` | `PVTR_AI_BASE_URL` | no | adapter default | Alternate endpoint: a proxy, gateway, or self-hosted deployment. | -| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Per-call timeout, as a Go duration string. | -| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Cap on the length of the model's response. | +| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Which AI service to use: `openai` or `anthropic`. | +| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name, spelled the way your provider spells it. | +| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Your API key for that provider. | +| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | A different endpoint: a proxy, a gateway, or a deployment you host yourself. | +| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | How long to wait for one answer, for example `30s` or `2m`. | +| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Longest answer to allow back from the model. | -The default endpoints are `https://api.openai.com/v1` for `openai` and -`https://api.anthropic.com/v1` for `anthropic`. +If you do not set `ai_base_url`, the scanner uses `https://api.openai.com/v1` +for `openai` and `https://api.anthropic.com/v1` for `anthropic`. -`ai_max_tokens` must be an integer; the remaining keys must be strings. An -`ai_timeout` value that is not a valid Go duration is a hard error rather than a -silent fallback to the default. +`ai_max_tokens` must be a whole number; the rest must be text. An `ai_timeout` +the scanner cannot read stops the run instead of quietly using the default. -Lowering `ai_max_tokens` much below the default is not recommended. The response -has to carry a short message, a long-form explanation, and any citations, so an -aggressive cap can truncate the answer and cause the step to fall back to manual -review. +Avoid setting `ai_max_tokens` much lower than the default. The answer has to +carry a short message, a longer explanation, and any citations, so a tight limit +can cut the answer off and send the check to manual review. -### Where To Put The Keys +### Where To Put The Settings -AI settings may be declared at the top level of the config file, in which case -every service inherits them, or inside a single service's `vars` block. For a -given key, the scanner resolves the first of these that is present: +You can put AI settings at the top level of the config file, where every service +picks them up, or inside one service's `vars` block. For each setting, the +scanner uses the first of these it finds: -1. `services..vars.` -2. top-level `vars.` -3. a top-level `` entry -4. the corresponding `PVTR_*` environment variable +1. `services..vars.` +2. top-level `vars.` +3. a top-level `` entry +4. the matching `PVTR_*` environment variable -A per-service value therefore overrides an inherited one, which is how a single +So a value set on a service beats one set at the top level, which is how one service can use a different model or endpoint from the rest. See [Running Only Some Services With AI](#running-only-some-services-with-ai) before -trying to use an override to disable AI for one service. +trying to use this to turn AI off for one service. -### Keeping The Credential Out Of The Config File +### Keeping The API Key Out Of The Config File -Prefer supplying the API key through `PVTR_AI_API_KEY`, or from whatever secret +Prefer passing the API key through `PVTR_AI_API_KEY`, or from whatever secret store your CI already uses, rather than writing it into `config.yml`: ```sh @@ -91,18 +88,17 @@ export PVTR_AI_API_KEY='' ## Provider Examples -The scanner contains no provider-specific code. It asks the SDK for a client and -the SDK selects an adapter from `ai_provider`, so the available providers are -whichever ones the pinned SDK registers — currently `openai` and `anthropic`. -Each adapter is covered by its own test suite in the SDK, so neither is a -second-class path. +The scanner has no code specific to any one provider. It asks the SDK for a +client and the SDK picks the provider from `ai_provider`, so the choices are +whichever ones the pinned SDK supports — currently `openai` and `anthropic`. +Each one has its own test suite in the SDK, so neither is an afterthought. ### OpenAI ```yaml ai_provider: openai ai_model: gpt-4o-mini -# Supply the credential via PVTR_AI_API_KEY rather than this file. +# Pass the API key via PVTR_AI_API_KEY rather than writing it here. services: my-scan: @@ -118,7 +114,7 @@ services: ```yaml ai_provider: anthropic ai_model: # check the provider's current model list -# Supply the credential via PVTR_AI_API_KEY rather than this file. +# Pass the API key via PVTR_AI_API_KEY rather than writing it here. services: my-scan: @@ -131,10 +127,10 @@ services: ### A Gateway Or Self-Hosted Endpoint -Use `ai_base_url` when calls must travel through a corporate gateway, an -observability proxy, or a self-hosted deployment that speaks the selected -provider's wire protocol. Keep `ai_provider` set to the protocol the endpoint -implements. +Use `ai_base_url` when calls have to go through a company gateway, a monitoring +proxy, or a deployment you host yourself that accepts the same requests as the +provider you picked. Leave `ai_provider` set to whichever provider's request +format the endpoint accepts. ```yaml ai_provider: openai @@ -144,8 +140,8 @@ ai_base_url: https://ai-gateway.internal.example.com/v1 ### Per-Service Overrides -A service's own `vars` win over anything inherited, so one service can use a -different model from the rest: +A setting on a service beats anything it would otherwise pick up from the top +level, so one service can use a different model from the rest: ```yaml ai_provider: openai @@ -170,45 +166,45 @@ services: ### Running Only Some Services With AI -If only a subset of your services should use AI, configure AI **inside those +If only some of your services should use AI, put the AI settings **inside those services** rather than at the top level. Services that say nothing about AI then -run with it disabled. +run with it off. -Opting a single service out of inherited settings is possible but easy to get -wrong. An explicitly empty value does override an inherited one, but AI counts -as "not configured" only when *every* `ai_*` key resolves to empty. Blanking -`ai_provider` and `ai_model` while an `ai_api_key` is still reachable — from a -top-level entry or from `PVTR_AI_API_KEY` — leaves the service configured but -invalid, and its AI-assisted requirements report `Needs Review`. Scoping AI to -the services that need it avoids the problem entirely. +Turning AI off for one service that would otherwise pick up top-level settings +is possible but easy to get wrong. Setting a value to empty does override the +top-level one, but AI counts as "off" only when *every* `ai_*` setting ends up +empty. Blanking `ai_provider` and `ai_model` while an `ai_api_key` is still +reachable — from a top-level entry or from `PVTR_AI_API_KEY` — leaves the +service switched on but broken, and its AI-assisted requirements report +`Needs Review`. Setting AI only on the services that need it avoids this. -## Validating Your Configuration Without Provider Spend +## Checking Your Configuration Without Paying For Calls The scanner has no dry-run mode. AI dry-run existed briefly in the SDK ([privateer-sdk#227](https://github.com/privateerproj/privateer-sdk/pull/227)) -and was removed when the AI packages were restructured in +and was removed when the AI packages were reorganized in [privateer-sdk#252](https://github.com/privateerproj/privateer-sdk/pull/252). -This scanner never implemented dry-run itself, so there is nothing to enable on +This scanner never had a dry-run of its own, so there is nothing to turn on with the current SDK. You can still check most of a configuration cheaply: -- **Typos cost nothing.** The provider name, model, and credential are - validated locally before any network call is made. An unsupported - `ai_provider`, an empty `ai_model`, an `ai_api_key` left unset while other AI - keys are present, or an unparseable `ai_timeout` all fail without contacting a - provider. +- **Typos cost nothing.** The provider name, model, and API key are checked on + your machine before any network call happens. A provider name that is not + supported, an empty `ai_model`, an `ai_api_key` left unset while other AI + settings are present, or an `ai_timeout` the scanner cannot read all fail + without contacting a provider. - **Watch the logs.** Abandoned AI assessments are logged at `warn` level with the requirement id and the reason, so keep `loglevel` at `info` or lower (the - default in `example-config.yml`) — at `error` these warnings are suppressed. - A run that produces no such warning and still reports manual review for the - requirements below means AI was never picked up at all, which usually points - at the keys being in the wrong place. -- **Point at a local endpoint.** Set `ai_base_url` to a local - OpenAI-compatible mock server to exercise the full AI code path, including - evidence gathering and response validation, without any provider usage. -- **Scope the run.** Use `--service=` to run a single service - while you are iterating on the configuration. + default in `example-config.yml`) — at `error` these warnings are hidden. A run + that logs no such warning and still asks for manual review on the requirements + below means AI was never picked up at all, which usually means the settings + are in the wrong place. +- **Point at a local endpoint.** Set `ai_base_url` to a local mock server that + accepts OpenAI-style requests to exercise the whole AI path, including + gathering content and checking the answer, without using your provider account. +- **Run one service.** Use `--service=` to run a single service + while you work on the configuration. ## Reading `[AI-Assisted]` Results @@ -219,75 +215,76 @@ message: [AI-Assisted] CONTRIBUTING.md tells contributors to run `go test ./...` before opening a pull request. ``` -When the model returns a verdict but no usable message, the prefix is followed -by the verdict itself: +When the model gives an answer but no usable message, the prefix is followed by +the answer itself, labelled `verdict`: ```text [AI-Assisted] verdict: needs_review (medium confidence) ``` -The message is always a single line and is capped at 160 characters, so it reads -like every other assessment message. The model's longer reasoning is not -discarded; it is kept in the evidence record described below. +The message is always a single line, and anything past 160 characters is cut +with an ellipsis, so it reads like every other result message. The model's +longer reasoning is not thrown away; it is kept in the evidence described below +(up to 1500 characters). -### Verdicts +### Answers -The model answers with `pass`, `fail`, or `needs_review`, which map to `Passed`, -`Failed`, and `Needs Review`. Anything else the model might return — an -unexpected value, a missing field, a malformed payload — maps to `Needs Review`. +The model replies `pass`, `fail`, or `needs_review`, which become `Passed`, +`Failed`, and `Needs Review`. Anything else the model might send back — an +unexpected value, a missing field, a garbled reply — becomes `Needs Review`. -An AI-assisted check therefore **never silently passes a requirement**. The -worst case is that a human is asked to look at it. +So an AI-assisted check **never silently passes a requirement**. The worst case +is that a person is asked to look at it. ### Confidence -Confidence is a `low`, `medium`, or `high` enum reported by the model, not a -numeric score. Treat it as a triage aid: a `Passed` result at `low` confidence -deserves a spot check before you rely on it. +The model reports its confidence as `low`, `medium`, or `high` — not as a +number. Treat it as a hint about how much to trust the answer: a `Passed` result +at `low` confidence is worth a quick look before you rely on it. -For `OSPS-AC-04.02` the scanner applies an extra guard of its own. Unless the -model returns a definite verdict at `high` confidence, the result is recorded as -`Needs Review` at `low` confidence, with the model's summary appended to the -deterministic finding. +For `OSPS-AC-04.02` the scanner adds a check of its own. Unless the model gives +a clear answer at `high` confidence, the result is recorded as `Needs Review` at +`low` confidence, with the model's summary added to what the regular check +found. ### When AI Cannot Answer -Every failure path degrades to `Needs Review` at `low` confidence, logs a -warning naming the requirement, records no AI evidence, and lets the scan -continue. This covers: +Every failure ends the same way: `Needs Review` at `low` confidence, a warning +in the log naming the requirement, no AI evidence saved, and the scan carries +on. This covers: -- AI is configured incorrectly and the client cannot be built. -- The repository content needed for the question could not be retrieved. -- The content exceeds the size limits described below. +- AI settings are wrong and the client cannot be built. +- The repository content needed for the question could not be fetched. +- The content is larger than the limits described below. - The provider returned an error, timed out, rate-limited the request, or - rejected the credential. -- The response did not conform to the expected verdict schema. + rejected the API key. +- The answer did not match the expected format. A failed AI call never turns into a `Failed` requirement. ## Evidence And Auditing -When the model answers, the scanner records a `gemara` evidence entry of type -`ai-assessment` in the normal evaluation log. There is no separate evidence file -or directory to collect. +When the model answers, the scanner saves an entry of type `ai-assessment` in +the results file the scan already writes. There is no extra evidence file or +directory to collect. -The entry contains: +The entry holds: -- the verdict, confidence, short message, long-form explanation, and any - citations the model supplied; -- the exact prompt and the exact material the model was shown; -- provenance: the provider, the model actually used, and the provider's request - identifier; -- a description naming the files the assessment was based on, as a permalink to - each file at the scanned commit where one can be constructed, and otherwise as - a repository-absolute path such as `/README.md`. +- the answer, the confidence, the short message, the longer explanation, and any + citations the model gave; +- the exact question the model was asked and the exact content it was shown; +- where the answer came from: the provider, the model actually used, and the + provider's request id; +- a description naming the files the answer was based on — as a permanent link + to each file at the scanned commit where one can be built, and otherwise as a + path from the repository root such as `/README.md`. -That is enough for a reviewer to judge, reproduce, or dispute the answer without -access to provider-side logs. +That is enough for a reviewer to judge, repeat, or dispute the answer without +access to the provider's own logs. > **Do not point AI-assisted checks at content you would not publish.** The -> prompt and the material are written verbatim into the results file, and no -> redaction is performed on them. Anything that should not appear in a results +> question and the content are written word for word into the results file, and +> nothing is removed or masked. Anything that should not appear in a results > file should not be sent to an AI provider in the first place. ## Which Checks Use AI @@ -296,46 +293,46 @@ access to provider-side logs. | Requirement | Question asked | When AI is consulted | | --- | --- | --- | -| `OSPS-QA-06.02` | Does project documentation explain when and how tests are run? | Whenever AI is configured. Without AI the requirement reports `Needs Review`. | -| `OSPS-QA-06.03` | Does project documentation state a policy for maintaining tests? | Whenever AI is configured. Without AI the requirement reports `Needs Review`. | -| `OSPS-AC-04.02` | Are the permissions a CI/CD job grants itself the minimum it needs? | Only when the deterministic check is inconclusive. | +| `OSPS-QA-06.02` | Does project documentation explain when and how tests are run? | Whenever AI is set up. Without AI the requirement reports `Needs Review`. | +| `OSPS-QA-06.03` | Does project documentation state a policy for maintaining tests? | Whenever AI is set up. Without AI the requirement reports `Needs Review`. | +| `OSPS-AC-04.02` | Are the permissions a CI/CD job grants itself the minimum it needs? | Only when the regular check cannot decide. | -`OSPS-AC-04.02` is evaluated deterministically first. A `write-all` grant fails +`OSPS-AC-04.02` is checked by the regular rules first. A `write-all` grant fails outright, permissions set to `none` or left empty pass, and a workflow with no -explicit `permissions:` block is not applicable — none of those consult a model. -AI is only asked about the remaining case, where a job holds a specific grant -whose necessity depends on what the job actually does. +`permissions:` block at all is not applicable — none of those involve a model. +AI is only asked about what is left: a job holding a specific grant where +whether it is needed depends on what the job actually does. ### Size Limits -The scanner bounds what it will send: +The scanner limits what it will send: -- README and CONTRIBUTING material for the `OSPS-QA-06` requirements is capped - at 64 KiB combined. -- Workflow material for `OSPS-AC-04.02` is capped at 50 workflow files and - 64 KiB. Both caps apply only to the workflows that actually need semantic - review, not to every workflow in the repository, so a repository with many - workflows can still be assessed as long as few of them are ambiguous. +- README and CONTRIBUTING content for the `OSPS-QA-06` requirements is limited + to 64 KiB in total. +- Workflow content for `OSPS-AC-04.02` is limited to 50 workflow files and + 64 KiB. Both limits count only the workflows that actually need a judgment + call, not every workflow in the repository, so a repository with many + workflows can still be checked as long as few of them are unclear. -Exceeding a limit defers to manual review instead of truncating the input. -Truncation could drop the very passage the verdict depends on and produce a -confidently wrong answer, so the scanner refuses to guess. +Going over a limit sends the requirement to manual review instead of cutting the +content short. Cutting it short could drop the very passage the answer depends +on and produce a confident but wrong answer, so the scanner does not guess. ## Cost And Operational Notes -- At most one provider call is made per applicable requirement per scan. With - the checks listed above, a single scan makes at most three calls, and fewer - when a deterministic check already answered. -- Request size is bounded by the caps above, and response size by +- Each applicable requirement makes at most one call to the provider per scan. + With the checks listed above, one scan makes at most three calls, and fewer + when a regular check already answered. +- Request size is limited by the caps above, and answer size by `ai_max_tokens`. -- Only the specific files a question needs are sent — documentation and - workflow definitions — never the whole repository or its source code. -- Usage charges are yours. A small, inexpensive model is generally sufficient - for these questions. -- If a provider is unreachable or over quota, scans still complete; the affected - requirements report `Needs Review`. -- The repository content is sent to whichever provider you configure. Confirm - that is acceptable for the repositories you scan, particularly for private - ones, before enabling AI. +- Only the specific files a question needs are sent — documentation and workflow + files — never the whole repository or its source code. +- The provider charges you for these calls. A small, inexpensive model is + usually good enough for these questions. +- If a provider is unreachable or you are over quota, scans still finish; the + affected requirements report `Needs Review`. +- Repository content is sent to whichever provider you configure. Confirm that + is acceptable for the repositories you scan, especially private ones, before + turning AI on. diff --git a/example-config.yml b/example-config.yml index 5157fc68..6b00fb0d 100644 --- a/example-config.yml +++ b/example-config.yml @@ -25,18 +25,17 @@ services: repo: token: - # AI-assisted checks are opt-in and disabled while these are unset. - # They may also be declared at the top level of this file, in which case - # every service inherits them, or supplied as PVTR_AI_* environment - # variables. See docs/ai-assisted-checks.md. + # AI-assisted checks are off unless these are set. They can also go at the + # top level of this file, where every service picks them up, or be passed + # as PVTR_AI_* environment variables. See docs/ai-assisted-checks.md. - # --- Required to enable AI --- + # --- Required to turn AI on --- # ai_provider: openai # openai or anthropic - # ai_model: gpt-4o-mini # provider-specific model id + # ai_model: gpt-4o-mini # model name, as your provider spells it # ai_api_key: # prefer the PVTR_AI_API_KEY env var instead # --- Optional (defaults shown) --- # ai_base_url: https://api.openai.com/v1 # proxy, gateway, or self-hosted - # ai_timeout: 30s # per-call timeout - # ai_max_tokens: 1024 # response length cap + # ai_timeout: 30s # how long to wait for one answer + # ai_max_tokens: 1024 # longest answer to allow From f4299575a5e7080a754997e894d1ac301e8791b7 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Tue, 25 Aug 2026 12:06:55 -0700 Subject: [PATCH 04/13] docs: explain why some requirements need AI and unpin SDK version Two problems with the previous draft. The claim that some requirements 'cannot be answered by pattern matching' was asserted rather than shown. Give concrete examples instead, drawn from the prompts the scanner actually sends: documentation that says 'run the tests' without a command, a 'make test' snippet that never says when tests are expected, and testing guidance aimed at end users rather than contributors -- all of which contain the keyword and none of which satisfy OSPS-QA-06.02. Cover the describe-versus-require distinction behind OSPS-QA-06.03 and the context dependence of a grant like 'contents: write' for OSPS-AC-04.02. Naming the pinned SDK version also dated the document. A scheduled workflow bumps that pin weekly, so the number would be stale almost immediately. Point at go.mod instead, which is the source of truth and needs no doc change. Also generalize the OSPS-AC-04.02 summary: granting every scope at its highest level fails in the same way as the write-all shorthand. Signed-off-by: vinayada1 --- README.md | 17 +++++++--- docs/ai-assisted-checks.md | 63 ++++++++++++++++++++++++++++---------- 2 files changed, 60 insertions(+), 20 deletions(-) diff --git a/README.md b/README.md index 9e88e190..0a7e5db4 100644 --- a/README.md +++ b/README.md @@ -29,10 +29,19 @@ You may have to adjust the plugin name in the config.yaml file to match them. ## AI-Assisted Checks -A few OSPS Baseline requirements ask whether a project *documents* something, or -whether a setting is *appropriate for what it is used for*. Searching for -keywords cannot answer those questions, so the scanner can optionally ask an AI -model and record its answer as evidence. +Most Baseline requirements can be answered by looking for something specific: a +file exists, a setting has a certain value. A few ask whether something is *good +enough* rather than whether it is *present*, and those cannot be answered by +searching for keywords. + +`OSPS-QA-06.02`, for example, asks whether a project documents when and how its +tests are run. A README saying "run the tests before submitting" contains the +word "test" but gives no command; a `make test` snippet explains how but never +says when it is expected. Both mention testing and neither satisfies the +requirement, so the check has to read the documentation rather than search it. + +For requirements like these, the scanner can optionally ask an AI model and +record its answer as evidence. AI is **opt-in**. With no `ai_*` settings the scanner behaves exactly as it always has, contacts no provider, and sends nothing anywhere. diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 278136fd..4d1c362b 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -3,26 +3,55 @@ This guide explains how to turn on AI-assisted checks in the Privateer GitHub repository scanner, how to set up a provider, and how to read the results. -What is described here matches -[`privateer-sdk`](https://github.com/privateerproj/privateer-sdk) **v1.32.2**, -the version this scanner currently builds against. The SDK owns the AI client, -the answer format, and the `ai_*` settings, so a later SDK version can change +The AI client, the answer format, and the `ai_*` settings all come from +[`privateer-sdk`](https://github.com/privateerproj/privateer-sdk). See `go.mod` +for the version this scanner currently builds against; an SDK upgrade can change any of this. For a short introduction, see the [AI-Assisted Checks](../README.md#ai-assisted-checks) section of the README. -## What AI Assistance Does +## Why Some Requirements Need AI -A handful of OSPS Baseline requirements ask whether a project *documents* -something, or whether a setting is *appropriate for what a job actually does*. -Searching for keywords cannot answer those questions. For those requirements, -the scanner can send a limited amount of repository content to an AI model and -ask it to answer in a fixed format. +Most Baseline requirements can be answered by looking for something specific: a +file exists, a setting has a certain value, a release has an attached SBOM. A +few cannot, because they ask whether something is *good enough* rather than +whether it is *present*. + +`OSPS-QA-06.02` is the clearest example. It asks whether a project documents +when and how its tests are run. Searching for the word "test" finds a match in +almost every repository, and finding one proves nothing: + +- A README saying "run the tests before submitting" mentions tests and gives no + command. A contributor still cannot run them. +- A README with `make test` under a heading about releases explains *how* but + never says *when* — is it expected on every change, or only at release time? +- A page explaining how end users report failing tests is about tests, but says + nothing to a contributor. + +Each of these contains the keyword. None of them satisfies the requirement. The +difference is in meaning, not in the presence of a word, so the check has to +read the documentation rather than search it. + +`OSPS-QA-06.03` turns on an even finer distinction: the difference between +*describing* and *requiring*. "The project has an automated test suite" and +"changes to functionality must add or update tests" can use nearly identical +words, but only the second is a policy. That is what the requirement asks for. + +`OSPS-AC-04.02` is ambiguous in a different way. It asks whether a CI/CD job +grants itself no more permission than it needs, and the identical setting can be +correct or wrong depending on the job. `contents: write` is necessary for a job +that publishes a release and excessive for one that only runs a linter. No rule +that inspects the permission alone can separate the two; you have to consider +what the job does with it. + +For requirements like these, the scanner can send the relevant files to an AI +model and ask it to answer in a fixed format. AI never replaces a regular check. It is only used where a regular check does not exist or cannot reach a conclusion, and its answer is saved as evidence -alongside every other observation the scan records. +alongside every other observation the scan records. Where the rules can decide, +they do — see [Which Checks Use AI](#which-checks-use-ai). ## AI Is Opt-In @@ -299,11 +328,13 @@ access to the provider's own logs. -`OSPS-AC-04.02` is checked by the regular rules first. A `write-all` grant fails -outright, permissions set to `none` or left empty pass, and a workflow with no -`permissions:` block at all is not applicable — none of those involve a model. -AI is only asked about what is left: a job holding a specific grant where -whether it is needed depends on what the job actually does. +`OSPS-AC-04.02` is checked by the regular rules first. Granting everything fails +outright — whether written as `write-all` or as every scope listed at its +highest level. Permissions set to `none` or left empty pass, and a workflow with +no `permissions:` block at all is not applicable. None of those involve a model. +AI is only asked about what is left: a job holding a specific grant, such as +`contents: write`, where whether it is needed depends on what the job actually +does. ### Size Limits From a6d01ce19847ecf759e5bce586fe9409f408bd91 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Tue, 25 Aug 2026 13:39:21 -0700 Subject: [PATCH 05/13] docs: shorten the AI rationale and move scope note to its own section The rationale section made the same point three times, once per AI-assisted requirement, and the table below already lists them. Keep the clearest example (OSPS-QA-06.02) and drop the rest, roughly halving the section. The sentence about AI never replacing a regular check describes when AI is used, not why it is needed, so move it to the head of 'Which Checks Use AI' where the rest of that scope is defined. Signed-off-by: vinayada1 --- README.md | 16 ++++++------ docs/ai-assisted-checks.md | 51 +++++++++++--------------------------- 2 files changed, 23 insertions(+), 44 deletions(-) diff --git a/README.md b/README.md index 0a7e5db4..24080492 100644 --- a/README.md +++ b/README.md @@ -31,14 +31,14 @@ You may have to adjust the plugin name in the config.yaml file to match them. Most Baseline requirements can be answered by looking for something specific: a file exists, a setting has a certain value. A few ask whether something is *good -enough* rather than whether it is *present*, and those cannot be answered by -searching for keywords. - -`OSPS-QA-06.02`, for example, asks whether a project documents when and how its -tests are run. A README saying "run the tests before submitting" contains the -word "test" but gives no command; a `make test` snippet explains how but never -says when it is expected. Both mention testing and neither satisfies the -requirement, so the check has to read the documentation rather than search it. +enough* rather than whether it is *present*. + +`OSPS-QA-06.02` asks whether a project documents when and how its tests are run. +Searching for the word "test" matches almost every repository and proves +nothing: "run the tests before submitting" leaves a contributor without a +command, and a `make test` snippet never says whether tests are expected on +every change or only at release time. Both mention testing; neither satisfies +the requirement. For requirements like these, the scanner can optionally ask an AI model and record its answer as evidence. diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 4d1c362b..b7d4361d 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -14,44 +14,19 @@ For a short introduction, see the ## Why Some Requirements Need AI Most Baseline requirements can be answered by looking for something specific: a -file exists, a setting has a certain value, a release has an attached SBOM. A -few cannot, because they ask whether something is *good enough* rather than -whether it is *present*. - -`OSPS-QA-06.02` is the clearest example. It asks whether a project documents -when and how its tests are run. Searching for the word "test" finds a match in -almost every repository, and finding one proves nothing: - -- A README saying "run the tests before submitting" mentions tests and gives no - command. A contributor still cannot run them. -- A README with `make test` under a heading about releases explains *how* but - never says *when* — is it expected on every change, or only at release time? -- A page explaining how end users report failing tests is about tests, but says - nothing to a contributor. - -Each of these contains the keyword. None of them satisfies the requirement. The -difference is in meaning, not in the presence of a word, so the check has to -read the documentation rather than search it. - -`OSPS-QA-06.03` turns on an even finer distinction: the difference between -*describing* and *requiring*. "The project has an automated test suite" and -"changes to functionality must add or update tests" can use nearly identical -words, but only the second is a policy. That is what the requirement asks for. - -`OSPS-AC-04.02` is ambiguous in a different way. It asks whether a CI/CD job -grants itself no more permission than it needs, and the identical setting can be -correct or wrong depending on the job. `contents: write` is necessary for a job -that publishes a release and excessive for one that only runs a linter. No rule -that inspects the permission alone can separate the two; you have to consider -what the job does with it. +file exists, a setting has a certain value, a release has an SBOM attached. A +few ask whether something is *good enough* rather than whether it is *present*. -For requirements like these, the scanner can send the relevant files to an AI -model and ask it to answer in a fixed format. +`OSPS-QA-06.02` asks whether a project documents when and how its tests are run. +Searching for the word "test" matches almost every repository and proves +nothing: "run the tests before submitting" leaves a contributor without a +command, and a `make test` snippet never says whether tests are expected on +every change or only at release time. Both mention testing; neither satisfies +the requirement. The difference is in meaning, not in the presence of a word, so +the check has to read the documentation rather than search it. -AI never replaces a regular check. It is only used where a regular check does -not exist or cannot reach a conclusion, and its answer is saved as evidence -alongside every other observation the scan records. Where the rules can decide, -they do — see [Which Checks Use AI](#which-checks-use-ai). +For requirements like these, the scanner can send the relevant files to an AI +model and ask for an answer in a fixed format. ## AI Is Opt-In @@ -318,6 +293,10 @@ access to the provider's own logs. ## Which Checks Use AI +AI never replaces a regular check. It is used only where a regular check does +not exist or cannot reach a conclusion, and its answer is saved as evidence +alongside every other observation the scan records. + | Requirement | Question asked | When AI is consulted | From be7d2254154a5efa0a569953d675b55eb068f36e Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Tue, 25 Aug 2026 13:46:38 -0700 Subject: [PATCH 06/13] docs: restructure AI guide around examples Replace the abstract settings-precedence list with a worked config example, and merge the four separate sections that all described where settings go into one example-led section. Move the on/off boundary note out of the opt-in section and reframe it as part of turning AI on, and lead the configuration-checking section with what you can do rather than with the history of the removed dry-run flag. Correct the config-versus-environment precedence: the environment overrides a top-level setting, but not one inside a vars block. Signed-off-by: vinayada1 --- docs/ai-assisted-checks.md | 172 ++++++++++++++++++------------------- 1 file changed, 83 insertions(+), 89 deletions(-) diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index b7d4361d..f396ddbe 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -28,16 +28,32 @@ the check has to read the documentation rather than search it. For requirements like these, the scanner can send the relevant files to an AI model and ask for an answer in a fixed format. -## AI Is Opt-In +## Turning AI On -When none of the `ai_*` settings are set, AI is off. The scanner makes no +AI is off until you configure it. With no `ai_*` settings the scanner makes no requests to any provider, and every AI-capable check gives the same answer it gave before AI support existed. -Setting *any* `ai_*` setting while a required one is missing counts as a mistake -in the configuration rather than as "turned off" — including setting only an -optional one such as `ai_base_url`. The affected check reports `Needs Review` -and logs a warning; the scan itself still finishes. +The smallest working configuration is a provider, a model, and an API key: + +```yaml +ai_provider: openai +ai_model: gpt-4o-mini +# Pass the API key via PVTR_AI_API_KEY rather than writing it here. + +services: + my-scan: + plugin: github-repo + vars: + owner: + repo: + token: +``` + +AI counts as on as soon as *any* `ai_*` setting is present. If the required ones +are not all there, the affected checks report `Needs Review` and log a warning +rather than quietly carrying on without AI, so a half-finished configuration is +visible instead of silent. The scan itself still finishes. ## Configuration Settings @@ -64,21 +80,35 @@ Avoid setting `ai_max_tokens` much lower than the default. The answer has to carry a short message, a longer explanation, and any citations, so a tight limit can cut the answer off and send the check to manual review. -### Where To Put The Settings +### Where Settings Go -You can put AI settings at the top level of the config file, where every service -picks them up, or inside one service's `vars` block. For each setting, the -scanner uses the first of these it finds: +Settings at the top level apply to every service. Settings inside a service's +`vars` block apply to that service only, and win over the top-level value: + +```yaml +ai_provider: openai +ai_model: gpt-4o-mini # every service uses this model... + +services: + routine-scan: + plugin: github-repo + vars: + owner: + repo: + token: -1. `services..vars.` -2. top-level `vars.` -3. a top-level `` entry -4. the matching `PVTR_*` environment variable + careful-scan: + plugin: github-repo + vars: + owner: + repo: + token: + ai_model: # ...except this one +``` -So a value set on a service beats one set at the top level, which is how one -service can use a different model or endpoint from the rest. See -[Running Only Some Services With AI](#running-only-some-services-with-ai) before -trying to use this to turn AI off for one service. +Any setting can also come from its `PVTR_AI_*` environment variable, which is +the usual way to pass the API key. The environment overrides a top-level setting +like the `ai_provider:` above, but not one inside a `vars:` block. ### Keeping The API Key Out Of The Config File @@ -90,6 +120,17 @@ export PVTR_AI_API_KEY='' ./pvtr run --binaries-path . ``` +### Running Only Some Services With AI + +Put the AI settings inside the services that need them rather than at the top +level. A service that says nothing about AI runs with it off. + +Going the other way — inheriting top-level settings and then switching them off +for one service — is easy to get wrong. AI counts as off only when *every* +`ai_*` setting ends up empty, so blanking `ai_provider` and `ai_model` while an +`ai_api_key` is still reachable from the top level or the environment leaves the +service switched on but broken, and its AI-assisted checks report `Needs Review`. + ## Provider Examples The scanner has no code specific to any one provider. It asks the SDK for a @@ -97,36 +138,12 @@ client and the SDK picks the provider from `ai_provider`, so the choices are whichever ones the pinned SDK supports — currently `openai` and `anthropic`. Each one has its own test suite in the SDK, so neither is an afterthought. -### OpenAI - -```yaml -ai_provider: openai -ai_model: gpt-4o-mini -# Pass the API key via PVTR_AI_API_KEY rather than writing it here. - -services: - my-scan: - plugin: github-repo - vars: - owner: - repo: - token: -``` - -### Anthropic +The example above uses OpenAI. To use Anthropic instead, change the provider and +the model: ```yaml ai_provider: anthropic ai_model: # check the provider's current model list -# Pass the API key via PVTR_AI_API_KEY rather than writing it here. - -services: - my-scan: - plugin: github-repo - vars: - owner: - repo: - token: ``` ### A Gateway Or Self-Hosted Endpoint @@ -142,56 +159,33 @@ ai_model: ai_base_url: https://ai-gateway.internal.example.com/v1 ``` -### Per-Service Overrides - -A setting on a service beats anything it would otherwise pick up from the top -level, so one service can use a different model from the rest: - -```yaml -ai_provider: openai -ai_model: gpt-4o-mini - -services: - routine-scan: - plugin: github-repo - vars: - owner: - repo: - token: - - careful-scan: - plugin: github-repo - vars: - owner: - repo: - token: - ai_model: -``` - -### Running Only Some Services With AI - -If only some of your services should use AI, put the AI settings **inside those -services** rather than at the top level. Services that say nothing about AI then -run with it off. +## Checking Your Configuration Without Paying For Calls -Turning AI off for one service that would otherwise pick up top-level settings -is possible but easy to get wrong. Setting a value to empty does override the -top-level one, but AI counts as "off" only when *every* `ai_*` setting ends up -empty. Blanking `ai_provider` and `ai_model` while an `ai_api_key` is still -reachable — from a top-level entry or from `PVTR_AI_API_KEY` — leaves the -service switched on but broken, and its AI-assisted requirements report -`Needs Review`. Setting AI only on the services that need it avoids this. +There is no dry-run mode, but you can still check most of a configuration +cheaply: -## Checking Your Configuration Without Paying For Calls +- **Typos cost nothing.** The provider name, model, and API key are checked on + your machine before any network call happens. A provider name that is not + supported, an empty `ai_model`, an `ai_api_key` left unset while other AI + settings are present, or an `ai_timeout` the scanner cannot read all fail + without contacting a provider. +- **Watch the logs.** Abandoned AI assessments are logged at `warn` level with + the requirement id and the reason, so keep `loglevel` at `info` or lower (the + default in `example-config.yml`) — at `error` these warnings are hidden. A run + that logs no such warning and still asks for manual review on the requirements + below means AI was never picked up at all, which usually means the settings + are in the wrong place. +- **Point at a local endpoint.** Set `ai_base_url` to a local mock server that + accepts OpenAI-style requests to exercise the whole AI path, including + gathering content and checking the answer, without using your provider account. +- **Run one service.** Use `--service=` to run a single service + while you work on the configuration. -The scanner has no dry-run mode. AI dry-run existed briefly in the SDK -([privateer-sdk#227](https://github.com/privateerproj/privateer-sdk/pull/227)) +If you are looking for a dry-run flag you saw mentioned elsewhere: it existed +briefly in the SDK ([privateer-sdk#227](https://github.com/privateerproj/privateer-sdk/pull/227)) and was removed when the AI packages were reorganized in [privateer-sdk#252](https://github.com/privateerproj/privateer-sdk/pull/252). -This scanner never had a dry-run of its own, so there is nothing to turn on with -the current SDK. - -You can still check most of a configuration cheaply: +This scanner never had one of its own. - **Typos cost nothing.** The provider name, model, and API key are checked on your machine before any network call happens. A provider name that is not From 67902136938607d9158a3b1b801fd28fcc952927 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Tue, 25 Aug 2026 13:54:50 -0700 Subject: [PATCH 07/13] docs: rewrite AI guide for users Replace the implementation-focused document with a concise task-based guide. Remove dry-run and SDK history, simplify configuration and results guidance, and make the lack of secret redaction prominent. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: vinayada1 --- README.md | 87 ++------- docs/ai-assisted-checks.md | 375 ++++++++++--------------------------- example-config.yml | 10 +- 3 files changed, 117 insertions(+), 355 deletions(-) diff --git a/README.md b/README.md index 24080492..0d15ae7f 100644 --- a/README.md +++ b/README.md @@ -29,84 +29,27 @@ You may have to adjust the plugin name in the config.yaml file to match them. ## AI-Assisted Checks -Most Baseline requirements can be answered by looking for something specific: a -file exists, a setting has a certain value. A few ask whether something is *good -enough* rather than whether it is *present*. - -`OSPS-QA-06.02` asks whether a project documents when and how its tests are run. -Searching for the word "test" matches almost every repository and proves -nothing: "run the tests before submitting" leaves a contributor without a -command, and a `make test` snippet never says whether tests are expected on -every change or only at release time. Both mention testing; neither satisfies -the requirement. - -For requirements like these, the scanner can optionally ask an AI model and -record its answer as evidence. - -AI is **opt-in**. With no `ai_*` settings the scanner behaves exactly as it -always has, contacts no provider, and sends nothing anywhere. - -### Configuration - - - -| Setting | Environment variable | Required | Default | Purpose | -| --- | --- | :---: | --- | --- | -| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Which AI service to use: `openai` or `anthropic`. | -| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name, spelled the way your provider spells it. | -| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Your API key for that provider. | -| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | A different endpoint, such as a proxy or gateway. | -| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | How long to wait for one answer, for example `30s` or `2m`. | -| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Longest answer to allow back from the model. | - - - -Put the settings at the top level so every service picks them up, or inside a -single service's `vars` block to apply them to just that service: - -```yaml -ai_provider: openai -ai_model: gpt-4o-mini -# Pass the API key via PVTR_AI_API_KEY rather than writing it here. - -services: - my-scan: - plugin: github-repo - vars: - owner: - repo: - token: -``` - -### Reading The Results +Some requirements need judgment rather than a simple file or setting check. The +scanner can use an AI model for these requirements and store the assessment as +evidence. AI is disabled by default. -A requirement answered with AI help is prefixed with `[AI-Assisted]`: +Enable AI with a provider, model, and API key: -```text -[AI-Assisted] CONTRIBUTING.md tells contributors to run `go test ./...` before opening a pull request. +```sh +export PVTR_AI_PROVIDER='openai' +export PVTR_AI_MODEL='gpt-4o-mini' +export PVTR_AI_API_KEY='' ``` -The model answers `Passed`, `Failed`, or `Needs Review`, with a confidence of -`low`, `medium`, or `high`. Anything unexpected — a malformed answer, a provider -error, a timeout, a missing API key — becomes `Needs Review` at low confidence -and the scan continues. **An AI-assisted check never silently passes a -requirement.** - -The model's full reasoning, the exact question it was asked, the content it was -shown, and the model used are all saved as evidence in the results file, so a -reviewer can check or dispute the answer. - -Three requirements use AI today: `OSPS-QA-06.02`, `OSPS-QA-06.03`, and -`OSPS-AC-04.02`. Each one makes at most one call to the provider per scan, and -only the specific documentation or workflow files a question needs are sent — -never the whole repository. +The scanner supports OpenAI and Anthropic. AI-assisted results include the +`[AI-Assisted]` prefix and a confidence level. Provider errors and invalid +responses produce `Needs Review`; the scan continues. -Note that repository content is sent to whichever provider you configure, and -that provider charges you for the calls. +> **The scanner does not detect or redact secrets in repository files before +> sending those files to the provider or storing them in AI evidence.** -For provider examples, where to put the settings, how to check a configuration -without paying for provider calls, size limits, and the full evidence format, -see [docs/ai-assisted-checks.md](docs/ai-assisted-checks.md). +See [AI-Assisted Checks](docs/ai-assisted-checks.md) for configuration, +security guidance, supported checks, and result details. ## Docker Usage diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index f396ddbe..d1d6391f 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -1,45 +1,46 @@ # AI-Assisted Checks -This guide explains how to turn on AI-assisted checks in the Privateer GitHub -repository scanner, how to set up a provider, and how to read the results. +AI-assisted checks help evaluate requirements that need judgment rather than a +simple file or setting check. -The AI client, the answer format, and the `ai_*` settings all come from -[`privateer-sdk`](https://github.com/privateerproj/privateer-sdk). See `go.mod` -for the version this scanner currently builds against; an SDK upgrade can change -any of this. +For example, `OSPS-QA-06.02` asks whether a project explains when and how to run +its tests. Finding the word "test" is not enough: the documentation must provide +useful instructions. An AI model can review the relevant documentation and +assess whether it meets the requirement. -For a short introduction, see the -[AI-Assisted Checks](../README.md#ai-assisted-checks) section of the README. +AI is disabled by default. -## Why Some Requirements Need AI +## Enabling AI -Most Baseline requirements can be answered by looking for something specific: a -file exists, a setting has a certain value, a release has an SBOM attached. A -few ask whether something is *good enough* rather than whether it is *present*. +Set an AI provider, model, and API key. The scanner supports `openai` and +`anthropic`. -`OSPS-QA-06.02` asks whether a project documents when and how its tests are run. -Searching for the word "test" matches almost every repository and proves -nothing: "run the tests before submitting" leaves a contributor without a -command, and a `make test` snippet never says whether tests are expected on -every change or only at release time. Both mention testing; neither satisfies -the requirement. The difference is in meaning, not in the presence of a word, so -the check has to read the documentation rather than search it. +The recommended setup uses environment variables so the API key is not stored +in the configuration file: -For requirements like these, the scanner can send the relevant files to an AI -model and ask for an answer in a fixed format. +```sh +export PVTR_AI_PROVIDER='openai' +export PVTR_AI_MODEL='gpt-4o-mini' +export PVTR_AI_API_KEY='' + +./pvtr run --binaries-path . +``` -## Turning AI On +For Anthropic, change the provider and model: -AI is off until you configure it. With no `ai_*` settings the scanner makes no -requests to any provider, and every AI-capable check gives the same answer it -gave before AI support existed. +```sh +export PVTR_AI_PROVIDER='anthropic' +export PVTR_AI_MODEL='' +``` -The smallest working configuration is a provider, a model, and an API key: +Environment variables apply to every service in the run. Use +`--service=` to run only one configured service. + +You can also add non-secret settings to the configuration file: ```yaml ai_provider: openai ai_model: gpt-4o-mini -# Pass the API key via PVTR_AI_API_KEY rather than writing it here. services: my-scan: @@ -50,293 +51,111 @@ services: token: ``` -AI counts as on as soon as *any* `ai_*` setting is present. If the required ones -are not all there, the affected checks report `Needs Review` and log a warning -rather than quietly carrying on without AI, so a half-finished configuration is -visible instead of silent. The scan itself still finishes. - -## Configuration Settings - - - -| Setting | Environment variable | Required | Default | Purpose | -| --- | --- | :---: | --- | --- | -| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | Which AI service to use: `openai` or `anthropic`. | -| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name, spelled the way your provider spells it. | -| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | Your API key for that provider. | -| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | A different endpoint: a proxy, a gateway, or a deployment you host yourself. | -| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | How long to wait for one answer, for example `30s` or `2m`. | -| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Longest answer to allow back from the model. | - - - -If you do not set `ai_base_url`, the scanner uses `https://api.openai.com/v1` -for `openai` and `https://api.anthropic.com/v1` for `anthropic`. - -`ai_max_tokens` must be a whole number; the rest must be text. An `ai_timeout` -the scanner cannot read stops the run instead of quietly using the default. - -Avoid setting `ai_max_tokens` much lower than the default. The answer has to -carry a short message, a longer explanation, and any citations, so a tight limit -can cut the answer off and send the check to manual review. - -### Where Settings Go - -Settings at the top level apply to every service. Settings inside a service's -`vars` block apply to that service only, and win over the top-level value: +Top-level settings apply to every service. To override a setting for one +service, add it to that service's `vars` block: ```yaml -ai_provider: openai -ai_model: gpt-4o-mini # every service uses this model... - services: - routine-scan: - plugin: github-repo - vars: - owner: - repo: - token: - - careful-scan: + my-scan: plugin: github-repo vars: - owner: - repo: - token: - ai_model: # ...except this one -``` - -Any setting can also come from its `PVTR_AI_*` environment variable, which is -the usual way to pass the API key. The environment overrides a top-level setting -like the `ai_provider:` above, but not one inside a `vars:` block. - -### Keeping The API Key Out Of The Config File - -Prefer passing the API key through `PVTR_AI_API_KEY`, or from whatever secret -store your CI already uses, rather than writing it into `config.yml`: - -```sh -export PVTR_AI_API_KEY='' -./pvtr run --binaries-path . + ai_model: ``` -### Running Only Some Services With AI - -Put the AI settings inside the services that need them rather than at the top -level. A service that says nothing about AI runs with it off. - -Going the other way — inheriting top-level settings and then switching them off -for one service — is easy to get wrong. AI counts as off only when *every* -`ai_*` setting ends up empty, so blanking `ai_provider` and `ai_model` while an -`ai_api_key` is still reachable from the top level or the environment leaves the -service switched on but broken, and its AI-assisted checks report `Needs Review`. +## Configuration -## Provider Examples + -The scanner has no code specific to any one provider. It asks the SDK for a -client and the SDK picks the provider from `ai_provider`, so the choices are -whichever ones the pinned SDK supports — currently `openai` and `anthropic`. -Each one has its own test suite in the SDK, so neither is an afterthought. +| Setting | Environment variable | Required | Default | Description | +| --- | --- | :---: | --- | --- | +| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | AI provider: `openai` or `anthropic`. | +| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name from the provider. | +| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | API key for the provider. | +| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | URL for a compatible gateway, proxy, or self-hosted endpoint. | +| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Maximum time to wait for a response. | +| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Maximum response size. | -The example above uses OpenAI. To use Anthropic instead, change the provider and -the model: + -```yaml -ai_provider: anthropic -ai_model: # check the provider's current model list -``` +All three required settings must be present. If one is missing, the affected +checks report `Needs Review`, log a warning, and allow the scan to continue. +`ai_timeout` must use a duration such as `30s` or `2m`; an invalid duration stops +the run. `ai_max_tokens` must be a whole number. -### A Gateway Or Self-Hosted Endpoint +### Custom Endpoints -Use `ai_base_url` when calls have to go through a company gateway, a monitoring -proxy, or a deployment you host yourself that accepts the same requests as the -provider you picked. Leave `ai_provider` set to whichever provider's request -format the endpoint accepts. +Set `ai_base_url` to use a compatible gateway, proxy, or self-hosted endpoint. +The endpoint must accept the request format for the selected provider. ```yaml ai_provider: openai -ai_model: -ai_base_url: https://ai-gateway.internal.example.com/v1 +ai_model: +ai_base_url: https://ai-gateway.example.com/v1 ``` -## Checking Your Configuration Without Paying For Calls - -There is no dry-run mode, but you can still check most of a configuration -cheaply: - -- **Typos cost nothing.** The provider name, model, and API key are checked on - your machine before any network call happens. A provider name that is not - supported, an empty `ai_model`, an `ai_api_key` left unset while other AI - settings are present, or an `ai_timeout` the scanner cannot read all fail - without contacting a provider. -- **Watch the logs.** Abandoned AI assessments are logged at `warn` level with - the requirement id and the reason, so keep `loglevel` at `info` or lower (the - default in `example-config.yml`) — at `error` these warnings are hidden. A run - that logs no such warning and still asks for manual review on the requirements - below means AI was never picked up at all, which usually means the settings - are in the wrong place. -- **Point at a local endpoint.** Set `ai_base_url` to a local mock server that - accepts OpenAI-style requests to exercise the whole AI path, including - gathering content and checking the answer, without using your provider account. -- **Run one service.** Use `--service=` to run a single service - while you work on the configuration. - -If you are looking for a dry-run flag you saw mentioned elsewhere: it existed -briefly in the SDK ([privateer-sdk#227](https://github.com/privateerproj/privateer-sdk/pull/227)) -and was removed when the AI packages were reorganized in -[privateer-sdk#252](https://github.com/privateerproj/privateer-sdk/pull/252). -This scanner never had one of its own. - -- **Typos cost nothing.** The provider name, model, and API key are checked on - your machine before any network call happens. A provider name that is not - supported, an empty `ai_model`, an `ai_api_key` left unset while other AI - settings are present, or an `ai_timeout` the scanner cannot read all fail - without contacting a provider. -- **Watch the logs.** Abandoned AI assessments are logged at `warn` level with - the requirement id and the reason, so keep `loglevel` at `info` or lower (the - default in `example-config.yml`) — at `error` these warnings are hidden. A run - that logs no such warning and still asks for manual review on the requirements - below means AI was never picked up at all, which usually means the settings - are in the wrong place. -- **Point at a local endpoint.** Set `ai_base_url` to a local mock server that - accepts OpenAI-style requests to exercise the whole AI path, including - gathering content and checking the answer, without using your provider account. -- **Run one service.** Use `--service=` to run a single service - while you work on the configuration. - -## Reading `[AI-Assisted]` Results - -An assessment answered with AI help carries an `[AI-Assisted]` prefix on its -message: +## Security And Privacy -```text -[AI-Assisted] CONTRIBUTING.md tells contributors to run `go test ./...` before opening a pull request. -``` +The scanner sends repository content to the configured AI provider. It sends +only the files needed by the check: README, CONTRIBUTING, or relevant GitHub +Actions workflow files. -When the model gives an answer but no usable message, the prefix is followed by -the answer itself, labelled `verdict`: +> **The scanner does not detect or redact secrets in repository files.** Do not +> enable AI for repositories whose relevant files contain secrets or other data +> that must not be sent to the provider. -```text -[AI-Assisted] verdict: needs_review (medium confidence) -``` - -The message is always a single line, and anything past 160 characters is cut -with an ellipsis, so it reads like every other result message. The model's -longer reasoning is not thrown away; it is kept in the evidence described below -(up to 1500 characters). - -### Answers - -The model replies `pass`, `fail`, or `needs_review`, which become `Passed`, -`Failed`, and `Needs Review`. Anything else the model might send back — an -unexpected value, a missing field, a garbled reply — becomes `Needs Review`. - -So an AI-assisted check **never silently passes a requirement**. The worst case -is that a person is asked to look at it. - -### Confidence +The same content is stored without redaction in the scan evidence, together with +the question sent to the model and its response. Protect the results file +accordingly. -The model reports its confidence as `low`, `medium`, or `high` — not as a -number. Treat it as a hint about how much to trust the answer: a `Passed` result -at `low` confidence is worth a quick look before you rely on it. +Keep the provider API key in `PVTR_AI_API_KEY` or your CI secret store. Do not +commit it to the configuration file. -For `OSPS-AC-04.02` the scanner adds a check of its own. Unless the model gives -a clear answer at `high` confidence, the result is recorded as `Needs Review` at -`low` confidence, with the model's summary added to what the regular check -found. +Confirm that your organization permits the selected provider to process the +repository content. -### When AI Cannot Answer +## Reading Results -Every failure ends the same way: `Needs Review` at `low` confidence, a warning -in the log naming the requirement, no AI evidence saved, and the scan carries -on. This covers: +Results produced with AI include the `[AI-Assisted]` prefix: -- AI settings are wrong and the client cannot be built. -- The repository content needed for the question could not be fetched. -- The content is larger than the limits described below. -- The provider returned an error, timed out, rate-limited the request, or - rejected the API key. -- The answer did not match the expected format. - -A failed AI call never turns into a `Failed` requirement. - -## Evidence And Auditing +```text +[AI-Assisted] CONTRIBUTING.md explains how and when contributors should run the tests. +``` -When the model answers, the scanner saves an entry of type `ai-assessment` in -the results file the scan already writes. There is no extra evidence file or -directory to collect. +The result is `Passed`, `Failed`, or `Needs Review`, with `low`, `medium`, or +`high` confidence. Review low-confidence results before relying on them. -The entry holds: +An invalid response, provider error, timeout, missing setting, or oversized +input produces `Needs Review` at low confidence. The scan continues and logs a +warning. An AI failure never becomes a `Failed` requirement. -- the answer, the confidence, the short message, the longer explanation, and any - citations the model gave; -- the exact question the model was asked and the exact content it was shown; -- where the answer came from: the provider, the model actually used, and the - provider's request id; -- a description naming the files the answer was based on — as a permanent link - to each file at the scanned commit where one can be built, and otherwise as a - path from the repository root such as `/README.md`. +The scanner stores successful AI assessments as `ai-assessment` evidence in: -That is enough for a reviewer to judge, repeat, or dispute the answer without -access to the provider's own logs. +```text +//.yaml +``` -> **Do not point AI-assisted checks at content you would not publish.** The -> question and the content are written word for word into the results file, and -> nothing is removed or masked. Anything that should not appear in a results -> file should not be sent to an AI provider in the first place. +The evidence includes the result, confidence, explanation, question, content, +provider, model, request ID, and source file references. -## Which Checks Use AI +## Checks That Use AI -AI never replaces a regular check. It is used only where a regular check does -not exist or cannot reach a conclusion, and its answer is saved as evidence -alongside every other observation the scan records. +AI is used only where a regular check does not exist or cannot answer the +requirement. -| Requirement | Question asked | When AI is consulted | -| --- | --- | --- | -| `OSPS-QA-06.02` | Does project documentation explain when and how tests are run? | Whenever AI is set up. Without AI the requirement reports `Needs Review`. | -| `OSPS-QA-06.03` | Does project documentation state a policy for maintaining tests? | Whenever AI is set up. Without AI the requirement reports `Needs Review`. | -| `OSPS-AC-04.02` | Are the permissions a CI/CD job grants itself the minimum it needs? | Only when the regular check cannot decide. | +| Requirement | AI assessment | +| --- | --- | +| `OSPS-QA-06.02` | Whether project documentation explains when and how tests are run. | +| `OSPS-QA-06.03` | Whether project documentation defines a policy for maintaining tests. | +| `OSPS-AC-04.02` | Whether specific GitHub Actions permissions are required by the job. | -`OSPS-AC-04.02` is checked by the regular rules first. Granting everything fails -outright — whether written as `write-all` or as every scope listed at its -highest level. Permissions set to `none` or left empty pass, and a workflow with -no `permissions:` block at all is not applicable. None of those involve a model. -AI is only asked about what is left: a job holding a specific grant, such as -`contents: write`, where whether it is needed depends on what the job actually -does. - -### Size Limits - -The scanner limits what it will send: - -- README and CONTRIBUTING content for the `OSPS-QA-06` requirements is limited - to 64 KiB in total. -- Workflow content for `OSPS-AC-04.02` is limited to 50 workflow files and - 64 KiB. Both limits count only the workflows that actually need a judgment - call, not every workflow in the repository, so a repository with many - workflows can still be checked as long as few of them are unclear. - -Going over a limit sends the requirement to manual review instead of cutting the -content short. Cutting it short could drop the very passage the answer depends -on and produce a confident but wrong answer, so the scanner does not guess. - -## Cost And Operational Notes - -- Each applicable requirement makes at most one call to the provider per scan. - With the checks listed above, one scan makes at most three calls, and fewer - when a regular check already answered. -- Request size is limited by the caps above, and answer size by - `ai_max_tokens`. -- Only the specific files a question needs are sent — documentation and workflow - files — never the whole repository or its source code. -- The provider charges you for these calls. A small, inexpensive model is - usually good enough for these questions. -- If a provider is unreachable or you are over quota, scans still finish; the - affected requirements report `Needs Review`. -- Repository content is sent to whichever provider you configure. Confirm that - is acceptable for the repositories you scan, especially private ones, before - turning AI on. +`OSPS-QA-06.02` and `OSPS-QA-06.03` report `Needs Review` when AI is disabled. +For `OSPS-AC-04.02`, regular checks handle clear cases first; AI reviews only +permissions that require context. + +Each requirement makes at most one provider request per scan. Documentation sent +for `OSPS-QA-06` is limited to 64 KiB. `OSPS-AC-04.02` is limited to 50 workflow +files and 64 KiB. Inputs over these limits report `Needs Review`. diff --git a/example-config.yml b/example-config.yml index 6b00fb0d..b8d8300e 100644 --- a/example-config.yml +++ b/example-config.yml @@ -25,11 +25,12 @@ services: repo: token: - # AI-assisted checks are off unless these are set. They can also go at the - # top level of this file, where every service picks them up, or be passed - # as PVTR_AI_* environment variables. See docs/ai-assisted-checks.md. + # AI-assisted checks are disabled unless these settings are present. + # Prefer PVTR_AI_* environment variables, especially for the API key. + # Secrets in repository files are not detected or redacted. + # See docs/ai-assisted-checks.md. - # --- Required to turn AI on --- + # --- Required to enable AI --- # ai_provider: openai # openai or anthropic # ai_model: gpt-4o-mini # model name, as your provider spells it # ai_api_key: # prefer the PVTR_AI_API_KEY env var instead @@ -38,4 +39,3 @@ services: # ai_base_url: https://api.openai.com/v1 # proxy, gateway, or self-hosted # ai_timeout: 30s # how long to wait for one answer # ai_max_tokens: 1024 # longest answer to allow - From 7e02d431ad39bc4e6c8c62b96d9bc241f1397fde Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Wed, 26 Aug 2026 10:34:36 -0700 Subject: [PATCH 08/13] docs: clarify optional role of AI State that deterministic checks remain the baseline and that AI is an optional enhancement used only when those checks cannot answer a requirement. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: vinayada1 --- README.md | 5 ++--- docs/ai-assisted-checks.md | 6 ++---- 2 files changed, 4 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 0d15ae7f..2abb2d03 100644 --- a/README.md +++ b/README.md @@ -29,9 +29,8 @@ You may have to adjust the plugin name in the config.yaml file to match them. ## AI-Assisted Checks -Some requirements need judgment rather than a simple file or setting check. The -scanner can use an AI model for these requirements and store the assessment as -evidence. AI is disabled by default. +Deterministic checks remain the baseline. AI is an optional enhancement used +only when those checks cannot answer a requirement. AI is never required. Enable AI with a provider, model, and API key: diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index d1d6391f..6837a559 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -1,15 +1,13 @@ # AI-Assisted Checks -AI-assisted checks help evaluate requirements that need judgment rather than a -simple file or setting check. +Deterministic checks remain the baseline. AI is an optional enhancement used +only when those checks cannot answer a requirement. AI is never required. For example, `OSPS-QA-06.02` asks whether a project explains when and how to run its tests. Finding the word "test" is not enough: the documentation must provide useful instructions. An AI model can review the relevant documentation and assess whether it meets the requirement. -AI is disabled by default. - ## Enabling AI Set an AI provider, model, and API key. The scanner supports `openai` and From f5e7b1a300c5c5571942e51c8fdb9f87d40a40fc Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Wed, 26 Aug 2026 10:35:34 -0700 Subject: [PATCH 09/13] docs: lead AI guide with user need Explain why some requirements need AI before describing the optional AI behavior. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: vinayada1 --- README.md | 5 +++-- docs/ai-assisted-checks.md | 12 ++++++------ 2 files changed, 9 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index 2abb2d03..9848908e 100644 --- a/README.md +++ b/README.md @@ -29,8 +29,9 @@ You may have to adjust the plugin name in the config.yaml file to match them. ## AI-Assisted Checks -Deterministic checks remain the baseline. AI is an optional enhancement used -only when those checks cannot answer a requirement. AI is never required. +Some requirements cannot be answered by checking for a file or setting. AI can +assess these requirements when a deterministic check cannot. AI is optional and +never replaces the deterministic checks. Enable AI with a provider, model, and API key: diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 6837a559..14c1d607 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -1,12 +1,12 @@ # AI-Assisted Checks -Deterministic checks remain the baseline. AI is an optional enhancement used -only when those checks cannot answer a requirement. AI is never required. +Some requirements cannot be answered by checking for a file or setting. For +example, `OSPS-QA-06.02` asks whether a project explains when and how to run its +tests. Finding the word "test" is not enough; the documentation must provide +useful instructions. -For example, `OSPS-QA-06.02` asks whether a project explains when and how to run -its tests. Finding the word "test" is not enough: the documentation must provide -useful instructions. An AI model can review the relevant documentation and -assess whether it meets the requirement. +AI can assess requirements like this when a deterministic check cannot. AI is +optional and never replaces the deterministic checks. ## Enabling AI From 1ad9b5142663712debcbf59449393aaf7e1e3342 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Wed, 26 Aug 2026 10:42:40 -0700 Subject: [PATCH 10/13] docs: make AI setup example-first Lead the enabling section with a runnable example, then explain its settings and alternatives. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: vinayada1 --- docs/ai-assisted-checks.md | 23 ++++++++++++----------- 1 file changed, 12 insertions(+), 11 deletions(-) diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 14c1d607..6f5efad0 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -1,6 +1,6 @@ # AI-Assisted Checks -Some requirements cannot be answered by checking for a file or setting. For +Some requirements are subjective and cannot be answered by implementing checks for a file, setting or an API call in the code. For example, `OSPS-QA-06.02` asks whether a project explains when and how to run its tests. Finding the word "test" is not enough; the documentation must provide useful instructions. @@ -10,11 +10,7 @@ optional and never replaces the deterministic checks. ## Enabling AI -Set an AI provider, model, and API key. The scanner supports `openai` and -`anthropic`. - -The recommended setup uses environment variables so the API key is not stored -in the configuration file: +This example enables AI with OpenAI and runs the scanner: ```sh export PVTR_AI_PROVIDER='openai' @@ -24,17 +20,22 @@ export PVTR_AI_API_KEY='' ./pvtr run --binaries-path . ``` -For Anthropic, change the provider and model: +All three settings are required. `PVTR_AI_PROVIDER` selects the provider, +`PVTR_AI_MODEL` selects one of its models, and `PVTR_AI_API_KEY` authenticates +the request. The scanner supports `openai` and `anthropic`. + +Keep the API key in an environment variable or CI secret store. The environment +variables apply to every service in the run. Use `--service=` to +run only one configured service. + +To use Anthropic instead, change the provider and model: ```sh export PVTR_AI_PROVIDER='anthropic' export PVTR_AI_MODEL='' ``` -Environment variables apply to every service in the run. Use -`--service=` to run only one configured service. - -You can also add non-secret settings to the configuration file: +Non-secret settings can also go in the configuration file: ```yaml ai_provider: openai From 180ef0506601b281ca51ff594a16f9b41c5a1a03 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Wed, 26 Aug 2026 10:43:51 -0700 Subject: [PATCH 11/13] docs: simplify AI enablement steps Show how to add AI settings to an existing service configuration and remove unnecessary placement and override examples. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: vinayada1 --- README.md | 5 ++-- docs/ai-assisted-checks.md | 50 ++++++++++---------------------------- 2 files changed, 15 insertions(+), 40 deletions(-) diff --git a/README.md b/README.md index 9848908e..89760ebd 100644 --- a/README.md +++ b/README.md @@ -33,11 +33,10 @@ Some requirements cannot be answered by checking for a file or setting. AI can assess these requirements when a deterministic check cannot. AI is optional and never replaces the deterministic checks. -Enable AI with a provider, model, and API key: +Add `ai_provider` and `ai_model` to the service's `vars` in `config.yml`, then +set the API key: ```sh -export PVTR_AI_PROVIDER='openai' -export PVTR_AI_MODEL='gpt-4o-mini' export PVTR_AI_API_KEY='' ``` diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 6f5efad0..001bd7dc 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -10,37 +10,10 @@ optional and never replaces the deterministic checks. ## Enabling AI -This example enables AI with OpenAI and runs the scanner: - -```sh -export PVTR_AI_PROVIDER='openai' -export PVTR_AI_MODEL='gpt-4o-mini' -export PVTR_AI_API_KEY='' - -./pvtr run --binaries-path . -``` - -All three settings are required. `PVTR_AI_PROVIDER` selects the provider, -`PVTR_AI_MODEL` selects one of its models, and `PVTR_AI_API_KEY` authenticates -the request. The scanner supports `openai` and `anthropic`. - -Keep the API key in an environment variable or CI secret store. The environment -variables apply to every service in the run. Use `--service=` to -run only one configured service. - -To use Anthropic instead, change the provider and model: - -```sh -export PVTR_AI_PROVIDER='anthropic' -export PVTR_AI_MODEL='' -``` - -Non-secret settings can also go in the configuration file: +Add the provider and model to the service's `vars` in `config.yml`. This example +enables OpenAI for `my-scan`: ```yaml -ai_provider: openai -ai_model: gpt-4o-mini - services: my-scan: plugin: github-repo @@ -48,19 +21,22 @@ services: owner: repo: token: + ai_provider: openai + ai_model: gpt-4o-mini ``` -Top-level settings apply to every service. To override a setting for one -service, add it to that service's `vars` block: +Set the API key in an environment variable or CI secret store, then run the +scanner: -```yaml -services: - my-scan: - plugin: github-repo - vars: - ai_model: +```sh +export PVTR_AI_API_KEY='' +./pvtr run --binaries-path . ``` +All three settings are required. The scanner supports `openai` and `anthropic`. +To use Anthropic, set `ai_provider` to `anthropic` and use an Anthropic model +name for `ai_model`. + ## Configuration From eea62c54964c882532c97425caf63efd17b3cb87 Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Wed, 26 Aug 2026 10:53:31 -0700 Subject: [PATCH 12/13] docs: merge AI setup and configuration Describe the AI settings first, then show one complete configuration with every setting populated. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: vinayada1 --- docs/ai-assisted-checks.md | 52 +++++++++++++++----------------------- 1 file changed, 21 insertions(+), 31 deletions(-) diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 001bd7dc..622f00e6 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -10,8 +10,21 @@ optional and never replaces the deterministic checks. ## Enabling AI -Add the provider and model to the service's `vars` in `config.yml`. This example -enables OpenAI for `my-scan`: + + +| Setting | Environment variable | Required | Default | Description | +| --- | --- | :---: | --- | --- | +| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | AI provider: `openai` or `anthropic`. | +| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name from the provider. | +| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | API key for the provider. | +| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | URL for a compatible gateway, proxy, or self-hosted endpoint. | +| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Maximum time to wait for a response. | +| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Maximum response size. | + + + +Add the settings to the service's `vars` in `config.yml`. This example shows +every setting and enables OpenAI for `my-scan`: ```yaml services: @@ -23,6 +36,9 @@ services: token: ai_provider: openai ai_model: gpt-4o-mini + ai_base_url: https://api.openai.com/v1 + ai_timeout: 30s + ai_max_tokens: 1024 ``` Set the API key in an environment variable or CI secret store, then run the @@ -33,41 +49,15 @@ export PVTR_AI_API_KEY='' ./pvtr run --binaries-path . ``` -All three settings are required. The scanner supports `openai` and `anthropic`. -To use Anthropic, set `ai_provider` to `anthropic` and use an Anthropic model -name for `ai_model`. - -## Configuration - - - -| Setting | Environment variable | Required | Default | Description | -| --- | --- | :---: | --- | --- | -| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | AI provider: `openai` or `anthropic`. | -| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name from the provider. | -| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | API key for the provider. | -| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | URL for a compatible gateway, proxy, or self-hosted endpoint. | -| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Maximum time to wait for a response. | -| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Maximum response size. | - - +To use Anthropic, set `ai_provider` to `anthropic`, use an Anthropic model name, +and omit `ai_base_url` to use the provider default. Set `ai_base_url` only for a +compatible gateway, proxy, or self-hosted endpoint. All three required settings must be present. If one is missing, the affected checks report `Needs Review`, log a warning, and allow the scan to continue. `ai_timeout` must use a duration such as `30s` or `2m`; an invalid duration stops the run. `ai_max_tokens` must be a whole number. -### Custom Endpoints - -Set `ai_base_url` to use a compatible gateway, proxy, or self-hosted endpoint. -The endpoint must accept the request format for the selected provider. - -```yaml -ai_provider: openai -ai_model: -ai_base_url: https://ai-gateway.example.com/v1 -``` - ## Security And Privacy The scanner sends repository content to the configured AI provider. It sends From ccadecaac4ce902a8825bfa9300ec1d65b53494d Mon Sep 17 00:00:00 2001 From: vinayada1 Date: Wed, 26 Aug 2026 10:58:25 -0700 Subject: [PATCH 13/13] docs: focus AI settings on config file Remove the environment-variable column and document environment use only for securely supplying the provider API key. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Signed-off-by: vinayada1 --- docs/ai-assisted-checks.md | 25 +++++++++++++++---------- example-config.yml | 4 ++-- 2 files changed, 17 insertions(+), 12 deletions(-) diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md index 622f00e6..593ec2c0 100644 --- a/docs/ai-assisted-checks.md +++ b/docs/ai-assisted-checks.md @@ -12,14 +12,16 @@ optional and never replaces the deterministic checks. -| Setting | Environment variable | Required | Default | Description | -| --- | --- | :---: | --- | --- | -| `ai_provider` | `PVTR_AI_PROVIDER` | yes | -- | AI provider: `openai` or `anthropic`. | -| `ai_model` | `PVTR_AI_MODEL` | yes | -- | Model name from the provider. | -| `ai_api_key` | `PVTR_AI_API_KEY` | yes | -- | API key for the provider. | -| `ai_base_url` | `PVTR_AI_BASE_URL` | no | provider default | URL for a compatible gateway, proxy, or self-hosted endpoint. | -| `ai_timeout` | `PVTR_AI_TIMEOUT` | no | `30s` | Maximum time to wait for a response. | -| `ai_max_tokens` | `PVTR_AI_MAX_TOKENS` | no | `1024` | Maximum response size. | +Configure AI with these settings in `config.yml`: + +| Setting | Required | Default | Description | +| --- | :---: | --- | --- | +| `ai_provider` | yes | -- | AI provider: `openai` or `anthropic`. | +| `ai_model` | yes | -- | Model name from the provider. | +| `ai_api_key` | yes | -- | API key for the provider. Do not store it in `config.yml`. | +| `ai_base_url` | no | provider default | URL for a compatible gateway, proxy, or self-hosted endpoint. | +| `ai_timeout` | no | `30s` | Maximum time to wait for a response. | +| `ai_max_tokens` | no | `1024` | Maximum response size. | @@ -41,14 +43,17 @@ services: ai_max_tokens: 1024 ``` -Set the API key in an environment variable or CI secret store, then run the -scanner: +Store the API key in an environment variable or CI secret rather than in +`config.yml`. The scanner reads it from `PVTR_AI_API_KEY`: ```sh export PVTR_AI_API_KEY='' ./pvtr run --binaries-path . ``` +A CI secret may use any name as long as its value is exposed to the scanner as +`PVTR_AI_API_KEY`. + To use Anthropic, set `ai_provider` to `anthropic`, use an Anthropic model name, and omit `ai_base_url` to use the provider default. Set `ai_base_url` only for a compatible gateway, proxy, or self-hosted endpoint. diff --git a/example-config.yml b/example-config.yml index b8d8300e..56df2e74 100644 --- a/example-config.yml +++ b/example-config.yml @@ -26,14 +26,14 @@ services: token: # AI-assisted checks are disabled unless these settings are present. - # Prefer PVTR_AI_* environment variables, especially for the API key. + # Store the API key in an environment variable or CI secret. # Secrets in repository files are not detected or redacted. # See docs/ai-assisted-checks.md. # --- Required to enable AI --- # ai_provider: openai # openai or anthropic # ai_model: gpt-4o-mini # model name, as your provider spells it - # ai_api_key: # prefer the PVTR_AI_API_KEY env var instead + # The API key is also required; expose it as PVTR_AI_API_KEY. # --- Optional (defaults shown) --- # ai_base_url: https://api.openai.com/v1 # proxy, gateway, or self-hosted