diff --git a/README.md b/README.md index 88257a40..89760ebd 100644 --- a/README.md +++ b/README.md @@ -27,6 +27,29 @@ If the binaries and the config files are in different directories specify the co You may have to adjust the plugin name in the config.yaml file to match them. +## AI-Assisted Checks + +Some requirements cannot be answered by checking for a file or setting. AI can +assess these requirements when a deterministic check cannot. AI is optional and +never replaces the deterministic checks. + +Add `ai_provider` and `ai_model` to the service's `vars` in `config.yml`, then +set the API key: + +```sh +export PVTR_AI_API_KEY='' +``` + +The scanner supports OpenAI and Anthropic. AI-assisted results include the +`[AI-Assisted]` prefix and a confidence level. Provider errors and invalid +responses produce `Needs Review`; the scan continues. + +> **The scanner does not detect or redact secrets in repository files before +> sending those files to the provider or storing them in AI evidence.** + +See [AI-Assisted Checks](docs/ai-assisted-checks.md) for configuration, +security guidance, supported checks, and result details. + ## Docker Usage ```sh diff --git a/docs/ai-assisted-checks.md b/docs/ai-assisted-checks.md new file mode 100644 index 00000000..593ec2c0 --- /dev/null +++ b/docs/ai-assisted-checks.md @@ -0,0 +1,131 @@ +# AI-Assisted Checks + +Some requirements are subjective and cannot be answered by implementing checks for a file, setting or an API call in the code. For +example, `OSPS-QA-06.02` asks whether a project explains when and how to run its +tests. Finding the word "test" is not enough; the documentation must provide +useful instructions. + +AI can assess requirements like this when a deterministic check cannot. AI is +optional and never replaces the deterministic checks. + +## Enabling AI + + + +Configure AI with these settings in `config.yml`: + +| Setting | Required | Default | Description | +| --- | :---: | --- | --- | +| `ai_provider` | yes | -- | AI provider: `openai` or `anthropic`. | +| `ai_model` | yes | -- | Model name from the provider. | +| `ai_api_key` | yes | -- | API key for the provider. Do not store it in `config.yml`. | +| `ai_base_url` | no | provider default | URL for a compatible gateway, proxy, or self-hosted endpoint. | +| `ai_timeout` | no | `30s` | Maximum time to wait for a response. | +| `ai_max_tokens` | no | `1024` | Maximum response size. | + + + +Add the settings to the service's `vars` in `config.yml`. This example shows +every setting and enables OpenAI for `my-scan`: + +```yaml +services: + my-scan: + plugin: github-repo + vars: + owner: + repo: + token: + ai_provider: openai + ai_model: gpt-4o-mini + ai_base_url: https://api.openai.com/v1 + ai_timeout: 30s + ai_max_tokens: 1024 +``` + +Store the API key in an environment variable or CI secret rather than in +`config.yml`. The scanner reads it from `PVTR_AI_API_KEY`: + +```sh +export PVTR_AI_API_KEY='' +./pvtr run --binaries-path . +``` + +A CI secret may use any name as long as its value is exposed to the scanner as +`PVTR_AI_API_KEY`. + +To use Anthropic, set `ai_provider` to `anthropic`, use an Anthropic model name, +and omit `ai_base_url` to use the provider default. Set `ai_base_url` only for a +compatible gateway, proxy, or self-hosted endpoint. + +All three required settings must be present. If one is missing, the affected +checks report `Needs Review`, log a warning, and allow the scan to continue. +`ai_timeout` must use a duration such as `30s` or `2m`; an invalid duration stops +the run. `ai_max_tokens` must be a whole number. + +## Security And Privacy + +The scanner sends repository content to the configured AI provider. It sends +only the files needed by the check: README, CONTRIBUTING, or relevant GitHub +Actions workflow files. + +> **The scanner does not detect or redact secrets in repository files.** Do not +> enable AI for repositories whose relevant files contain secrets or other data +> that must not be sent to the provider. + +The same content is stored without redaction in the scan evidence, together with +the question sent to the model and its response. Protect the results file +accordingly. + +Keep the provider API key in `PVTR_AI_API_KEY` or your CI secret store. Do not +commit it to the configuration file. + +Confirm that your organization permits the selected provider to process the +repository content. + +## Reading Results + +Results produced with AI include the `[AI-Assisted]` prefix: + +```text +[AI-Assisted] CONTRIBUTING.md explains how and when contributors should run the tests. +``` + +The result is `Passed`, `Failed`, or `Needs Review`, with `low`, `medium`, or +`high` confidence. Review low-confidence results before relying on them. + +An invalid response, provider error, timeout, missing setting, or oversized +input produces `Needs Review` at low confidence. The scan continues and logs a +warning. An AI failure never becomes a `Failed` requirement. + +The scanner stores successful AI assessments as `ai-assessment` evidence in: + +```text +//.yaml +``` + +The evidence includes the result, confidence, explanation, question, content, +provider, model, request ID, and source file references. + +## Checks That Use AI + +AI is used only where a regular check does not exist or cannot answer the +requirement. + + + +| Requirement | AI assessment | +| --- | --- | +| `OSPS-QA-06.02` | Whether project documentation explains when and how tests are run. | +| `OSPS-QA-06.03` | Whether project documentation defines a policy for maintaining tests. | +| `OSPS-AC-04.02` | Whether specific GitHub Actions permissions are required by the job. | + + + +`OSPS-QA-06.02` and `OSPS-QA-06.03` report `Needs Review` when AI is disabled. +For `OSPS-AC-04.02`, regular checks handle clear cases first; AI reviews only +permissions that require context. + +Each requirement makes at most one provider request per scan. Documentation sent +for `OSPS-QA-06` is limited to 64 KiB. `OSPS-AC-04.02` is limited to 50 workflow +files and 64 KiB. Inputs over these limits report `Needs Review`. diff --git a/example-config.yml b/example-config.yml index 32eb0c7c..56df2e74 100644 --- a/example-config.yml +++ b/example-config.yml @@ -19,9 +19,23 @@ services: # - Maturity Level 2 # - Maturity Level 3 - # All variables are required to run the evaluation + # The owner, repo, and token variables are required to run the evaluation vars: owner: repo: token: + # AI-assisted checks are disabled unless these settings are present. + # Store the API key in an environment variable or CI secret. + # Secrets in repository files are not detected or redacted. + # See docs/ai-assisted-checks.md. + + # --- Required to enable AI --- + # ai_provider: openai # openai or anthropic + # ai_model: gpt-4o-mini # model name, as your provider spells it + # The API key is also required; expose it as PVTR_AI_API_KEY. + + # --- Optional (defaults shown) --- + # ai_base_url: https://api.openai.com/v1 # proxy, gateway, or self-hosted + # ai_timeout: 30s # how long to wait for one answer + # ai_max_tokens: 1024 # longest answer to allow