Skip to content

docs: document AI configuration and usage - #462

Open
vinayada1 wants to merge 13 commits into
ossf:mainfrom
vinayada1:vinayada1-docs-ai-configuration-and-usage
Open

docs: document AI configuration and usage#462
vinayada1 wants to merge 13 commits into
ossf:mainfrom
vinayada1:vinayada1-docs-ai-configuration-and-usage

Conversation

@vinayada1

@vinayada1 vinayada1 commented Aug 25, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Closes #319

Documents the scanner's AI-assisted checks: why a few requirements need AI at
all, how to turn them on, how to configure a provider, and how to read the
results.

What's here

  • docs/ai-assisted-checks.md (new) — why these requirements can't be
    answered by searching, all six ai_* settings, where they can be placed and
    which location wins, provider examples for openai and anthropic,
    gateway/self-hosted endpoints, per-service overrides, how to check a
    configuration without paying for provider calls, how to read [AI-Assisted]
    results, what happens when AI cannot answer, the evidence format, size
    limits, and cost notes.
  • README.md — a short section covering the settings table and result
    interpretation, linking to the deep dive.
  • example-config.yml — a commented AI block split into required and
    optional settings.

Docs only; no behaviour change.

Why AI, concretely

Rather than asserting that these requirements "can't be pattern matched", the
doc shows why, using the criteria from the prompts the scanner actually sends.
For OSPS-QA-06.02: documentation saying "run the tests" with no command, a
make test snippet that never says when tests are expected, and testing
guidance aimed at end users rather than contributors. All three contain the
keyword; none satisfies the requirement. OSPS-QA-06.03 rests on the
difference between describing a test suite and requiring tests with changes,
and OSPS-AC-04.02 on the fact that contents: write is right for a release
job and excessive for a linter.

A note on dry-run

The issue originally asked for dry-run guidance. Dry-run does not exist in the
pinned SDK: it was added in privateer-sdk#227 and removed in privateer-sdk#252
when the AI packages were reorganised. This scanner never had a dry-run of its
own, so there was nothing to document. The issue was updated to ask instead for
guidance on validating a configuration without provider spend, and the doc has
a section on that — plus an explanation of the history so the next person
asking doesn't have to re-derive it.

Verification

Claims were checked against the pinned privateer-sdk and this repo's step
implementations rather than assumed. A few were verified by running code, which
corrected several drafting errors:

  • Blanking ai_provider/ai_model for one service does not opt it out if
    an ai_api_key is still reachable — the service stays enabled but invalid.
    The doc now recommends scoping AI to the services that need it.
  • Misconfiguration triggers when any ai_* setting is present and a required
    one is missing, including when only an optional one such as ai_base_url is
    set.
  • The OSPS-AC-04.02 size limits apply only to workflows that need a judgment
    call, not to every workflow in the repository.
  • Granting every scope at its highest level fails the same way as the
    write-all shorthand, so the summary covers both spellings.

The doc deliberately does not name an SDK version. A scheduled workflow
bumps that pin weekly, so a hardcoded version would go stale almost
immediately; it points at go.mod instead.

go build ./..., go test ./..., and golangci-lint run (0 issues) all pass.

Note: docs/best-practices-badge.md was used as the format model for the new
doc.

@vinayada1
docs: document AI configuration and usage
4670160 
AI-assisted assessments have shipped for three requirements, but nothing told
users how to turn them on, which providers work, what the settings mean, or how
to read an [AI-Assisted] verdict.

Add a concise "AI-Assisted Checks" section to the README covering the six ai_*
keys and their PVTR_AI_* environment variables, a minimal configuration, and
how to interpret results. Add docs/ai-assisted-checks.md for the detail:
provider examples for openai and anthropic, configuration precedence, verdict
and confidence semantics, failure behavior, the evidence record, per-
requirement coverage, size limits, and cost considerations. Add a commented
AI block to example-config.yml, split into required and optional keys.

The keys, defaults, verdict schema, and failure paths documented here were
verified against privateer-sdk v1.32.2 and the step implementations.

Dry-run is deliberately not documented. It briefly existed in the SDK
(privateerproj/privateer-sdk#227) and was removed when the ai package was
restructured (privateerproj/privateer-sdk#252); this scanner never implemented
it. The docs instead explain how to validate a configuration without provider
spend, relying on the SDK validating provider, model, and credential locally
before any network call.

Closes ossf#319

Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: clarify that provider selection is SDK-driven
61943ba 
The scanner has no provider-specific code; it calls ai.NewClient and the SDK
picks an adapter from ai_provider. Say so explicitly, and note that each
adapter has its own SDK test suite so readers can weigh the examples fairly.

Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: simplify wording in AI documentation
2c27876 
Replace jargon with plain language throughout: 'deterministic check' becomes
'regular check', 'provenance' becomes 'where the answer came from', 'adapter',
'schema', 'enum' and 'wire protocol' are dropped in favour of ordinary terms.
Use 'settings' consistently instead of alternating with 'keys', and describe
evidence as landing in the results file rather than alternating between that
and 'evaluation log'.

Also make two claims exact: the short message is truncated with an ellipsis at
160 characters rather than simply capped, and the retained explanation is
bounded at 1500 characters.

Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
vinayada1 requested a review from a team as a code owner August 25, 2026 19:01
@github-actions github-actions Bot added the documentation Improvements or additions to documentation label Aug 25, 2026
vinayada1 and others added 10 commits August 25, 2026 12:06
@vinayada1
docs: explain why some requirements need AI and unpin SDK version
f429957 
Two problems with the previous draft.

The claim that some requirements 'cannot be answered by pattern matching' was
asserted rather than shown. Give concrete examples instead, drawn from the
prompts the scanner actually sends: documentation that says 'run the tests'
without a command, a 'make test' snippet that never says when tests are
expected, and testing guidance aimed at end users rather than contributors --
all of which contain the keyword and none of which satisfy OSPS-QA-06.02.
Cover the describe-versus-require distinction behind OSPS-QA-06.03 and the
context dependence of a grant like 'contents: write' for OSPS-AC-04.02.

Naming the pinned SDK version also dated the document. A scheduled workflow
bumps that pin weekly, so the number would be stale almost immediately. Point
at go.mod instead, which is the source of truth and needs no doc change.

Also generalize the OSPS-AC-04.02 summary: granting every scope at its highest
level fails in the same way as the write-all shorthand.

Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: shorten the AI rationale and move scope note to its own section
a6d01ce 
The rationale section made the same point three times, once per AI-assisted
requirement, and the table below already lists them. Keep the clearest example
(OSPS-QA-06.02) and drop the rest, roughly halving the section.

The sentence about AI never replacing a regular check describes when AI is
used, not why it is needed, so move it to the head of 'Which Checks Use AI'
where the rest of that scope is defined.

Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: restructure AI guide around examples
be7d225 
Replace the abstract settings-precedence list with a worked config example,
and merge the four separate sections that all described where settings go
into one example-led section.

Move the on/off boundary note out of the opt-in section and reframe it as
part of turning AI on, and lead the configuration-checking section with what
you can do rather than with the history of the removed dry-run flag.

Correct the config-versus-environment precedence: the environment overrides
a top-level setting, but not one inside a vars block.

Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: rewrite AI guide for users
6790213 
Replace the implementation-focused document with a concise task-based guide. Remove dry-run and SDK history, simplify configuration and results guidance, and make the lack of secret redaction prominent.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: clarify optional role of AI
7e02d43 
State that deterministic checks remain the baseline and that AI is an optional enhancement used only when those checks cannot answer a requirement.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: lead AI guide with user need
f5e7b1a 
Explain why some requirements need AI before describing the optional AI behavior.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: make AI setup example-first
1ad9b51 
Lead the enabling section with a runnable example, then explain its settings and alternatives.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: simplify AI enablement steps
180ef05 
Show how to add AI settings to an existing service configuration and remove unnecessary placement and override examples.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: merge AI setup and configuration
eea62c5 
Describe the AI settings first, then show one complete configuration with every setting populated.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
@vinayada1
docs: focus AI settings on config file
ccadeca 
Remove the environment-variable column and document environment use only for securely supplying the provider API key.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Signed-off-by: vinayada1 <vinayada1@users.noreply.github.com>
eddie-knight
eddie-knight reviewed Aug 29, 2026
View reviewed changes
Comment thread docs/ai-assisted-checks.md
every setting and enables OpenAI for `my-scan`:

```yaml
services:

@eddie-knight eddie-knight Aug 29, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
services:
targets:

We're trying to deprecate use of services due to the confusion it was causing among users

jmeridth
jmeridth approved these changes Aug 30, 2026
View reviewed changes

@jmeridth jmeridth left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 (summary) Verified every documented config key, env var, requirement list, model default, and fallback behavior against the SDK v1.33.0 source and the scanner code, and it all checks out, including the post-#458 behavior-keyed prompts and post-#469 spellings. One small accuracy fix inline. The branch base is 9 commits behind main but the hunks land in regions #469 left untouched, so a routine rebase applies cleanly.

Comment thread docs/ai-assisted-checks.md

The result is `Passed`, `Failed`, or `Needs Review`, with `low`, `medium`, or
`high` confidence. Review low-confidence results before relying on them.

@jmeridth jmeridth Aug 30, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🤖 (low, non-blocking) This path is only true for the default output: yaml. With output: json (documented as supported in example-config.yml) the file is <service>.json, and with output: sarif the AI evidence payload has no place in the SARIF schema and likely does not appear at all. Worth qualifying so users on json/sarif do not conclude the AI assessment silently failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@eddie-knight eddie-knight eddie-knight left review comments
@jmeridth jmeridth jmeridth approved these changes

At least 1 approving review is required to merge this pull request.

Assignees

No one assigned

Labels

documentation Improvements or additions to documentation

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

docs: document AI configuration and usage [Draft]

3 participants

@vinayada1 @jmeridth @eddie-knight