ci: automatic advisory Quality Keeper runs on same-repo PRs (#146)

[sophiecarreras]

What this changes

Implements the SDK pilot in backblaze-labs/quality-keeper#146: Quality Keeper now runs automatically on same-repo pull requests to main, and stays advisory / non-blocking. Extends the existing .github/workflows/quality-run-demo.yml (no new file).

• Trigger: pull_request (NOT pull_request_target) on branches: [main], types opened/synchronize/reopened/ready_for_review. workflow_dispatch retained for manual demo/debug.
• Same-repo only / fork-safe: both jobs carry if: event == workflow_dispatch || head.repo.full_name == github.repository, so fork PRs skip entirely — untrusted fork code never runs in a job that can reach secrets or PR-write authority. (GitHub also withholds secrets / forces a read-only token on fork pull_request runs — defense in depth.)
• Two-job split preserved: quality-run (contents: read, runs the target's own code, QK_INSTALL_TOKEN only, no App key) → comment (pull-requests: write + actions: read, runs no target code, mints the App token downscoped to pull-requests: write).
• Advisory / non-blocking: run-found-checks exits non-zero when a safe check fails; that exit code is captured as advisory evidence (execution.json is written first), so a failing SDK check does not turn QK into a red/blocking check — while genuine QK errors (assess / post-execute) still fail. This workflow is not a required check and does not gate merge.
• Injection-safe: comment target = inputs.issue_url (manual) or github.event.pull_request.html_url (auto), passed via env: and quoted — no PR-controlled text expanded inline.
• Idempotent one-comment upsert, --expect-repo, and Repository Activation preserved. Per-PR concurrency with cancel-in-progress. All actions SHA-pinned with version comments.

No seeds, branches, issues, PRs, or auto-fixes are created.

Security / verification

• zizmor clean at regular and pedantic personas.
• YAML valid; triggers = pull_request + workflow_dispatch (no pull_request_target); all uses: SHA-pinned.
• The repo's GitHub Actions security check (security.yml → gha-security) runs on this PR and should pass.
• This PR is itself the first live automatic proof (a same-repo PR, not Gonzalo's): opening it triggers the advisory QK run, which should post/update one comment as quality-keeper[bot]. The run + comment will be linked back to quality-keeper#146.

Residual risk (documented, follow-up proposed)

QK_INSTALL_TOKEN is reachable by same-repo (trusted-collaborator) PR code in quality-run, because quality-keeper is private and must be pip-installed there. It must remain a read-only, repo-scoped credential. Safer follow-up: eliminate the install secret from the code-running job by publishing quality-keeper to an internal registry / prebuilt image / making it public.

Refs backblaze-labs/quality-keeper#146

🤖 Generated with Claude Code

[quality-keeper]

Quality Keeper — Advisory Review

Source: backblaze-labs/b2-sdk-typescript

Advisory only — this is not a required status check. Quality Keeper reviewed this repository snapshot and is reporting what it found. It changes no status and merges nothing; suggested scaffolds are starting points, not proof.

Quality Keeper assessed the source, executed checks, and collected results, covering 14 quality expectations.

Result

• 10 expectations are covered by acceptable evidence.
• 4 still need attention — Quality Keeper cannot count them as covered yet.

Covered

Quality Keeper found acceptable evidence for these Testing Types.

• Build Verification — Exercised this run.
  Confirms the software actually builds/packages into a runnable or publishable artifact.
• Build Verification — Exercised this run.
  Confirms the software actually builds/packages into a runnable or publishable artifact.
• Compatibility Testing — Exercised this run.
  Checks the software works across the consumers, platforms, or versions it claims to support.
• Compatibility Testing — Exercised this run.
  Checks the software works across the consumers, platforms, or versions it claims to support.
• Configuration & Artifact Review — Verified before execution.
  Checks that config files, manifests, and build artifacts are present, well-formed, and say what they should.
• Metadata & Discoverability Review — Verified before execution.
  Checks that a repository or package is findable and understandable — description, topics, README, keywords.
• Specification Conformity Review — Exercised this run.
  Checks that documented assets actually match the specification or documented behavior they claim.
• Specification Conformity Review — Exercised this run.
  Checks that documented assets actually match the specification or documented behavior they claim.
• Static Security Analysis — Verified before execution.
  Scans code and configuration for security problems without running it — secrets, unsafe patterns, misconfigurations.
• Unit Testing — Exercised this run.
  Exercises individual functions or modules in isolation.

Needs attention

Quality Keeper does not yet have acceptable evidence for these Testing Types.

Some have partial checks that need to run, be fixed, or be connected. Others need new proof or seeded tests. In both cases, Quality Keeper cannot count them as covered yet.

• Contract Testing
  Checks that a public or network boundary (API, SDK surface, schema) matches its agreed shape.
• Environmental Isolation Testing
  Checks that tests set up and tear down their own state and do not depend on leftover data, shared buckets, or each other.
• Integration Testing
  Checks that two or more components work together — a module plus a real dependency, service, or datastore.
• Static Security Analysis
  Scans code and configuration for security problems without running it — secrets, unsafe patterns, misconfigurations.

Quality Keeper changes no Evidence State. Pending, missing, executed-but-not-exercised, and review items are not passing, and seeded scaffolds are next actions, not evidence, until applied and re-evaluated.

Full breakdown (14 expectations)

Area	Status	Expectation	Why it applies
Build Verification	Exercised this run	packaged_or_runnable_assets_need_build_verification	The interpreted asset has package, script, workflow, UI, service, CLI, edge-runtime, or game-runtime signals, so it needs a reproducible build or equivalent construction check.
Build Verification	Exercised this run	published_packages_need_sdk_build_artifact_review	The interpreted asset is a published package with source code, so the distribution manifest completeness, pack tarball integrity, and declared file list need verification as part of the release artifact.
Compatibility Testing	Exercised this run	consumer_or_platform_boundaries_need_compatibility_testing	The interpreted asset is published or consumed across environments (published package, exported API, command-line tool, or edge target), so compatibility across platforms, versions, and consumers needs verification.
Compatibility Testing	Exercised this run	published_typescript_packages_need_type_declaration_check	The interpreted asset is a published TypeScript package (tsconfig.json or has_types in package.json detected), so generated .d.ts declaration files and type resolution must be verified from a consumer project's perspective.
Configuration & Artifact Review	Verified before execution	every_asset_needs_configuration_review	Every interpreted asset needs a static review of its declared files, manifests, configuration, and artifacts so Quality Keeper can explain its shape before evidence is harvested.
Contract Testing	Missing — no seed proposal	public_or_network_boundaries_need_contract_testing	The interpreted asset publishes a boundary that other software depends on (network API, CLI, or package exports), so the public interface contract needs verification.
Environmental Isolation Testing	Selected, not executed	external_resource_assets_need_environmental_isolation_testing	The interpreted asset uses external resources, deployment configuration, infrastructure-as-code, or automated workflows, so repeated runs need isolated names, cleanup, and teardown verification.
Integration Testing	Selected, not executed	published_sdk_packages_need_integration_testing	The interpreted asset is a published package or library that creates an integration boundary for consumers, so the package's integration paths, API contracts, and external service behaviors need integration-level verification.
Metadata & Discoverability Review	Verified before execution	every_asset_needs_discoverability_review	Every interpreted asset needs enough metadata and documentation for humans and systems to identify what it is and how to work with it.
Specification Conformity Review	Exercised this run	documented_assets_need_specification_conformity_review	The asset carries documentation, so documented commands, setup instructions, examples, and claims need to stay aligned with the implementation.
Specification Conformity Review	Exercised this run	published_packages_with_docs_need_example_validation	The interpreted asset is a published package with detected documentation, so code examples in README, docs, or examples directories should be compiled or type-checked to confirm they match the actual public API.
Static Security Analysis	Missing — no seed proposal	code_and_configuration_assets_need_static_security_analysis	The interpreted asset contains code, configuration, dependencies, secrets coordinates, infrastructure-as-code, or service boundaries, so static security analysis is required before runtime evidence is trusted.
Static Security Analysis	Verified before execution	published_packages_need_provenance_proof	The interpreted asset is a published package with source code, so published release artifacts should carry provenance metadata (npm provenance, SLSA attestation, or sigstore signature) to protect the supply chain.
Unit Testing	Exercised this run	code_assets_need_unit_testing	The interpreted asset contains runnable or reusable source code, so the smallest testable units need isolated verification.

---

New to these terms? See the Quality Keeper glossary.

[sophiecarreras]

Review findings addressed (commit a9556c7)

• ci(deps): bump the github-actions group with 7 updates #1 Draft PRs (Low) — fixed. Both job guards now also require github.event.pull_request.draft == false, so QK comments only once the PR is marked ready (ready_for_review re-triggers with draft == false). Manual dispatch unaffected.
• build(deps-dev): bump the dev-dependencies group with 4 updates #2 Inaccurate inline guarantee (Low) — fixed. Replaced "execution.json is always written first" with the true contract: check findings don't fail the job, but a genuine QK error (no execution.json) still fails step 3 (job red).
• build(deps-dev): bump vitest from 3.2.4 to 4.1.7 #3 Exit-code contract (altitude) — filed as a quality-keeper follow-up: backblaze-labs/quality-keeper#147 (give run-found-checks a findings-vs-error exit contract so advisory callers don't need the shell set +e wrapper). Out of scope for this SDK PR.
• build(deps-dev): bump @biomejs/biome from 1.9.4 to 2.4.16 #4 Redundant comment guard (cleanup) — kept deliberately. It's the only write-capable job; the mirrored guard is defense-in-depth against a future refactor dropping needs. Added a comment explaining why it's retained despite being redundant today.
• build(deps-dev): bump @vitest/coverage-v8 from 3.2.4 to 4.1.7 #5 cancel-in-progress trade-off (info) — documented. Noted that a superseded intermediate commit may not get its own comment (latest SHA wins), which is correct for an advisory signal.

Verified: YAML valid; both guards carry same-repo + non-draft; zizmor clean (regular + pedantic). This push re-triggers the advisory run on this (non-draft) PR, confirming the draft gate doesn't affect normal same-repo PRs.

[sophiecarreras]

Quality Keeper workflow warning cleanup is fixed on commit 000d6a7.

What changed:

• Updated the Quality Keeper demo workflow actions to Node 24-compatible pinned releases.
• Switched actions/create-github-app-token v3 from deprecated app-id to the public client-id input; the private key remains in QK_APP_PRIVATE_KEY.
• Kept the existing security shape: same-repo/non-draft guards, two-job split, read-only quality-run job, PR-write only in the comment job.

Verification:

• Old Node 20 action SHAs and app-id: scan clean.
• Workflow YAML parses cleanly.
• git diff --check clean.
• Quality Keeper run is green with the updated workflow: https://github.com/backblaze-labs/b2-sdk-typescript/actions/runs/29273358557
• The existing QK advisory comment was updated in place with the current main renderer: ci: automatic advisory Quality Keeper runs on same-repo PRs (#146) #127 (comment)

I did not merge this PR; it remains open for SDK review.