docs(problems): add a concrete trust scorecard artifact shape

[Benkapner]

Summary

trustworthiness-evidence.md describes the five evidence types and how they compose into a trust decision (threshold / weighted / tiered), but it never says what the decision is actually recorded as. This adds a concrete "trust scorecard" artifact: the serialized record that carries the evidence plus the resulting decision, so a trust decision is reproducible, auditable, and portable across the dispatch pipeline instead of living only in a reviewer's head.

Related Issue

Extends the "Trustworthiness Evidence" problem area. No tracking issue; opening as an additive contribution to the problem doc.

Changes

• New ## A concrete artifact: the trust scorecard section between the composition models and the relationship section, with a proposed (not fixed) JSON shape that:
  • binds the whole card to a config_hash (so a config change invalidates its signals, matching the drift-detection requirement);
  • gives every signal its own source + as_of provenance/freshness, so a stale or missing signal is visible rather than silently treated as a pass;
  • keeps status separate from score (present-but-weak vs. never-measured);
  • records the composition outcome and what blocking it, so an insufficient decision names the missing evidence instead of returning an opaque no.
• One new open question: where a scorecard lives and how long it stays valid.

Scope is deliberately narrow: the scorecard makes the inputs and outcome explicit; it does not decide weights or thresholds (those stay with the composition model and governance.md).

Testing

Docs-only change. Rendered locally; internal links reuse targets already referenced by this doc (governance.md, testing-agents.md, etc.).

Checklist

• One problem area per PR
• Additive only; no existing content reworded
• DCO sign-off
• Conventional Commits title

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[qodo-code-review]

PR Summary by Qodo

Document trust scorecards and lint agent configurations in CI

📝 Documentation ⚙️ Configuration changes ✨ Enhancement 🕐 10-20 Minutes

AI Description

• Proposes config-bound trust scorecards carrying evidence provenance, freshness, status, and
 composition outcomes.
• Records blockers and rationale so autonomy decisions remain reproducible, auditable, and portable.
• Adds pinned harness-eval linting for agent configurations to the CI test job.

Diagram

graph TD
  CI["CI workflow"] --> HE["Harness lint"] --> AC["Agent config"]
  AC --> EV["Evidence signals"] --> SC["Trust scorecard"] --> CD{"Composition decision"} --> AG["Autonomy gate"]
  AC -->|config hash| SC

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Embed scorecards in run records

• ➕ Keeps evidence beside the execution audit trail
• ➕ Provides simple per-run retention and provenance
• ➖ Duplicates evidence shared by runs using the same configuration
• ➖ Makes config-level caching and portability harder

2. Store scorecards by configuration hash

• ➕ Allows reuse across runs with identical configurations
• ➕ Makes configuration invalidation explicit and efficient
• ➖ Requires a registry, retention policy, and access controls
• ➖ Must separately associate scorecards with individual run decisions

Recommendation: Keep the PR's proposed, non-fixed artifact shape because it makes the data contract concrete without prematurely choosing storage or lifecycle policy. A config-hash-keyed scorecard referenced by run records is a promising eventual model, but validity periods, refresh rules, and governance should be resolved before formalizing a schema.

Files changed (2) +47 / -0

Documentation (1) +42 / -0

trustworthiness-evidence.mdDefine a concrete trust scorecard artifact proposal +42/-0

Define a concrete trust scorecard artifact proposal

• Adds a proposed JSON scorecard that binds five evidence categories to a configuration hash and preserves per-signal provenance, freshness, status, and measurements. It records composition decisions, blockers, and rationale, while leaving thresholds, storage, and validity policy open.

docs/problems/trustworthiness-evidence.md

Other (1) +5 / -0

lint.ymlEnforce harness-eval agent configuration linting in CI +5/-0

Enforce harness-eval agent configuration linting in CI

• Installs pinned harness-eval 6.2.0 in the main test job and runs its recommended lint preset across the repository. Lint findings configured as errors now fail CI.

.github/workflows/lint.yml

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Scorecard lacks analyzed alternatives ✓ Resolved 📘 Rule violation ⌂ Architecture

Description

The new scorecard section presents one JSON artifact shape and only lists its intended benefits; it
does not compare at least one alternative representation or provide explicit trade-offs for the
proposed shape. The storage choices at line 154 are merely an open-question list without analysis,
which the rule identifies as insufficient.

Code

docs/problems/trustworthiness-evidence.md[R100-102]

+The shape below is a proposal for that artifact, not a fixed schema. It composes the five evidence types above into a single record bound to the configuration hash they were measured against (per "Configuration drift detection", evidence is only valid for one configuration):
+
+```json

Relevance

●●● Strong

Recent docs precedent explicitly requires multiple options with labeled trade-offs; this finding
matches that accepted repository rule.

PR-#5941
PR-#3058

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

PR Compliance 1062035 requires multiple options with explicit trade-off analysis. The added section
calls the single JSON record the shape below, follows it only with affirmative design points, and
the only alternatives added are unexamined storage choices in the final open question.

Rule 1062035: Problem docs must present multiple options with trade-offs, not a single prescribed solution
docs/problems/trustworthiness-evidence.md[96-135]
docs/problems/trustworthiness-evidence.md[154-154]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new trust scorecard proposal introduces a single concrete artifact shape without comparing alternative approaches or documenting explicit pros, cons, and risks for each option.

## Issue Context
PR Compliance 1062035 requires problem documents to present at least two distinct approaches with clearly described trade-offs. Briefly listing possible storage locations as an open question does not satisfy that comparison requirement.

## Fix Focus Areas
- docs/problems/trustworthiness-evidence.md[96-135]
- docs/problems/trustworthiness-evidence.md[154-154]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Signals omit required provenance ✓ Resolved 🐞 Bug ≡ Correctness

Description

The proposed scorecard omits source from both track_record and drift, even though the design
explicitly says every signal carries source and as_of. Consumers following this artifact shape
cannot attribute those measurements, undermining the stated reproducibility and auditability of the
trust decision.

Code

docs/problems/trustworthiness-evidence.md[R115-116]

+    "track_record":    { "status": "partial", "sample_size": 42, "revert_rate": 0.02,       "as_of": "..." },
+    "drift":           { "status": "pass",    "baseline_config_hash": "sha256:...",         "as_of": "..." }

Relevance

●●● Strong

Concrete schema omissions contradict its stated provenance contract; repository history accepts
documentation correctness and consistency fixes.

PR-#5269
PR-#38

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The section defines the scorecard as reproducible and auditable, but the example leaves two
measurements unattributed while its design points explicitly identify source and as_of as fields
carried by every signal.

docs/problems/trustworthiness-evidence.md[98-100]
docs/problems/trustworthiness-evidence.md[112-116]
docs/problems/trustworthiness-evidence.md[128-130]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The concrete scorecard example omits `source` for `track_record` and `drift`, contradicting the documented requirement that every evidence signal include provenance and freshness.

## Issue Context
The artifact is intended to make trust decisions reproducible, auditable, and attributable. Keep the example aligned with that stated invariant by supplying meaningful source identifiers for both entries.

## Fix Focus Areas
- docs/problems/trustworthiness-evidence.md[115-116]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

✅ Compliance rules (platform): 62 rules

✅ Skills: writing-user-docs, writing-adrs

Review mode: ⚖️ Balanced: Although the main change is documentation, it also modifies CI behavior by adding a pinned external lint installation and a new failing workflow step, warranting a careful review.

Tip of the day

💡 Did you know, you can describe a rule in plain language on the Rules page and Qodo drafts it for you

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[github-actions]

Site preview

Preview: https://55e8f6c5-site.fullsend-ai.workers.dev

Commit: 9cf3eb4fec095ffb6e678e72294fcdae04c42595

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[waynesun09]

Review sweep on the new trust-scorecard section: 6 findings, all MEDIUM, posted inline.

They cluster around one theme — the section states four properties the artifact is supposed to guarantee (reproducible decisions, never-measured distinguishable from measured-weak, drift detection matching the per-dimension rules above, provenance per section 3), and in each case the shape as drawn does not yet carry the field that would make the property hold. Each comment names the specific line pair that disagrees and proposes either a field to add or a wording change to match what the shape actually does. All are docs-level; none is a merge blocker.

Comments only — no approval or change request implied.

[rh-hemartin]

This reads as an ADR to me. Are you trying to make a design decision or just show an example?

Key words are raising this for me are basically "proposal" and "design". Frame it as an example used on other products, not as a design or proposal.

[Benkapner]

@rh-hemartin i've reframed it away from reading as a design or proposal. it now presents the record as an illustrative example, grounded in existing artifacts (OpenSSF Scorecard, in-toto/SLSA provenance), rather than a shape this repo is deciding on. dropped the "proposal" wording, the "design points" list, and the "alternatives considered" section. the doc's job here is to show what such a record can look like, not to pick one.

[waynesun09]

Follow-up review at head 895bfa4: 1 finding, posted inline.