feat(trust-rollup): add trust scorecard rollup collector

[Benkapner]

Summary

Adds a stdlib-only collector that rolls up trust scorecards into a per-agent CSV. The scorecard artifact shape is the one proposed in the companion fullsend-ai/fullsend PR to trustworthiness-evidence.md (a record binding the five evidence types to a config hash plus a composition decision). This repo already tracks per-agent SDLC signals (PR type, rework rate); trust rollups are the natural home for "is this agent accumulating the evidence it needs for more autonomy?"

Changes

• scripts/collect-trust-rollup.py — reads a JSON array of scorecards and writes docs/trust-rollup.csv, one row per (repo, agent_role):
  • scorecards, latest_decision, signal_pass_rate, avg_config_health, avg_behavioral_eval, avg_track_record_revert_rate, blocking_signals.
  • stdlib only (argparse, csv, json, pathlib, collections), matching collect-pr-type.py conventions (ROOT-relative paths, *_HEADER constant, DictWriter).
• docs/trust-evidence.sample.json — a documented sample fixture (4 scorecards across 2 repos / 2 roles, including a failing and a partial signal) so the collector is runnable today.
• scripts/test_collect_trust_rollup.py — unittest, no network, mirrors test_collect_pr_type.py (loads the hyphenated module via importlib.util).

Why a fixture instead of a live source

There is no trust-evidence feed in the org yet. Rather than ship a collector that can't run, this ships a documented fixture and a --input flag; point it at a real feed when one exists and the rollup logic is unchanged.

Testing

$ python3 scripts/test_collect_trust_rollup.py
.........
Ran 9 tests in 0.005s
OK

$ python3 scripts/collect-trust-rollup.py
Wrote 3 rollup rows from 4 scorecards to docs/trust-rollup.csv

Sample output:

repo,agent_role,scorecards,latest_decision,signal_pass_rate,avg_config_health,avg_behavioral_eval,avg_track_record_revert_rate,blocking_signals
fullsend-ai/agents,review,2,sufficient,0.9,0.95,0.89,0.015,track_record
fullsend-ai/agents,triage,1,sufficient,1.0,0.9,0.83,,
fullsend-ai/fullsend,review,1,insufficient,0.6,0.4,0.8,0.08,config_health;track_record

Follow-up (not in this PR)

Wire into collect.yml / lib.sh once a real scorecard source is available. Kept out here to stay self-contained.

Checklist

• stdlib only, no new dependencies
• Tests pass, no network in tests
• DCO sign-off
• Conventional Commits title

[qodo-code-review]

PR Summary by Qodo

Add trust scorecard rollup collector

✨ Enhancement 🧪 Tests 🕐 10-20 Minutes

AI Description

• Aggregate trust scorecards into per-repository, per-agent CSV metrics for autonomy decisions.
• Support sample or supplied JSON inputs without introducing runtime dependencies.
• Cover grouping, recency, rates, averages, and blocking-signal behavior with unit tests.

Diagram

graph TD
  A[("Scorecard JSON")] --> B["JSON Loader"] --> C["Per-Agent Rollup"] --> D["CSV Writer"] --> E[("Rollup CSV")]
  T["Unit Tests"] -.-> B
  T -.-> C

Loading

High-Level Assessment

The fixture-backed, input-agnostic collector is the best current approach because no live trust-evidence feed exists. Keeping the implementation stdlib-only and deferring workflow wiring avoids premature source coupling while preserving a stable CLI for future automation.

Files changed (3) +269 / -0

Enhancement (1) +117 / -0

collect-trust-rollup.pyAggregate trust scorecards into per-agent CSV rows +117/-0

Aggregate trust scorecards into per-agent CSV rows

• Adds a stdlib-only CLI that accepts scorecard JSON, groups records by repository and agent role, and calculates decision, pass-rate, average-score, revert-rate, and blocking-signal metrics. It writes a stable CSV schema to a configurable output path.

scripts/collect-trust-rollup.py

Tests (1) +103 / -0

test_collect_trust_rollup.pyTest trust rollup loading and aggregation +103/-0

Test trust rollup loading and aggregation

• Adds network-free unittest coverage for wrapped JSON loading, numeric averaging, grouping, latest-decision selection, signal pass rates, score averages, and sorted blocker unions.

scripts/test_collect_trust_rollup.py

Documentation (1) +49 / -0

trust-evidence.sample.jsonAdd representative trust scorecard fixture +49/-0

Add representative trust scorecard fixture

• Provides four documented scorecards spanning multiple repositories and agent roles, including passing, partial, and failing evidence. The fixture makes the collector runnable before a live evidence source exists.

docs/trust-evidence.sample.json

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

1. Missing wrapper erases rollup ✓ Resolved 🐞 Bug ☼ Reliability

Description

load_scorecards() converts any JSON object lacking a scorecards key into an empty list, so a
mistyped or wrong feed shape succeeds and overwrites the output with a header-only CSV. The
collector should reject such input instead of reporting a successful zero-card run.

Code

scripts/collect-trust-rollup.py[R40-43]

+    if isinstance(data, dict):
+        data = data.get("scorecards", [])
+    if not isinstance(data, list):
+        raise ValueError(f"{path}: expected a JSON array of scorecards")

Relevance

●●● Strong

Accepted bug precedents favor rejecting malformed inputs and preventing silent empty outputs in
collectors.

PR-#7

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The parser defaults a missing wrapper member to []; main() then writes the resulting empty rows
and prints success. The supplied documented format is a top-level array, so an arbitrary object is
not a valid empty instance of that format.

scripts/collect-trust-rollup.py[38-44]
scripts/collect-trust-rollup.py[110-113]
docs/trust-evidence.sample.json[1-2]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
An object input without a `scorecards` member is silently interpreted as an empty feed and can replace a valid rollup with an empty CSV.

## Issue Context
Keep support for an intentional object wrapper if needed, but require `scorecards` to exist and be a list. Add a regression test for an object missing that key.

## Fix Focus Areas
- scripts/collect-trust-rollup.py[38-44]
- scripts/test_collect_trust_rollup.py[44-53]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Timezone offsets misorder decisions ✓ Resolved 🐞 Bug ≡ Correctness

Description

rollup() compares raw ISO-8601 strings, so valid timestamps with different UTC offsets can select
an older scorecard as latest. This makes latest_decision report a stale composition decision
even though all timestamps are valid.

Code

scripts/collect-trust-rollup.py[R62-63]

+        # Most recent by generated_at (ISO-8601 sorts lexically).
+        latest = max(cards, key=lambda c: (c.get("subject") or {}).get("generated_at") or "")

Relevance

●●● Strong

Recent collector precedent accepts date/time correctness fixes; parsing offsets is a deterministic
reliability improvement.

PR-#7

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The collector explicitly passes the raw generated_at string to max, then publishes the selected
card's decision. ISO-8601 lexical order is not chronological across offsets; for example,
2026-08-24T00:30:00+01:00 sorts after 2026-08-23T23:45:00Z despite representing an earlier
instant.

scripts/collect-trust-rollup.py[62-63]
scripts/collect-trust-rollup.py[80-85]
docs/trust-evidence.sample.json[4-16]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
`latest_decision` is selected by lexical comparison of `generated_at`, which does not preserve chronological order when ISO-8601 timestamps use different offsets.

## Issue Context
Parse each timestamp into a timezone-aware datetime and compare normalized instants. Reject missing, malformed, or timezone-naive timestamps rather than silently ordering them as empty strings.

## Fix Focus Areas
- scripts/collect-trust-rollup.py[62-63]
- scripts/test_collect_trust_rollup.py[77-79]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Context sources

Review mode: ⚖️ Balanced

Tip of the day

💡 Did you know, you can start a comment with 'qodo' or '@qodo' to chat about any finding

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗