feat(st5): harness scoreboard — 9 public harnesses, foreign-layout registry, 8 vendored fixtures

.gitignore

@@ -1,19 +1,20 @@
-# Ephemeral workbench
-.tmp/
+# Ephemeral workbench (root-anchored: examples/*-run fixtures may legitimately
+# contain same-named dirs — .claude/, .taskmaster/, memory/ — that MUST track)
+/.tmp/
 
 # Loop run telemetry (operating-contract state for the self-improvement run; not plugin content)
-.loop/
+/.loop/
 
 # Machine-local Claude Code config + GSD audit telemetry (not plugin content)
-.claude/
-.gsd/
+/.claude/
+/.gsd/
 
 # Local session memory / private build narrative (not plugin content)
-memory/
+/memory/
 
 # Workbench dirs (review scratch + roadmap drafts; not plugin content)
-review/
-roadmap/
+/review/
+/roadmap/
 
 # Python
 __pycache__/

docs/gap-reports/superpowers.md

@@ -98,10 +98,11 @@ That path is worked end-to-end for a real engine in
 
 ## This is a seed
 
-This is the first entry in an "inspect N public harnesses" scoreboard: read a
-foreign layout read-only, name the gaps a contract would close, keep every
-claim checkable against a vendored fixture. Contributions of further gap
-reports are welcome — this file is the template. See the drafted contributor
+This was the first entry in the "inspect N public harnesses" scoreboard —
+now live at [`scoreboard.md`](scoreboard.md) with nine harnesses read the same
+way: a foreign layout read read-only, the gaps a contract would close named,
+every claim checkable against a vendored fixture. Contributions of further gap
+reports are welcome — this file is the deep-dive template. See the drafted contributor
 issue
 [`docs/contributing/issues/06-help-wanted-gap-reports.md`](../contributing/issues/06-help-wanted-gap-reports.md)
 (`help wanted: gap reports`, filed on GitHub at release).

docs/superpowers/plans/2026-07-09-st5-harness-scoreboard.md

@@ -0,0 +1,39 @@
+# ST5 — harness scoreboard (plan)
+
+Spec: `../specs/2026-07-09-st5-harness-scoreboard-design.md`. Branch:
+`feat/st5-scoreboard`. Clones pinned in the session scratchpad; SHAs recorded
+in the spec table.
+
+## Stages
+
+1. **Prep (main loop).** Root-anchor `.gitignore` workbench patterns; create
+   the branch.
+2. **Per-harness pipeline (workflow, 8×).**
+   - *Analyst (opus, read-only):* read the pinned clone; return structured
+     layout facts — run-artifact paths, detect signature, success-criteria
+     vocabulary, any verification/terminal machinery (with file evidence),
+     fixture file plan, caveats.
+   - *Builder (opus):* write `examples/<name>-run/` — the shared fictional
+     CSV-dedupe task in the harness's documented layout + provenance README.
+     Zero verbatim template prose. ≤ 9 files.
+3. **Registry (workflow, 1×, barrier).** One engineer rewrites
+   `loop/foreign.py` as a layout registry (all 9 layouts; superpowers signal
+   tightened), extends `scripts/test_foreign_inspect.py`, runs the suite.
+4. **Score (main loop).** `loop inspect` + `loop doctor` on all fixtures;
+   capture verbatim JSON.
+5. **Adversarial verify (workflow, 8×).** Per row: fixture fidelity vs cloned
+   docs; no verbatim prose; notes capture machinery our signals miss; row
+   claims accurate. Fix-or-flag.
+6. **Post (main loop).** `docs/gap-reports/scoreboard.md` + HN draft in
+   `roadmap/launch/`.
+7. **Gates + PR.** validate_frontmatter · self_eval · full pytest (≥400 pass,
+   no new failures) · fixture inspect determinism. Atomic commits; PR without
+   auto-merge (human review gate).
+
+## Global constraints
+
+- Fixtures: content fictional and ours; structure theirs; provenance README
+  per fixture; no wikilinks needed (constraint currently scoped to skills/).
+- No scoring changes in `scripts/inspect_loop.py`.
+- Workflow `agent()` calls carry explicit `model:` (HARD CONTRACT); writes are
+  disjoint per harness; `foreign.py`/tests single-writer.

docs/superpowers/specs/2026-07-09-st5-harness-scoreboard-design.md

@@ -0,0 +1,104 @@
+# ST5 — "Inspect N public harnesses" scoreboard (design)
+
+**Date:** 2026-07-09 · **Slice:** ST5 (follows ST4's contributor funnel; the
+Superpowers gap report declared itself "the first entry in an 'inspect N public
+harnesses' scoreboard" — this slice builds the scoreboard.)
+
+## Problem
+
+`docs/gap-reports/superpowers.md` proved the pattern: read a foreign harness
+layout read-only, score what it *structurally cannot prove*, keep every claim
+checkable against a vendored fixture. One row is a seed, not a scoreboard. The
+launch plan (Show HN, human-gated) needs the N-harness post.
+
+Scoring foreign repos **without layout mappers would be dishonest**: the
+inspector reads contract-owned files (SPEC/WORKFLOW/TASKS/verify-*/RUNLOG), so
+a harness whose run leaves `specs/001-x/spec.md` on disk would score near-zero
+simply because our tool didn't read its files — a parsing artifact presented
+as a finding. `loop/foreign.py`'s own docstring states the principle: **a
+mapper, never a scorer.**
+
+## The 8 new harnesses (pinned 2026-07-09)
+
+| Harness | Repo | Stars | Pinned SHA |
+|---|---|---|---|
+| Spec Kit | github/spec-kit | 119k | `3f7392a` |
+| Agent OS | buildermethods/agent-os | 5.0k | `cae8e66` |
+| BMAD-METHOD | bmad-code-org/BMAD-METHOD | 50k | `49069b8` |
+| Task Master | eyaltoledano/claude-task-master | 28k | `c0c98d3` |
+| CCPM | automazeio/ccpm | 8.3k | `7d7e462` |
+| PRPs | Wirasm/PRPs-agentic-eng | 2.2k | `ada2f5b` |
+| OpenSpec | Fission-AI/OpenSpec | 60k | `93e27a7` |
+| ruflo (né claude-flow) | ruvnet/ruflo | 64k | `7ef4d4e` |
+
+Plus the existing Superpowers row (fixture, score 12) and calibration rows
+from this repo's own examples (`naive-loop` 0, `flaky-test-triage` 90).
+
+**Selection criteria:** public, active, substantial adoption, and the harness
+prescribes an on-disk layout for a run (specs/plans/tasks/state). Platforms
+whose run state lives off-repo (OpenHands, SWE-agent trajectories) are a
+methodology note, not scored rows.
+
+## Deliverables
+
+1. **`examples/<name>-run/` × 8** — vendored fixtures, each instantiating the
+   harness's *documented* run layout for the **same fictional CSV-dedupe task**
+   as `superpowers-run` (comparability: one task, nine layouts). Each fixture
+   carries a README with provenance (layout per `<repo>@<sha>`; all content
+   fictional). **Zero verbatim template prose** — BMAD / Task Master / PRPs are
+   not MIT-licensed; structure and headings follow their docs, every sentence
+   is ours.
+2. **`loop/foreign.py` → layout registry** — data-driven table of foreign
+   layouts (signature paths → LoopPaths mapping), superpowers + 8 new. Native
+   `.loop/state.json` always wins; detection precedence is deterministic;
+   `doctor` stays unmapped. **Scoring stays layout-blind** — the registry only
+   points existing signals at foreign files; no scoring changes in this slice.
+   Includes the roadmap follow-up: tighten the superpowers signature so this
+   repo's own root (specs/plans but no journal) no longer false-positives.
+3. **Tests** — per-fixture detect + advisory + deterministic-score tests in
+   `scripts/test_foreign_inspect.py`; a repo-root-negative regression.
+4. **`docs/gap-reports/scoreboard.md`** — the scoreboard post: table (stars,
+   SHA, target, score, verdict, terminal coverage), methodology, fairness
+   rules, per-harness sections with verbatim inspect JSON + a **notes** field
+   for verification machinery the harness has that our conservative signals
+   don't credit.
+5. **`.gitignore` root-anchoring** — `.loop/`→`/.loop/` etc., so fixture
+   dot-dirs (`.claude/epics/`, `.taskmaster/`) are trackable; resolves the
+   `!examples/*/.loop/**` follow-up.
+6. **HN post draft** → `roadmap/launch/` (gitignored; publication human-gated).
+
+## Fairness invariants (the post lives or dies on these)
+
+- **Advisory, always.** Every foreign row carries `advisory: true`. The score
+  measures *proof-of-done machinery visible on disk*, never project quality.
+- **Composes, doesn't compete.** Same framing as the Superpowers report: each
+  harness drives *how the agent works*; the contract proves *how the work
+  ended*. Any harness can emit the contract at its finish line.
+- **Version-specific claims only.** Every row is pinned to a SHA and a vendored
+  fixture; reproduce = `python3 -m loop inspect examples/<name>-run`.
+- **Conservative signals must not read as absences.** Where a harness defines
+  success under a different heading, or has verification machinery our gate
+  tokens can't see, the row's notes say so explicitly. Verifier agents hunt
+  for exactly this failure mode before any row ships.
+- **Fixtures are faithful.** A fixture must not omit an artifact a real run of
+  that harness would leave (especially verification artifacts), and must not
+  add anything the harness doesn't prescribe.
+
+## Non-goals
+
+- No scoring/weight changes to `inspect_loop.py`.
+- No claims about harness quality, community, or roadmap.
+- No mapper for off-disk-state platforms (methodology note instead).
+- Publication (HN/blog) is out of scope — human gate.
+
+## Risks
+
+- **Reputational:** naming popular projects with "weak" verdicts. Mitigation:
+  fairness invariants above + per-row adversarial verification + the same
+  respectful framing that shipped for Superpowers.
+- **Fixture infidelity:** a wrong fixture poisons a row. Mitigation: analyst →
+  builder → independent adversarial verifier per harness, all reading the
+  pinned clone.
+- **Registry false positives:** generic signatures (`docs/prd.md`) colliding.
+  Mitigation: require distinctive multi-path signatures; explicit precedence;
+  negative tests.

examples/agent-os-run/README.md

@@ -0,0 +1,52 @@
+# agent-os-run — a vendored foreign-harness fixture
+
+A minimal, sanitized run directory in the layout that
+[Agent OS](https://github.com/buildermethods/agent-os) v3.0 leaves behind
+(layout per `buildermethods/agent-os@cae8e664fb59a01869718c3151e0f45b7a06a2fb`,
+MIT). All content is fictional and every sentence is original — none of the
+harness's own template text is copied. It exists so `python3 -m loop inspect
+examples/agent-os-run` can score this layout read-only, holding the *same*
+fictional csv-dedupe task the other fixtures use, dated 2026-07-09.
+
+## The shared task
+
+Identical scenario to [`examples/superpowers-run`](../superpowers-run/): dedupe
+`import_contacts.py` so the two sample files collapse from 57 rows to 41 unique
+contacts, a second import inserts 0 new rows, and dropped duplicates are logged
+with their source line numbers. The scoreboard compares one task across nine
+harness layouts, so the story here matches — only the on-disk shape differs.
+
+## What a completed run looks like on disk (the differentiating finding)
+
+Agent OS v3 was refocused onto standards and spec-shaping; it retired the
+implementation, orchestration, and verification phases earlier versions shipped,
+along with the post-build recap. As a result **it records "done" nowhere on
+disk.** The spec folder under `agent-os/specs/2026-07-09-1030-csv-dedupe/` is
+authored up front — `plan.md` Task 1 is always "Save Spec Documentation" — and
+is never mutated afterward, so a finished run and an abandoned plan are
+byte-identical here. Completion for this run lived in two places the fixture
+cannot show: the git history of the code changes (outside the `agent-os/` tree
+entirely) and Claude Code's ephemeral in-tool todo list (never serialized).
+`plan.md` therefore lists tasks as plain `## Task N` headers with no checkboxes
+and no status field — faithful to the template, unlike the superpowers fixture's
+`- [x]` marks.
+
+## Composes, not competes
+
+Agent OS is a **complement**: a standards catalog plus a plan-mode spec-shaping
+workflow that governs how an agent starts work. Loop Engineer is the contract
+layer that proves how work ended. The honest low score here is not a verdict on
+Agent OS — it measures what a standards-and-shaping layout *structurally cannot
+prove*: there is no held-out gate, no typed terminal record, and no evidence
+trail, so nothing on disk distinguishes a verified finish from a claimed one.
+Its only gates are procedural (`/shape-spec` refuses to run outside plan mode)
+and human (AskUserQuestion confirm-before-create loops) — neither leaves an
+artifact. Emitting the contract is exactly what closes that gap.
+
+## Reproduce
+
+```bash
+uv run --with pyyaml python3 -B -m loop inspect examples/agent-os-run
+```
+
+A weak verdict (exit 1) is the expected, faithful result for this layout.

examples/agent-os-run/agent-os/product/mission.md

@@ -0,0 +1,23 @@
+# Product Mission
+
+> Fictional sample content for the vendored fixture. Not a real project.
+
+## Problem
+
+Small teams import contacts from spreadsheet exports that overlap. The same
+person appears in two files with different casing, so the importer writes the
+same contact twice and the address book fills with near-duplicates that are
+tedious to reconcile by hand.
+
+## Target Users
+
+Operators and sales assistants who bulk-load contact lists into a lightweight
+CRM and expect the import to be safe to run more than once without doubling
+their records.
+
+## Solution
+
+Every import normalizes and de-duplicates before it writes. A contact is keyed
+on its lowercased email and phone, the first occurrence wins, and later
+collisions are dropped and logged. Re-running the same source is a no-op, so
+imports become idempotent instead of destructive.

examples/agent-os-run/agent-os/product/roadmap.md

@@ -0,0 +1,16 @@
+# Product Roadmap
+
+> Fictional sample content for the vendored fixture. Not a real project.
+
+## Phase 1: MVP
+
+- CSV import from common spreadsheet exports
+- **De-duplication on import** — collapse overlapping rows into unique contacts
+- A drop log that records every discarded duplicate with its source line
+- Manual review of the imported list before it is committed
+
+## Phase 2: Post-Launch
+
+- Fuzzy matching on names to catch typo-level duplicates the key misses
+- Scheduled re-imports that stay idempotent across runs
+- Per-source merge rules so newer fields can override older ones

examples/agent-os-run/agent-os/specs/2026-07-09-1030-csv-dedupe/plan.md

@@ -0,0 +1,31 @@
+# CSV Dedupe — Plan
+
+> Fictional sample content for the vendored fixture. Not a real project.
+>
+> Tasks are plain headers. Agent OS v3 tracks progress in the tool's ephemeral
+> todo list, not on disk, so this file is never marked up as tasks complete —
+> it reads the same after the run as before it.
+
+## Task 1: Save Spec Documentation
+
+Create `agent-os/specs/2026-07-09-1030-csv-dedupe/` holding this plan plus
+`shape.md`, `standards.md`, and `references.md`. This runs first so the shaping
+work is captured before any code changes begin.
+
+## Task 2: Normalization key
+
+Add `normalize_key(email, phone)` returning `(email.strip().lower(),
+phone.strip().lower())`. Cover it with mixed-case and surrounding-whitespace
+fixtures so equal contacts collapse to one key regardless of how they were typed.
+
+## Task 3: Idempotent import
+
+Track seen keys during the load and skip the insert when a key already exists,
+counting each skip. Result: the two sample files produce 41 unique contacts from
+57 rows read, and a second import of the same files inserts 0 new rows.
+
+## Task 4: Dedupe log
+
+Append every dropped duplicate to `dedupe.log` as `<source>:<lineno> dropped
+(kept <first_lineno>)`. Result: each discarded row is traceable back to the
+source line it came from, and kept-plus-dropped equals rows read.

examples/agent-os-run/agent-os/specs/2026-07-09-1030-csv-dedupe/references.md

@@ -0,0 +1,23 @@
+# References for CSV Dedupe
+
+> Fictional sample content for the vendored fixture. Not a real project.
+
+## Similar Implementations
+
+### Existing contact importer
+
+- **Location:** `import_contacts.py`
+- **Relevance:** This is the module being changed. Its current load path parses
+  each CSV row and inserts it directly, with no membership check, which is the
+  source of the duplicate rows.
+- **Key patterns:** Reuse the existing CSV parsing and the row-to-record
+  mapping; wrap only the raw insert with the new key-membership guard so the
+  parsing behavior is unchanged.
+
+### Sample source files
+
+- **Location:** `fixtures/contacts_a.csv`, `fixtures/contacts_b.csv`
+- **Relevance:** The two overlapping exports that motivate the work — together
+  57 rows that should collapse to 41 unique contacts.
+- **Key patterns:** The overlap is same-person, different-casing, which is why
+  the key lowercases email and phone rather than comparing raw strings.

examples/agent-os-run/agent-os/specs/2026-07-09-1030-csv-dedupe/shape.md

@@ -0,0 +1,32 @@
+# CSV Dedupe — Shaping Notes
+
+> Fictional sample content for the vendored fixture. Not a real project.
+
+## Scope
+
+Make `import_contacts.py` de-duplicate as it loads. Today it writes one row per
+parsed line, so two overlapping source files produce doubles. After this change,
+importing the two sample files yields 41 unique contacts where 57 rows were
+read, a second import of the same files inserts 0 new rows, and every dropped
+duplicate is written to `dedupe.log` with the line number it came from. Those
+three outcomes are the definition of done for this work; Agent OS keeps them
+here as scope prose rather than under a separate acceptance heading.
+
+## Decisions
+
+- Key each contact on `(lower(email), lower(phone))` after trimming whitespace.
+- Keep the first row seen for a key; discard every later collision.
+- Log drops to `dedupe.log`; do not fail the import when duplicates appear.
+- Guard the existing insert with a key-membership check so re-runs are no-ops.
+
+## Context
+
+- **Visuals:** None — this is a backend import path with no UI surface.
+- **References:** the current `import_contacts.py` insert loop (see references.md).
+- **Product alignment:** the "De-duplication on import" line in Phase 1 of the
+  product roadmap; imports must be safe to run more than once.
+
+## Standards Applied
+
+- backend/data-imports — this is an importer, so its idempotency, normalized-key,
+  and drop-logging rules govern the whole change.

examples/agent-os-run/agent-os/specs/2026-07-09-1030-csv-dedupe/standards.md

@@ -0,0 +1,25 @@
+# Standards for CSV Dedupe
+
+> Fictional sample content for the vendored fixture. Not a real project.
+
+These are the standards that govern this change. Their full text is inlined
+below so the spec folder stands alone as build context.
+
+---
+
+## backend/data-imports
+
+Every importer is idempotent and de-duplicates before it writes.
+
+- Build a normalization key from the lowercased, trimmed identifying fields
+- Keep the first row seen for a key; drop any later row that collides
+- Never insert a row whose key already exists in the target table
+- Append each dropped row to a log with its source file and line number
+
+```python
+key = (email.strip().lower(), phone.strip().lower())
+```
+
+- Re-running an import over the same source inserts zero new rows
+- Drop-log format: one line per discard — `<source>:<lineno> dropped (kept <first_lineno>)`
+- The count of dropped rows plus the count of kept rows equals the rows read