feat(st5): harness scoreboard — 9 public harnesses, foreign-layout registry, 8 vendored fixtures - #41
Conversation
…ch gitignore patterns Root-anchoring .loop/ .claude/ .gsd/ memory/ review/ roadmap/ .tmp/ lets examples/*-run fixtures track same-named dot-dirs (resolves the '!examples/*/.loop/**' roadmap follow-up at the root).
…un fixtures loop/foreign.py becomes a data-driven registry — detection predicate + per-role resolvers per layout; mapper-never-scorer preserved; scoring stays layout-blind; doctor stays unmapped; native .loop/state.json always wins. Superpowers detection tightened per the roadmap follow-up (journal required), fixing the own-root false positive. tasks maps only where the ledger is genuine JSON (task-master); markdown/YAML ledgers surface via runlog. Fixtures instantiate the same fictional csv-dedupe task as a completed run per harness's documented layout at a pinned SHA; structure theirs, every sentence ours (BMAD/Task Master/PRPs are not MIT). +31 tests: per-fixture detect/advisory/determinism + repo-root-negative regression. Suite 431/9.
docs/gap-reports/scoreboard.md: 9 public harnesses (~590k combined stars, pinned SHAs) + 2 calibration rows, scored advisory-only against the vendored fixtures. Headline finding: 0/7 typed terminal states on every foreign row — done is always expressible, typed failure never is. Per-row fairness notes from adversarial passes over each pinned source name the real verification machinery the conservative signals cannot credit (console-only gates, off-disk state, self-asserted verdicts). Adds the prp-run fixture (verifier-corrected: real stdout run log, pre-archive plan paths, licensing rewording) and links the scoreboard from the superpowers seed report.
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 193bbe5ed0
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
|
|
||
| import csv | ||
|
|
||
| from src.import_contacts import import_contacts, normalize_key |
There was a problem hiding this comment.
When this new fixture test is collected from the repository root, e.g. python -m pytest -q examples/bmad-run/tests/test_import_contacts.py, collection fails with ModuleNotFoundError: No module named 'src' because pytest puts the repo root on sys.path, not examples/bmad-run. This prevents anyone running or validating the vendored BMAD sample tests outside that fixture directory; add the fixture root to sys.path, package the sample, or import the module by path.
Useful? React with 👍 / 👎.
What
The roadmap's "inspect N public harnesses" scoreboard post (the seed was
docs/gap-reports/superpowers.md, PR #33). Three commits:a01bccc— ST5 spec+plan; root-anchor workbench .gitignore patterns (unanchored.claude//.loop//memory/matched at any depth and would have silently swallowed fixture dot-dirs; resolves the!examples/*/.loop/**follow-up).8dbbb03—loop/foreign.py→ data-driven layout registry (9 layouts: superpowers + spec-kit, agent-os, bmad, task-master, ccpm, prp, openspec, ruflo). Mapper-never-scorer preserved; scoring layout-blind; doctor unmapped; native contract always wins. Superpowers detection tightened (journal required) — fixes the own-root false-positive follow-up. 7 vendored fixtures + 31 tests.193bbe5—docs/gap-reports/scoreboard.md(the post) + the verifier-correctedprp-runfixture + seed-report link.Methodology / fairness
verdict.json).Results
Headline: 0/7 typed terminal states on all nine harnesses (~590k stars). The 0-vs-12 split is a heading-vocabulary match, not a discipline ranking — stated prominently in the doc.
Evidence
self_eval.py13/13;validate_frontmatter.py9/9.foreign_layoutdetection.Held for the operator
roadmap/launch/show-hn-scoreboard.md; publication remains human-gated, andpip install loop-engineerin the draft is only true after the PyPI tags/publisher gate from the v0.8.0 release.Follow-ups (non-blocking)
## Success Measures/## Acceptance Criteriavocabulary extension to the defines_success signal — deliberately NOT done here (no scoring changes in a scoreboard slice).