Skip to content

Real-OSS-repo case study #18

Description

@Connorrmcd6

STATUS (2026-06-14): deferred to stretch / v3. Per the approved v2 plan, the real-OSS case study is the highest-effort, hardest-to-grade item and is out of the core published study — which lands multi-turn (#102–#106) + multi-provider (#107–#108) + diverse synthetic scenarios (#104) first. Revisit as an external-validity credibility anchor once the core result holds. Original scope below.


Real-OSS-repo case study.

The Connorrmcd6/surface#11 suite is synthetic for control; this adds external validity with at least one real project.

  • pick a real repo + a genuine documented invariant; anchor it with a real hub
  • inject a logic drift surf check catches; generate the genuine surf_report.json
  • author a realistic task + a deterministic (or carefully-rubric'd) grader
  • run + report; note the higher noise vs synthetic and how it's handled

Metadata

Metadata

Assignees

No one assigned

    Labels

    researchSurface validation study

    Projects

    No projects

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions