Skip to content

Process idea: field testing a QEP while in Draft #14

Description

@mmcky

Idea

Add field testing to the QEP process: once a QEP is in Draft, run one or more real decisions against the draft text and record the outcome in the discussion PR, before the decision deadline. This would land as an in-place amendment to QEP-1's "How a QEP is decided" section if adopted.

Where the idea comes from

QEP-3 (#7) got two rounds of field testing during its comment window, and both were more informative than any amount of reading the text:

  • A validation. A live placement decision (project-reporting vs workspace-reporting) was run against the draft's goal-vs-fleet boundary rules, which decided it without interpretation. That is direct evidence the rules work in practice — exactly what a reviewer wants to know before accepting a standard, and something a read-through cannot establish.
  • A defect found while the draft was still cheap to change. Pulling on one thread the test exposed (reports-activity doesn't parse under the registry) revealed that the workflow-* type names a mechanism rather than a role, and produced a concrete restructure proposal (reporter-*/task-*) before acceptance rather than as a post-acceptance amendment.
  • An earlier field report on the same PR (the translation program's bench placement) surfaced two text gaps that were fixed in-flight (8150620).

The pattern generalises: a standard-type QEP exists to settle recurring decisions, so the natural test of a draft is to settle one real decision with it and see whether the text was sufficient. Every outcome is useful — a clean decision becomes validation evidence, a strained one becomes an amendment while amending is free.

Proposal sketch

Amend QEP-1's decision process with an optional step between Draft and Decide:

(Optional, encouraged for standards) Field test. Apply the draft to at least one real decision it claims to govern, and record the outcome as a field report comment on the discussion PR: what was decided, whether the text sufficed, and any amendments it motivated. Field reports are review evidence at the deadline.

Deliberately lightweight: no template, no gate, no new status — just a named, expected activity so authors and reviewers know it counts as review.

Points to decide

  1. Optional vs expected. Fully optional, or expected for type: standard QEPs (which govern recurring decisions and therefore always have a testable case) while remaining optional for process/informational QEPs?
  2. Interaction with the deadline. Leave the one-to-two-week window unchanged, or note that a substantive amendment arising from a field test should reset a short fresh window (as is happening on QEP-3: Repository Naming and Types #7)?
  3. Who runs it. The author testing their own draft is still valuable (both QEP-3: Repository Naming and Types #7 rounds were author-run), but a field test by a second maintainer is stronger evidence; worth encouraging without requiring.
  4. Naming. "Field test" / "field report" as used on QEP-3: Repository Naming and Types #7, unless someone prefers another term.

If there's support, I'll draft the QEP-1 amendment (version bump per QEP-1's own rules) as a PR.

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussDiscussion / decision threadqepQuantEcon Enhancement Proposal

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions