-
Notifications
You must be signed in to change notification settings - Fork 0
docs: prepare M01 learner-validation evidence gate #3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: feature/m01-learning-validation
Are you sure you want to change the base?
Changes from all commits
d538184
9b68ef2
1130e22
b838294
261f260
82c4418
8d79022
ae769d5
ec4b33d
3ce50de
d869f96
f589a9f
f7a749e
158c404
00cce1c
d3001b3
4c8a02f
2fdd515
a49d3e8
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,231 @@ | ||
| # M01 Cohort Review Template | ||
|
|
||
| Use this document after the first learner cohort is complete. The first review should normally include at least **5 completed sessions**. | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a cohort has fewer than five completed sessions, “normally” still permits reviewers to complete this template and select A, whose authorization section then allows Phase 2. This contradicts the mandatory minimum in the validation protocol and Useful? React with 👍 / 👎. |
||
|
|
||
| This is a product/learning decision record, not a statistical significance report. | ||
|
|
||
| ## Cohort metadata | ||
|
|
||
| - **Review date:** | ||
| - **Sessions included:** | ||
| - **Sessions excluded:** | ||
| - **Reason for exclusions:** | ||
| - **Experience mix:** | ||
| - **Facilitator(s):** | ||
|
|
||
| ## Per-session summary | ||
|
|
||
| | Participant ID | Baseline | Post | Delta | Improved dimensions | Individual signal | Field transfer | Major friction | Delayed follow-up | | ||
| |---|---:|---:|---:|---:|---|---|---|---| | ||
| | | | | | | | | | | | ||
| | | | | | | | | | | | ||
| | | | | | | | | | | | ||
| | | | | | | | | | | | ||
| | | | | | | | | | | | ||
|
|
||
| ## L1 — Diagnostic reasoning delta | ||
|
|
||
| Review the direction and distribution of change rather than only the average. | ||
|
|
||
| - **Positive individual signals (`+3` and ≥2 improved dimensions):** | ||
| - **No-signal sessions:** | ||
| - **Negative-delta sessions:** | ||
| - **Median delta:** | ||
| - **Range:** | ||
|
|
||
| ### Dimension-level pattern | ||
|
|
||
| | Dimension | Improved | Unchanged | Worse | Interpretation | | ||
| |---|---:|---:|---:|---| | ||
| | Mechanism | | | | | | ||
| | Evidence | | | | | | ||
| | Trade-offs | | | | | | ||
| | Intervention | | | | | | ||
| | Change condition | | | | | | ||
|
|
||
| Questions: | ||
|
|
||
| 1. Is improvement broad across reasoning dimensions or concentrated in one item? | ||
| 2. Is any item too easy/hard to discriminate before and after learning? | ||
| 3. Is wording ambiguity a plausible alternative explanation for score movement? | ||
| 4. Does free-text reasoning support the same conclusion as multiple-choice scores? | ||
|
|
||
| ## L2 — Transfer | ||
|
|
||
| For participants who studied M01: | ||
|
|
||
| - **Credible field applications:** | ||
| - **Partial applications:** | ||
| - **Restatement-only / no transfer:** | ||
| - **Not completed:** | ||
|
|
||
| Prototype transfer rate: | ||
|
|
||
| ```text | ||
| credible or policy-defined completed field applications | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When a participant submits the field application before observing an outcome, the session template explicitly permits Useful? React with 👍 / 👎. |
||
| ------------------------------------------------------- | ||
| participants who studied the module | ||
| ``` | ||
|
|
||
| Do not interpret a filled form as transfer when intervention/evidence does not follow from the diagnosis. | ||
|
|
||
| ### Common transfer patterns | ||
|
|
||
| - Mechanisms learners identified: | ||
| - Interventions learners attempted: | ||
| - Signals/evidence learners used: | ||
| - Where learners reverted to symptom/person-level reasoning: | ||
|
|
||
| ## L3 — Reflection quality | ||
|
|
||
| Count/classify whether reflections show: | ||
|
|
||
| - changed diagnosis; | ||
| - changed planned action; | ||
| - new evidence requirement; | ||
| - testable remaining uncertainty; | ||
| - lesson restatement without mental-model change. | ||
|
|
||
| Key qualitative patterns: | ||
|
|
||
| ## L4 — Delayed retrieval / application | ||
|
|
||
| - **Delayed checks run:** | ||
| - **Independent transfer retained:** | ||
| - **Partial:** | ||
| - **Absent:** | ||
|
|
||
| Observations: | ||
|
|
||
| - Did participants reconstruct the model without course vocabulary prompts? | ||
| - Did transfer survive a different project domain? | ||
| - Did immediate post-case gains persist? | ||
|
|
||
| Do not introduce a `mastered` rule from a small cohort. Record evidence only. | ||
|
|
||
| ## Decision Drill review | ||
|
|
||
| ### `m01-drill-system` | ||
|
|
||
| - Preferred choice selected immediately by most learners: yes / no / unclear | ||
| - Meaningful misconception exposed: yes / no / mixed | ||
| - Feedback changed reasoning: yes / no / mixed | ||
| - Transfer contribution observed: yes / no / unclear | ||
| - Recommendation: keep / revise / remove | ||
| - Reason: | ||
|
|
||
| ### `m01-drill-diagnostic` | ||
|
|
||
| - Preferred choice selected immediately by most learners: yes / no / unclear | ||
| - Meaningful misconception exposed: yes / no / mixed | ||
| - Feedback changed reasoning: yes / no / mixed | ||
| - Transfer contribution observed: yes / no / unclear | ||
| - Recommendation: keep / revise / remove | ||
| - Reason: | ||
|
|
||
| ## Integrative case review | ||
|
|
||
| - Case felt structurally similar but non-identical to baseline: yes / no / mixed | ||
| - Case tested multiple concepts rather than recall: yes / no / mixed | ||
| - Scoring dimensions remained interpretable: yes / no / mixed | ||
| - Free-text diagnosis added useful evidence beyond choices: yes / no / mixed | ||
| - Recommendation: keep / revise / replace | ||
|
|
||
| ## Rubric review | ||
|
|
||
| For each dimension, note ambiguity, ceiling/floor effects, or mismatch between option score and observed reasoning. | ||
|
|
||
| | Dimension | Keep | Revise | Evidence | | ||
| |---|---|---|---| | ||
| | Mechanism | | | | | ||
| | Evidence | | | | | ||
| | Trade-offs | | | | | ||
| | Intervention | | | | | ||
| | Change condition | | | | | ||
|
|
||
| ### Promotion threshold review | ||
|
|
||
| The current `+3 total / ≥2 dimensions` rule is a development heuristic. | ||
|
|
||
| - Did it classify sessions in a way consistent with qualitative reasoning evidence? | ||
| - Did it create obvious false positives? | ||
| - Did it create obvious false negatives? | ||
| - Recommendation: keep for next prototype cycle / revise / stop using | ||
|
|
||
| Do not convert this threshold into mastery semantics. | ||
|
|
||
| ## Reliability / UX review | ||
|
|
||
| Count and classify: | ||
|
|
||
| - response/state loss; | ||
| - reload/navigation recovery failures; | ||
| - persistence errors; | ||
| - keyboard/accessibility blockers; | ||
| - route/sequence confusion; | ||
| - baseline contamination; | ||
| - abandonment points; | ||
| - wording/UI interfering with reasoning. | ||
|
|
||
| ### Blocking defects | ||
|
|
||
| List defects that invalidate or materially distort learner evidence: | ||
|
|
||
| ## Content-contract fit review | ||
|
|
||
| Start from `M01-READINESS-AUDIT.md` and record whether learner-driven revisions create a real new content-domain requirement. | ||
|
|
||
| ### Expected migration enrichments | ||
|
|
||
| These are not special cases by themselves: | ||
|
|
||
| - competency/outcome metadata; | ||
| - structured drill analysis; | ||
| - explicit rubric IDs/versioning; | ||
| - explicit field-application artifact/evidence/privacy metadata. | ||
|
|
||
| ### New exceptions discovered | ||
|
|
||
| List only requirements that cannot be represented cleanly by the current canonical entities: | ||
|
|
||
| ## Phase 1 exit decision | ||
|
|
||
| Choose exactly one: | ||
|
|
||
| ### A. Promote to Phase 2 | ||
|
|
||
| Use only when: | ||
|
|
||
| - learner evidence indicates meaningful reasoning improvement or useful discrimination; | ||
|
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
When cases distinguish stronger from weaker learners but the cohort shows flat or negative pre/post results, “useful discrimination” alone satisfies this learning criterion and can authorize Phase 2. The roadmap's Phase 1 exit gate requires the module to reveal and improve reasoning, while the existing protocol uses discrimination only to decide whether an individual interaction is useful. Keep discrimination as drill evidence, but require evidence of improvement for the module-level promotion decision. Useful? React with 👍 / 👎. |
||
| - field transfer is credible enough to preserve as a requirement; | ||
| - major learner friction is known and non-blocking; | ||
| - no unresolved content-domain ambiguity remains. | ||
|
|
||
| ### B. Revise and retest | ||
|
|
||
| Use when the learning mechanism looks promising but: | ||
|
|
||
| - wording/rubric distorts measurement; | ||
| - an interaction is weak; | ||
| - transfer is incomplete; | ||
| - UX/reliability interferes with evidence; | ||
| - content-contract assumptions changed materially. | ||
|
|
||
| ### C. Reject mechanism | ||
|
|
||
| Use when an interaction or module format adds complexity without producing useful reasoning or transfer evidence. | ||
|
|
||
| ## Recorded decision | ||
|
|
||
| - **Decision:** A / B / C | ||
| - **Rationale:** | ||
| - **Evidence supporting the decision:** | ||
| - **Required changes before next gate:** | ||
| - **Owner:** | ||
| - **Date:** | ||
|
|
||
| ## Phase 2 authorization | ||
|
|
||
| Phase 2 contract freeze is authorized only if the recorded decision is **A. Promote to Phase 2**. | ||
|
|
||
| If the decision is B or C, do not begin v1 domain freeze or framework selection. | ||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When a learner enters
#/validation/m01from a scrolled page or while the mobile drawer is open, this early return skips the base router'sscrollTo(0, 0)and mobile-menu reset.renderValidationRoute()only focuses#mainwithpreventScroll: trueand does not perform either cleanup, so the validation flow can open midway down the page or remain covered by the drawer; move the shared transition cleanup before this return or reproduce it in the extension.Useful? React with 👍 / 👎.