Replace adaptive-difficulty aggregate lookback with a rolling-window … - #42
Merged
llinsss merged 1 commit intoAug 31, 2026
Merged
Conversation
…policy Practice difficulty was recomputed on every attempt from each word's lifetime successes/attempts counters over the 10 most-recently-seen distinct words, so old attempts dominated recent ones, repeated attempts on one word never formed a real rolling window, and the level could flap on alternating outcomes with no evidence floor. - agent/profiler.py: _compute_difficulty now derives from a rolling window over the immutable attempt_log event stream, gated by minimum evidence, evaluation cadence, and a cooldown/hysteresis window before a level can change again; every evaluation is logged to a new difficulty_log with a reason code and the algorithm_version that produced it. record_attempt's mutation logic is extracted into a pure apply_attempt() so it can be driven identically by disk-backed and in-memory (replay) callers. - agent/experiments.py: registers the new rolling-window parameters and an algorithm_version per variant; documents why "control"'s difficulty semantics were intentionally redefined rather than held bit-identical. - agent/recommender.py: recommendations now carry machine-readable reason codes (due_review, phonics_gap, preferred_theme, target_difficulty, frustration_intervention) and a stable (score, word) tie-break, without exposing raw struggle/theme aggregate counts. Frustration intervention is split into its own function that only affects the transient recommendation-time target, never the persisted practice difficulty. - agent/replay.py (new): deterministic, disk-free replay of an attempt-event stream against a pinned algorithm version, reporting level changes, review load, coverage, and oscillation counts. - dashboard/experiment_report.py: per-variant reports now include algorithm_version and aggregate difficulty-dynamics (increases, decreases, oscillations). - Existing profiles migrate safely: missing fields backfill to empty/zero, and the evidence gate means a legacy profile with no attempt_log history holds its difficulty rather than guessing from stale aggregates. - ADAPTIVE_DIFFICULTY_DESIGN.md documents the policy, its assumptions, and what still needs educator/product validation. Tests: timeline-based coverage of the rolling-window policy (evidence, cadence, cooldown, boundary behavior, migration fallback), recommender reason codes and tie-break determinism, and replay determinism across repeated runs and variants.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adaptive difficulty is described as reacting to recent session performance, but _compute_difficulty selects the ten most recently seen distinct words and calculates success from each word's lifetime aggregate counters. Old attempts therefore continue to dominate future decisions, repeated attempts on one word do not form a true rolling window, and difficulty can step again on every request even when little new evidence exists. The recommender also has no explanation payload or outcome-oriented offline evaluation.
This makes personalization hard to reason about and risks oscillation or inappropriate jumps for young learners.
closes #24