Problem
Adaptive difficulty is described as reacting to recent session performance, but _compute_difficulty selects the ten most recently seen distinct words and calculates success from each word's lifetime aggregate counters. Old attempts therefore continue to dominate future decisions, repeated attempts on one word do not form a true rolling window, and difficulty can step again on every request even when little new evidence exists. The recommender also has no explanation payload or outcome-oriented offline evaluation.
This makes personalization hard to reason about and risks oscillation or inappropriate jumps for young learners.
Proposed change
- Derive adaptation from immutable recent attempt events (or explicit sessions), not lifetime per-word aggregates.
- Define a minimum evidence threshold, evaluation cadence, hysteresis/cooldown, and boundary behavior before changing levels.
- Separate placement difficulty, practice difficulty, and frustration intervention so one mechanism does not overwrite another unexpectedly.
- Return machine-readable recommendation reasons (due review, phonics gap, preferred theme, target difficulty) and stable tie-breaking.
- Add replay tooling that runs anonymized/synthetic attempt sequences through an algorithm version and reports level changes, review load, coverage, and oscillation.
- Version the algorithm and persist the version used for each decision so reports and experiments remain reproducible.
- Specify pedagogical invariants with an educator/product review rather than declaring thresholds optimal from unit tests alone.
Acceptance criteria
- A documented rolling-window/session policy is implemented and covered with timeline-based tests.
- Difficulty cannot change without the configured amount of new evidence and does not oscillate on alternating outcomes within the cooldown.
- Replaying the same event stream and algorithm version produces identical decisions and recommendation order.
- Recommendation responses expose reason codes/scores without leaking sensitive aggregate data.
- Existing profiles migrate safely, with an explicit fallback when historical event data is insufficient.
- A short design note documents assumptions, metrics, and educator validation still required.
Relevant code
agent/profiler.py (record_attempt, _compute_difficulty)agent/recommender.pyagent/experiments.pydashboard/experiment_report.py
Problem
Adaptive difficulty is described as reacting to recent session performance, but
_compute_difficultyselects the ten most recently seen distinct words and calculates success from each word's lifetime aggregate counters. Old attempts therefore continue to dominate future decisions, repeated attempts on one word do not form a true rolling window, and difficulty can step again on every request even when little new evidence exists. The recommender also has no explanation payload or outcome-oriented offline evaluation.This makes personalization hard to reason about and risks oscillation or inappropriate jumps for young learners.
Proposed change
Acceptance criteria
Relevant code
agent/profiler.py(record_attempt,_compute_difficulty)agent/recommender.pyagent/experiments.pydashboard/experiment_report.py