Skip to content

fix: MMM 时间 holdout + 标准化 Ridge/Lasso + Hill 饱和预算优化#1

Merged
MeaFew merged 3 commits into
mainfrom
fix/mmm-holdout-and-saturation
Jun 24, 2026
Merged

fix: MMM 时间 holdout + 标准化 Ridge/Lasso + Hill 饱和预算优化#1
MeaFew merged 3 commits into
mainfrom
fix/mmm-holdout-and-saturation

Conversation

@MeaFew

@MeaFew MeaFew commented Jun 20, 2026

Copy link
Copy Markdown
Owner

改动

核心问题:MMM 的 R² 是 in-sample 重代入、三模型对比是假的、+1.8% 预算提升是线性外推虚高。

HIGH

  • H1 无 holdout:MMM 是日时间序列,旧代码在全量数据拟合后报 R²(纯 in-sample)。加按日期留出尾部 20% 的 holdout,同时报 in-sample 与 holdout R²/MAE。
  • H2 Ridge/Lasso 与 OLS 几乎相同:根因是未标准化特征 + alpha=1.0/0.1 小惩罚 → 系数几乎不动。改为 StandardScaler + TimeSeriesSplit 在 log 网格上选 alpha,三模型显著有别。
  • H6 线性预算优化:目标函数是线性的 → 最优解退化为平凡贪心重分配,外推到训练范围外得 +1.8% 虚高。改为 Hill 饱和响应函数,当前运营点几乎无重分配空间(≈0%),是诚实的结论。

其他

  • run_cross_brand_elasticity 同步用 chronological split。
  • 新增 TestChronologicalSplit、Ridge 收缩回归测试。

结果

  • MMM in-sample R² 0.569 → 0.542,holdout R² 0.440(此前无此数);Ridge 选 alpha=1000,系数与 OLS 显著不同。
  • 预算优化 +1.8% → ≈0%(饱和模型诚实结论)。
  • pytest 37/37 全绿,ruff 全绿。

Checklist

  • 代码改动有回归测试守护
  • README 数字来自重跑后的 data/processed/models/*.json
  • pytest + ruff 全绿

MeaFew added 3 commits June 21, 2026 01:46
…et response

- H1: add a chronological (time-respecting) train/test split for MMM; report
  both in-sample and holdout R^2/MAE so the generalization gap is visible.
  MMM is a daily time series, never random-split.
- H2: standardize features + select Ridge/Lasso alpha via TimeSeriesSplit CV
  over a log grid. Earlier, tiny alphas (1.0/0.1) on unscaled features left
  Ridge/Lasso coefficients ~identical to OLS (one model dressed as three).
- H6: replace the linear budget-response objective with a Hill saturation
  function (diminishing returns). The old +1.8% gain was a trivial linear-
  extrapolation artifact; the saturating model honestly shows ~0% reallocation
  headroom at the current operating point.
- Add chronological-split + shrinkage regression tests.
- Update README R^2 (in-sample 0.54 / holdout 0.44) and budget table.
test_preprocess.py reads the gitignored mmm_cleaned.parquet, which is absent
in CI (the mock-data step only stubs the model JSONs for test_attribution/
test_mmm). These tests now skip when the artifact is missing, matching the
pattern in the other repos. Note: this CI step was already failing on main.
@MeaFew MeaFew merged commit 1f6648c into main Jun 24, 2026
3 checks passed
@MeaFew MeaFew deleted the fix/mmm-holdout-and-saturation branch June 24, 2026 13:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant