feat(data-analytics-demo): T-06 churn + T-07 upsell ML pipelines#87
Merged
Merged
Conversation
Phase 3 of the data-analytics-demo bolt-on. Trains two propensity models on the dbt marts shipped in #86 and saves the resulting model artifacts + SHAP summary that the narrative layer (T-08) consumes next. T-06 — Churn pipeline (AC-3.1〜3.5): - ml/churn.py — fits a LogisticRegression baseline AND an XGBoost classifier on `churn_features`, picks the higher hold-out ROC-AUC, and saves model.pkl + metadata.json + shap_summary.json. - ml/explain.py — SHAP wrapper used by both the churn and (later) narrative paths. TreeExplainer first, falls back to model-agnostic. - ml/_io.py — shared mart loader, fails with clear errors when the warehouse / mart is missing (AC-3.4). T-07 — Upsell propensity (AC-3.6〜3.7): - ml/upsell.py — fits a LogisticRegression propensity model on `upsell_opportunities`, measures hold-out ROC-AUC and lift @ top-10%, raises if the lift falls below the 1.5× floor. Data-generator amendment: the churn signal in `data/generate.py` was under-engineered (best ROC-AUC was 0.6972, just below the AC-3.2 0.70 floor). Reworked the event generator so churned customers (a) get 4× lower event weight and (b) have their timestamps biased into the older half of the history window. The mart's `recent_to_lifetime_ratio` feature now correlates cleanly with the cancel label, pushing churn ROC-AUC to 0.7448 on a seed=42 / n_customers=1000 run. Local verify (Python 3.12 venv, deterministic seed=42): - `make data` + `make dbt` + `make ml` end-to-end OK - Churn ROC-AUC = 0.7448 (LR wins; XGBoost 0.7196), AC-3.2 PASS - Upsell lift @ top-10% = 2.81× (vs 1.5× floor), AC-3.7 PASS - ruff OK / mypy OK / pytest 15 PASS, coverage 86.75% - doc-drift 0 fail / adr-claims 77/77 Test infra: switched from `subprocess.run(["dbt", ...])` to `dbt.cli.main.dbtRunner` so the fixtures work on Windows without venv Scripts being on PATH. DuckDB rw-mode for both dbt + ml avoids the "different configuration" connection error when both run in-process.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
leagames0221-sys
added a commit
that referenced
this pull request
May 17, 2026
…#88) Phase 4 of the data-analytics-demo bolt-on. Reads the SHAP summary that the ML layer (#87) writes, sends a templated prompt to a local Ollama daemon, and saves an executive-facing markdown narrative — never touching a cloud LLM API. What lands: - narrative/ollama_client.py — env-var-gated host/model resolution (defaults: localhost:11434 + llama3.1:8b-instruct-q4_K_M), AC-4.3 assertion that no cloud-LLM credentials are present at invocation, AC-4.2 remediation hint when Ollama is unreachable. - narrative/prompts.py — the executive-brief prompt template; SHAP-summary rendering is the only call point. - narrative/generate.py — orchestration. Reads shap_summary.json, builds the prompt, calls Ollama, wraps the body with provenance metadata (model id, SHAP source path, timestamp, "external calls: 0" assertion-enforced advertisement) — satisfies AC-4.1, AC-4.4, AC-4.5. - tests/test_narrative.py — 7 cases covering AC-4.1〜4.5 plus missing-data and prompt-builder paths. Uses monkeypatch to stub `ollama.Client` so no network is required in CI. - Makefile narrative target + cli.py narrative subcommand wire-up. AC coverage (mock-Ollama tests + real-Ollama smoke locally): - AC-4.1 produces output.md PASS - AC-4.2 unreachable Ollama PASS (clear RuntimeError with "ollama serve" hint) - AC-4.3 external API guard PASS (raises before any client call) - AC-4.4 cites shap_summary.json PASS - AC-4.5 model identifier in output PASS Local verify: - ruff OK / mypy OK (14 source files) / pytest 22 PASS / coverage 87.20% - Real smoke vs Ollama (gemma3:4b, env-var override): 3-paragraph executive narrative produced end-to-end, all metadata fields present. Design note: the literal AC-4.5 default model name is preserved as the package default; deployments running a different quantized variant can override via the OLLAMA_MODEL env var without code changes. Co-authored-by: leagames0221-sys <leagames0221@users.noreply.github.com>
2 tasks
leagames0221-sys
added a commit
that referenced
this pull request
May 21, 2026
Phase 3 of the data-analytics-demo bolt-on. Trains two propensity models on the dbt marts shipped in #86 and saves the resulting model artifacts + SHAP summary that the narrative layer (T-08) consumes next. T-06 — Churn pipeline (AC-3.1〜3.5): - ml/churn.py — fits a LogisticRegression baseline AND an XGBoost classifier on `churn_features`, picks the higher hold-out ROC-AUC, and saves model.pkl + metadata.json + shap_summary.json. - ml/explain.py — SHAP wrapper used by both the churn and (later) narrative paths. TreeExplainer first, falls back to model-agnostic. - ml/_io.py — shared mart loader, fails with clear errors when the warehouse / mart is missing (AC-3.4). T-07 — Upsell propensity (AC-3.6〜3.7): - ml/upsell.py — fits a LogisticRegression propensity model on `upsell_opportunities`, measures hold-out ROC-AUC and lift @ top-10%, raises if the lift falls below the 1.5× floor. Data-generator amendment: the churn signal in `data/generate.py` was under-engineered (best ROC-AUC was 0.6972, just below the AC-3.2 0.70 floor). Reworked the event generator so churned customers (a) get 4× lower event weight and (b) have their timestamps biased into the older half of the history window. The mart's `recent_to_lifetime_ratio` feature now correlates cleanly with the cancel label, pushing churn ROC-AUC to 0.7448 on a seed=42 / n_customers=1000 run. Local verify (Python 3.12 venv, deterministic seed=42): - `make data` + `make dbt` + `make ml` end-to-end OK - Churn ROC-AUC = 0.7448 (LR wins; XGBoost 0.7196), AC-3.2 PASS - Upsell lift @ top-10% = 2.81× (vs 1.5× floor), AC-3.7 PASS - ruff OK / mypy OK / pytest 15 PASS, coverage 86.75% - doc-drift 0 fail / adr-claims 77/77 Test infra: switched from `subprocess.run(["dbt", ...])` to `dbt.cli.main.dbtRunner` so the fixtures work on Windows without venv Scripts being on PATH. DuckDB rw-mode for both dbt + ml avoids the "different configuration" connection error when both run in-process. Co-authored-by: leagames0221-sys <leagames0221@users.noreply.github.com>
leagames0221-sys
added a commit
that referenced
this pull request
May 21, 2026
…#88) Phase 4 of the data-analytics-demo bolt-on. Reads the SHAP summary that the ML layer (#87) writes, sends a templated prompt to a local Ollama daemon, and saves an executive-facing markdown narrative — never touching a cloud LLM API. What lands: - narrative/ollama_client.py — env-var-gated host/model resolution (defaults: localhost:11434 + llama3.1:8b-instruct-q4_K_M), AC-4.3 assertion that no cloud-LLM credentials are present at invocation, AC-4.2 remediation hint when Ollama is unreachable. - narrative/prompts.py — the executive-brief prompt template; SHAP-summary rendering is the only call point. - narrative/generate.py — orchestration. Reads shap_summary.json, builds the prompt, calls Ollama, wraps the body with provenance metadata (model id, SHAP source path, timestamp, "external calls: 0" assertion-enforced advertisement) — satisfies AC-4.1, AC-4.4, AC-4.5. - tests/test_narrative.py — 7 cases covering AC-4.1〜4.5 plus missing-data and prompt-builder paths. Uses monkeypatch to stub `ollama.Client` so no network is required in CI. - Makefile narrative target + cli.py narrative subcommand wire-up. AC coverage (mock-Ollama tests + real-Ollama smoke locally): - AC-4.1 produces output.md PASS - AC-4.2 unreachable Ollama PASS (clear RuntimeError with "ollama serve" hint) - AC-4.3 external API guard PASS (raises before any client call) - AC-4.4 cites shap_summary.json PASS - AC-4.5 model identifier in output PASS Local verify: - ruff OK / mypy OK (14 source files) / pytest 22 PASS / coverage 87.20% - Real smoke vs Ollama (gemma3:4b, env-var override): 3-paragraph executive narrative produced end-to-end, all metadata fields present. Design note: the literal AC-4.5 default model name is preserved as the package default; deployments running a different quantized variant can override via the OLLAMA_MODEL env var without code changes. Co-authored-by: leagames0221-sys <leagames0221@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Phase 3 of the data-analytics-demo bolt-on. Trains two propensity models on the dbt marts that #86 shipped, and saves the model + SHAP summary that the next phase (T-08 LLM narrative) will consume.
What lands
src/data_analytics_demo/ml/_io.pysrc/data_analytics_demo/ml/explain.pytop_featuresJSON the narrative layer can read directlysrc/data_analytics_demo/ml/churn.pysrc/data_analytics_demo/ml/upsell.pytests/test_ml_churn.py(5 cases) +tests/test_ml_upsell.py(2 cases)ml+cli.py mlAC coverage (seed=42, n_customers=1000)
FileNotFoundErrorw/ remediation hintGenerator amendment
The original churn signal in
data/generate.pywas under-engineered — best ROC-AUC was 0.6972, narrowly below the 0.70 floor. Reworked_generate_events:The
recent_to_lifetime_ratiofeature inchurn_featuresnow carries a real signal, lifting churn ROC-AUC to 0.7448.Test-infra notes
subprocess.run(["dbt", ...])todbt.cli.main.dbtRunnerso the test works on Windows without the venv Scripts dir on PATH.read_only=Trueso it can coexist with dbt's in-process adapter — DuckDB refuses to open the same file with mismatched configurations.Local verify
make data+make dbt+make ml— end-to-end OKruff+mypy --strict— OK on 10 source filespytest— 15 passed, coverage 86.75% (≥ 80% floor)check-doc-drift.mjs— 0 failure / 0 warningcheck-adr-claims.mjs— 77/77 PASSTest plan