Fix VEI crash on degenerate market proxy (qcut 'Bin edges must be unique') - #369
Fix VEI crash on degenerate market proxy (qcut 'Bin edges must be unique')#369ABtheMD wants to merge 1 commit into
Conversation
…rate market proxy get_vertical_equity_scores crashed with "ValueError: Bin edges must be unique" whenever the VEI market proxy was degenerate but had >= 20 rows (so the small-sample guard didn't apply). This aborted the entire modeling run. Two triggers: - valuation all-zero/all-NaN (e.g. a tax-exempt model group with $0 assessed values) -> median_ratio == 0 -> market_proxy = sale*0.5 + val/0 = NaN for every row -> pd.qcut raises on all-NaN edges. - concentrated values (many identical/capped amounts, common in assessment rolls) -> repeated quantile edges -> pd.qcut raises even with enough distinct values overall. Fix: (1) return the NaN result dict (as already done for < 20 obs) when median_ratio is 0/non-finite or the proxy has < 2 distinct finite values; (2) pass duplicates="drop" to pd.qcut and index the actual top/bottom tiers (min/max labels) so concentrated data forms as many tiers as it can instead of crashing, returning NaN if fewer than 2 tiers result. Valid inputs are unaffected. Adds tests/test_vertical_equity.py covering all three degenerate cases plus a healthy-data control. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
Thank you for your contribution. I affirm that this contributor has signed the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
|
recheck |
|
The PR was generated by Claude, so I just wanted to add my own human understanding here. I was working on VEI metrics for each of the land classes, when the library crashed on exempt. Because the vast majority of the exempts have an assessment of $0, which ultimately gives us a whole lot of NaNs when we are trying to calculate the market_proxy. I suppose there's an alternative path where we exclude Exempts from VEI calculation, but this seems cleaner and more foolproof. |
Problem
get_vertical_equity_scores(openavmkit/vertical_equity_study.py) raisesValueError: Bin edges must be uniqueand aborts the entire modeling runwhenever the VEI "market proxy" is degenerate but the sample has ≥ 20 rows (so
the existing small-sample guard doesn't apply). Two real triggers:
group, whose assessed values are
$0. Thenmedian_ratio == 0, somarket_proxy = sale*0.5 + valuation/median_ratio = sale*0.5 + 0/0 = NaNforevery row, and
pd.qcutraises on all-NaN bin edges.(common in assessment rolls) put several quantile boundaries on the same
edge, so
pd.qcut(q=N)raises even when there are enough distinct valuesoverall.
Either way, one degenerate group takes down a run where every other group
modeled fine.
Fix
Purely defensive — valid inputs are unaffected; degenerate inputs now degrade to
a
NaNresult (exactly as the function already does for< 20observations)instead of raising:
NaNresult dict whenmedian_ratiois0/non-finite or themarket proxy has
< 2distinct finite values.duplicates="drop"topd.qcutand index the actual top/bottomtiers (
group_stats.index.min()/max()) rather than assuming labels0..percentile_group_count-1; returnNaNif fewer than 2 tiers can be formed.Tests
Adds
tests/test_vertical_equity.py:test_all_zero_valuation_returns_nan_not_crash(the tax-exempt repro)test_constant_market_proxy_returns_nan_not_crashtest_concentrated_values_do_not_crashtest_healthy_data_still_returns_finite_vei(control — happy path unchanged)All four pass;
test_ensemble.py,test_modeling.py,test_horizontal_equity.pystill green (no regression).
Found while running a full all-class county model (Berks County, PA) on v0.6.0.