Fix: dupe-date-price heuristic no longer drops distinct parcels of a multi-parcel deed - #349
Conversation
|
Thank you for your contribution. I affirm that this contributor has signed the CLA You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot. |
…e deed run_heuristics' flag_dupe_date_price keyed duplicate detection on (sale_date, sale_price) alone. Distinct parcels conveyed together in a single multi-parcel deed share a sale date and price (per-parcel records frequently carry the deed total), so every lot but one was flagged as a duplicate and dropped -- silently discarding most comps of a subdivision sale. Include the parcel key in the signature so only true same-parcel repeats are flagged; distinct parcels are kept. Backward-compatible: only applied when a `key` column is present. Adds tests/test_sales_scrutiny_heuristics.py. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
f7c0647 to
66f7c2f
Compare
|
A data point in support of this PR and #351 together. We applied both on |
Problem
run_heuristics'sflag_dupe_date_pricekeyed duplicate detection on(sale_date, sale_price)alone (optionally prefixed byjurisdiction). Distinct parcels conveyed together in a single multi-parcel deed share a sale date and price — and per-parcel records frequently carry the deed total — so every lot but one of a subdivision sale was flagged as a "duplicate" and (whendrop=True) dropped. This silently discards most comparable sales of a subdivision, which is especially damaging for vacant-land / rural models that depend on those comps.Fix
A genuine duplicate is the same parcel reported more than once at the same date and price (duplicate data entry / shell trade); distinct parcels that merely share a date and price are not. The fix appends the parcel
keyto the duplicate signature when akeycolumn is present, so only true same-parcel repeats are flagged. Null keys fall back to the unique per-salekey_saleso they never collide.Behavior change (deliberate, not a no-op)
Multi-parcel-deed lots that were previously dropped now survive, increasing the sale count feeding downstream ratio studies and models. The previous behavior was over-aggressive; the new behavior is correct.
Scope
Fixes
flag_dupe_date_price(heuristic #2) only.flag_dupe_deed_date(heuristic #1) intentionally flags multi-parcel deeds bydeed_id + sale_date— that is its purpose (multi-parcel-sale detection), distinct from duplicate-report detection — so it is left unchanged.Tests
Adds
tests/test_sales_scrutiny_heuristics.py: distinct parcels in one deed survive; a same-parcel repeat is still dropped; one parcel sold twice on different dates is kept; the jurisdiction-scoped branch composes correctly. Backward-compatible: only applies when akeycolumn is present (always the case for hydrated OpenAVMKit sales).Note for reviewer
The duplicate signature is built by string concatenation to match the sibling
flag_dupe_deed_dateheuristic's existing style; happy to switch to agroupby(...).transform("size")form if preferred.