Skip to content

Fix: dupe-date-price heuristic no longer drops distinct parcels of a multi-parcel deed - #349

Open
mividtim wants to merge 4 commits into
larsiusprime:masterfrom
fabrica-land:panforte/parcel-aware-dupe-date-price
Open

Fix: dupe-date-price heuristic no longer drops distinct parcels of a multi-parcel deed#349
mividtim wants to merge 4 commits into
larsiusprime:masterfrom
fabrica-land:panforte/parcel-aware-dupe-date-price

Conversation

@mividtim

@mividtim mividtim commented Jun 4, 2026

Copy link
Copy Markdown

Problem

run_heuristics's flag_dupe_date_price keyed duplicate detection on (sale_date, sale_price) alone (optionally prefixed by jurisdiction). Distinct parcels conveyed together in a single multi-parcel deed share a sale date and price — and per-parcel records frequently carry the deed total — so every lot but one of a subdivision sale was flagged as a "duplicate" and (when drop=True) dropped. This silently discards most comparable sales of a subdivision, which is especially damaging for vacant-land / rural models that depend on those comps.

Fix

A genuine duplicate is the same parcel reported more than once at the same date and price (duplicate data entry / shell trade); distinct parcels that merely share a date and price are not. The fix appends the parcel key to the duplicate signature when a key column is present, so only true same-parcel repeats are flagged. Null keys fall back to the unique per-sale key_sale so they never collide.

Behavior change (deliberate, not a no-op)

Multi-parcel-deed lots that were previously dropped now survive, increasing the sale count feeding downstream ratio studies and models. The previous behavior was over-aggressive; the new behavior is correct.

Scope

Fixes flag_dupe_date_price (heuristic #2) only. flag_dupe_deed_date (heuristic #1) intentionally flags multi-parcel deeds by deed_id + sale_date — that is its purpose (multi-parcel-sale detection), distinct from duplicate-report detection — so it is left unchanged.

Tests

Adds tests/test_sales_scrutiny_heuristics.py: distinct parcels in one deed survive; a same-parcel repeat is still dropped; one parcel sold twice on different dates is kept; the jurisdiction-scoped branch composes correctly. Backward-compatible: only applies when a key column is present (always the case for hydrated OpenAVMKit sales).

Note for reviewer

The duplicate signature is built by string concatenation to match the sibling flag_dupe_deed_date heuristic's existing style; happy to switch to a groupby(...).transform("size") form if preferred.

@github-actions

github-actions Bot commented Jun 4, 2026

Copy link
Copy Markdown
Contributor

Thank you for your contribution.
Please sign our CLA at the following link:
Click here to sign the CLA.

No action is required from you in this PR thread. Once you have signed the CLA externally, a maintainer will verify your signature and record it here on your behalf by commenting:


I affirm that this contributor has signed the CLA


You can retrigger this bot by commenting recheck in this Pull Request. Posted by the CLA Assistant Lite bot.

…e deed

run_heuristics' flag_dupe_date_price keyed duplicate detection on
(sale_date, sale_price) alone. Distinct parcels conveyed together in a single
multi-parcel deed share a sale date and price (per-parcel records frequently
carry the deed total), so every lot but one was flagged as a duplicate and
dropped -- silently discarding most comps of a subdivision sale. Include the
parcel key in the signature so only true same-parcel repeats are flagged;
distinct parcels are kept. Backward-compatible: only applied when a `key`
column is present.

Adds tests/test_sales_scrutiny_heuristics.py.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@fabrica-cc-engineering-agent
fabrica-cc-engineering-agent Bot force-pushed the panforte/parcel-aware-dupe-date-price branch from f7c0647 to 66f7c2f Compare June 22, 2026 20:40
@nikhilsi

nikhilsi commented Jul 4, 2026

Copy link
Copy Markdown

A data point in support of this PR and #351 together. We applied both on
top of 7952236 and ran them against another county's data: roughly 130K
parcels with about 24K sales in the modeling window. The (date, price)
dupe heuristic had been dropping around 12% of those sales as false
duplicates; with the parcel key from this PR they are retained, and the
orphaned bulk deeds that then surface are caught by the #351 scrutiny
check rather than leaking through. The ratio-study COD improved on the
larger sample. The two PRs complement each other and we would suggest
reviewing them as a pair.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants