fix(etl): flag multi-EIK authority rows so they don't form phantom authorities (#196) - #205
Closed
StanislavBG wants to merge 1 commit into
Closed
fix(etl): flag multi-EIK authority rows so they don't form phantom authorities (#196)#205StanislavBG wants to merge 1 commit into
StanislavBG wants to merge 1 commit into
Conversation
…thorities (midt-bg#196) ~406 of ~4874 authority_eik values are a list of several EIK joined by "; " (joint procurements naming multiple co-authorities on one notice). Keying 'auth:' || authority_eik on such a list minted one phantom authority for the whole list. Exclude list-valued rows from the authorities dedup (WHERE authority_eik NOT LIKE '%;%'); the tenders and contracts EXISTS-on-authority guards then keep those joint-procurement rows out of every served table and rollup. Raw staging is untouched, so they stay traceable. Full joint-procurement split (proposal 1) is deferred pending a product decision on modelling/value double-counting.
StanislavBG
added a commit
to StanislavBG/sigma-pr
that referenced
this pull request
Jul 3, 2026
Contributor
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes #196 (conservative option — proposal 2).
~406 of ~4874
authority_eikvalues (~8%) are a list of several ЕИК joined by „; " — joint procurements where one notice names multiple co-authorities, e.g.000590540; 000582020; 829091529(sourcedata/admin-tenders-load.sql:47202). Inscripts/normalize-raw.sqlstep 1, keying'auth:' || authority_eikon such a list minted one phantom authority for the whole list — a ЕИК that exists nowhere — and every downstream tender/contract/rollup then misattributed value to it.Fix
Exclude list-valued rows from the authorities dedup:
Because tenders (2a/2b) and contracts (5) only attach to an authority that
EXISTSin theauthoritiestable, dropping the phantom node here transitively keeps those joint-procurement rows out of the served tenders/contracts and every rollup — no fake single authority, no misattributed value. The raw staging rows (raw_contracts/raw_tenders) are left untouched, so the joint procurements stay fully traceable for the follow-up.Verification (real local D1, domain-only)
Applying the fix predicate to the produced
authorities.bulstat(=authority_eik):;Single-EIK authorities are unaffected; only the 405 phantom list nodes are removed.
Deferred (follow-up, proposal 1)
The full joint-procurement split — attributing a joint procurement to each real ЕИК (or a lead authority) without double-counting value in the totals — needs a product decision on how joint procurements are modelled, so it is intentionally not done here. A code comment in step 1 marks this deferral.