fix(etl): eik_valid enforces Bulstat checksum (#195) - #207
Closed
StanislavBG wants to merge 1 commit into
Closed
Conversation
The bidder identity key trusted a purely SYNTACTIC ЕИК check (digits-only, length 9 or 13). The fake/service code „000000001" and wrong-digit typo twins passed it, so unrelated foreign suppliers (Elsevier + Clarivate/Web of Science + a gas consultancy + a 102 EUR construction line) collapsed onto one node labelled „Elsevier B. V." (14 contracts, 45.7M EUR), and real publishers grew a wrong-checksum twin (131106522 „Просвета" vs typo 131106552). Add the Bulgarian ЕИК/Булстат control-digit algorithm to eik_valid: - 9-digit: weight positions 1..8 by 1..8 (fallback 3..10; second 10 → 0) - 13-digit: valid leading 9-digit ЕИК, then positions 9..12 by 2,7,3,5 (fallback 4,9,5,7; second 10 → 0) Invalid codes now get eik_valid = 0 and fall back to the name-based key, which correctly splits the merged suppliers. Verified against the live corpus: 64 of 13134 previously-valid EIK nodes fail the checksum (000000001 and the Просвета twin among them); real EIKs (131106522, 2016195800141, 204799888, …) still pass. Alias-merge (proposal 2 in midt-bg#195) is intentionally left as follow-up.
Contributor
Author
|
Folded into #203 (checksum now lives in both normalize-raw.sql and refresh-slice.sql). Closing to keep the ETL identity work in a single PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem (#195)
eik_validinscripts/normalize-raw.sqlwas only a syntactic check — digits-only, length 9 or 13, no control digit:So the fake/service code „000000001" passed and became a valid
eik:key, merging unrelated foreign suppliers — Elsevier + Clarivate/Web of Science + an IGB gas consultancy + a 102 EUR construction line — into one node labelled „Elsevier B. V." (eik:000000001= 14 contracts, 45.7M EUR). Real publishers also grew a wrong-checksum twin (131106522„Просвета" vs typo131106552).Fix
Add the Bulgarian ЕИК/Булстат control-digit (checksum) algorithm to
eik_valid:Invalid codes now get
eik_valid = 0and fall back to the name-based key (name:), which correctly splits Elsevier from Clarivate from the gas consultancy, and the typo twin from the real publisher. Implemented as a pure SQLite expression over the digit positions;eik_validis computed once and the identity key references it.Verification (live corpus, real data)
Cross-checked the SQL expression against an independent JS reference over all 13,134 currently-valid EIK nodes: 0 mismatches; 64 previously-valid nodes correctly fail the checksum.
eik_valideik_valid0000000011311065521027809198339001868313648713110652220161958001418319178340012204799888End-to-end run of the actual edited
INSERTon a seededraw_contracts: the singleeik:000000001node disappears and its four unrelated suppliers split into distinctname:nodes; valid EIKs keep theireik:node.Scope
Checksum only. The optional alias-merge (proposal 2 in #195) is intentionally left as follow-up — it needs product input and careful handling.
Closes #195.