Skip to content

fix(etl): eik_valid enforces Bulstat checksum (#195) - #207

Closed
StanislavBG wants to merge 1 commit into
midt-bg:mainfrom
StanislavBG:fix/eik-checksum-validation
Closed

fix(etl): eik_valid enforces Bulstat checksum (#195)#207
StanislavBG wants to merge 1 commit into
midt-bg:mainfrom
StanislavBG:fix/eik-checksum-validation

Conversation

@StanislavBG

Copy link
Copy Markdown
Contributor

Problem (#195)

eik_valid in scripts/normalize-raw.sql was only a syntactic check — digits-only, length 9 or 13, no control digit:

CASE WHEN eik_clean NOT GLOB '*[^0-9]*' AND LENGTH(eik_clean) IN (9, 13) THEN 1 ELSE 0 END

So the fake/service code „000000001" passed and became a valid eik: key, merging unrelated foreign suppliers — Elsevier + Clarivate/Web of Science + an IGB gas consultancy + a 102 EUR construction line — into one node labelled „Elsevier B. V." (eik:000000001 = 14 contracts, 45.7M EUR). Real publishers also grew a wrong-checksum twin (131106522 „Просвета" vs typo 131106552).

Fix

Add the Bulgarian ЕИК/Булстат control-digit (checksum) algorithm to eik_valid:

  • 9-digit — weight positions 1..8 by 1..8; control = sum % 11. If that is 10, re-weight by 3..10; a second 10 → 0. The 9th digit must equal the control.
  • 13-digit — the leading 9 digits must themselves be a valid 9-digit ЕИК, then weight positions 9..12 by 2,7,3,5 (fallback 4,9,5,7; second 10 → 0); the 13th digit must equal that control.

Invalid codes now get eik_valid = 0 and fall back to the name-based key (name:), which correctly splits Elsevier from Clarivate from the gas consultancy, and the typo twin from the real publisher. Implemented as a pure SQLite expression over the digit positions; eik_valid is computed once and the identity key references it.

Verification (live corpus, real data)

Cross-checked the SQL expression against an independent JS reference over all 13,134 currently-valid EIK nodes: 0 mismatches; 64 previously-valid nodes correctly fail the checksum.

EIK label old eik_valid new eik_valid
000000001 fake/service (Elsevier merge node) 1 0
131106552 „Просвета" wrong-checksum typo twin 1 0
1027809198339 Russian ОАД (foreign 13-digit) 1 0
0018683136487 foreign 13-digit 1 0
131106522 „Просвета - София" АД (real) 1 1
2016195800141 ДГС Батак (real 13-digit branch) 1 1
8319178340012 ТУ-НИС (real 13-digit branch) 1 1
204799888 ЕМДИ Пропърти ООД (real 9-digit) 1 1

End-to-end run of the actual edited INSERT on a seeded raw_contracts: the single eik:000000001 node disappears and its four unrelated suppliers split into distinct name: nodes; valid EIKs keep their eik: node.

Scope

Checksum only. The optional alias-merge (proposal 2 in #195) is intentionally left as follow-up — it needs product input and careful handling.

Closes #195.

The bidder identity key trusted a purely SYNTACTIC ЕИК check (digits-only,
length 9 or 13). The fake/service code „000000001" and wrong-digit typo twins
passed it, so unrelated foreign suppliers (Elsevier + Clarivate/Web of Science
+ a gas consultancy + a 102 EUR construction line) collapsed onto one node
labelled „Elsevier B. V." (14 contracts, 45.7M EUR), and real publishers grew a
wrong-checksum twin (131106522 „Просвета" vs typo 131106552).

Add the Bulgarian ЕИК/Булстат control-digit algorithm to eik_valid:
- 9-digit: weight positions 1..8 by 1..8 (fallback 3..10; second 10 → 0)
- 13-digit: valid leading 9-digit ЕИК, then positions 9..12 by 2,7,3,5
  (fallback 4,9,5,7; second 10 → 0)

Invalid codes now get eik_valid = 0 and fall back to the name-based key, which
correctly splits the merged suppliers. Verified against the live corpus: 64 of
13134 previously-valid EIK nodes fail the checksum (000000001 and the Просвета
twin among them); real EIKs (131106522, 2016195800141, 204799888, …) still pass.

Alias-merge (proposal 2 in midt-bg#195) is intentionally left as follow-up.
@StanislavBG

Copy link
Copy Markdown
Contributor Author

Folded into #203 (checksum now lives in both normalize-raw.sql and refresh-slice.sql). Closing to keep the ETL identity work in a single PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

1 participant