feat(etl): канонична идентичност на субектите — име и вид по мода by todorkolev · Pull Request #251 · midt-bg/sigma

todorkolev · 2026-07-17T01:31:35Z

Част от разделянето на #203 на фокусирани PR-и по проблем. Този PR прави идентичността на субектите канонична и консистентна: име на възложител, име на фирма и вид на възложителя вече се избират по честотна мода, а не с MIN()/MAX().

Проблемът

normalize-raw.sql избираше показвания надпис с агрегат, който изхвърля всички други варианти:

Име на възложител — MIN(authority_name) връщаше азбучно първия низ. При споделен ЕИК (второстепенни разпоредители под ЕИК на родителя) министерството излизаше под името на училище. Например под ЕИК 000695114 МОН (285 договора, ~296 млн. €) се показваше като „БСУ Д-р Петър Берон".
Име на фирма — MIN(contractor_name) (normalize-raw.sql:309) е същият бъг, оставен жив за изпълнителите: 995 от 13 091 ЕИК-фирми показват немодален (често с артефактни кавички/интервали) етикет. Изхвърлените варианти носят и реална информация - членове на консорциуми, представлявани чужди принципали.
Вид на възложителя — MAX(authority_type) (normalize-raw.sql:105) дава на субекта името на едно лице и вида на друго; изкривява bucket-а „държавна компания" (~11.2 млрд. €).

Решението

Един шаблон навсякъде - честотна мода с детерминистична верига за равенство:

ROW_NUMBER() OVER (PARTITION BY <ключ> ORDER BY
  COUNT(*) DESC,                                        -- мода: най-честият вариант
  CASE WHEN name GLOB '*[a-zа-я]*' THEN 0 ELSE 1 END,   -- смесен регистър пред ВСИЧКО-ГЛАВНИ (кирилица-safe; UPPER не сгъва кирилица)
  LENGTH(name) DESC, name)                              -- по-описателно, после детерминистичен ключ

Възложител (име): scratch authority_canonical_name заменя MIN(authority_name).
Фирма (име): нов scratch bidder_canonical_name над същия почистен subquery, който дава bidder_key, заменя MIN(contractor_name).
Вид: модален вид per ЕИК, избран измежду редовете с модалното име (име и вид от един и същ субект); „държавна компания" се извежда през EXISTS над суровия набор типове, не от една свита стойност.

Ключовете остават ЕИК-базирани → нула промяна на URL-и и връзки; сменят се само надписите. Всяка промяна е огледана в refresh-slice.sql (parity тест го заковава).

Как работи след фикса

Профилите на фирми и възложители показват най-често записаното (обикновено най-чисто) име и коректния вид; идентификаторите и сумите не мърдат. Проверено в браузър за случая МОН - показва министерството, не училището.

Кредит: диагнозата и SQL-ът за каноничното име на възложителя са от #203 (@StanislavBG); тук се изважда като фокусиран PR и се добавят двата асиметрично пропуснати близнака (име на фирма, вид).

@StanislavBG

Authority name, bidder name, and authority type now use the most-frequent value per ЕИК with a deterministic tiebreak (mixed-case over ALL-CAPS via a Cyrillic-aware GLOB, then longer, then lexical), mirrored in normalize-raw.sql and refresh-slice.sql. Keys stay ЕИК-based so URLs are unaffected. Extracts the canonical-authority-name fix from #203 (@StanislavBG) and adds the two asymmetrically-missed siblings: bidder name (was MIN(contractor_name)) and authority type (was MAX(authority_type), which also mis-drove the state-company bucket).

lyubomir-bozhinov

Прегледах стриктно на връх ce14433. Модовата логика е издържана.

Каноничният избор е коректен и детерминиран: ROW_NUMBER() OVER (PARTITION BY authority_eik ORDER BY cnt DESC, <mixed-case преди ALL-CAPS>, LENGTH DESC, name) — честотна мода, после презентационни tiebreak-ове. Заменя MIN(), който подбираше подвеждащ надпис при споделен ЕИК (#194 — поръчки на МОН под име на училище). Видът се държи модален в рамките на редовете с печелившото име, тъй че двете канонични полета описват един субект. Добре.

Трябва да се съгласува с #252 (пресича се): bidder_canonical_name тук деривира bidder_key със СТАРАТА синтактична ЕИК-проверка (NOT GLOB … AND LENGTH IN (9,13)), а #252 сменя същата деривация на checksum-валидна. Двата PR-а редактират един и същ bidder-блок в normalize-raw.sql. При merge каноничното-име стъпката трябва да приеме checksum-ключа на #252 — иначе checksum-невалиден ЕИК получава различен bidder_key в двете стъпки и LEFT JOIN bcn.bidder_key = b.bidder_key се разминава. Дефинирай реда на merge на серията (#251/#252/#253) и rebase-ни.

Логиката е добра. Одобрявам с уговорката за реда спрямо #252.

…cal authority Under #194 the authority display name is canonicalized to the frequency mode per ЕИК, which collapses genuine sub-units that share one ЕИК (e.g. a school placed under МОН's ЕИК keeps the ministry name). Additionally keep the raw per-row "Възложител" verbatim so no ordering unit is lost. - add contracts.ordering_unit_name / tenders.ordering_unit_name, filled from the raw authority_name in normalize-raw and both refresh-slice paths - getContract surfaces it on the contract page only when it differs from the canonical name after a Cyrillic-safe fold (case + whitespace), so spelling variants stay hidden and real sub-units show - /contracts/:id.json sourceNames.authority is now genuinely verbatim; it previously served the canonical name despite the "verbatim" contract Extends #251; credit StanislavBG's #203 identity work.

lyubomir-bozhinov

Прегледах новия комит 45b100a (verbatim „Възложител по документа" до каноничното име). Продуктовата идея е добра и коректно изолирана — каноничният избор, който одобрих на ce14433, е непокътнат: authority_id пак е по ЕИК, authorities.name пак е по mode, а ordering_unit_name е чисто additive и се показва само когато foldName(source) !== foldName(canonical). Тестът добавя правилните под-единични случаи. Дотук — добре.

Блокер преди merge — миграцията липсва, ETL-ът ще падне на прод. Новата колона ordering_unit_name е добавена само в 0000_init.sql (на tenders и contracts) и няма нов номериран migration файл. 0000_init вече е приложен на обслужващото D1 (проследява се по име в d1_migrations), тъй че редакцията му е no-op за прод — колоната никога не се създава там. А новият SQL я пише на две места срещу мигрираното D1:

Cron/slice път (import.mjs default): wrangler d1 migrations apply → после runSliceDerive/runFullDerive изпълняват refresh-slice.sql/normalize-raw.sql през wrangler d1 execute директно срещу обслужващото D1. Новият INSERT INTO contracts (... ordering_unit_name ...) → table contracts has no column named ordering_unit_name → целият import пада.
ship-domain.mjs път: списъкът с колони се чете от локалното work-DB (tableColumns → PRAGMA table_info върху workDb, ship-domain.mjs:107/194), а work-DB е построено от 0000_init → включва ordering_unit_name. Полученият INSERT INTO "contracts" (...,"ordering_unit_name",...) (ship-domain.mjs:143) се изпълнява срещу мигрираното D1, което няма колоната → същата грешка, ship-ът пада на non-zero exit.

Т.е. следващият production import след merge спира — не частична грешка, а пълен провал на обновяването.

Посока за фикс (по установената конвенция). 0001_flow_pairs_bidder_index.sql показва модела: нови обекти влизат с нов номериран migration, а 0000_init се пази замразен. Точно както #257 добави current_value_currency през отделен migration (ALTER TABLE … ADD COLUMN), тук трябва нов файл, който да ALTER-не tenders и contracts. Внимание към дупликацията, която тук хапе: 0000_init храни и work-DB-то (runWorkBackfill) и SQL-тестовете, които не прилагат migrations, тъй че колоната им трябва в 0000_init; но при fresh deploy миграциите ще пуснат 0000_init (с колоната) и после ALTER-а → duplicate column. SQLite няма ADD COLUMN IF NOT EXISTS, тъй че двете места не се комбинират наивно — избери разрешението, което екипът ползва за колони, не като за индекс. Ключовото е обслужващото D1 да получи колоната по миграционния път.

Каноничната логика е чиста; блокирам само заради миграционната дупка. Оправи миграцията и одобрявам.

(Серийна бележка: #251/#252/#253 пипат общо normalize-raw.sql/refresh-slice.sql → нужен е дефиниран merge ред + rebase; canonical-name стъпката тук да поеме checksum bidder-key-а от #252 на merge.)

A raw authority_eik can carry a joint-procurement composite ('EIK1; EIK2'). The rewritten canonical name/kind sources and the main authority insert consumed it verbatim, minting orphan 'auth:EIK1; EIK2' authorities referenced by no tender, contract, rollup or search row - a full 2020-2026 rebuild measured 404 of them. Guard every minting source with NOT LIKE '%;%'; joint tenders are attributed through their individual members.

todorkolev · 2026-07-23T19:31:48Z

Правилно наблюдение за принципа, но тук поемаме по друг път и той е безопасен - проверено, не по усет:

Продукционният ETL Worker е замразен на таг v1.0.2 (deploy към production става само през release таг; merge към main деплойва единствено staging). Тоест production cron-ът носи СТАРИЯ refresh-slice.sql без ordering_unit_name срещу старата prod D1 - консистентно, не може да гръмне.
Релийзът на тази промяна е планиран като rebuild + swap: нова база, изградена от 0000_init.sql (която вече носи колоната), после смяна на слота. Тоест към момента, в който production Worker получи новия SQL, базата под него вече има колоната.
Дори staging cron да удари прозореца между merge и staging-ребилда: всяка група на refresh-slice се прилага през db.batch() (атомарно) - провалът връща групата, пропуска един цикъл и се самолекува след ребилда. Никакво повреждане.

Затова не добавяме in-place миграция тук - тя би дублирала схемата в два източника точно когато пресъздаваме от единия. Ако стратегията някога се обърне към in-place ъпгрейд, миграцията става задължителна - записано в runbook-а на релийза.

midt-admin

Одобрено - канонична идентичност по мода + ordering_unit_name + composite-EIK guard. Блокерът за миграцията е адресиран в коментара (env-specific deploy: prod замразен на v1.0.2, релийз чрез rebuild+swap). Локална сюита 270/270, CI зелен.

midt-admin

Одобрено - канонична идентичност по мода + ordering_unit_name + composite-EIK guard. Миграционният блокер е адресиран в коментара (env-specific deploy, rebuild+swap). Локална сюита 270/270, CI зелен.

Resolution: bidder_canonical_name is rebuilt ON TOP of contractor_identity (mode re-derived from raw_contracts, keyed by the checksum bidder_key) in both normalize-raw and refresh-slice - the validated integration resolution. The canonicalization test's bidder EIK moves to a checksum-valid one (400000004) now that the mode fixture must survive Bulstat validation.

…resolution)

@StanislavBG

* fix(etl): enforce bulstat checksum for eik identity eik_valid now computes the real Bulstat control digit (9-digit weights 1..8 mod 11 with the remainder-10 reweight, 13-digit second pass), and rejects the degenerate all-zero code. A checksum-invalid ЕИК routes the bidder to a name: key instead of eik:, so falsely-merged distinct companies split apart; a valid ЕИК stays merged correctly. The checksum block is repeated across both ETL paths and pinned identical by a consistency test; fixtures updated to checksum-valid ЕИК-та. Extracts the checksum fix from #203 (@StanislavBG). * feat(etl): canonical entity fields by frequency mode, not MIN/MAX Authority name, bidder name, and authority type now use the most-frequent value per ЕИК with a deterministic tiebreak (mixed-case over ALL-CAPS via a Cyrillic-aware GLOB, then longer, then lexical), mirrored in normalize-raw.sql and refresh-slice.sql. Keys stay ЕИК-based so URLs are unaffected. Extracts the canonical-authority-name fix from #203 (@StanislavBG) and adds the two asymmetrically-missed siblings: bidder name (was MIN(contractor_name)) and authority type (was MAX(authority_type), which also mis-drove the state-company bucket). * feat(etl,web): preserve verbatim ordering-unit name beside the canonical authority Under #194 the authority display name is canonicalized to the frequency mode per ЕИК, which collapses genuine sub-units that share one ЕИК (e.g. a school placed under МОН's ЕИК keeps the ministry name). Additionally keep the raw per-row "Възложител" verbatim so no ordering unit is lost. - add contracts.ordering_unit_name / tenders.ordering_unit_name, filled from the raw authority_name in normalize-raw and both refresh-slice paths - getContract surfaces it on the contract page only when it differs from the canonical name after a Cyrillic-safe fold (case + whitespace), so spelling variants stay hidden and real sub-units show - /contracts/:id.json sourceNames.authority is now genuinely verbatim; it previously served the canonical name despite the "verbatim" contract Extends #251; credit StanislavBG's #203 identity work. * fix(etl,web): preserve identity-poor contracts via an unknown-bidder bucket The bidder key fell to NULL when a contractor had an invalid-checksum ЕИК AND no name; `WHERE bidder_key IS NOT NULL` then silently dropped the whole contract. Never drop a contract for identity reasons — downgrade the key. - add a third key rung: invalid ЕИК + empty/NULL name -> one labelled `unknown:анонимен` bucket ("Неизвестен изпълнител", kind=unknown), so every previously-dropped contract stays in the corpus - compute the checksum + the three-rung key ONCE in a scratch `contractor_identity` table (NULL-safe raw-pair join), replacing the ~7 copy-pasted checksum blocks; delete the consistency test that pinned them - keep the bucket inside company_totals so money still reconciles; hide it only from browsable rankings (search_index, /companies, home top suppliers) - resolve the bucket's /companies slug so the contract-page link works - mirror all of it in refresh-slice; the checksum algorithm is byte-identical A new SQL test asserts no eligible contract is dropped and SUM(amount_eur) is preserved across both the normalize and refresh paths. * fix(etl): fold contractor name-key over case, quotes and dash variants Name-keyed bidders (no valid ЕИК) split the same company across surface variants because SQLite UPPER() is ASCII-only and quote/dash encodings vary. Normalise the name in the `contractor_identity` scratch table so Cyrillic case, quote styles and dash encodings collapse to one key — while dash vs space stays distinct, so genuinely different names are not over-merged. - staged `name_norm` UPDATEs (whitespace + quotes + dashes + Cyrillic case), each a shallow expression: a single ~60-deep nested REPLACE overflows the sqlite3 CLI parser (v3.40.1) on the import path, so the fold is split into stages of <=15 REPLACE calls - the fold is byte-identical in normalize-raw and refresh-slice - tests assert quote/case/space variants merge to one key, en-dash == hyphen, and hyphen != space (over-fold guard); SUM(amount_eur) still reconciles * fix(db): exclude the unknown-bidder bucket from authority top-suppliers getAuthority's top-suppliers query lacked the AND b.kind <> 'unknown' guard the other three surfaces (home, companies, search_index) have, so the synthetic 'Неизвестен изпълнител' bucket could rank in an authority's public top-7. Add the same guard. * fix(etl): exclude composite joint EIKs from authority minting A raw authority_eik can carry a joint-procurement composite ('EIK1; EIK2'). The rewritten canonical name/kind sources and the main authority insert consumed it verbatim, minting orphan 'auth:EIK1; EIK2' authorities referenced by no tender, contract, rollup or search row - a full 2020-2026 rebuild measured 404 of them. Guard every minting source with NOT LIKE '%;%'; joint tenders are attributed through their individual members. * chore: drop accidentally committed node_modules symlinks The .gitignore pattern node_modules/ matches directories only; a dev-environment symlink slipped past it into the merge commit and broke CI's pnpm install (ENOTDIR). Remove from tracking. * style: prettier-format companies.test.ts --------- Co-authored-by: t <t@e.com>

@StanislavBG

* fix(etl): enforce bulstat checksum for eik identity eik_valid now computes the real Bulstat control digit (9-digit weights 1..8 mod 11 with the remainder-10 reweight, 13-digit second pass), and rejects the degenerate all-zero code. A checksum-invalid ЕИК routes the bidder to a name: key instead of eik:, so falsely-merged distinct companies split apart; a valid ЕИК stays merged correctly. The checksum block is repeated across both ETL paths and pinned identical by a consistency test; fixtures updated to checksum-valid ЕИК-та. Extracts the checksum fix from #203 (@StanislavBG). * feat(etl): canonical entity fields by frequency mode, not MIN/MAX Authority name, bidder name, and authority type now use the most-frequent value per ЕИК with a deterministic tiebreak (mixed-case over ALL-CAPS via a Cyrillic-aware GLOB, then longer, then lexical), mirrored in normalize-raw.sql and refresh-slice.sql. Keys stay ЕИК-based so URLs are unaffected. Extracts the canonical-authority-name fix from #203 (@StanislavBG) and adds the two asymmetrically-missed siblings: bidder name (was MIN(contractor_name)) and authority type (was MAX(authority_type), which also mis-drove the state-company bucket). * feat(db): attribute joint procurements to each co-authority Contracts whose authority_eik is a '; '-joined list of co-authorities are no longer excluded/dropped (was 579 contracts / 302M EUR vanishing). A lead authority is chosen per joint tender (co-author with the modal canonical name, fallback first ЕИК) so the contract attaches to a real authority; all co-authorities are recorded in a new contract_co_authorities bridge (migration 0002). Full contract value is attributed to the lead only in authority_totals.spent_eur (no split, no double-count); co-participation lives in a separate authority_joint_participation rollup that never enters spent_eur. Builds on the #196 diagnosis in #203 (@StanislavBG); related to #250 (@atanasster). * feat(etl,web): preserve verbatim ordering-unit name beside the canonical authority Under #194 the authority display name is canonicalized to the frequency mode per ЕИК, which collapses genuine sub-units that share one ЕИК (e.g. a school placed under МОН's ЕИК keeps the ministry name). Additionally keep the raw per-row "Възложител" verbatim so no ordering unit is lost. - add contracts.ordering_unit_name / tenders.ordering_unit_name, filled from the raw authority_name in normalize-raw and both refresh-slice paths - getContract surfaces it on the contract page only when it differs from the canonical name after a Cyrillic-safe fold (case + whitespace), so spelling variants stay hidden and real sub-units show - /contracts/:id.json sourceNames.authority is now genuinely verbatim; it previously served the canonical name despite the "verbatim" contract Extends #251; credit StanislavBG's #203 identity work. * fix(etl,web): preserve identity-poor contracts via an unknown-bidder bucket The bidder key fell to NULL when a contractor had an invalid-checksum ЕИК AND no name; `WHERE bidder_key IS NOT NULL` then silently dropped the whole contract. Never drop a contract for identity reasons — downgrade the key. - add a third key rung: invalid ЕИК + empty/NULL name -> one labelled `unknown:анонимен` bucket ("Неизвестен изпълнител", kind=unknown), so every previously-dropped contract stays in the corpus - compute the checksum + the three-rung key ONCE in a scratch `contractor_identity` table (NULL-safe raw-pair join), replacing the ~7 copy-pasted checksum blocks; delete the consistency test that pinned them - keep the bucket inside company_totals so money still reconciles; hide it only from browsable rankings (search_index, /companies, home top suppliers) - resolve the bucket's /companies slug so the contract-page link works - mirror all of it in refresh-slice; the checksum algorithm is byte-identical A new SQL test asserts no eligible contract is dropped and SUM(amount_eur) is preserved across both the normalize and refresh paths. * fix(etl): fold contractor name-key over case, quotes and dash variants Name-keyed bidders (no valid ЕИК) split the same company across surface variants because SQLite UPPER() is ASCII-only and quote/dash encodings vary. Normalise the name in the `contractor_identity` scratch table so Cyrillic case, quote styles and dash encodings collapse to one key — while dash vs space stays distinct, so genuinely different names are not over-merged. - staged `name_norm` UPDATEs (whitespace + quotes + dashes + Cyrillic case), each a shallow expression: a single ~60-deep nested REPLACE overflows the sqlite3 CLI parser (v3.40.1) on the import path, so the fold is split into stages of <=15 REPLACE calls - the fold is byte-identical in normalize-raw and refresh-slice - tests assert quote/case/space variants merge to one key, en-dash == hyphen, and hyphen != space (over-fold guard); SUM(amount_eur) still reconciles * feat(db): pick joint leads by УНП prefix and value the participation Three refinements to the joint-procurement attribution: - Lead selection gains a top tier: the УНП prefix. УНП is `<AOP authority number>-<year>-<sequence>`, so the prefix identifies the authority that REGISTERED the procedure, i.e. the organiser. The map is learned from single-authority rows, where 4030 prefixes resolve to exactly one authority. Measured against the 354 joint tenders in the corpus, the prefix agrees with the previous first-EIK fallback 292 times, corrects it twice, and never points outside the member list (60 have no usable prefix). Name match and first EIK stay as the lower tiers. - `authority_joint_participation` gains `joint_contract_value_eur`: the total value of the joint procurements an authority took part in. It is informational only and is never summed into `authority_totals.spent_eur` or any national/leaderboard total, so a non-lead co-authority still shows zero spend for the contract while its page can state the amount involved. - `contract_co_authorities` and `authority_joint_participation` were defined both in `0000_init.sql` and in `0002_contract_co_authorities.sql`. The ETL work DB is built from 0000 only, so 0000 is the single home and the redundant 0002 is deleted; its table assertions stay in migrations.test.ts, now covering 0000. refresh-slice mirrors all of it. It learns the prefix map from the existing tender history rather than raw staging, which holds only the touched slice. * fix(db): exclude the unknown-bidder bucket from authority top-suppliers getAuthority's top-suppliers query lacked the AND b.kind <> 'unknown' guard the other three surfaces (home, companies, search_index) have, so the synthetic 'Неизвестен изпълнител' bucket could rank in an authority's public top-7. Add the same guard. * fix(etl): exclude composite joint EIKs from authority minting A raw authority_eik can carry a joint-procurement composite ('EIK1; EIK2'). The rewritten canonical name/kind sources and the main authority insert consumed it verbatim, minting orphan 'auth:EIK1; EIK2' authorities referenced by no tender, contract, rollup or search row - a full 2020-2026 rebuild measured 404 of them. Guard every minting source with NOT LIKE '%;%'; joint tenders are attributed through their individual members. * fix(etl): skip undecomposable composite EIKs when seeding joint members A ';'-bearing member EIK is a composite the splitter could not decompose - it is not a real single authority, and member_defaults would mint an orphan 'auth:EIK1; EIK2' row from it. Guard both the normalize and refresh member inserts. * style: prettier-format companies.test.ts --------- Co-authored-by: t <t@e.com>

Trio content taken from the validated per-stage resolutions; the #261 fold (amount_eur through the amendment currency, trusted_currency companion, current-amount-parity gating) re-applied as the exact integration diff. Comment- only conflicts resolved to the fuller phrasing.

@StanislavBG

…ent currency (#245) (#261) * fix(etl): enforce bulstat checksum for eik identity eik_valid now computes the real Bulstat control digit (9-digit weights 1..8 mod 11 with the remainder-10 reweight, 13-digit second pass), and rejects the degenerate all-zero code. A checksum-invalid ЕИК routes the bidder to a name: key instead of eik:, so falsely-merged distinct companies split apart; a valid ЕИК stays merged correctly. The checksum block is repeated across both ETL paths and pinned identical by a consistency test; fixtures updated to checksum-valid ЕИК-та. Extracts the checksum fix from #203 (@StanislavBG). * feat(etl): canonical entity fields by frequency mode, not MIN/MAX Authority name, bidder name, and authority type now use the most-frequent value per ЕИК with a deterministic tiebreak (mixed-case over ALL-CAPS via a Cyrillic-aware GLOB, then longer, then lexical), mirrored in normalize-raw.sql and refresh-slice.sql. Keys stay ЕИК-based so URLs are unaffected. Extracts the canonical-authority-name fix from #203 (@StanislavBG) and adds the two asymmetrically-missed siblings: bidder name (was MIN(contractor_name)) and authority type (was MAX(authority_type), which also mis-drove the state-company bucket). * feat(db): attribute joint procurements to each co-authority Contracts whose authority_eik is a '; '-joined list of co-authorities are no longer excluded/dropped (was 579 contracts / 302M EUR vanishing). A lead authority is chosen per joint tender (co-author with the modal canonical name, fallback first ЕИК) so the contract attaches to a real authority; all co-authorities are recorded in a new contract_co_authorities bridge (migration 0002). Full contract value is attributed to the lead only in authority_totals.spent_eur (no split, no double-count); co-participation lives in a separate authority_joint_participation rollup that never enters spent_eur. Builds on the #196 diagnosis in #203 (@StanislavBG); related to #250 (@atanasster). * feat(etl,web): preserve verbatim ordering-unit name beside the canonical authority Under #194 the authority display name is canonicalized to the frequency mode per ЕИК, which collapses genuine sub-units that share one ЕИК (e.g. a school placed under МОН's ЕИК keeps the ministry name). Additionally keep the raw per-row "Възложител" verbatim so no ordering unit is lost. - add contracts.ordering_unit_name / tenders.ordering_unit_name, filled from the raw authority_name in normalize-raw and both refresh-slice paths - getContract surfaces it on the contract page only when it differs from the canonical name after a Cyrillic-safe fold (case + whitespace), so spelling variants stay hidden and real sub-units show - /contracts/:id.json sourceNames.authority is now genuinely verbatim; it previously served the canonical name despite the "verbatim" contract Extends #251; credit StanislavBG's #203 identity work. * fix(etl,web): preserve identity-poor contracts via an unknown-bidder bucket The bidder key fell to NULL when a contractor had an invalid-checksum ЕИК AND no name; `WHERE bidder_key IS NOT NULL` then silently dropped the whole contract. Never drop a contract for identity reasons — downgrade the key. - add a third key rung: invalid ЕИК + empty/NULL name -> one labelled `unknown:анонимен` bucket ("Неизвестен изпълнител", kind=unknown), so every previously-dropped contract stays in the corpus - compute the checksum + the three-rung key ONCE in a scratch `contractor_identity` table (NULL-safe raw-pair join), replacing the ~7 copy-pasted checksum blocks; delete the consistency test that pinned them - keep the bucket inside company_totals so money still reconciles; hide it only from browsable rankings (search_index, /companies, home top suppliers) - resolve the bucket's /companies slug so the contract-page link works - mirror all of it in refresh-slice; the checksum algorithm is byte-identical A new SQL test asserts no eligible contract is dropped and SUM(amount_eur) is preserved across both the normalize and refresh paths. * fix(etl): fold contractor name-key over case, quotes and dash variants Name-keyed bidders (no valid ЕИК) split the same company across surface variants because SQLite UPPER() is ASCII-only and quote/dash encodings vary. Normalise the name in the `contractor_identity` scratch table so Cyrillic case, quote styles and dash encodings collapse to one key — while dash vs space stays distinct, so genuinely different names are not over-merged. - staged `name_norm` UPDATEs (whitespace + quotes + dashes + Cyrillic case), each a shallow expression: a single ~60-deep nested REPLACE overflows the sqlite3 CLI parser (v3.40.1) on the import path, so the fold is split into stages of <=15 REPLACE calls - the fold is byte-identical in normalize-raw and refresh-slice - tests assert quote/case/space variants merge to one key, en-dash == hyphen, and hyphen != space (over-fold guard); SUM(amount_eur) still reconciles * feat(db): pick joint leads by УНП prefix and value the participation Three refinements to the joint-procurement attribution: - Lead selection gains a top tier: the УНП prefix. УНП is `<AOP authority number>-<year>-<sequence>`, so the prefix identifies the authority that REGISTERED the procedure, i.e. the organiser. The map is learned from single-authority rows, where 4030 prefixes resolve to exactly one authority. Measured against the 354 joint tenders in the corpus, the prefix agrees with the previous first-EIK fallback 292 times, corrects it twice, and never points outside the member list (60 have no usable prefix). Name match and first EIK stay as the lower tiers. - `authority_joint_participation` gains `joint_contract_value_eur`: the total value of the joint procurements an authority took part in. It is informational only and is never summed into `authority_totals.spent_eur` or any national/leaderboard total, so a non-lead co-authority still shows zero spend for the contract while its page can state the amount involved. - `contract_co_authorities` and `authority_joint_participation` were defined both in `0000_init.sql` and in `0002_contract_co_authorities.sql`. The ETL work DB is built from 0000 only, so 0000 is the single home and the redundant 0002 is deleted; its table assertions stay in migrations.test.ts, now covering 0000. refresh-slice mirrors all of it. It learns the prefix map from the existing tender history rather than raw staging, which holds only the touched slice. * fix(db,web): convert current_value_eur from the amendment currency, not signing currency (#245) contracts.current_value is populated from the latest amendment's value_after, denominated in THAT amendment's own currency — but every EUR conversion of current_value used contracts.currency (the contract's original signing currency), silently re-converting an already-EUR amendment (e.g. one recorded after ЦАИС ЕОП's 2026 BGN->EUR feed switch) and halving the reported value. Add contracts.current_value_currency to track which currency last set current_value, and route every current_value -> EUR conversion (refresh-slice, normalize-raw, precompute, the contract detail query) through it instead of contracts.currency. signing_value/signing_value_eur are unaffected — signing really is denominated in the contract's own currency. * fix(etl): route amount_eur through the amendment currency, not the signing currency (#245) current_value is minted from the latest amendment, which can be denominated in a different currency than the contract's signing currency. amount_eur — the column every rollup sums — still converted via contracts.currency, so a BGN contract with a EUR amendment carried roughly half its true value across all aggregates while the detail page showed the corrected current_value_eur, a self-contradicting DB. Pair a trusted_currency column with trusted_native so the conversion always uses the currency of the value actually chosen: the amendment currency when current_value was selected, the contract currency on the signing_value fallback (incl. annex_suspect). Mirror the same routing into eff_eur and add an integrity check asserting amount_eur and current_value_eur agree within a cent for value_flag='ok' rows with a current value — the invariant whose absence let this ship. * fix(etl): apply the whole migration chain to the work DB and served D1 (#245) The work-DB rebuild seeded schema from 0000_init.sql only, so a numbered migration (0002_current_value_currency) never reached it and normalize-raw aborted on the missing column. Apply every packages/db/migrations/*.sql in sorted order, which also lets 0001's index reach rebuilt work DBs for the first time. The served D1 gets the column via deploy.yml: probe the real table (wrangler's migration ledger is empty because the base schema was created out-of-band), then ALTER + backfill + precompute, gated on a completion-marker table so a partial first deploy resumes without a double ALTER. backfill-current-value-currency.sql repairs the ~2,510 already-shipped rows and self-checks parity. * fix(db): current-amount-parity must skip before precompute (#245) current_value_eur is populated by precompute on the served D1, not by normalize on the work DB. The integrity gate also runs on the work DB (import.mjs), where every current_value_eur is still NULL — so the parity check failed on every amended contract and aborted the rebuild. Gate it on home_totals like rollup-reconciliation: skip pre-precompute, assert on the served DB. Found by a full local rebuild; the served-DB data satisfies the invariant with 0 mismatches. * fix(db): exclude the unknown-bidder bucket from authority top-suppliers getAuthority's top-suppliers query lacked the AND b.kind <> 'unknown' guard the other three surfaces (home, companies, search_index) have, so the synthetic 'Неизвестен изпълнител' bucket could rank in an authority's public top-7. Add the same guard. * fix(etl): exclude composite joint EIKs from authority minting A raw authority_eik can carry a joint-procurement composite ('EIK1; EIK2'). The rewritten canonical name/kind sources and the main authority insert consumed it verbatim, minting orphan 'auth:EIK1; EIK2' authorities referenced by no tender, contract, rollup or search row - a full 2020-2026 rebuild measured 404 of them. Guard every minting source with NOT LIKE '%;%'; joint tenders are attributed through their individual members. * fix(etl): skip undecomposable composite EIKs when seeding joint members A ';'-bearing member EIK is a composite the splitter could not decompose - it is not a real single authority, and member_defaults would mint an orphan 'auth:EIK1; EIK2' row from it. Guard both the normalize and refresh member inserts. * style: prettier-format companies.test.ts * fix(merge): carry the full fold set for the amendment-currency composition The fold diff was applied only to the six textually-conflicted files; four more files it touches (async current-amount-parity in integrity-checks.mjs, migration 0002 in the two SQL-execution tests) auto-merged to stale versions - the sync parity check against async helpers returned NaN and failed the gate. Apply the fold to them too. * test(etl): teach the gate fixture about current-amount-parity #261 adds current-amount-parity to the shared roster; the fakeD1 from #156 did not answer its query (NaN) and the skip-count expectation assumed only rollup-reconciliation un-skips with rollups present. Answer the parity query clean and expect two fewer skips. --------- Co-authored-by: Bilko <StanislavBG@gmail.com> Co-authored-by: t <t@e.com>

…ty changes After rebasing midt-bg#183 onto upstream/main, the masking test fixtures needed two adaptations to upstream's new APIs (no behaviour change to the production masking logic): - Add `getDb` to the `@sigma/db` mocks in the three loader tests. Upstream's read-only D1 chokepoint (midt-bg#199/midt-bg#225) means loaders now call `getContract(getDb(env), …)` instead of `getContract(env.DB, …)`; the mock passes the env's DB through so the stubbed `getContract` still resolves. - Add the new required `orderingUnit: null` (canonical-identity midt-bg#251) and `amendments: []` (annex history midt-bg#165) fields to the `ContractParty` / `ContractRecord` test builders so they satisfy the widened types. All masking assertions unchanged. `pnpm --filter @sigma/web test` → 424 passing; `pnpm --filter @sigma/db test` → 297 passing; typecheck exit 0.

Pulls the euro-annex conversion fix (midt-bg#245/midt-bg#261), canonical value base (midt-bg#259), identity canonicalization + Bulstat checksum + joint procurement (midt-bg#251-253), app-layer read-only D1 guard (midt-bg#225), JSON-LD escaping (midt-bg#212), react-router 7.18.0. Non-trivial resolutions: - normalize-raw.sql: kept upstream's amendment_winner currency CTE + our is_synthetic column (both additive, one column-list collision). - integrity-checks.mjs: upstream's (await rows()) wrapper carrying our is_synthetic != 1 filter on auth/bidder attribution. - Migration collision: our 0002_contracts_is_synthetic renumbered to 0006 (upstream took 0002 for current_value_currency); tests load all migrations. - refresh-slice.test seedReattrContract: upstream's authority params + our real-tender-header seed so reattr contracts stay non-synthetic and reconcile. - root.tsx/assistant.chat.tsx: adopted getDb read-only chokepoint + kept dock. - describe-schema DATA_TRAPS: upstream canonical-base rule + our NULL detail.

…ty changes After rebasing midt-bg#183 onto upstream/main, the masking test fixtures needed two adaptations to upstream's new APIs (no behaviour change to the production masking logic): - Add `getDb` to the `@sigma/db` mocks in the three loader tests. Upstream's read-only D1 chokepoint (midt-bg#199/midt-bg#225) means loaders now call `getContract(getDb(env), …)` instead of `getContract(env.DB, …)`; the mock passes the env's DB through so the stubbed `getContract` still resolves. - Add the new required `orderingUnit: null` (canonical-identity midt-bg#251) and `amendments: []` (annex history midt-bg#165) fields to the `ContractParty` / `ContractRecord` test builders so they satisfy the widened types. All masking assertions unchanged. `pnpm --filter @sigma/web test` → 424 passing; `pnpm --filter @sigma/db test` → 297 passing; typecheck exit 0.

todorkolev changed the title ~~feat(etl): канонична идентичност на субектите (мода, не MIN/MAX) [DRAFT]~~ feat(etl): канонична идентичност на субектите — име и вид по мода Jul 17, 2026

midt-admin force-pushed the feat/identity-canonicalization branch from d5623e4 to edca1ae Compare July 17, 2026 02:16

midt-admin force-pushed the feat/identity-canonicalization branch from edca1ae to ce14433 Compare July 17, 2026 02:44

lyubomir-bozhinov mentioned this pull request Jul 17, 2026

feat(etl): съвместни поръчки — приписване към всеки съвъзложител #253

Merged

lyubomir-bozhinov reviewed Jul 17, 2026

View reviewed changes

lyubomir-bozhinov mentioned this pull request Jul 20, 2026

feat(etl): контролен код по Булстат за идентичност по ЕИК #252

Merged

lyubomir-bozhinov reviewed Jul 20, 2026

View reviewed changes

This was referenced Jul 20, 2026

fix(etl): authority/executor identity — canonical name, EIK checksum, multi-ЕИК guard (#194 #195 #196) #203

Closed

fix(etl): canonical authority name by frequency-mode + curated override, not MIN() (#194) #215

Closed

lyubomir-bozhinov mentioned this pull request Jul 21, 2026

feat: индекс на качеството на договорите (ETL оценка 0..1 + страница) #188

Open

todorkolev and others added 2 commits July 23, 2026 19:44

merge main (#246 #263 #156 #259)

f773680

merge main (#260)

ebb0081

midt-admin approved these changes Jul 23, 2026

View reviewed changes

Merge branch 'main' into feat/identity-canonicalization

324918b

midt-admin approved these changes Jul 23, 2026

View reviewed changes

todorkolev marked this pull request as ready for review July 23, 2026 19:56

todorkolev merged commit af4917e into main Jul 23, 2026
1 check passed

todorkolev added a commit that referenced this pull request Jul 23, 2026

merge main (#251 squashed; identical content, ours carries the +#252 …

8a150c4

…resolution)

LyuboslavLyubenov mentioned this pull request Jul 28, 2026

fix(privacy): apply noindex+mask policy to machine-readable outputs (#173) #183

Open

3 tasks

todorkolev mentioned this pull request Jul 28, 2026

fix(etl): каноничното име на възложителя (MIN на името) подвежда при споделен ЕИК — поръчки на МОН излизат под училище #194

Closed

todorkolev mentioned this pull request Jul 28, 2026

feat: свързани лица — детерминистична основа за данни за конфликт на интереси #226

Merged

Navigation Menu

Uh oh!

feat(etl): канонична идентичност на субектите — име и вид по мода - #251

feat(etl): канонична идентичност на субектите — име и вид по мода#251
todorkolev merged 6 commits into
mainfrom
feat/identity-canonicalization

todorkolev commented Jul 17, 2026 •

edited

Loading

Uh oh!

lyubomir-bozhinov left a comment

Uh oh!

lyubomir-bozhinov left a comment

Uh oh!

todorkolev commented Jul 23, 2026

Uh oh!

midt-admin left a comment

Uh oh!

midt-admin left a comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Uh oh!

Conversation

todorkolev commented Jul 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Проблемът

Решението

Как работи след фикса

Uh oh!

lyubomir-bozhinov left a comment

Choose a reason for hiding this comment

Uh oh!

lyubomir-bozhinov left a comment

Choose a reason for hiding this comment

Uh oh!

todorkolev commented Jul 23, 2026

Uh oh!

midt-admin left a comment

Choose a reason for hiding this comment

Uh oh!

midt-admin left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

todorkolev commented Jul 17, 2026 •

edited

Loading