feat: индекс на качеството на договорите (ETL оценка 0..1 + страница) - #188
Open
StanislavBG wants to merge 40 commits into
Open
feat: индекс на качеството на договорите (ETL оценка 0..1 + страница)#188StanislavBG wants to merge 40 commits into
StanislavBG wants to merge 40 commits into
Conversation
lyubomir-bozhinov
left a comment
lyubomir-bozhinov
left a comment
Collaborator
There was a problem hiding this comment.
Adversarial review на feat/contract-health-index (@ 4796a75) — пуснах миграциите и health derive-а локално (node:sqlite), не само четох diff-а.
Два блокера, и двата възпроизведени локално:
- Свежо
wrangler d1 migrations applyпада — виж inline на0002_contract_health.sql. derive-health.sqlаборти́ра целият при CPV дивизия със сума 0 — виж inline наderive-health.sql.
Потвърдено чисто (проверих, не са проблеми):
- value_suspect: коректно NULL-gated —
score_overall/score_cса NULL за value_suspect (derive-contract-features.sql:209,478,568), плюс self-check на ред 602 иvalidate-health.mjsт.2. Обратното на капана от #182 — тук инвариантът е верен и тестван. - cache-key drift: новите /quality параметри (
contract,csort,grain,sel) са вCACHE_QUERY_PARAMS— няма CWE-349 дрифт. - Staleness: health derive-ът СЕ пуска след slice-а (
import.mjs:runSliceDerive→runHealthDerive, иship-domain.mjsслед precompute) — rollup-ите не застояват. Проверих нарочно, защотоrefresh-slice.sqlсам по себе си не ги пипа.
Извън обхвата на този преглед: не одитирах петте pillar формули в derive-contract-features.sql (795 реда, тегла/leaves) — заслужава отделен pass.
Присъда: промени преди merge (2 блокера). От Triage — не е формален approve/reject.
This was referenced Jul 2, 2026
Contributor
Author
|
Двата блокера са затворени, миграцията е преномерирана. Нов връх: 86f89d1.
|
Collaborator
|
@StanislavBG — не намирам
Push-ни |
Contributor
|
Both blockers reproduced locally at the PR head ( Ревю на PR #188 — Индекс на качеството на договоритеБлагодаря за сериозната работа — методологията е добре документирана, а ETL слоят е внимателно оформен. Прегледах не само diff-а, а възпроизведох миграциите и derive-а локално (SQLite) на текущия head на PR-а ( 🔴 Блокер 1 — свежа миграционна верига пада (дублирани колони)Деветте нови колони живеят едновременно в Възпроизведено локално (чист SQLite, точната верига): Всяка свежа инсталация на D1 ще пропадне. Съществуващият Поправката, описана от автора (колоните само в миграцията + преномериране на 🔴 Блокер 2 —
|
Adds 9 columns to contracts, amendments, tenders, and flow_pairs as the schema and ETL foundation for the contract quality / health index. No scoring logic — columns are populated by the existing pipeline (normalize-raw.sql, promote-amendments.sql, precompute.sql, refresh-slice.sql). New columns: - contracts.exemption_legal_basis TEXT -- правно основание за изключение - contracts.outside_zop INTEGER -- извън ЗОП - contracts.dps_contract INTEGER -- договор по ДСП - amendments.reason TEXT -- причини за изменение - amendments.circumstances TEXT -- обстоятелства - tenders.corrections_count INTEGER -- брой поправки (corrigenda) - tenders.estimated_value_eur REAL -- estimated_value → EUR (BGN÷1.95583) - flow_pairs.first_date TEXT -- MIN(signed_at) over pair's contracts - flow_pairs.last_date TEXT -- MAX(signed_at) over pair's contracts Migration 0002_contract_health.sql applies the columns to existing local D1 databases. 0000_init.sql updated inline so fresh installs need no migration. The CREATE TABLE IF NOT EXISTS flow_pairs guard in precompute.sql kept in sync.
Add scripts/derive-health.sql building authority_health_rollup, bidder_health_rollup, sector_concentration, and health_percentiles on the served D1 (docs/contract-quality-spec.local.md §7.2/§8), and wire a new --derive=health mode into scripts/import.mjs so these can be rebuilt standalone against an existing local corpus without the full ~25-minute re-import. This is the entity-grain foundation the per-contract scoring (next PRD group) joins against.
Phase 5a+5b of the Contract Quality / Health Index (docs/contract-quality-spec.local.md §5-§7.3, §12): scripts/derive-contract-features.sql builds contract_features (one row per contract, 194,484 rows), populating raw leaf values, the effective peer key (§5.6 fine→mid→coarse→GLOBAL fallback), and the [0,1] coverage score. Scoring UPDATEs (score_a..score_overall) are left NULL for the next PRD (group 338). Verified against the local corpus: contract_features_rows = contracts_rows = 194,484; score_coverage non-NULL and in [0,1] for all rows; effective_peer_key non-NULL with peer_n >= 30 or 'GLOBAL'; scoring_regime='framework' on 3,238 rows (ДСП/КС/DPS regime, contract-grain); single_offer=1 on 78,739 rows (matches the corpus bids_received=1 tally).
…gning values Code review on the contract_features PRD caught two gaps in the NEW-C6 first-amendment- shock leaf: it fell back to 0 (not NULL) when signing_value was NULL/<=0 instead of staying unknown, and compared amendments.value_delta (amendments.currency) directly against contracts.signing_value (contracts.currency) with no currency check. Both now resolve to NULL — re-verified against the local corpus (194,484/194,484 rows, first_amend_shock: 179,100 NULL / 15,151 zero / 233 flagged).
… wiring Pillar scores A-E and score_overall = 0.6*wmean + 0.4*worst on the [0,1] scale with the five-state value_flag gate; six *_quality_totals rollup grains; health derive wired into full/slice derive and ship-domain; scripts/validate-health.mjs runs the spec §10 checks (18/18 pass on the 194k-contract local corpus, 194,481 scored, 3 value_suspect unknown).
Re-run safety: drop scoring temp tables up front so a lock-retried batch re-executes cleanly; reject --catchup --derive=health instead of silently downgrading and remove the unreachable post-load health branch; guard /quality against missing health tables (pre-first-derive or mid-rebuild prod window); NULL-currency guard in first_amend_shock; >=0 floor on the B5 bid-window penalty; un-pin validate-health from the current corpus (value_suspect count, year range); surface spawn errors in the wrangler lock-retry wrapper; document the C2 linear-band choice and the §5.6 fallback-cohort limitation; honor the ranking top param; refresh stale scoring comments.
A fresh `wrangler d1 migrations apply` hit "duplicate column": the nine health-index columns lived both in 0000_init.sql and as ADD COLUMNs in the health migration. SQLite has no ADD COLUMN IF NOT EXISTS, so the columns now live ONLY in the migration, removed from 0000_init; the health rollup tables stay in 0000_init (and are rebuilt idempotently by the ETL derives for already-migrated DBs). The migration is renumbered 0002 -> 0003 to leave 0002 to 0002_contracts_overrun_index (PRs midt-bg#170/midt-bg#171). 0000_init's second role (direct schema load for the work-DB backfill and the sqlite-backed tests) is preserved by applying the FULL migration chain there too: scripts/import.mjs and the db test fixtures now read every migration in apply order, exactly like a fresh D1. migrations.test.ts applies the full fresh chain and asserts the columns and rollup tables exist — the regression test for this blocker.
A CPV division whose priced contracts sum to 0 EUR (a single amount_eur=0 contract, or exact +/- offsets) made win_share 0/0 = NULL and aborted the whole derive-health.sql on sector_concentration.win_share NOT NULL — one such row stopped the daily health refresh. div_totals now carries HAVING SUM(c.amount_eur) <> 0, so a zero-sum division is skipped at the source; downstream (derive-contract-features.sql LEFT JOIN) its contracts get sector_win_share NULL — an honest unknown, never a fabricated 0 score. New derive-health.test.ts proves the derive completes on a zero-sum fixture, the division lands no row, and a healthy division still rolls up.
Local D1 runs the file as one batch: the bare diagnostic SELECT on tmp_b1 left a cursor that made the later DROP TABLE fail with SQLITE_LOCKED, and the flat 6-term UNION ALL exceeded the local SQLITE_MAX_COMPOUND_SELECT. Stash the diagnostic into tmp_diag (surfaced by the final summary SELECT) and split the min/max compounds into nested 3+3 chains.
Crumb has no 'href' — typecheck failed; also aligns the empty-state trail
with the loaded page ('Начало').
Contributor
Author
|
Rebase-нат върху main с css split — всички q-/ov- стилове са в styles/pages.css, app.css остава само @imports. Двата блокера остават затворени: свежа миграционна верига (health колоните само в 0003 + fresh-chain тест) и HAVING SUM≠0 за win_share (zero-sum тест). Prettier-чист, typecheck 7/7, 214 db теста зелени. @lyubomir-bozhinov — готов за re-check; @todorkolev — merge след стека #169–#172. |
…opovers Разпределение на оценките: the chart grows from a 116px to a 182px plot; every bin and zone label is a plain GET link (?band=0–19 | weak|mid|good) filtering the „Договори · оценки" list to that exact score range, with a selected state, ✕ chips at the chart and the list, and an sr-only status announcing the range. Native SVG <title> tooltips on bins/zones/mean marker and a ⓘ on the section heading explain how the histogram is built (scored contracts only — unscored are never zero); the confidence legend gets one-sentence hover explanations too. band ships in CACHE_QUERY_PARAMS in the same commit (CWE-349) with behavioral asserts; the DB filter is bound-param, validated at the query boundary, and covered by exact-count narrowing tests. The ⓘ uses the shared MetricInfo popover, hardened so text can never overflow: white-space reset + overflow-wrap: anywhere, 320px card clamped to the viewport, JS shift-into-viewport, coarse-pointer 44px hit area.
Contributor
Author
|
Histogram click-filter (?band, кеш-ключ + тестове) + подобрен ⓘ (MetricInfo без преливане, обяснения на хистограмата/зоните/увереността) — нов head 2fff429. |
…e filter The /quality „Разбивка" ranking gains richer faceting, all server-side over the *_quality_totals rollups: - ?rdir=asc|desc flips the ranking for both sort keys (индекс and договори); the default stays the historical order (score asc — най-слабите отгоре, contracts desc). The section hint now reads the current direction. - ?rfrom/?rto (ints 0–100, От/До number inputs in a no-JS GET form) filter the rollup rows to avg index within [from, to]; bounds are inclusive, divided to the stored [0,1] scale at the SQL boundary, swapped when inverted, dropped when malformed. Composes with grain, sort and direction; the active range shows a clear ✕ chip and an sr-only status line announces the row count. - Param hygiene: rdir/rfrom/rto validated in the shared qualityRankingControls parser and re-checked at the query boundary; added to CACHE_QUERY_PARAMS with behavioral cache-key tests (CWE-349). - The /quality grain/sort/direction/filter controls keep the viewport anchored (preventScrollReset) instead of jumping to the page top.
Contributor
Author
|
Разбивката вече има посока на подреждане (?rdir — „най-слабите/най-добрите отгоре“, за индекс и договори) и диапазон по среден индекс (?rfrom/?rto, 0–100, GET форма без JS, валидирани + в CACHE_QUERY_PARAMS). |
StanislavBG
force-pushed
the
feat/contract-health-index
branch
from
July 3, 2026 18:55
4796a75 to
e2da348
Compare
Contributor
Author
|
Прав си — фиксовете бяха качени на грешния fork (StanislavBG/sigma вместо StanislavBG/sigma-pr, откъдето е отворен PR-ът). Вече е поправено: head-ът е e2da348 и носи всичко обявено:
|
Добавя раздел „Индексът за здраве на договора" в /methodology: петте измерения и теглата (30/15/25/20/10), формулата с думи (0,6 × претеглена средна + 0,4 × най-слабото), пренормирането при липсващо измерение, праговете на покритие, стойностните флагове и претеглянето по грейн (таван 15% за институция/доставчик, непретеглено по година) — сверено срещу quality.ts и derive-contract-features.sql. Само документация; без промяна в поведението.
Collaborator
|
Ре-верифицирах на текущия head (
От моя страна блокерите падат — благодаря за бързия fix (и за изясняването с грешния fork). Останалото (CI/maintainer approve) е извън Triage. |
Contributor
Author
|
@todorkolev готов за ревю 🙏 — rebase-нат на main, CI зелен, prettier-чист, CSS промените в styles/* (app.css само @import). Резолвнати нишки. Approve-ни когато ти е удобно. |
- import.mjs: gate checkContractFeaturesIntegrity in runFullDerive/runSliceDerive too, matching ship-domain.mjs, so a local derive enforces the same contract_features invariants as the daily prod ETL. - precompute.sql: stop guessing an FX rate from today's date for tenders with a NULL published_at — leave estimated_value_eur NULL explicitly instead. - validate-health.mjs: exclude the synthetic NA bucket from the >60%-NULL threshold check (it's informational-only, out of the check's stated 2020-2026 scope); guard the decile-correlation check against zero-variance pillars instead of printing NaN. - trendAxis.ts: drop the stale TODO(midt-bg#170), already tracked by the issue itself. Reply drafts for the 4 confirmation-only threads (no code change needed): - filters.ts qualityRankingControls: confirmed — quality.ts's rankSql call binds every rank param exclusively via `.prepare(...).bind(minScored, ...rangeParams, top)` (packages/db/src/queries/quality.ts:235-236), never string-interpolated; rankFrom/rankTo/rankDir are re-validated in qualityRankingControls (regex + numeric clamp) before reaching the query. - analytics.tsx getQualitySummary(...).catch(...): thanks, confirmed. - 0003_contract_health.sql numbering: confirmed both — (1) 0002_contracts_overrun_index (PR midt-bg#170/midt-bg#171, not yet merged) adds no column/table these ALTERs read, so applying 0003 first is safe; (2) every environment applies migrations only via `wrangler d1 migrations apply` (tracked, once-only), never raw re-execution. - ship-domain.mjs DROP+CREATE window: acknowledged as a real operational risk, tracked as a separate follow-up per PRD 530 — not re-litigated here.
…index # Conflicts: # apps/web/app/lib/filters.test.ts # apps/web/app/lib/filters.ts # apps/web/app/styles/components.css # apps/web/workers/cache-key.ts
…AL_QUERY_PARAMS The origin/main merge brought in a stale CANONICAL_QUERY_PARAMS (predating the g->step trends rename and missing the /quality params this PR adds). Restore the full param set so withParams/cache-key don't silently drop them from generated links and cache keys.
This was referenced Jul 19, 2026
fix(db,web): convert current_value_eur from the amendment currency, not signing currency (#245)
#257
Closed
ydimitrof
reviewed
Jul 20, 2026
ydimitrof
left a comment
ydimitrof
left a comment
Contributor
There was a problem hiding this comment.
Ревю на PR: feat: индекс на качеството на договорите (ETL оценка 0..1 + страница)
Обобщение
ВЕРДИКТ: COMMENT — прегледът обхваща 8 партиди; няма блокиращи проблеми със сигурността или коректността. Има няколко въпроса за потвърждение и незадължителни бележки за полиране.
Какво прави PR-ът
PR въвежда изчислен от ETL индекс на качеството на договорите (нормализирана оценка в диапазон 0..1) и нова публична страница за визуализацията му. Промяната обхваща цялата верига:
- ETL слой — нови SQL скриптове (
derive-contract-features.sql,derive-health.sql) с формула за оценка по стълбове, пренормализация на теглата и твърди integrity gates, които проваля ETL при нарушен инвариант. - Схема и типове — адитивни, обратно-съвместими миграции (
0003_contract_health.sql, нови nullable колони) и разширения вapi-contract(TrendGranularity, документиран инвариант „NULL = недостатъчно данни, никога 0"). - Слой за заявки — нов read-only
queries/quality.tsи разширения наqueries/trend.ts(тримесечна гранулация, разпределения по CPV групи). - UI — нова страница
/quality, разширения на/trends, споделена ос за графики, SVG визуализации без chart библиотека, достъпни компоненти (MetricInfo), CSS и документация (methodology.tsx,docs/etl.md). - Операторски инструмент —
scripts/validate-health.mjsза валидация на локалния D1/SQLite файл.
Силни страни
- Сигурност — чиста през всички партиди. Няма зашити тайни, нови външни URL адреси, нови зависимости, обфускация или backdoor шаблони. SQL достъпът е параметризиран (
.bind()/?placeholders); единствените интерполирани в SQL стойности са вътрешни литерали след строга валидация (allow-list заdir,grain; regex заband,cpv,rankFrom/rankTo). Целият потребителски вход в UI се екранира от JSX (без XSS). Валидацията на ranking контролите (CWE-349) е образцова и целенасочено тествана срещу SQL-подобни входове. - Защита срещу cache poisoning — новите query параметри са добавени синхронно и в
CANONICAL_QUERY_PARAMS, и вPARAM_ORDER, с тестове, доказващи различни cache ключове. - Целост на данните — неутралната позиция от спецификацията е спазена последователно навсякъде: липсваща оценка →
—, никога фалшива нула;value_suspectи покритие < 0,40 се изключват от средните. - Изключително силно тестово покритие — integrity gates, guard срещу дрейф на схемата (тройно сравнение на DDL), гранични случаи; тестовете са смислени, не тривиални.
Въпроси за потвърждение (не блокиращи)
- Съвместимост на URL: параметърът
gе премахнат отCANONICAL_QUERY_PARAMSв полза наstep. Стари линкове/trends?g=yearвече изпускат параметъра тихо — потвърдете дали е добавен редирект или толерантно четене, или дребният пробив в съвместимостта е приемлив. - Тиха нормализация на филтри: невалидни
year/cpv/непознатиangle/stepсе свеждат мълчаливо към стойности по подразбиране; предишният callout „Непознат филтър" е премахнат. Потвърдете, че липсата на обратна връзка при сгрешен филтър е умишлена. - CDN cache-ключ: страницата
/qualityе публично кеширана (max-age=1800), а съдържанието зависи от query параметри (band,sel,contract) — уверете се, че cache-ключът включва query string.
Наблюдения за follow-up (не блокиращи)
- Крехкост при номерирането на миграциите — липсва
0002; смекчено с документация иmigrations.test.ts, който заковава точния набор файлове.EXPECTED_MIGRATION_FILESизисква ръчна синхронизация при бъдещи миграции. - Неатомарен ребилд на прод D1 —
DROP TABLE IF EXISTS contract_features+ нов build създава прозорец на неналичност (документирано като follow-up в кода). INNER JOIN bidders/tendersможе да изпусне договор с NULL/orphanbidder_id, което после проваля целия ETL през паритетния gate — умишлен fail-loud дизайн; потвърдете, чеcontracts.bidder_idникога не е NULL в корпуса.- Гранична неточност „(показани първите 24)" в
trends.tsx— текстът се показва и при точно 24 договора; заявка сlimit: 25и показване само при> 24би било по-коректно. qualityOverview— при хипотетичен оценен ред сscore_coverage IS NULLconfidence кофите няма да сумират доscored;COALESCE(score_coverage, 0)или CHECK инвариант би направил стойността експлицитна.- Тип на
yearвvalidate-health.mjs(проверка №3) — възможно несъответствие низ спрямо стойност от базата; за изясняване. - DRY нит — повтарящ се grid шаблон в
pages.css(.ov-cpv-head/.ov-cpv-rowи.ov-cpv-foot) би могъл да се обедини. - Незадължителен тест, който да докаже, че една и съща логическа заявка дава същия cache ключ (стабилност на подредбата), за да се хване регресия към прекалено фрагментиран кеш.
Заключение
Качествена, добре структурирана, обстойно документирана и изключително добре тествана промяна, следваща правилата от CLAUDE.md (без частична имплементация, TODO-та, дублиран или мъртъв код). Няма открити уязвимости по сигурността или проблеми с целостта на данните в нито една партида. Блокиращи концерни няма; преди merge препоръчвам да се адресират трите въпроса за потвърждение по-горе.
lyubomir-bozhinov
left a comment
lyubomir-bozhinov
left a comment
Collaborator
There was a problem hiding this comment.
Прегледах в дълбочина на текущия HEAD (121c5f22) — data-слоя, миграционния модел, скоринга и рамката. Силна, спец-водена работа. Какво проверих, че държи:
Миграционен модел — коректен (обратното на капана в #251). Нови таблици → 0000_init.sql + идемпотентен CREATE TABLE IF NOT EXISTS в derive-health/derive-contract-features (за вече-мигрираното served D1); нови колони на съществуващи таблици → само в 0003 през ALTER (защото SQLite няма ADD COLUMN IF NOT EXISTS, а fresh D1 пуска цялата верига). runWorkBackfill вече прилага цялата миграционна верига към work-DB-то (import.mjs:327-335), тъй че 0003-колоните съществуват и там. Взехте 0003, за да не влизате в четворната 0002 колизия. Подредбата 0002↔0003 е обмислена и документирана. Чисто.
Скоринг — методологически издържан и репутационно внимателен. score_overall = 0.6·wmean + 0.4·worst, ренормализиран само по наличните стълбове (липсващ стълб се изключва от средното, НЕ се брои като 0 — worst е MIN с COALESCE(...,1.0)); всеки под-скор е клампнат MAX(0,MIN(1,…)), деленето е пазено (wsum=0 → NULL); value_suspect е изключен/НЕОЦЕНЕН. Адверсариалните self-checks (value_suspect_leak_rows, a1_floor_violations…) са вързани в твърд ship-gate (checkContractFeaturesIntegrity + assertIntegrity, ship-domain.mjs:242-248). Health се деривира на served D1 по двата пътя (D1 full-derive + ship-domain след ship), а DROP/rebuild прозорецът е толериран от /quality.
Рамката на /quality е образцова: „Сигнал за преглед, не присъда", изричните ограничения („не открива картели… конфликт на интереси"), — при NULL (никога фабрикувана 0), уговорки по покритие. Точно както трябва за индекс, който слага число до име на публична страница.
Две бележки (не блокират):
-
Зависимост от #257 (евро-анекси). Стълб C (стойност) и
derive-health.avg_cost_overrun/перцентилите четатcurrent_value_eur/signing_value_eur. Договорите в лева с евро-анекс от 2026 са с наполовиненcurrent_value_eur(и НЕ се хващат от annex_suspect, ratio≈0,51<100), тъй че реалното превишение изчезва → тези договори излизат фалшиво по-здрави по C, докато #257 не поправиcurrent_value_eur. Само за merge-координация (C ще е коректен след #257). -
Физически лица на /quality. Страницата показва именувани изпълнители със скор (quality.tsx:1006/1044 →
entityName, който не маскира ЕТ) и нямаnoindex/ЕТ-обработка. ЕТ би излязъл по име под съставна „здравна" оценка на индексируема страница. Не е регресия на този PR (flows/competition правят същото), но #237 току-що въведеnoindexза ЕТ на страницата на договора — редно е същият модел да се приложи и тук (и на /overruns), като политика за целия сайт.
Иначе — одобрявам. Наистина добра работа.
Collaborator
|
Уточнение към бележка (1) по-горе — механизмът, който посочих, беше неточен. |
…op orphan rows safely Build the new scores into a disposable contract_features_next staging table and swap it into the live contract_features name with a back-to-back DROP+RENAME in the same wrangler batch, so served D1 never has contract_features missing/empty mid-rebuild. Also LEFT JOIN tenders/bidders instead of INNER JOIN so a dangling contracts.tender_id/bidder_id scores the row as unknown instead of silently dropping it from the feature store and failing the contract_features_rows == contracts_rows integrity gate.
… fx lookups precompute.sql's tender estimated_value_eur conversion silently leaves foreign- currency estimates NULL when no fx_rate row falls inside the 10-day lookback. Track that count in a new pipeline_diag table and print it in the run summary (fx_rate_gap_rows) so a systemic fx_rates coverage gap is visible instead of invisible.
…NTEGER year Normalize both sides of the year_quality_totals coverage comparison through String() before comparing, so the check passes regardless of whether the driver hands back `year` as text or a number — previously a type mismatch would silently always-FAIL. Extracts the comparison into an exported missingYears() and adds a unit test covering the INTEGER-column case.
precompute.sql's CREATE TABLE IF NOT EXISTS pipeline_diag was bootstrap-only, missing from packages/db/migrations/0000_init.sql unlike every sibling rollup table (home_totals, *_quality_totals, etc.) per the file's own stated "canonical definitions live in migrations" convention.
Collaborator
|
Обобщение след 07-23 вълната (main мръдна съществено):
Логиката на индекса си остава силна (прегледът ми от 07-21). Само rebase-ът е нетривиален заради застъпването във въпросните SQL файлове. |
…, suppress RSC-only CSRF Bump postcss to ^8.5.18 (GHSA-r28c-9q8g-f849) and valibot to ^1.4.2 (GHSA-5qjj-4xww-7phc) via pnpm overrides - both patch-level, non-breaking fixes. Bump react-router/@react-router/dev to ^7.18.0 via override, fixing 4 real advisories (SSR hydration constructor injection, unauthenticated DoS, RSCErrorHandler XSS, open-redirect via backslash) - all fixed within the 7.x line, no major bump needed. Add a time-boxed osv-scanner.toml suppression for the one remaining advisory, GHSA-qwww-vcr4-c8h2, a CSRF flaw scoped to unstable RSC APIs this app does not use (verified via repo-wide grep) with no fix in the 7.x line; bumping to 8.x is out of scope for this patch.
…flicts
Resolves conflicts across 12 files created by main's independent evolution
(read-only D1 chokepoint getDb(), CVE suppressions, migration/backfill
scripts) diverging from this branch's contract health index work:
- osv-scanner.toml: keep both independent CVE suppressions
- apps/web/app/routes/{analytics,trends,quality}.tsx: keep this branch's
quality/health-index features and trends explorer rewrite, adopt main's
getDb(context.cloudflare.env) chokepoint in place of raw env.DB
- packages/db/src/*.test.ts, scripts/import.mjs, scripts/normalize-raw.sql:
keep this branch's full migration-chain loader (0000-0003) and the
Contract Quality / Health Index derive/gate wiring, combine with main's
async integrity-check runner and amendment-currency CTE
- scripts/integrity-checks.mjs: merge main's async runner/checks (D1
support) with this branch's checks-param narrowing; fix
checkContractFeaturesIntegrity, which relied on a since-async'd
tableExists()/rows() without awaiting them (always-false self-skip)
- packages/db/src/{contractor-identity,etl-entity-canonicalization}-sql.test.ts,
migrations.test.ts: extend fixture migration chains to include 0003 now
that normalize-raw.sql/precompute.sql depend on its health-index columns
- apps/web/app/routes/trends.test.ts: mock the new getDb export
- pnpm-lock.yaml: regenerated via pnpm install
Full typecheck + test matrix (all 7 packages) green after resolution.
…dvisories doc Lint and Docs integrity checks were red on CI. prettier --check flagged unformatted code in scripts/integrity-checks.mjs; check-docs flagged docs/security-advisories.md as an orphan doc not linked from docs/README.md.
This was referenced Jul 28, 2026
todorkolev
added a commit
that referenced
this pull request
Jul 29, 2026
… + hardening) (#212) * perf(db): ordering indexes for the non-default list sorts The list pages keyset-paginate with ORDER BY <sortExpr> <dir>, <id> <dir> LIMIT N. Six user-selectable sorts had no matching index, so the planner fell back to a full table SCAN + temp-B-tree ORDER BY on every page (D1 bills rows scanned): /contracts date-desc, date-asc (idx_contracts_signed is on the bare column, not the COALESCE(signed_at, ...) expr the query uses) /companies count, authorities /authorities count, avg Add one index per missing sort, matching the exact ORDER BY expression plus the keyset id tiebreak, so SQLite walks the index and stops at LIMIT. Additive, idempotent; rollup tables are DELETE+INSERT-refreshed so the indexes survive ships. A sqlite3 EXPLAIN QUERY PLAN test proves each sort full-scans before and index-walks after. * fix(web): escape < in the JSON-LD data island (defense-in-depth) root.tsx embeds JSON-LD via dangerouslySetInnerHTML with a raw JSON.stringify. JSON.stringify does not escape '<', so a '</script>' in any string value would close the <script> element early (stored XSS) — the exact sink the project's own review standard (docs/review-security.md) requires be escaped. Today only the request origin reaches the graph (new URL() cannot make it carry '</script>'), so this is not currently exploitable; the jsonLdScript helper closes the sink pre-emptively for any DB/user-derived field added later. A unit test proves '<' is escaped, U+2028/U+2029 are escaped, and the output stays JSON-equivalent. * chore(db): renumber list-sort-indexes migration 0002 → 0005 De-conflict the migration number: 0002 is claimed by the contracts_overrun_index family (#169/#170/#171/#172), 0003 by #188 (contract_health), and 0004 by #210 (cpv_division_stats). 0005 is the next free number. Additive/idempotent, so final merge order stays the maintainer's call; this just removes the known 0002 clash. * test(db): apply all migrations + cover keyset pages; guard jsonLdScript(undefined) Address the review notes on the list-sort-indexes PR: 1. The sort-index test now applies EVERY migration on the branch (discovered from the migrations dir), not a hardcoded 0000/0001/000N subset. The "BEFORE" base is exactly the real served schema minus this PR's index, and the test survives any renumbering. (Confirmed: company_totals/authority_totals are created in 0000 and nothing between affects these sort plans.) 2. Each sort now asserts the plan on the keyset page too - the real paginated path `WHERE (expr <cmp> ? OR (expr = ? AND id <cmp> ?))`, not only the first page. Full-scans BEFORE and index-walks (no temp B-tree) AFTER, on both pages. 3. jsonLdScript now returns "null" when JSON.stringify yields undefined (undefined / function / symbol) instead of throwing on the following .replace - defense-in-depth for the documented "safe for any future field" helper. Covered by a test. * refactor(web): rename jsonLdScript → serializeJsonForScript + document sort-index sync Address the (non-blocking) review nits: - Rename jsonLdScript to serializeJsonForScript: the helper returns a serialized JSON string safe to embed in an inline <script>, not a <script> element (review ydimitrof). Updates root.tsx and the test. - Document the sentinel sync: the COALESCE defaults in queries/contracts.ts SORTS ('' / '9999-99') must stay byte-identical to the expression indexes, or SQLite silently drops the index and falls back to a full scan + temp-B-tree sort. Added reciprocal SYNC comments in the migration and the SORTS map, both noting that list-sort-indexes.test.ts's EXPLAIN assertions catch a drift. * docs(db): state the boundaries of the sort-index guarantee (review) Document the two known limits of the EXPLAIN-plan proof, per review: (1) the local sqlite3 CLI planner is not version-identical to Cloudflare D1's (a strong indication, not a bit-exact production proof; the binary itself is a pre-existing suite-wide dependency), and (2) the index-walk guarantee covers the UNFILTERED sort paths - with an active filter the planner may prefer the filter's index and temp-sort the much smaller filtered set, which is the correct trade. Comment-only. * chore: drop internal review-marker traces from code comments Remove the '(review ydimitrof)' attribution artifacts from json-ld.ts and list-sort-indexes.test.ts comments; the explanations stay. Comment-only. * refactor(web): share one JSON-for-script serializer between the JSON-LD island and .json route The .json contract endpoint had its own safeJson escaper, a second implementation of the same <script>/separator escaping as serializeJsonForScript - a DRY smell the comment itself admitted, and a drift risk (one could add a U+2028 escape the other lacks). Route it through the shared serializer instead. It escapes every `<` (vs the old `</`-only form) - JSON-equivalent, harmless for the JSON body, strictly safer. Also document, in the shared helper, why `>` and `&` are deliberately left unescaped (only `<` can start a token in a script raw-text context), with a test that locks it. * fix(web): set nosniff on the .json route; add planner-independent sentinel-sync test - contract.json.tsx: the actual MIME-sniffing defense is X-Content-Type-Options: nosniff, not the content escaping. The worker already sets it globally (baseSecurityHeaders); set it explicitly on this resource route too so it is safe on its own, and correct the comment that over-credited the escaping (review). - Add sort-index-sentinel-sync.test.ts: the date-sort index only matches while its COALESCE sentinel is byte-identical to SORTS in queries/contracts.ts. A .sql migration can't import a TS constant, so guard the coupling with a static cross-file check of the sentinels ('' and '9999-99') that fails on drift regardless of the DB engine - independent of the local sqlite3 planner the EXPLAIN test relies on (review). * chore: drop stray review-marker artifacts from this PR's comments Remove the bare '(review ...)' attribution notes I left in contract.json.tsx and sort-index-sentinel-sync.test.ts; the explanations stay. The pre-existing '(review #80)' issue references elsewhere are an established convention and are untouched. Comment-only. * test(db): cover filtered list sorts and guard the sqlite3 dependency Two review follow-ups on the ordering-index test: - Filtered sorts were documented as out of scope, leaving the reader unable to tell whether an active list filter makes the ordering index redundant. It does not: with a sector (tenders.cpv_code) or eu-funded filter the planner still walks idx_contracts_signed_desc and drops the sort step, while the pre-index baseline sorts the whole table. Asserted both directions. - A missing sqlite3 CLI surfaced as an opaque ENOENT. Probe it in beforeAll and fail with the fix. Deliberately not a skip: this is a perf/cost gate, and silently passing it on an image without sqlite3 would retire the gate. --------- Co-authored-by: Rumen Slavov <26761822+B353N@users.noreply.github.com> Co-authored-by: todorkolev <tkolev@obecto.com>
todorkolev
added a commit
that referenced
this pull request
Jul 29, 2026
…аницата на договора (#210) * feat(db): per-cpv-division value percentiles rollup + cohort benchmark query * feat(web): similar-contracts value benchmark on the contract page * fix(web): harden similar-contracts cohort labels + ETL placement (review) Addresses the Request-Changes review on the „Подобни договори" benchmark: Label correctness (cohortBand, was `>=` cascade over a shared, self-inclusive grid): - #1 ties: strict `>` for the top bands + require each anchor to be strictly above the next coarser one, so a tie-collapsed cohort (p99=p95=…=median) or a value merely equal to an anchor never reads as „top 1%". Symmetric guard on „bottom 25%". - #2 tiny cohorts: per-band minimum cohort sizes (top1≥100, top5≥40, top10≥20, top25≥12), so the single most-expensive contract in a 12-row cohort is not „top 1%". The self-inclusion caveat is now stated in the UI + methodology, not only in code. - #3 median: a value at the nearest-rank median maps to a new `at-median` band instead of „above median"; strict comparisons on both sides. - #5: UI + methodology note that this uses a different method than the anomaly report, so the two figures may differ. Redundant read (review comment 3): the cohort is now computed inside getContract from the row it already read (stats load in the existing Promise.all), returned as `ContractDetail.cohort` — no second contract scan, no added latency. ETL (#4): the ~150k-row window recompute moved out of the shared `globals` batch into its own `@refresh-batch cohort-stats` step, so its CPU cost can't fail the whole globals step. Coordination: renumbered this migration 0002 → 0003 to de-conflict with the 0002 in the list-sort-indexes branch. Regression tests cover each labelling case (ties, tiny cohort, at-median, non-distinct anchors) and the pure contractCohort gate. * fix(web): drop the false „2020 г. - днес" scope claim from the cohort label The cpv_division_stats cohort has no signed_at bound — it spans every priced clean-value contract in the division (matching sector_totals and the /contracts list, which are also not date-bounded), incl. pre-2020-signed and undated rows. The UI hint and methodology claimed „(2020 г. - днес)" / „от 2020 г. насам", promising a window the query never enforces (review lyubomir-bozhinov). Fix the copy rather than add a date bound: bounding only the cohort would break the cohort↔rollup membership invariant (a shown contract must be a real member of the cohort it is compared against) and diverge from the site-wide value basis. The methodology now states the comparison is over the full corpus, same scope as the sector totals. * chore(db): renumber cpv-division-stats migration 0003 → 0004 De-conflict with the 0003 already claimed by #188 (contract_health). 0002 is taken by the list-sort-indexes branch, so the next free number is 0004. Final merge order stays the maintainer's call; this just removes the known 0003 clash. * perf(db): skip the cohort rollup read for non-clean-value contracts getContract gated the cpv_division_stats PK read only on CPV presence, but contractCohort returns null anyway when value_flag <> 'ok' or amount_eur is null/<=0 - so a suspect/valueless contract paid for a read it always discarded, against the PR's own "one extra rollup read, not a second scan" goal (review ydimitrof). Apply the clean-value gate to the read itself; behaviour is identical. Also close the integration gap in details.test.ts: a case with a real cpv_division_stats row now asserts detail.cohort is populated end-to-end (catches an argument-order swap into contractCohort), plus a case proving the read is skipped entirely for a non-clean value. * style(web): use the Bulgarian closing quote (U+201C) in the cohort sector label The „Подобни договори" section opened with „ (U+201E) but closed with a straight " (U+0022). Match the Bulgarian pair „…“ (review ydimitrof). Cosmetic only. * fix(web,db): repair duplicated imports and pin-free migration list Two breakages surfaced once CI actually ran on this branch (the fork workflow runs were awaiting approval, so neither had been reported): - contract.tsx carried both sides of an earlier main merge, so contractIdFromSlug/getContract/ContractDetail were each imported twice and typecheck failed with TS2300. Keep the main-side @sigma/db import (it has contractSlug and getDb) and the branch-side type import (it adds CohortBand). - precompute-cohort.test.ts pinned the migration list to 0000/0001/0004, skipping 0002_current_value_currency. precompute.sql reads that column, so the test DB no longer matched the served schema and sqlite3 aborted with 'no such column: current_value_currency'. Read every migration from the directory instead, sorted - the same approach the sort-index test uses. --------- Co-authored-by: Rumen Slavov <26761822+B353N@users.noreply.github.com> Co-authored-by: Todor Kolev <tkolev@obecto.com>
todorkolev
pushed a commit
that referenced
this pull request
Aug 8, 2026
safeD1 tested err.message for "no such table", but wrangler prints the SQLite error as JSON on stdout and leaves the exception message as a bare "Command failed: wrangler d1 execute ...". The guard therefore never matched, safeD1 rethrew instead of returning [], and the data_freshness fallback in latestLoadedDate was unreachable. The visible effect is that `pnpm run import --catchup --plan-only` crashes whenever the transient staging tables are absent, which is their normal steady state: drop-transient-staging removes them in a finally, and --plan-only exits before the main flow recreates them. Ordinary runs are unaffected, since they create an empty raw_contracts first and so reach the fallback. Cherry-picked verbatim from e3fc6b3 in #188 by Bilko (StanislavBG), split out so the fix can land without the contract health index feature.
todorkolev
added a commit
that referenced
this pull request
Aug 8, 2026
safeD1 tested err.message for "no such table", but wrangler prints the SQLite error as JSON on stdout and leaves the exception message as a bare "Command failed: wrangler d1 execute ...". The guard therefore never matched, safeD1 rethrew instead of returning [], and the data_freshness fallback in latestLoadedDate was unreachable. The visible effect is that `pnpm run import --catchup --plan-only` crashes whenever the transient staging tables are absent, which is their normal steady state: drop-transient-staging removes them in a finally, and --plan-only exits before the main flow recreates them. Ordinary runs are unaffected, since they create an empty raw_contracts first and so reach the fallback. Cherry-picked verbatim from e3fc6b3 in #188 by Bilko (StanislavBG), split out so the fix can land without the contract health index feature. Co-authored-by: Bilko <StanislavBG@gmail.com>
1 task
Collaborator
|
Този клон е в конфликт с |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
Contract quality index (ETL score 0..1 +
/qualitypage). Latest round addresses ydimitrof's 2026-07-13 review (9 threads):scripts/import.mjs:runFullDerive/runSliceDerivenow gate oncheckContractFeaturesIntegrityin addition to the standardCHECKS, so the local full/slice/health ETL path enforces the samecontract_featuresinvariants (row parity, A1 floor, score_b=0 on direct award, no value_suspect leakage) asship-domain.mjs.scripts/precompute.sql: dropped theCOALESCE(tenders.published_at, date('now'))FX-rate fallback — aNULLpublished_atnow resolvesestimated_value_eurtoNULLinstead of silently borrowing a same-day rate that could mis-date historical foreign-currency estimates.scripts/validate-health.mjs: fixed a false-FAIL where the synthetic'NA'bucket (undated/out-of-range contracts) tripped the >60%-NULL threshold check meant only for 2020–2026 strata; and fixed the Spearman-lite decile-correlation check printingNaNon zero-variance samples (now logs and skips).apps/web/app/lib/trendAxis.ts: removed the in-codeTODO(#170)note — the third-copy consolidation is tracked solely via issue feat(web): договори — обзор (лещи време/CPV/кръстосано) #170 per repo convention.filters.tsranking params (rdir/rfrom/rto) are fully parameterized via.bind(...)inpackages/db/src/queries/quality.ts(no string interpolation);analytics.tsx'sgetQualitySummary(...).catch(...)correctly no-ops on the expected pre-derive "no such table" case; migration0003_contract_health.sql's out-of-numeric-order columns are purely additive and independent of the still-pending0002_contracts_overrun_index, and all target environments apply viawrangler d1 migrations apply(tracked by name, not raw re-execution); theship-domain.mjsnon-atomic DROP/CREATE availability window oncontract_featuresis a known, already-documented follow-up (staging-table swap), out of scope here.How it was tested
pnpm --filter web testandpnpm --filter db test— both pass, including regression coverage added for the axis/relLabel edge cases and the schema-drift guard extension from the prior round.node:sqlitefixture trace re-confirmed thevalidate-health.mjsNA-bucket and zero-variance fixes against a synthetic dataset.Quality checks
normalize-raw.sqlandrefresh-slice.sql.