[Данни]: Двойно броене на стойността в анексите (нова обща стойност се добавя към старата)

[todorkolev]

Какъв е проблемът с данните

Когато анексът обявява нова обща стойност на договора, тя се записва като добавка към старата, вместо да я замени. Изходната стойност се брои двойно.

Измерено на пълен локален пребилд от 12.08.2026 (период 2020 - 11.08.2026, 26 921 анекса).

Доказателството - един запис, сверен с текста на самия анекс. УНП 00010-2023-0006, договор 145652:

Какво казва анексът	Какво трябва да е	Какво има в базата
„442 000.00 лв. без ДДС се актуализира с 22% или общата стойност на договора ще възлезе на 539 240.00 лв. без ДДС"	стойност след: 539 240 лв. (ръст +22%)	стойност след: 981 240 лв. (отчетен ръст +122%), тоест 442 000 + 539 240

Мащаб. Дефектът има математически подпис: щом новата обща стойност е по-голяма от старата, сборът винаги дава ръст от 100% нагоре.

• 7 335 анекса вдигат цената;
• от тях 686 са с ръст 100% или повече - заподозрените;
• от тези 686 текущите флагове хващат само 20;
• остават 666 незасечени, на стойност 475.4 млн. EUR.

Защо #299 не ги хваща. Правилото от #299 (единична стъпка >=10x при сбор >=5x) се цели в друга повреда - сбъркано число, надуло договора десетки пъти. То иска сборът да стигне 5 пъти подписаната стойност, а двойното броене дава около 2 пъти (в примера горе - 2.22). Тоест стои трайно под прага и по конструкция не може да бъде хванато от него. #299 си свърши работата за своя случай: на ниво договор вдигна засечените от 3 на 19.

Подслучай: 199 записа точно на +100%. Когато анексът обявява същата обща стойност, сборът дава точно удвояване. Двата най-големи са недвусмислени:

• УНП 00210-2024-0024, договор 189325: 77 000 000 -> 154 000 000 лв. Текстът на анекса: „Считано от 01.10.2025 г., отпечатваната върху акцизните бандероли за тютюневи изделия продажна цена се променя от лева в евро." Тоест анексът само сменя валутата, а стойността е удвоена.
• УНП 00080-2023-0001, договор 84818: 76 769 540.87 -> 153 539 081.74 EUR. Анексът само преструктурира курсове за обучение в рамките на гаранционния период.

Само тези 199 записа внасят 193.4 млн. EUR фиктивен ръст.

Сляпо петно на подписа. Ако анексът обявява обща стойност, която е по-ниска от старата, двойното броене дава ръст под 100% и се скрива сред „чистите" записи. Тоест прагът от 100% е предпазна мярка, а не доказателство за чистота - не може да се твърди, че анексите под 100% не са засегнати.

Флаговете лекуват договора, не анекса. Дори за 20-те засечени записа value_flag поправя показаната стойност на договора, но редът в amendments продължава да носи удвоеното value_after. Времевата линия на анексите на страницата на договора показва грешната стойност.

Източникът е консистентен, дефектът идва от входа. value_delta съвпада с value_after - value_before при 26 826 от 26 864 записа (99.9%), тоест ЦАИС ЕОП е вътрешно непротиворечив и грешното число пристига отвън, а не се получава при сметка в СИГМА.

Засегнати консуматори надолу по веригата:

• PR feat(anomalies): add automated price-anomaly screen #239 (/anomalies) - сигналът „Ръст чрез анекси" (текуща стойност >= +20% над подписаната) брои и тези записи;
• PR feat(web): overruns dashboard #171 (/overruns) - същото;
• сумите по компании и възложители и износът към CSV.

Възможна посока за поправка. Числата в свободния текст на основанието са четими: в 355 записа стойността, равна на делтата, стои в контекст на „обща / общата / крайна / възлиза", което е сигнатурата на обявена обща стойност. Проверка от вида „ако value_after приблизително е равно на value_before + число, срещано в текста като обща стойност, тогава замени вместо да събереш" изглежда изпълнима, но иска внимание - евристиките върху свободен текст лесно дават лъжливи попадения. Минималната мярка е флаг, който сваля тези записи от агрегатите, както #299 прави за своя случай.

Конкретен пример (адрес / ЕИК / УНП)

00010-2023-0006 (договор 145652); също 00210-2024-0024 и 00080-2023-0001

Източник за правилната стойност (по желание)

Текстът на самия анекс в ЦАИС ЕОП: „442 000.00 лв. без ДДС се актуализира с 22% или общата стойност на договора ще възлезе на 539 240.00 лв. без ДДС"

Преди да изпратите

• Потърсих съществуващ issue за същия запис или проблем.

_{Проверени за дублиране: #248 (затворен, за сбъркано число в анекс - друг случай), #286 (затворен, за OCID вместо УНП), #245 (затворен, за двойна конверсия на евро анекси), #247 и PR #304 (стотинки), както и отворените PR-и по пътя на анексите.}

[cefothe]

Investigation, fix, and coverage for #305

Verified the double-count on the real corpus (sigma-dev) and root-caused it in code, then implemented a two-tier fix. Summary below; a fuller-text coverage re-measure is running and I'll follow up with those numbers.

Root cause (confirmed)

packages/ingest/src/base.ts stores the annex values verbatim from ЦАИС ЕОП (value_after ← currentContractValue, value_delta ← contractValueDifference) — no arithmetic. For the affected annexes the authority put the new total into the change field, so currentContractValue = lastContractValue + newTotal arrives already doubled. The 99.9% value_delta = value_after − value_before consistency is the tell that the source is self-consistent but the "difference" is semantically a total. The three named records all confirm: 189325 77M→154M, 84818 76.77M→153.54M (EUR), 145652 442k→981,240 (where value_delta = 539,240 is the announced total).

Fix (two tiers)

• Tier 1 — flag & exclude. New value_flag = 'annex_total_suspect' for a single annex whose value_after ≥ 2×value_before (ЗОП чл.116 caps a single amendment at +50%, so a ≥2× step is a defect). Threaded through the value fallback exactly like annex_suspect, so these contracts fall back to signing_value and drop out of every EUR aggregate (amount_eur, current_value_eur, totals, CSV, contract page) — the "minimal measure" the issue calls for.
• Tier 2 — text-confirmed correction. A heuristic reads the основание text and classifies each annex by the preposition in front of the figure: "…на <N>" / "обща стойност … <N>" = a new total → the value is restated to N; "…с <N>" = a genuine increment → the value is correct and is not flagged; a currency re-denomination ("лева в евро" + exact 2×) → restated to the before-value. The corrected value drives current_value and the served amendment row (with a value_restated marker), so the amendment timeline shows the right number too — addressing the issue's point that flags fix the contract but not the amendment row.

This follows the issue's own proposed direction ("if value_after ≈ before + a number the text calls a total, replace instead of add") and is deliberately conservative — it only rewrites a value it can corroborate from text (the 108677-class genuine +100% increase is exonerated, not mis-flagged).

Coverage on the real corpus (lower bound — description field only)

Over the 642 doubled (value_after ≥ 2×value_before) amendments on sigma-dev:

Outcome	Count	~value_after (EUR)
Corrected — restated total	145	€130.2M
Corrected — currency re-denomination	1	€78.7M
Exonerated — genuine increment	9	€0.28M
No text signal → Tier-1 arithmetic flag	487	—
…of which Tier-1 flags (excluded from aggregates)	~241	—

Net: 146 corrected (~€209M restated to true totals), 9 legit increases exonerated, ~241 more flagged vs the "existing flags catch 20". The served table drops reason/circumstances, so this measures description only — the real ingest uses all three text fields, so production coverage is higher (re-measuring now).

Honest residuals (as the issue itself anticipates)

• Sub-100% blind spot (a new total lower than the old value → <100% growth): not detectable by this signal — out of scope.
• No-text-signal doublings (e.g. 84818's restructuring text, or 103903 where the true total ≠ delta): left to the Tier-1 flag (excluded), not rewritten — better to flag than to mis-restate.

Downstream

Company/authority/home totals, CSV, and the contract page all read amount_eur, which now corrects (Tier 2) or excludes (Tier 1). /anomalies (#239) and /overruns (#171) aren't on main yet; they'll inherit the corrected base when they land.

[cefothe]

Follow-up: full three-field text coverage (measured)

Re-ran the coverage through the real ingest over all 26,865 EOP annexes (2020→2026), using all three основание fields (description + reason + circumstances; confirmed 100% populated), not just description. The doubled set (value_after ≥ 2×value_before) is 686 — exactly the number in the issue.

Treatment (full 3-field text)	Count	Outcome
total_restated	159	value_after corrected to the announced total ("…на <N>")
currency_restated	2	corrected to the before-value (re-denomination)
genuine_increment	10	exonerated — a real increase, not flagged
none	515	no text signal → left to the Tier-1 arithmetic flag

Net: the text tier corrects 161 (~€210M restated to true totals) and exonerates 10 genuine increases — 171 / 686 (25%) resolved from text. The 515 none go to the Tier-1 annex_total_suspect flag, which excludes the ~signing-matching, same-currency subset (~241) from every EUR aggregate; the remainder (~270) are doublings with neither a textual total nor a clean arithmetic signature (multi-annex chains, currency mismatches) and stay uncaught.

Named records: 145652 → total_restated 539 240 ✓; 189325 → currency_restated 77 000 000 ✓; 84818 → none — its "курсове … се преструктурират" text carries no total figure, so it is genuinely uncorrectable from text (falls to the flag, not rewritten).

Note vs the earlier description-only estimate (146/1/9/487): using all three fields added only +15 corrected / +1 exonerated — the value-restating phrasing ("…на <total>") lives almost entirely in description; reason/circumstances (ЗОП grounds) rarely repeat the figure. So the previous lower bound was already close to reality.

Scope, stated plainly: this does not claim all 686. It reliably handles the text-confirmed quarter (corrected) and the arithmetic same-currency subset (flagged/excluded), and it deliberately avoids false-positiving genuine >100% increases — but ~270 doublings with no textual total and no clean arithmetic signature, plus the sub-100% blind spot, remain out of reach for this heuristic.

[cefothe]

How much of #305 is fixed — measured against the ticket

Accounting for all 686 annexes at ≥100% growth (the ticket's suspect set), measured through the real ingest over the full 2020→2026 annex corpus:

Bucket	Count	%	How
Corrected (right value shown)	362	53%	restated to the true total, or to the before-value for exact-2×
Flagged & excluded from aggregates	183	27%	annex_total_suspect (first-annex ≥2×) → falls back to signing
Handled by existing #299 (≥10× mis-keys)	106	15%	annex_suspect (already merged)
Exonerated (genuine increase, not a defect)	10	1.5%	text says "…с <N>"
Still uncaught (multi-annex, doubled in aggregates)	25	3.6%	escapes both tiers

Isolating the double-count band #299 cannot reach (686 − 106 = 580): 545 / 580 (94%) are handled (362 corrected + 183 flagged), 10 exonerated, 25 (4%) uncaught.

Point-by-point vs the ticket

#305 says	Status
New total added to old (double-count)	✅ fixed (362 corrected, 183 flagged)
Proof record 145652 → should be 539 240	✅ corrected to 539 240
199 exactly-+100% (currency/same-total), €193.4M — top two 189325 & 84818	✅ the 203 exact-2× restatements; both named records fixed
#299 can't catch (2× < 5×)	✅ new flag fires at ≥2×
"666 undetected, €475.4M"	✅ mostly — our undetected residual drops to 25; the €-inflation is corrected or excluded
Fix direction: "replace, don't add" + minimal flag	✅ implemented as proposed
Downstream totals / CSV / contract page inflated	✅ all read amount_eur → corrected or excluded
Flags fix the contract, not the amendment row (timeline)	◐ partial — fixed for the 362 corrected (value_restated marker); the 183 flag-only rows still show the doubled value_after in the timeline
Sub-100% blind spot (new total < old)	❌ out of scope — not detectable by this signal
/anomalies #239, /overruns #171	➖ not on main yet; will inherit amount_eur

Bottom line

~96% of the targeted double-count band is now corrected (53%) or excluded from every aggregate (27% + the 15% #299 already had); all three proof records resolve (incl. 84818). Residual: 25 multi-annex chains where the doubled step isn't the first, plus two gaps the ticket itself flags — the sub-100% blind spot and the 183 flag-only rows still displaying the doubled value in the amendment timeline (only corrected rows got the row rewritten). Those are left as documented follow-ups (multi-annex handling, v2 direct-total parsing, and cross-referencing the ЦАИС ЕОП contract feed for the no-figure annexes).

PR to follow.

[cefothe]

Update: precision fix + amendment-row (timeline) fix

Two improvements from a deeper pass (both in PR #307):

1. Precision fix — stopped over-correcting. The text detector's bare-"на" branch was reading Bulgarian "в размер на <N>" ("in the amount of N"), "ресурс … в размер на N" and "допълнителни … на обща стойност N" as the new total, when they name the change/added-work amount. Verified on the full corpus: this affected ~65 of 159 total_restated rows (incl. unit-price rows and understated genuine >100% increases). Added a veto (в размер / ресурс / допълнителн / увеличени / намалени): total_restated 159 → 94, "в размер на" contamination 69 → 4, all legitimate "от X на Y" / "възлиза на Y" totals retained (145652, 79382, 113291). The demoted rows fall to the arithmetic flag, so aggregate coverage holds while correction precision jumps.

2. Timeline fix — the flag-only rows no longer show the doubled value. This closes the "flags fix the contract, not the amendment row" gap. New per-amendment value_suspect marker (migration 0007), set at promote time for an untreated annex whose value_before ≈ signing_value and value_after ∈ [2×,10×); the contract-page timeline now blanks the after-value/delta and shows "— непотвърден тотал" instead of the doubled figure. Applies to the whole flag-only set, not just the 362 corrected rows.

Sub-100% blind spot — checked, staying out of scope. Quantified from the feed: of 6,649 sub-100% annexes a relaxed gate yields ~99% false positives (same "в размер на" trap); demanding a real totalizing verb leaves ~6 genuine rows (~€2.9M). Not worth the false-positive cost on financial values.

Net for the 686: ~304 corrected (high-precision), ~241 flagged & timeline-suppressed (annex_total_suspect/value_suspect), 106 already under #299, 10 exonerated, ~25 still uncaught (multi-annex chains). The remaining tail needs the ЦАИС ЕОП contract feed for the authoritative value rather than more free-text heuristics.

[cefothe]

Multi-annex double-counts now flagged — and where #305 hits a hard data limit

Landed (PR #307, commit 6709c57)

The annex_total_suspect arithmetic gate previously fired only when the doubled annex's value_before ≈ signing_value — i.e. it caught the double when it happened on the first annex. A later annex in a chain, whose value_before is the prior cumulative total (a preceding annex's value_after, not signing), slipped through.

The anchor is now relaxed across all gate sites — the contract-level flag (normalize-raw, refresh-slice, and the has_double CTE) and the per-row value_suspect marker (promote-amendments, refresh-slice):

value_before ≈ signing_value  OR  value_before ≈ any recorded prior annex value_after

The ≥2× single-step gate stays the defect signal — ЗОП чл. 116 caps a single amendment at +50 %, so a one-step ≥100 % jump is a defect wherever it sits in the chain. Slow legitimate multi-annex climbs never reach 2× per step, so they remain untouched. Positive and negative regression fixtures were added (they fail against the pre-change SQL and pass after), covering ~25 previously-missed later-in-chain rows.

Aggregate state of the double-count band

Bucket	Handling
Text-confirmed total restated (…на N)	corrected to the true total (total_restated)
Exact-2× echo / re-denomination / admin annex	corrected to value_before (unchanged_restated)
Text-confirmed increment (…с N)	exonerated, value kept (genuine_increment)
Arithmetic double, no usable text (single- and now multi-annex)	flagged annex_total_suspect → value falls back to signing, doubled figure blanked in the UI and excluded from every EUR aggregate
≥10× mis-key	handled by #299 (annex_suspect)

That covers the large majority of the affected rows: the value shown to users is either the corrected true total or a conservative signing-value floor — the doubled number is never displayed or summed.

Why the ticket can't be fully closed

The residual tail (rows flagged-but-not-corrected, annexes with no figure in the free text, and the sub-100 % blind spot) needs an authoritative post-amendment contract value to correct rather than merely suppress. A team-of-agents deep dive against the live sigma-dev corpus confirmed that value does not exist in any structured feed:

• ЦАИС ЕОП contract feed carries only contractValue (signing) + estimatedValue — there is no current/after-amendment value field.
• OCDS contracts[].value.amount is the pre-amendment / signing value. The true after-amendment total appears only in the unstructured Bulgarian description prose (the same text the Tier-2 heuristic already parses), and there are coverage gaps on several affected days.
• estimated_value cannot disambiguate total-vs-increment: the value_delta / estimated_value distributions for the two cases overlap almost completely (best achievable precision ≈ 0.45 — worse than a coin flip; 53 % of rows have an estimate ≥5× signing because it is a framework ceiling, not a delivered value).
• The sub-100 % blind spot (a restated new total that is below the old total) produces value_after < 2× value_before, so it does not trip the arithmetic signal at all and there is no text figure to anchor on. It is a genuinely undetectable residue (a handful of rows).

So the correctable set is bounded by what the основание free text states. Everything beyond that is suppressed, not corrected — the shown value is a safe signing-value floor and the untrusted figure is never surfaced, which protects the aggregates. Recovering the exact totals for the remainder would require a data source ЦАИС ЕОП does not publish in structured form.

Recommendation: merge #307 as the complete structured-data solution for #305, and treat the free-text-only residue as a documented, bounded limitation rather than an open defect.