fix(cacbg): фантомни редове и --limit обезсилваха гейта за пълнота - #281
Merged
Conversation
…ness gate Two ways the corpus completeness gate could report a verdict it had not earned. parseList announced any row with a truthy xmlFile. The register also emits placeholder rows that name no document — <Sent>False</Sent><xmlFile>U</xmlFile> with <Title>Уведомление</Title>, 87 of them across 15 of the 35 sets. 'U' passed the truthiness test, so the crawler announced a declaration that does not exist, failed to fetch it, and booked an error. Those 15 sets could therefore never satisfy the gate: a full crawl of the register ends in a non-zero exit and no ship, forever. Require the filename shape instead, reusing guard.mjs's — isXmlFile is the boolean twin of safeXmlFile, one source of truth for what a declaration file looks like. --limit truncated the work before `announced` was read, so a deliberately partial crawl reported announced == obtained. Read `announced` from the set, before the slice. That alone is not enough: rows never attempted produce no errors, and `incomplete` was derived from errors, so the gate still passed. Every announced row lands in exactly one bucket, so a full pass satisfies announced == obtained + sourceGaps + unfetched; a surplus means rows were never tried. Count it as notAttempted and fail on it. Verified against the full corpus: 256 286 announced rows, of which 87 phantoms; the remaining 256 199 are matched exactly by 255 582 obtained plus 617 source-side 404s, so a complete crawl now reports complete. Both fixes are mutation-checked — reverting either one fails exactly the tests written for it. Refs #279
Test coverage
✅ No workspace dropped below its baseline (tolerance 0.5pp). 📈 Coverage rose by more than 1pp — run |
midt-admin
approved these changes
Aug 4, 2026
lyubomir-bozhinov
added a commit
to lyubomir-bozhinov/sigma
that referenced
this pull request
Aug 4, 2026
…acbg gate midt-bg#281) Two commits landed on main while the midt-bg#226 sync was in flight — an undici override bump to ^7.29.0 and a cacbg completeness-gate fix (phantom rows / --limit). Clean merge; the surface is pnpm-workspace.yaml + scripts/cacbg, none of it inside the six measured workspaces, so the coverage baseline is untouched and the gate stays green. Verified against the merged tree: scripts tests 51 pass, cacbg pipeline tests 84 pass (run via the register-ts loader, as the CI job does), coverage ratchet green for all six workspaces.
10 tasks
LyuboslavLyubenov
added a commit
to LyuboslavLyubenov/sigma
that referenced
this pull request
Aug 5, 2026
Clean automatic merge: no conflicts. Upstream advanced 3 commits since the PR's previous rebase (related-persons midt-bg#226, undici bump midt-bg#282, cacbg fix midt-bg#281); none of those touch the PR's test-only surface (apps/web/test/integration/*, docs/spec/integration-testing.md, apps/web/vitest.integration.config.ts). The single auto-merged file is docs/README.md, which gained a new ADR entry in upstream (0032); the merge preserves the alphabetical/numerical ordering without re-flowing the PR's content. Verification (local): - pnpm typecheck → 7/7 packages clean - pnpm --filter @sigma/web test → 530 passing (52 files, 8 integration files, 41 integration tests, 0 .skip) - pnpm lint → (run separately)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Предпоставка за първото зареждане с данни към staging. Без нея пълният обход на регистъра завършва с ненулев изход и ship изобщо не се стига. Двете находки са от прегледа на #226 (моята за
parseList, тази за--limitе на @ydimitrof) и са описани в #279.1. Фантомни редове в
parseListparseListобявяваше всеки ред с истиненxmlFile. Регистърът обаче издава и редове-заместители, които не именуват документ:Има 87 такива в 15 от 35-те набора.
Uминаваше проверката за истинност, значи краулерът обявяваше декларация, която не съществува, после не успяваше да я изтегли и записваше грешка. Резултатът: тези 15 набора никога не можеха да удовлетворят гейта - пълен обход завършва с изход 1 и няма как да се публикува.Поправката иска формата на име на файл, взета от
guard.mjs:isXmlFileе булевият близнак наsafeXmlFile, тоест формата остава на едно място.parseListне бива да хвърля, защото липсата на име там не е грешка, а нормален заместител.2.
--limitзаобикаляше гейтаannouncedсе четеше след среза, значи нарочно частичен обход докладвашеannounced == obtainedи гейтът минаваше върху корпус, който никога не е бил опитван.Само преместването на
announcedпреди среза не стига: неопитаните редове не пораждат грешки, аincompleteсе извеждаше от грешките. Всеки обявен ред попада в точно една кофа, тоест пълен обход изпълняваannounced == obtained + sourceGaps + unfetched; излишъкът значи неопитани редове. Броим го катоnotAttemptedи гейтът пада на него.Проверка
Срещу пълния корпус: 256 286 обявени реда, от които 87 фантомни; останалите 256 199 се покриват точно от 255 582 изтеглени плюс 617 липсващи в източника (404). Тоест пълен обход вече докладва „пълен".
Тестовете са мутационно проверени - връщането на всяка от двете поправки поотделно чупи точно нейните тестове (2 и 1). Всичките 84 теста в
scripts/cacbgминават с командата на CI.Затваря частично #279 (§12 и последната точка от раздел 1).