Legislation: CTH sync fetched zero titles and parsed zero sections — fix the enum filter and the span-per-word extractor - #69
Merged
Conversation
…ns — enum filter + span-per-word markup (refs TailorAU/tailor-group#7)
Two independent breaks, both silent, both required for the weekly
legislation-sync cron to refill the corpus:
1. Titles filter. `status` and `collection` are OData enums on
api.prod.legislation.gov.au. `collection eq 'Act' and status eq
'InForce'` answers 400 "Could not find a property named 'InForce'"
(each alone parses; the conjunction mis-binds the literal). The loop
recorded it as one error string with docs_checked = 0 — under the
silent-zero alarm's threshold. `status in ('InForce')` binds correctly
(verified live 2026-09-18: 4,768 in-force Acts). Paging gets a stable
tiebreak (`year desc,number desc`); a Titles-fetch failure now also
increments parser_crash_count; the newest-first ceiling is env-tunable
(CTH_SYNC_MAX_ACTS, default 50); parser stamp → cth-parser@2.1.0.
2. Text extraction. The EPUB HTML now wraps every run of text in its own
<span> (indent spacers as `<span style=…> </span>` before the
words). The old extractor stopped at the first closing tag, harvested a
non-breaking space, and every section fell under the 10-char floor →
"No sections parsed" for every act. The extractor now takes each
provision-level <p> block whole to its </p>, strips markup, decodes
entities, and widens the markup slice (text is still capped at 4,000
chars).
Evidence: with both fixes the eight newest in-force Acts yield 3–35
sections each (35/38 headings for C2026A00086, 25/25 for the Combatting
Illicit Tobacco Act 2026). Offline tests pin the URL shape and parse a
real document_1.html excerpt (src/lib/fixtures/cth); CTH_LIVE=1 runs the
same against the live API and is skipped in CI.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_5a4aa8b4-4bfc-4706-b03d-141adb7fa859) |
Tailor-AUS
pushed a commit
that referenced
this pull request
Sep 18, 2026
…AU/tailor-group#7)
pr-check.yml's lint step still named the monorepo's
source-legislation-ingest.yml, source-pr-check.yml, cron-source.yml,
cd-source.yml and infra-source-setup.yml, none of which tailor-app#5949
re-rooted here, so actionlint exited 3 ("no such file or directory") on
every PR to rehome-review — PR #68 and #69 both red at this step with no
finding of their own. Lint pr-check.yml and cd-kg.yml instead, keeping
the two narrow ignores.
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Bugbot couldn't run - usage limit reachedBugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit. A user or team admin can review and increase usage limits in the Cursor dashboard. (requestId: serverGenReqId_42c8f730-f1a3-4cd6-9da4-262b71eba591) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Refs TailorAU/tailor-group#7 (step 2 of 3, code half). Base is
rehome-review;mainis untouched. Independent of #68 (no shared files).What was wrong
The weekly
legislation-synccron (tailor-appcron-source.yml, Sunday 06:00 UTC →pact.tailor.au/api/cron/legislation-sync) has produced no CTH documents for as long as the retained run history goes; last Sunday's job failed outright.pact.tailor.auserves 11 legislation documents today (QLD 3, SA 4, TAS 4) and zero CTH. Two independent breaks:statusandcollectionare OData enums onapi.prod.legislation.gov.au.collection eq 'Act' and status eq 'InForce'answers400 Could not find a property named 'InForce'(each clause parses alone; the conjunction mis-binds the literal).syncCthrecorded it as one error string withdocs_checked = 0, which sits below the silent-zero alarm's threshold — so "ran, 0 updated, 1 error" looked like a quiet week.<span>, with indent spacers as<span style=…> </span>ahead of the words. The old regex stopped at the first closing tag, harvested a non-breaking space, and every section fell under the 10-char floor →No sections parsed for …for every act. Even with (1) fixed, the corpus would not have refilled.What this does
collection eq 'Act' and status in ('InForce')— verified live 2026-09-18, 4,768 in-force Acts.$orderby=year desc,number descso$skippages are stable (ties were unordered).<p class="subsection|paragraph|…">block whole to its</p>, strip markup, decode&#xNN;/named entities, widen the markup slice (text still capped at 4,000 chars).parser_crash_count(not onlyerrors); parser stampcth-parser@2.1.0onlegislation_sync_log.CTH_SYNC_MAX_ACTS(default 50, newest-first) for a wider refill or a narrow smoke run.Evidence
Before this PR every one of these parsed to 0.
src/lib/parsers/cth-parser.test.ts: URL shape (neverstatus eq), stable paging, crash accounting, and a parse of a realdocument_1.htmlexcerpt (src/lib/fixtures/cth/, C2026A00003).CTH_LIVE=1additionally runs the live API and passed locally; CI skips it.npm test: 44 files / 738 pass (2 live tests skipped).npx tsc --noEmitclean. eslint clean on the changed files.Not in this PR
cron-source.yml). The consensus-sweep job in that workflow is failing every 30 minutes with HTTP 500 frompact.tailor.au/api/cron/auto-merge— unrelated to legislation, flagged in the tailor-group#7 report.ActHead6+,Item); the parser still ingestsActHead5sections only, as before.🤖 Generated with Claude Code
Note
Medium Risk
Changes external API query semantics and HTML parsing that drive legislation ingest volume and content quality; failures are better surfaced but a bad regex could still mis-parse or skip acts until the next run.
Overview
Fixes the Commonwealth legislation sync so weekly runs can again discover in-force Acts and extract section text from current EPUB HTML.
The Titles OData filter is corrected from
status eq 'InForce'(400 when combined withcollection eq 'Act') tostatus in ('InForce'), with$orderby=year desc,number descso paged$skipdoes not repeat or drop acts.buildTitlesUrl/fetchInForceActsare exported for tests; the per-run cap isCTH_SYNC_MAX_ACTS(default 50).parseActHtmlnow reads whole provision-level<p>blocks (span-heavy markup), decodes HTML entities, and widens the markup slice while keeping extracted text capped at 4,000 characters. A Titles API failure incrementsparser_crash_count(not onlyerrors), the parser stamp iscth-parser@2.1.0, and tests plus a realC2026A00003HTML fixture lock filter shape, paging, crash accounting, and parsing; optionalCTH_LIVE=1exercises the live API.Reviewed by Cursor Bugbot for commit b1852e7. Bugbot is set up for automated code reviews on this repo. Configure here.