Skip to content

Legislation: CTH sync fetched zero titles and parsed zero sections — fix the enum filter and the span-per-word extractor - #69

Merged
Tailor-AUS merged 2 commits into
rehome-reviewfrom
agent/open-graph-cth
Sep 18, 2026
Merged

Tailor-AUS merged 2 commits into
rehome-reviewfrom
agent/open-graph-cth

Conversation

@Tailor-AUS

@Tailor-AUS Tailor-AUS commented Sep 18, 2026

Copy link
Copy Markdown
Contributor

Refs TailorAU/tailor-group#7 (step 2 of 3, code half). Base is rehome-review; main is untouched. Independent of #68 (no shared files).

What was wrong

The weekly legislation-sync cron (tailor-app cron-source.yml, Sunday 06:00 UTC → pact.tailor.au/api/cron/legislation-sync) has produced no CTH documents for as long as the retained run history goes; last Sunday's job failed outright. pact.tailor.au serves 11 legislation documents today (QLD 3, SA 4, TAS 4) and zero CTH. Two independent breaks:

  1. Titles filter → 400. status and collection are OData enums on api.prod.legislation.gov.au. collection eq 'Act' and status eq 'InForce' answers 400 Could not find a property named 'InForce' (each clause parses alone; the conjunction mis-binds the literal). syncCth recorded it as one error string with docs_checked = 0, which sits below the silent-zero alarm's threshold — so "ran, 0 updated, 1 error" looked like a quiet week.
  2. Extractor → zero sections. The EPUB HTML now wraps every run of text in its own <span>, with indent spacers as <span style=…>&#xa0;</span> ahead of the words. The old regex stopped at the first closing tag, harvested a non-breaking space, and every section fell under the 10-char floor → No sections parsed for … for every act. Even with (1) fixed, the corpus would not have refilled.

What this does

  • Filter: collection eq 'Act' and status in ('InForce') — verified live 2026-09-18, 4,768 in-force Acts. $orderby=year desc,number desc so $skip pages are stable (ties were unordered).
  • Extractor: take each provision-level <p class="subsection|paragraph|…"> block whole to its </p>, strip markup, decode &#xNN;/named entities, widen the markup slice (text still capped at 4,000 chars).
  • Observability: a Titles-fetch failure increments parser_crash_count (not only errors); parser stamp cth-parser@2.1.0 on legislation_sync_log.
  • Ops knob: CTH_SYNC_MAX_ACTS (default 50, newest-first) for a wider refill or a narrow smoke run.

Evidence

Act headings sections with text
C2026A00086 Treasury Laws Amendment (Tax Adviser Misconduct…) 2026 38 35
C2026A00085 Wine and Other Legislation Amendment Act 2026 20 19
C2026A00084 Biosecurity Amendment (Improving Operational Efficiency) 6 6
C2026A00083 Online Safety Amendment (Strengthening Enforcement…) 12 12
C2026A00082 Combatting Illicit Tobacco Act 2026 25 25
C2026A00081 ASIO Amendment Act 2026 3 3
C2026A00080 Australia's Foreign Relations (State and Territory Arrangements…) 16 16
C2026A00079 Treasury Laws Amendment (News Media Bargaining)… 10 9

Before this PR every one of these parsed to 0.

  • src/lib/parsers/cth-parser.test.ts: URL shape (never status eq), stable paging, crash accounting, and a parse of a real document_1.html excerpt (src/lib/fixtures/cth/, C2026A00003). CTH_LIVE=1 additionally runs the live API and passed locally; CI skips it.
  • npm test: 44 files / 738 pass (2 live tests skipped). npx tsc --noEmit clean. eslint clean on the changed files.

Not in this PR

  • The cron itself still lives in the tailor-app repo (cron-source.yml). The consensus-sweep job in that workflow is failing every 30 minutes with HTTP 500 from pact.tailor.au/api/cron/auto-merge — unrelated to legislation, flagged in the tailor-group#7 report.
  • Amending Acts carry most of their substance in Schedules (ActHead6+, Item); the parser still ingests ActHead5 sections only, as before.
  • The #5581 write-protection for reviewed documents is still open; this PR does not touch the ingest path.

🤖 Generated with Claude Code


Note

Medium Risk
Changes external API query semantics and HTML parsing that drive legislation ingest volume and content quality; failures are better surfaced but a bad regex could still mis-parse or skip acts until the next run.

Overview
Fixes the Commonwealth legislation sync so weekly runs can again discover in-force Acts and extract section text from current EPUB HTML.

The Titles OData filter is corrected from status eq 'InForce' (400 when combined with collection eq 'Act') to status in ('InForce'), with $orderby=year desc,number desc so paged $skip does not repeat or drop acts. buildTitlesUrl / fetchInForceActs are exported for tests; the per-run cap is CTH_SYNC_MAX_ACTS (default 50).

parseActHtml now reads whole provision-level <p> blocks (span-heavy markup), decodes HTML entities, and widens the markup slice while keeping extracted text capped at 4,000 characters. A Titles API failure increments parser_crash_count (not only errors), the parser stamp is cth-parser@2.1.0, and tests plus a real C2026A00003 HTML fixture lock filter shape, paging, crash accounting, and parsing; optional CTH_LIVE=1 exercises the live API.

Reviewed by Cursor Bugbot for commit b1852e7. Bugbot is set up for automated code reviews on this repo. Configure here.

…ns — enum filter + span-per-word markup (refs TailorAU/tailor-group#7)

Two independent breaks, both silent, both required for the weekly
legislation-sync cron to refill the corpus:

1. Titles filter. `status` and `collection` are OData enums on
   api.prod.legislation.gov.au. `collection eq 'Act' and status eq
   'InForce'` answers 400 "Could not find a property named 'InForce'"
   (each alone parses; the conjunction mis-binds the literal). The loop
   recorded it as one error string with docs_checked = 0 — under the
   silent-zero alarm's threshold. `status in ('InForce')` binds correctly
   (verified live 2026-09-18: 4,768 in-force Acts). Paging gets a stable
   tiebreak (`year desc,number desc`); a Titles-fetch failure now also
   increments parser_crash_count; the newest-first ceiling is env-tunable
   (CTH_SYNC_MAX_ACTS, default 50); parser stamp → cth-parser@2.1.0.

2. Text extraction. The EPUB HTML now wraps every run of text in its own
   <span> (indent spacers as `<span style=…>&#xa0;</span>` before the
   words). The old extractor stopped at the first closing tag, harvested a
   non-breaking space, and every section fell under the 10-char floor →
   "No sections parsed" for every act. The extractor now takes each
   provision-level <p> block whole to its </p>, strips markup, decodes
   entities, and widens the markup slice (text is still capped at 4,000
   chars).

Evidence: with both fixes the eight newest in-force Acts yield 3–35
sections each (35/38 headings for C2026A00086, 25/25 for the Combatting
Illicit Tobacco Act 2026). Offline tests pin the URL shape and parse a
real document_1.html excerpt (src/lib/fixtures/cth); CTH_LIVE=1 runs the
same against the live API and is skipped in CI.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 18, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_5a4aa8b4-4bfc-4706-b03d-141adb7fa859)

Tailor-AUS pushed a commit that referenced this pull request Sep 18, 2026
…AU/tailor-group#7)

pr-check.yml's lint step still named the monorepo's
source-legislation-ingest.yml, source-pr-check.yml, cron-source.yml,
cd-source.yml and infra-source-setup.yml, none of which tailor-app#5949
re-rooted here, so actionlint exited 3 ("no such file or directory") on
every PR to rehome-review — PR #68 and #69 both red at this step with no
finding of their own. Lint pr-check.yml and cd-kg.yml instead, keeping
the two narrow ignores.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@cursor

cursor Bot commented Sep 18, 2026

Copy link
Copy Markdown

Bugbot couldn't run - usage limit reached

Bugbot is counted against Cursor usage for this user or team, and this run hit a usage or spend limit.

A user or team admin can review and increase usage limits in the Cursor dashboard.

(requestId: serverGenReqId_42c8f730-f1a3-4cd6-9da4-262b71eba591)

@Tailor-AUS
Tailor-AUS merged commit f9a4e91 into rehome-review Sep 18, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants