refactor: let the model own catalogue extraction - #133
Merged
Merged
Conversation
The course and structure converters each picked sections to keep, and the structure one dropped the key facts box, so college, length and selection rank never reached the model. One converter now keeps every visible part of the page in order and removes only page furniture.
Course and structure syncs no longer run a deterministic extractor or merge its output with the model's. The model reads the whole page and owns every field, including the structure title, sections and outcomes the old merge kept for itself. Whatever part of the response fits the contract is stored; a field that does not is left empty with an error for review, one bad array item costs only that item, and wording the page lacks is kept with a warning rather than rejected. The prompts now ask for tidied display text with the page's exact wording kept as evidence.
The stage and its artefact kind no longer exist in the pipeline. Their audit rows are removed and both are dropped from the allowed values.
Only the model's rule says what a course requires. The course page no longer parses the prerequisite wording, the course graph no longer recovers edges from older snapshots or draws bare codes found in the wording, the projection no longer scrapes codes from prose into rule references, and the unused automatic mapping in the requisite editor is gone. A draft preview now collects prerequisite codes the way the database does, and a course whose wording has not become a rule says so instead of claiming it has none.
A requirement tree lost every condition when one nested group broke the contract. A stray minimum count on a group whose operator does not use one is now cleared, and any node that still does not fit becomes a free-text condition holding its own wording, flagged for review, so the rest of the tree survives.
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
3 tasks
This branch was successfully deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
First of a stacked series that hands catalogue syncs to the model entirely.
Why prerequisites were stored as free text. FINM3006's rule ("FINM2001; FINM2002; and, FINM2003 or FINM3011") was lost twice: the deterministic grammar could not read ANU's semicolon lists, and the model's correct rule was then discarded because its evidence was filed under
requisitesrather thanrequisites.prerequisiteRule. The projection fell back to one free-text condition, which the graph drew as a single box.This PR removes the deterministic side instead of patching it:
Live check against the real pages with
google/gemini-3.1-flash-lite:all_of[FINM2001, FINM2002, one_of[FINM2003, FINM3011]], no flags, $0.004Verification
pnpm verify(format, lint, types, 520 unit and component tests, production build, 8 access tests, whitespace)Deployment notes
009_retire_deterministic_extraction.sqldeletes the audit rows of the retireddeterministic_extractstage anddeterministic_outputartefacts, then drops both values from their check constraints. It is safe on existing data. The matching storage objects are left in thecourse-import-artifactsbucket.deterministicuntil the catalogue content is cleared; that is handled separately.