Low-hanging improvements to the crawler APIs - #698
Merged
Merged
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## main #698 +/- ##
==========================================
+ Coverage 68.97% 69.00% +0.02%
==========================================
Files 46 46
Lines 2069 2071 +2
==========================================
+ Hits 1427 1429 +2
Misses 642 642 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
The three shared base crawlers asserted that the parser had found an image URL. CrawlerImage.url is deliberately optional, and add_image() validates it into an ImageURLNotFound, which is classified as CrawlerBroken and logged as an error naming the comic and date. The assert preempted that with a bare AssertionError, which fell through to the catch-all handler, and would vanish entirely under python -O.
Nothing in the codebase raises DoesNotExist, so the handler wrapping _get_all() could never run. Both the handler and the exception class go.
lxml returns str from both Element.get() and text_content(), so the bytes branch could never run. That it was applied in _get_all() but not in _get_one() showed nothing depended on it either way.
The implementation was annotated as returning list[str] | str | None, while both overloads and _get_one() only ever produce str | None.
The singular accessors raise MultipleElementsReturned when a selector matches more than one element. Pages that legitimately match several, where the comic is the first one, had to fall back to the plural accessor and index into it, guarding against the empty list by hand. first=True takes the first match in document order instead of raising, so those call sites collapse to the ordinary singular form.
Crawlers that needed to pick a container and then read its children had to reach past the parser into page.root and hand-roll the extraction, losing the default handling, the multiple-match guard and the CSS selectors. element() and elements() return parsers scoped to the matching elements, so the same extraction API keeps working one level down. Selectors on a scoped parser match the element itself as well as its descendants, since cssselect scopes them as descendant-or-self. Four crawlers move off page.root. This also fixes a latent bug in the Evil Inc crawler, which checked the result of xpath() against None, while xpath() returns an empty list when nothing matches. Awkward Zombie and Subnormality keep using page.root: they match on element text and sort by a parsed style attribute, which the selector API does not express.
link and title reach crawlers through __getattr__, so they typed as Any and nothing checked their use in the 41 feed crawlers. Declaring them alongside the existing summary and content0 annotations types them without changing lookup, as a bare annotation creates no class attribute.
feed.all() rebuilt the whole entry list on each call, and the crawler called it twice to test for and then take the first entry.
A zero-day history resolves to the same history_start as leaving both history attributes out, which already means only today can be crawled.
…s known These three crawlers already establish that the entry is the comic, by its link, its tags or its publishing date. Skipping to the next entry when the image selector then finds nothing reported the crawl as 'no release found' at info level, which is the answer for a day the comic did not publish on, not for a page whose markup moved. Letting the empty URL reach CrawlerImage raises ImageURLNotFound instead, naming the comic and the date at error level. The first of the two guards in the Joy of Tech crawler is the fallback between its two selectors, and stays.
LxmlParser fetched pages without checking the status, so an error page was parsed as if it were the comic's page. Finding no image in it, a crawler reported 'no release found', which is what a day the comic did not publish on looks like, and a site that moved or started refusing us stayed invisible. httpx.HTTPStatusError is an httpx.HTTPError, which get_release() already wraps into CrawlerHTTPError, so a gone page now reports as a transient failure while a page we did fetch but could not read reports as a broken crawler.
jodal
force-pushed
the
crawler-api-cleanups
branch
from
September 4, 2026 10:35
73fe356 to
e1c8f66
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Low-hanging improvements to the crawler APIs, found by reading
LxmlParser,FeedParserand all 60 hand-writtencrawl()implementations. One commit perchange, so they can be reviewed or dropped individually.
Dead code and wrong types in
LxmlParserDoesNotExistwas raised nowhere, so the handler wrapping_get_all()couldnever run. Both are gone.
_decode()could never take its bytes branch, as lxml returnsstrfrom bothElement.get()andtext_content(). That it was applied in_get_all()butnot
_get_one()showed nothing depended on it.text()was annotated as returninglist[str] | str | None, while itsoverloads and
_get_one()only ever producestr | None.Errors that were classified wrong
preempting the
ImageURLNotFoundthatadd_image()raises. That turned aCrawlerBrokennaming the comic and date into a bareAssertionErrorin thecatch-all, and would vanish under
python -O.LxmlParserfetched pages without checking the status, so an error page wasparsed as if it were the comic's page. Finding no image in it, a crawler
reported "no release found" — which is what a day the comic did not publish on
looks like — so a site that moved or started refusing us stayed invisible.
tags or publishing date, skipped to the next entry when the image selector
then found nothing. They now let the empty URL reach
CrawlerImage, whichreports
ImageURLNotFoundat error level.Two additions that shrink the crawlers
first=Trueon the singular accessors takes the first match in document orderinstead of raising
MultipleElementsReturned. Pages that legitimately matchseveral had to fall back to the plural accessor and index into it, guarding the
empty list by hand. Seven crawlers move over.
element()/elements()return parsers scoped to the matching elements, forcrawlers that need to pick a container and then read its children. Four
crawlers come off
page.root, where they hand-rolled the extraction and lostthe default handling, the multiple-match guard and the CSS selectors. This also
fixes a latent bug in the Evil Inc crawler, which tested
xpath()againstNonewhile it returns an empty list.Awkward Zombie and Subnormality keep using
page.root: they match on elementtext and sort by a parsed
styleattribute, which the selector API does notexpress.
Smaller things
Entry.titleandEntry.linkreach crawlers through__getattr__, so theytyped as
Anyand nothing checked their use in the 41 feed crawlers. Declaredalongside the existing
summaryandcontent0annotations.feed.all()twice, rebuilding the entry list to testfor and then take the first entry.
history_length_days = 0resolves to the samehistory_startas leaving bothhistory attributes out, which already means only today can be crawled.
Worth watching after deploy
raise_for_status()is the one change that can turn a working crawler into afailing one, if a site serves usable content with a non-2xx status. I checked the
crawlers that mention HTTP status: Penny Arcade's soft-404 comes with a 200 and
is unaffected, Dumbing of Age's 404 note is about the image server on the
downloader path, and Buttersafe's User-Agent dodges a 403 that would now surface
as a warning instead of being parsed silently.
Not touched, but the same shape
falling through.
CreatorsCrawlerBasealso callshttpx.get()without a status check and then.json(), so a non-2xx fails as a JSON decode error and lands in the catch-allrather than as a
CrawlerHTTPError.LxmlParserandFeedParserhave no tests, so the newfirst=Trueandelements()surface ships unit-tested only by hand against sample markup.https://claude.ai/code/session_013DLpoXp8z36AGE2ksWG58n