fix: separate crawlers from apps that merely name the vendor - #209
Merged
Conversation
Three bot-detection defects, each verified by a test that fails without the corresponding change. Headless Chrome went undetected. The list carried "chromeheadless", which is the Karma launcher label; Chrome itself reports "HeadlessChrome/<version>". Scrapy was absent entirely. The bare "google" pattern matched any user-agent carrying "Channel/googleplay" - that is, the WebView of any app shipped through the Play Store. TikTok already had a hard-coded exemption for exactly this, which covered one app and left the rest of the class flagged as crawlers. The pattern is replaced by the documented Google crawler tokens, and the exemption is removed as unreachable. A side effect: botName for Googlebot is now "googlebot" rather than "google". "pinterest" matched both the crawler and the in-app browser. The crawler sends "Pinterest/0.2", the browser sends "[Pinterest/iOS]"; a lookahead on the digit separates them. CONTRIBUTING advised using "google" over "Googlebot", which is the guidance that produced the first false positive, so that bullet is rewritten. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Merged
biggora
added a commit
that referenced
this pull request
Aug 22, 2026
The v2.2.3 tag was pushed onto the PR #209 merge commit while package.json still read 2.2.2, so npm rejected the publish with "You cannot publish over the previously published versions: 2.2.2". Bump package.json to 2.2.3 and close the Unreleased changelog section. Also add a pre-flight step to the publish workflow that compares the git tag against package.json before npm ci / tests / audit run, so a future mismatch fails in seconds with an actionable message instead of surfacing as a registry error at the dry-run step. Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Three bot-detection defects. Each is covered by a test that fails without its change — 8 of the 14 new cases fail on
master.1. Headless Chrome and Scrapy go undetected
The list carries
chromeheadless. That string is the Karma launcher label, not something a browser ever sends — Chrome reportsHeadlessChrome/<version>. So the one entry meant to catch headless browsing catches nothing. Scrapy was absent entirely.2.
'google'flags the WebView of any Play Store appThe pattern matches any user-agent carrying
Channel/googleplay. The repository already contains the evidence that this misfires in the wild:That exemption fixes one app and leaves the rest of the class flagged as crawlers. Reproduced on
masterwith an unrelated Play Store app:masterChannel/googleplay)false— via the exemptionfalseChannel/googleplay)true, botNamegooglefalsetrue, botNamegoogletrue, botNamegooglebottruetrue'google'is replaced by the documented Google crawler tokens (apis-google,feedfetcher-google,google-inspectiontool,google-read-aloud,googleother,storebot-google,googleweblight,google-site-verification, plus the Page Speed and Web Preview phrases).googlebot,adsbot,mediapartnersandgsa-crawlerwere already separate entries.The TikTok exemption then becomes unreachable —
botIdentifiercan no longer equal'google'— so it is removed. The existing TikTok test still passes, now structurally rather than by special case.Behavioural change worth noting:
botNamefor Googlebot becomesgooglebotinstead ofgoogle.3.
'pinterest'flags the in-app browserThe crawler sends
Pinterest/0.2 (+https://www.pinterest.com/bot.html); the in-app browser sends... Mobile/15E148 [Pinterest/iOS]. Onmasterboth are bots. The digit after the slash separates them, so the pattern becomespinterest(?=\\/\\d)pluspinterestbot.botNamestayspinterestfor the crawler.CONTRIBUTING
The guide currently advises "use
'google'instead of'Googlebot'" — which is the guidance that produced defect 2. That bullet is rewritten to say: match the crawler, not the vendor; prefer the crawler token or a lookahead pinning the crawler's shape.Verification
npm test— 201 passed (13 files), up from 187.npm run typecheck— clean.npm run lint— reports onlyDelete ␍on a Windows checkout;masterreports the same class of error (3407 of them), so it is acore.autocrlfartifact, not a change here. Worth confirming on CI.Not addressed here:
CHANGELOG.mdhas no entry for 2.2.2, so the new section is filed under Unreleased.