Skip to content

fix: separate crawlers from apps that merely name the vendor - #209

Merged
biggora merged 1 commit into
masterfrom
fix/bot-detection-false-positives
Aug 22, 2026
Merged

fix: separate crawlers from apps that merely name the vendor#209
biggora merged 1 commit into
masterfrom
fix/bot-detection-false-positives

Conversation

@biggora

@biggora biggora commented Aug 22, 2026

Copy link
Copy Markdown
Owner

Three bot-detection defects. Each is covered by a test that fails without its change — 8 of the 14 new cases fail on master.

1. Headless Chrome and Scrapy go undetected

The list carries chromeheadless. That string is the Karma launcher label, not something a browser ever sends — Chrome reports HeadlessChrome/<version>. So the one entry meant to catch headless browsing catches nothing. Scrapy was absent entirely.

2. 'google' flags the WebView of any Play Store app

The pattern matches any user-agent carrying Channel/googleplay. The repository already contains the evidence that this misfires in the wild:

// Handle false positives - TikTok WebView contains "googleplay" but isn't a bot
if (botIdentifier === 'google' && (source.includes('tiktok') || ...)) {

That exemption fixes one app and leaves the rest of the class flagged as crawlers. Reproduced on master with an unrelated Play Store app:

user-agent master this branch
TikTok WebView (Channel/googleplay) false — via the exemption false
Another app's WebView (Channel/googleplay) true, botName google false
Googlebot true, botName google true, botName googlebot
Google Page Speed Insights true true

'google' is replaced by the documented Google crawler tokens (apis-google, feedfetcher-google, google-inspectiontool, google-read-aloud, googleother, storebot-google, googleweblight, google-site-verification, plus the Page Speed and Web Preview phrases). googlebot, adsbot, mediapartners and gsa-crawler were already separate entries.

The TikTok exemption then becomes unreachable — botIdentifier can no longer equal 'google' — so it is removed. The existing TikTok test still passes, now structurally rather than by special case.

Behavioural change worth noting: botName for Googlebot becomes googlebot instead of google.

3. 'pinterest' flags the in-app browser

The crawler sends Pinterest/0.2 (+https://www.pinterest.com/bot.html); the in-app browser sends ... Mobile/15E148 [Pinterest/iOS]. On master both are bots. The digit after the slash separates them, so the pattern becomes pinterest(?=\\/\\d) plus pinterestbot. botName stays pinterest for the crawler.

CONTRIBUTING

The guide currently advises "use 'google' instead of 'Googlebot'" — which is the guidance that produced defect 2. That bullet is rewritten to say: match the crawler, not the vendor; prefer the crawler token or a lookahead pinning the crawler's shape.

Verification

  • npm test — 201 passed (13 files), up from 187.
  • npm run typecheck — clean.
  • npm run lint — reports only Delete ␍ on a Windows checkout; master reports the same class of error (3407 of them), so it is a core.autocrlf artifact, not a change here. Worth confirming on CI.

Not addressed here: CHANGELOG.md has no entry for 2.2.2, so the new section is filed under Unreleased.

Three bot-detection defects, each verified by a test that fails without the
corresponding change.

Headless Chrome went undetected. The list carried "chromeheadless", which is
the Karma launcher label; Chrome itself reports "HeadlessChrome/<version>".
Scrapy was absent entirely.

The bare "google" pattern matched any user-agent carrying
"Channel/googleplay" - that is, the WebView of any app shipped through the
Play Store. TikTok already had a hard-coded exemption for exactly this, which
covered one app and left the rest of the class flagged as crawlers. The
pattern is replaced by the documented Google crawler tokens, and the
exemption is removed as unreachable. A side effect: botName for Googlebot is
now "googlebot" rather than "google".

"pinterest" matched both the crawler and the in-app browser. The crawler
sends "Pinterest/0.2", the browser sends "[Pinterest/iOS]"; a lookahead on
the digit separates them.

CONTRIBUTING advised using "google" over "Googlebot", which is the guidance
that produced the first false positive, so that bullet is rewritten.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@biggora
biggora merged commit 10607cc into master Aug 22, 2026
7 checks passed
@biggora biggora mentioned this pull request Aug 22, 2026
biggora added a commit that referenced this pull request Aug 22, 2026
The v2.2.3 tag was pushed onto the PR #209 merge commit while
package.json still read 2.2.2, so npm rejected the publish with
"You cannot publish over the previously published versions: 2.2.2".

Bump package.json to 2.2.3 and close the Unreleased changelog section.

Also add a pre-flight step to the publish workflow that compares the
git tag against package.json before npm ci / tests / audit run, so a
future mismatch fails in seconds with an actionable message instead of
surfacing as a registry error at the dry-run step.

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant