Skip to content

chore: prefer <head> title, ignore SVG and iframe titles - #128

Merged
nikitachapovskii-dev merged 1 commit into
masterfrom
chore/title-extraction
Aug 3, 2026
Merged

chore: prefer <head> title, ignore SVG and iframe titles#128
nikitachapovskii-dev merged 1 commit into
masterfrom
chore/title-extraction

Conversation

@nikitachapovskii-dev

@nikitachapovskii-dev nikitachapovskii-dev commented Jul 29, 2026

Copy link
Copy Markdown
Collaborator

Closes #110.

  • Added the extractTitle helper from wcc: it takes the title from the head of the page and ignores titles inside SVG icons and inside Crawlee iframe placeholders.
  • Used it for the title in the output metadata and for the title passed to the transformer.
  • Tests: four unit tests for the helper, plus a title check in the existing Cheerio crawler test.

@nikitachapovskii-dev

nikitachapovskii-dev commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

Performance validation

Benchmarked extractTitle($) vs the old $('title').first().text() on an already-parsed $
(7 rounds × N iterations, median), in both parser modes — htmlparser2 (CheerioCrawler) and
parse5 (Playwright parseWithCheerio()). Numbers below are htmlparser2; parse5 matched it.

Case old new ratio
4 real pages (Wikipedia, GitHub, Apify Store, MDN — 181–508 KB) 0.125–0.161 ms 0.128–0.171 ms 1.0–1.1×
Real page with <title> stripped (2nd selector runs) 0.145 ms 0.291 ms 2.0×
Synthetic 36k elements, 2,000 SVG titles, no title anywhere 2.20 ms 5.08 ms 2.3×

It's called twice per page, so: +0.01 ms on real pages (0.02–0.04% of 14–34 ms of local
processing), +4.9 ms worst case (2% of 245 ms). The 2.3× only shows up when both selectors
miss and each walks the whole DOM i.e. only on pages with no <title> at all.

сс @nicklamonov

@nikitachapovskii-dev
nikitachapovskii-dev marked this pull request as ready for review August 3, 2026 08:50
@nikitachapovskii-dev nikitachapovskii-dev self-assigned this Aug 3, 2026
ruocco-l
ruocco-l approved these changes Aug 3, 2026
View reviewed changes

@ruocco-l ruocco-l left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a comment on the casting that doesn't really matter to the PR, just wanted to point it out

Comment thread tests/html-processing.test.ts
import { extractTitle } from '../src/website-content-crawler/html-processing.js';

// The `cheerio` version bundled with Crawlee differs from the top-level one, so the types don't match.
const parse = (html: string) => load(html) as unknown as CheerioAPI;

@ruocco-l ruocco-l Aug 3, 2026

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is an annoying cast, I think we can solve it by playing with Cheerio versions, I think it was discussed some time ago

@nicklamonov

nicklamonov commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator
  • In performance table are we really talking about milliseconds or seconds? If really milliseconds, then spending just a couple of extra milliseconds per title is totally not a big deal. :)
  • Isn't the title of the PR confusing (because there were more changes done)?

@nikitachapovskii-dev

nikitachapovskii-dev commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

In performance table are we really talking about milliseconds or seconds?

yes, real miliseconds, agree.
updated the pr title

@nikitachapovskii-dev
nikitachapovskii-dev merged commit 273d4f8 into master Aug 3, 2026
2 checks passed
@nikitachapovskii-dev
nikitachapovskii-dev deleted the chore/title-extraction branch August 3, 2026 10:29
@nicklamonov

nicklamonov commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator

I still don't see PR title update )

@nikitachapovskii-dev nikitachapovskii-dev changed the title chore: don't use SVG tooltips as page titles chore: prefer <head> title, ignore SVG and iframe titles Aug 3, 2026
@nikitachapovskii-dev

nikitachapovskii-dev commented Aug 3, 2026

Copy link
Copy Markdown
Collaborator Author

I still don't see PR title update )

damn, it wasn't saved. Re-updated now

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@ruocco-l ruocco-l ruocco-l approved these changes

Assignees

@nikitachapovskii-dev nikitachapovskii-dev

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Update title extraction

4 participants

@nikitachapovskii-dev @nicklamonov @ruocco-l @apify-service-account