Problem
While standard SEO metrics (Titles, H1s, Meta Descriptions) cover the basics, advanced SEO professionals and marketers often need to scrape custom data from web pages during a crawl—such as article publish dates, author names, e-commerce product prices, or "out of stock" labels.
Proposed Solution
Introduce a Custom Extraction feature that allows users to define custom rules before starting the crawl.
Key Features:
- XPath Extraction: Allow users to input custom XPath queries to scrape specific HTML elements.
- Regex Extraction: Allow users to use Regular Expressions to find specific patterns (e.g., Google Analytics IDs, specific keywords, or email addresses) in the page source.
- Dynamic Columns: Append the extracted data as new columns in the final
.csv / .xlsx export.
Value
This unlocks massive utility for e-commerce SEOs and data analysts. It bridges the gap between a standard SEO crawler and a fully-fledged custom web scraper, directly competing with the advanced features of desktop software like Screaming Frog.
Problem
While standard SEO metrics (Titles, H1s, Meta Descriptions) cover the basics, advanced SEO professionals and marketers often need to scrape custom data from web pages during a crawl—such as article publish dates, author names, e-commerce product prices, or "out of stock" labels.
Proposed Solution
Introduce a Custom Extraction feature that allows users to define custom rules before starting the crawl.
Key Features:
.csv/.xlsxexport.Value
This unlocks massive utility for e-commerce SEOs and data analysts. It bridges the gap between a standard SEO crawler and a fully-fledged custom web scraper, directly competing with the advanced features of desktop software like Screaming Frog.