Ahrefs Domain Rating & Keywords Scraper — Python Example

Pull Ahrefs Domain Rating, backlinks, organic keywords, keyword difficulty, SERP overviews, AI-search visibility, and broken-link audits for any domain — in Python, with no Ahrefs subscription required.

This repo is a ready-to-run Python project that uses the Ahrefs All-in-One SEO Scraper Apify actor to extract every Ahrefs surface a working SEO team typically buys access for — Domain Rating, URL Rating, Ahrefs Rank, backlink counts, referring domains, organic traffic & keywords, paid traffic, keyword ideas and difficulty, SERP overviews, AI-engine visibility, broken-link checks, and a top-websites leaderboard. One actor, thirteen searchType modes, pay-as-you-go pricing instead of Ahrefs' $129+/month Lite tier.

What this does

Ahrefs' API and dashboard subscriptions are powerful but expensive — the entry-level Lite plan is around $129/month with strict per-day report quotas, and the API tier is typically $500+/month with metered credits. This project lets you call all the same data through a managed Apify actor that handles residential IP rotation, Cloudflare bypass, retries, and parsing. You pay per result item; the free Apify tier ($5/month) covers ~1,000 domain audits before you ever pay anything.

Use cases

• Daily DR / backlink monitoring — run a scheduled job that records Domain Rating and referring-domain count for your customer book.
• Bulk competitive benchmarks — fetch DR, traffic, and keyword count for a hundred competitors in one go for a board deck.
• Keyword research — replace Ahrefs' Keywords Explorer with a Python pipeline that pulls volume + KD + CPC for thousands of seed terms.
• Broken-link building — find broken outbound links on high-DR sites in your niche and pitch your content as the replacement.
• AI-search visibility audit — see whether your site (or a competitor's) is being surfaced inside ChatGPT, Perplexity, Gemini, and other AI engines.
• SERP feature mapping — pull the full top-10 SERP for any keyword in any country and see exactly which sites and SERP features dominate.

Requirements

• Python 3.10+
• A free Apify account (no credit card; gives $5/month credit)
• No Ahrefs account

Quick start

git clone https://github.com/pro100chok/ahrefs-domain-rating-keywords-python.git
cd ahrefs-domain-rating-keywords-python
pip install -r requirements.txt
cp .env.example .env
# paste your APIFY_API_TOKEN
python main.py

main.py audits ten fintech competitors (Stripe, Adyen, Checkout.com, Braintree, PayPal, Wise, Revolut, N26, Mercury, Ramp), pulls each one's Domain Rating + backlinks + organic traffic + keywords, and prints a sorted leaderboard plus saves the raw response.

How it works

The actor exposes one knob — searchType — that switches between 13 different Ahrefs surfaces. The same input list of urls (or a single keyword) feeds each surface. Under the hood the actor uses residential proxies plus headless-browser fingerprinting to access Ahrefs' public-facing endpoints, parses the result into a flat JSON record, and pushes it to the run's dataset. Your Python script iterates the dataset and writes the data to disk, CSV, Google Sheets, BigQuery, or wherever else.

The 13 search types

searchType	What it returns
website_authority (default)	Domain Rating, URL Rating, Ahrefs Rank, backlinks, referring domains, organic + paid traffic and keywords.
backlinks_overview	Aggregated backlink stats — total, dofollow %, anchor distribution, country breakdown.
backlinks_list	Individual backlink records — source URL, target URL, anchor, DR, traffic.
broken_links	Broken outbound links on the target site (great for broken-link-building outreach).
traffic_overview	Organic traffic estimate, traffic value, traffic per country, top pages.
ai_visibility	Mentions inside AI engines (ChatGPT, Perplexity, Gemini, Copilot, Grok, Google AI Mode).
website_details	Combined deep-dive on a single domain.
keyword_ideas	Keyword ideas around a seed term — Keywords Explorer replacement.
keyword_difficulty	KD score, top-10 average DR, SERP-feature density.
keyword_metrics	Volume, CPC, clicks, return rate for an exact keyword list.
keyword_rank	Where a domain ranks for a given keyword today.
serp_overview	Live top-10 SERP for any keyword in any country.
top_websites	Highest-trafficked sites by Ahrefs rank or fastest-growing trending sites, with country + category filters.

Example: bulk fintech competitor audit

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_API_TOKEN"])

run = client.actor("pro100chok/ahrefs-seo-tools").call(run_input={
    "searchType": "website_authority",
    "urls": ["stripe.com", "adyen.com", "checkout.com", "braintreepayments.com"],
    "mode": "domain",
    "proxyConfiguration": {"useApifyProxy": True,
                            "apifyProxyGroups": ["RESIDENTIAL"]},
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(f"{item['url']:<24} DR {item['domainRating']:<3} "
          f"refdomains={item['refDomains']:,} keywords={item['organicKeywords']:,}")

Example output

{
  "url": "stripe.com",
  "domainRating": 91,
  "urlRating": 89,
  "ahrefsRank": 712,
  "backlinks": 18420330,
  "refDomains": 72540,
  "organicTraffic": 7642300,
  "organicKeywords": 412300,
  "paidTraffic": 22100,
  "paidKeywords": 1250
}

Input parameters

Parameter	Type	Used by	Description
searchType	string	all	One of 13 modes (see table above).
urls	string[]	most modes	Domains or URLs to analyze. Bulk-friendly.
mode	string	URL-based modes	How to interpret each URL: exact (one page), subdomains (incl. subs), prefix (path prefix), domain (entire domain).
keyword	string	keyword modes	The seed keyword for keyword_ideas / keyword_difficulty / keyword_metrics.
country	string	keyword modes	2-letter country code for keyword research (us, gb, de, etc.).
searchEngine	string	keyword_ideas	Google (default), Bing, YouTube, Amazon, plus AI engines: ChatGPT, Perplexity, Gemini, Copilot, Grok, GoogleAIMode.
topWebsitesMode	string	top_websites	ranking (by Ahrefs rank) or trending (fastest growing).
topWebsitesCountry	string	top_websites	Country slug (e.g. united-states, germany, worldwide).
topWebsitesCategory	string	top_websites	Vertical filter (finance, health, shopping, sports, etc.).
topWebsitesLimit	integer	top_websites	Up to 1,000 sites per call.
proxyConfiguration	object	all	Residential proxies recommended for Ahrefs.
maxRetries	integer	all	Retry attempts on transient errors. Default 5.

More examples

File	Demonstrates
examples/01_basic_usage.py	Single-domain DR lookup.
examples/02_keyword_research.py	Keyword ideas + KD + CPC for a seed term.
examples/03_broken_links_audit.py	Broken-link discovery for outreach.
examples/04_export_to_csv.py	DR tier bucketing + pandas export.
examples/05_export_to_google_sheets.py	Weekly DR snapshots into a shared Sheet.

FAQ

How does pricing work? The actor charges per result item (typically $0.001–$0.005 each depending on searchType). Apify's free $5/month covers a few thousand records before you pay.

Is the data the same as Ahrefs.com? Yes — the actor pulls from Ahrefs' public-facing endpoints, the same source that powers the Ahrefs UI. DR, traffic, keyword counts match what a paid Ahrefs user sees.

Why not the official Ahrefs API? Ahrefs API access is gated to higher-tier subscriptions ($500+/month) with a strict credits model where each report consumes credits. This actor lets you pay per result with no monthly minimum.

Does it work for fresh / unindexed domains? Newly-registered or low-traffic domains return null/zero for most fields — that's an Ahrefs data-coverage limitation, not a scraper issue.

Can I get keyword data for non-Google search engines? Yes. keyword_ideas supports Bing, YouTube, Amazon, Yahoo, Yandex, Baidu, and several AI engines (ChatGPT, Perplexity, Gemini, Copilot, Grok). See the searchEngine parameter.

How long does one run take? Most modes complete in 5–60 seconds. backlinks_list and broken_links for large sites can take a few minutes because they paginate through the entire link graph.

Can I run it on a schedule? Yes. Apify has a built-in scheduler — set a cron expression and the actor runs unattended. Combine with examples/05_export_to_google_sheets.py for an automated weekly competitor DR tracker.

What about AI search visibility? searchType: "ai_visibility" checks whether a URL is being cited inside ChatGPT, Perplexity, Gemini, Copilot, Grok, and Google AI Mode for prompts derived from the page's content. Useful as a leading indicator before Google starts losing share to LLM-driven search.

Related actors

• Semrush Domain Overview Scraper — same domain list through a different data source.
• SimilarWeb Traffic Data Scraper — traffic stats, rankings, traffic sources.
• Website Contact Scraper — enrich domains with emails + phones + socials.

See the full catalog at apify.com/pro100chok.

License

MIT — see LICENSE.

---

Built on top of the Ahrefs All-in-One SEO Scraper Apify actor.