Skip to content

Commit 1578540

Browse files
committed
docs: address review comments on Scrapling guide
1 parent 2ced5c5 commit 1578540

2 files changed

Lines changed: 141 additions & 12 deletions

File tree

docs/03_guides/07_scrapling.mdx

Lines changed: 47 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -61,6 +61,30 @@ Note that:
6161
- `response.urljoin(link_href)` resolves relative links against the page URL, so you can enqueue them directly.
6262
- The `impersonate='chrome'` and `stealthy_headers=True` options make the request look like it comes from a real Chrome browser, which, combined with Apify Proxy, reduces the chance of being blocked.
6363

64+
## Adaptive selectors
65+
66+
The example above uses plain CSS selectors. Scrapling can also track the elements you scrape and relocate them when a website changes its markup, so a redesign doesn't immediately break your scraper. This is most useful for scrapers that revisit the same pages over time, rather than one-off crawls.
67+
68+
1. Enable adaptive matching once on the fetcher:
69+
70+
```python
71+
AsyncFetcher.configure(adaptive=True)
72+
```
73+
74+
2. On the first run, pass `auto_save=True` when you select an element. Scrapling records a fingerprint of that element, keyed by the selector:
75+
76+
```python
77+
title = response.css('h1.product-title::text', auto_save=True).get()
78+
```
79+
80+
3. On a later run, if the selector no longer matches because the page changed, pass `adaptive=True` with the same selector. Scrapling uses the saved fingerprint to find the element in its new location:
81+
82+
```python
83+
title = response.css('h1.product-title::text', adaptive=True).get()
84+
```
85+
86+
Scrapling keeps these fingerprints in a local SQLite database. On the Apify platform the Actor's filesystem doesn't persist between runs, so to keep them across runs, store that database in a [key-value store](https://docs.apify.com/platform/storage/key-value-store) and restore it on startup. For details, see [Scrapling's adaptive parsing documentation](https://scrapling.readthedocs.io/en/latest/parsing/adaptive.html).
87+
6488
## Using Apify Proxy
6589

6690
Running on the Apify platform gives your scraper access to [Apify Proxy](https://docs.apify.com/platform/proxy), which rotates IP addresses to avoid rate limiting and blocking. In the example above, `main` creates a proxy configuration with `Actor.create_proxy_configuration` and passes a fresh proxy URL to `scrape_page` for every request, which forwards it to Scrapling's `proxy` argument.
@@ -75,13 +99,33 @@ Scrapling accepts the proxy as a URL string (for example `http://user:pass@proxy
7599
scrapling install
76100
```
77101

78-
Switching the example Actor from HTTP to a real browser takes only one code change. Swap the `AsyncFetcher.get` call in `scrape_page` for `DynamicFetcher.async_fetch`. The parsing API is identical, so the rest of the Actor stays the same:
102+
To switch the example from HTTP to a real browser, fetch each page through a browser session instead of `AsyncFetcher`. Opening a fresh browser for every page would be wasteful, so `main` enters an `AsyncDynamicSession` once and reuses it for the whole crawl, while `scrape_page` fetches with `session.fetch`. The parsing API is identical, so the extraction code stays the same:
79103

80104
<CodeBlock className="language-python">
81105
{ScraplingBrowserScraper}
82106
</CodeBlock>
83107

84-
To run this on the Apify platform, build on top of the [Apify Playwright base image](https://hub.docker.com/r/apify/actor-python-playwright), which already ships a browser together with all of its system-level dependencies, and run `scrapling install` during the Docker build to download the browser binaries that Scrapling expects.
108+
Note that:
109+
110+
- `AsyncDynamicSession` launches one browser and keeps it open across `session.fetch` calls, so the crawl doesn't pay the browser-startup cost on every page.
111+
- The proxy URL is passed per fetch, so each page can go through a fresh Apify Proxy IP while sharing the same browser.
112+
113+
To run this on the Apify platform, build on top of the [Apify Playwright base image](https://hub.docker.com/r/apify/actor-python-playwright), which already ships a browser together with all of its system-level dependencies, and run `scrapling install` during the Docker build to download the browser binaries that Scrapling expects:
114+
115+
```docker title="Dockerfile"
116+
FROM apify/actor-python-playwright:3.14
117+
118+
# Install the Actor's Python dependencies.
119+
COPY requirements.txt ./
120+
RUN pip install -r requirements.txt
121+
122+
# Download the browser binaries that Scrapling's browser fetchers need.
123+
RUN scrapling install
124+
125+
# Copy in the source code and launch the Actor as a module.
126+
COPY . ./
127+
CMD ["python", "-m", "src"]
128+
```
85129

86130
## Conclusion
87131

@@ -92,5 +136,6 @@ In this guide, you learned how to use Scrapling in your Apify Actors. You can no
92136
- [Scrapling: Official documentation](https://scrapling.readthedocs.io/)
93137
- [Scrapling: Fetchers](https://scrapling.readthedocs.io/en/latest/fetching/choosing/)
94138
- [Scrapling: Parsing and selecting elements](https://scrapling.readthedocs.io/en/latest/parsing/selection/)
139+
- [Scrapling: Adaptive parsing](https://scrapling.readthedocs.io/en/latest/parsing/adaptive.html)
95140
- [Scrapling: GitHub repository](https://github.com/D4Vinci/Scrapling)
96141
- [Apify: Proxy management](https://docs.apify.com/platform/proxy)
Lines changed: 94 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -1,21 +1,22 @@
1+
import asyncio
12
from typing import Any
3+
from urllib.parse import urlsplit
24

3-
from scrapling.fetchers import DynamicFetcher
5+
from scrapling.fetchers import AsyncDynamicSession
6+
7+
from apify import Actor, Request
8+
from apify.storages import RequestQueue
49

510

611
async def scrape_page(
12+
session: AsyncDynamicSession,
713
url: str,
814
*,
915
proxy_url: str | None = None,
1016
) -> tuple[dict[str, Any], list[str]]:
11-
"""Fetch a page in a real browser with Scrapling and return data and links."""
17+
"""Fetch a page through the shared browser session and return data and links."""
1218
# `network_idle` waits until the page stops making network requests.
13-
response = await DynamicFetcher.async_fetch(
14-
url,
15-
proxy=proxy_url,
16-
headless=True,
17-
network_idle=True,
18-
)
19+
response = await session.fetch(url, proxy=proxy_url, network_idle=True)
1920

2021
data = {
2122
'url': url,
@@ -25,11 +26,94 @@ async def scrape_page(
2526
'h3s': response.css('h3::text').getall(),
2627
}
2728

28-
# Collect absolute links from the page.
29+
# Keep only absolute links on the same host.
2930
links: list[str] = []
31+
host = urlsplit(url).netloc
3032
for href in response.css('a::attr(href)').getall():
3133
link_url = response.urljoin(href)
32-
if link_url.startswith(('http://', 'https://')):
34+
if not link_url.startswith(('http://', 'https://')):
35+
continue
36+
if urlsplit(link_url).netloc == host:
3337
links.append(link_url)
3438

3539
return data, links
40+
41+
42+
async def enqueue_links(
43+
request_queue: RequestQueue,
44+
links: list[str],
45+
*,
46+
depth: int,
47+
max_depth: int,
48+
) -> None:
49+
"""Enqueue the links one level deeper, unless max_depth was reached."""
50+
if depth >= max_depth:
51+
return
52+
53+
for link_url in links:
54+
Actor.log.info(f'Enqueuing {link_url} ...')
55+
request = Request.from_url(link_url)
56+
request.crawl_depth = depth + 1
57+
await request_queue.add_request(request)
58+
59+
60+
async def main() -> None:
61+
async with Actor:
62+
# Read the Actor input.
63+
actor_input = await Actor.get_input() or {}
64+
start_urls = actor_input.get('startUrls', [{'url': 'https://crawlee.dev'}])
65+
max_depth = actor_input.get('maxDepth', 1)
66+
67+
if not start_urls:
68+
Actor.log.info('No start URLs specified in Actor input, exiting...')
69+
await Actor.exit()
70+
71+
# Set up Apify Proxy and the request queue.
72+
proxy_configuration = await Actor.create_proxy_configuration()
73+
request_queue = await Actor.open_request_queue()
74+
75+
# Enqueue the start URLs (crawl depth defaults to 0).
76+
for start_url in start_urls:
77+
url = start_url.get('url')
78+
Actor.log.info(f'Enqueuing start URL: {url}')
79+
await request_queue.add_request(Request.from_url(url))
80+
81+
# Cap the crawl; raise or remove to follow more pages.
82+
max_requests = 50
83+
handled_requests = 0
84+
85+
# Open the browser once and reuse it for every page in the crawl.
86+
async with AsyncDynamicSession(headless=True) as session:
87+
while handled_requests < max_requests and (
88+
request := await request_queue.fetch_next_request()
89+
):
90+
handled_requests += 1
91+
url = request.url
92+
depth = request.crawl_depth
93+
Actor.log.info(f'Scraping {url} (depth={depth}) ...')
94+
95+
try:
96+
# Fresh proxy URL per request (None if no proxy).
97+
proxy_url = None
98+
if proxy_configuration:
99+
proxy_url = await proxy_configuration.new_url()
100+
101+
data, links = await scrape_page(session, url, proxy_url=proxy_url)
102+
await Actor.push_data(data)
103+
Actor.log.info(
104+
f'Stored data from {url} '
105+
f'(title={data["title"]!r}, {len(links)} links found).'
106+
)
107+
await enqueue_links(
108+
request_queue, links, depth=depth, max_depth=max_depth
109+
)
110+
111+
except Exception:
112+
Actor.log.exception(f'Cannot extract data from {url}.')
113+
114+
finally:
115+
await request_queue.mark_request_as_handled(request)
116+
117+
118+
if __name__ == '__main__':
119+
asyncio.run(main())

0 commit comments

Comments
 (0)