You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/03_guides/07_scrapling.mdx
+47-2Lines changed: 47 additions & 2 deletions
Display the source diff
Display the rich diff
Original file line number
Diff line number
Diff line change
@@ -61,6 +61,30 @@ Note that:
61
61
-`response.urljoin(link_href)` resolves relative links against the page URL, so you can enqueue them directly.
62
62
- The `impersonate='chrome'` and `stealthy_headers=True` options make the request look like it comes from a real Chrome browser, which, combined with Apify Proxy, reduces the chance of being blocked.
63
63
64
+
## Adaptive selectors
65
+
66
+
The example above uses plain CSS selectors. Scrapling can also track the elements you scrape and relocate them when a website changes its markup, so a redesign doesn't immediately break your scraper. This is most useful for scrapers that revisit the same pages over time, rather than one-off crawls.
67
+
68
+
1. Enable adaptive matching once on the fetcher:
69
+
70
+
```python
71
+
AsyncFetcher.configure(adaptive=True)
72
+
```
73
+
74
+
2. On the first run, pass `auto_save=True` when you select an element. Scrapling records a fingerprint of that element, keyed by the selector:
75
+
76
+
```python
77
+
title = response.css('h1.product-title::text', auto_save=True).get()
78
+
```
79
+
80
+
3. On a later run, if the selector no longer matches because the page changed, pass `adaptive=True` with the same selector. Scrapling uses the saved fingerprint to find the element in its new location:
81
+
82
+
```python
83
+
title = response.css('h1.product-title::text', adaptive=True).get()
84
+
```
85
+
86
+
Scrapling keeps these fingerprints in a local SQLite database. On the Apify platform the Actor's filesystem doesn't persist between runs, so to keep them across runs, store that database in a [key-value store](https://docs.apify.com/platform/storage/key-value-store) and restore it on startup. For details, see [Scrapling's adaptive parsing documentation](https://scrapling.readthedocs.io/en/latest/parsing/adaptive.html).
87
+
64
88
## Using Apify Proxy
65
89
66
90
Running on the Apify platform gives your scraper access to [Apify Proxy](https://docs.apify.com/platform/proxy), which rotates IP addresses to avoid rate limiting and blocking. In the example above, `main` creates a proxy configuration with `Actor.create_proxy_configuration` and passes a fresh proxy URL to `scrape_page` for every request, which forwards it to Scrapling's `proxy` argument.
@@ -75,13 +99,33 @@ Scrapling accepts the proxy as a URL string (for example `http://user:pass@proxy
75
99
scrapling install
76
100
```
77
101
78
-
Switching the example Actor from HTTP to a real browser takes only one code change. Swap the `AsyncFetcher.get` call in `scrape_page`for `DynamicFetcher.async_fetch`. The parsing API is identical, so the rest of the Actor stays the same:
102
+
To switch the example from HTTP to a real browser, fetch each page through a browser session instead of `AsyncFetcher`. Opening a fresh browser for every page would be wasteful, so `main` enters an `AsyncDynamicSession` once and reuses it for the whole crawl, while `scrape_page` fetches with `session.fetch`. The parsing API is identical, so the extraction code stays the same:
79
103
80
104
<CodeBlockclassName="language-python">
81
105
{ScraplingBrowserScraper}
82
106
</CodeBlock>
83
107
84
-
To run this on the Apify platform, build on top of the [Apify Playwright base image](https://hub.docker.com/r/apify/actor-python-playwright), which already ships a browser together with all of its system-level dependencies, and run `scrapling install` during the Docker build to download the browser binaries that Scrapling expects.
108
+
Note that:
109
+
110
+
-`AsyncDynamicSession` launches one browser and keeps it open across `session.fetch` calls, so the crawl doesn't pay the browser-startup cost on every page.
111
+
- The proxy URL is passed per fetch, so each page can go through a fresh Apify Proxy IP while sharing the same browser.
112
+
113
+
To run this on the Apify platform, build on top of the [Apify Playwright base image](https://hub.docker.com/r/apify/actor-python-playwright), which already ships a browser together with all of its system-level dependencies, and run `scrapling install` during the Docker build to download the browser binaries that Scrapling expects:
114
+
115
+
```docker title="Dockerfile"
116
+
FROM apify/actor-python-playwright:3.14
117
+
118
+
# Install the Actor's Python dependencies.
119
+
COPY requirements.txt ./
120
+
RUN pip install -r requirements.txt
121
+
122
+
# Download the browser binaries that Scrapling's browser fetchers need.
123
+
RUN scrapling install
124
+
125
+
# Copy in the source code and launch the Actor as a module.
126
+
COPY . ./
127
+
CMD ["python", "-m", "src"]
128
+
```
85
129
86
130
## Conclusion
87
131
@@ -92,5 +136,6 @@ In this guide, you learned how to use Scrapling in your Apify Actors. You can no
92
136
-[Scrapling: Official documentation](https://scrapling.readthedocs.io/)
0 commit comments