You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Polishes and modernizes the README, mirroring the recent
apify-client-python polish (apify/apify-client-python#776).
- Reworked intro and badge row (added a CI build-status badge), plus a
table of contents.
- Reordered sections, with a new **What are Actors?** up front and a
**What you can build** section (scraping, browser automation, AI agents,
MCP servers, web servers).
- Added **Documentation**, **Related projects**, **Support and
community**, and **Contributing** sections.
<ahref="https://discord.gg/jyEM2PRvMU"rel="nofollow"><imgsrc="https://img.shields.io/discord/801163717915574323?label=discord"alt="Chat on Discord"></a>
4
+
<strong>The official Python SDK for building <ahref="https://docs.apify.com/platform/actors">Apify Actors</a>.</strong>
9
5
</p>
10
6
11
-
The Apify SDK for Python is the official library to create [Apify Actors](https://docs.apify.com/platform/actors)
12
-
in Python. It provides useful features like Actor lifecycle management, local storage emulation, and Actor
<ahref="https://discord.gg/jyEM2PRvMU"><imgsrc="https://img.shields.io/discord/801163717915574323?label=discord"alt="Chat on Discord"></a>
15
+
</p>
16
+
17
+
`apify` is the official SDK for building [Apify Actors](https://docs.apify.com/platform/actors) in Python. It handles the Actor lifecycle, [storage](https://docs.apify.com/platform/storage) access, platform events, [Apify Proxy](https://docs.apify.com/platform/proxy), pay-per-event charging, and more.
14
18
15
-
If you just need to access the [Apify API](https://docs.apify.com/api/v2) from your Python applications,
16
-
check out the [Apify Client for Python](https://docs.apify.com/api/client/python) instead.
19
+
> If you only need to **consume** the [Apify API](https://docs.apify.com/api/v2) from Python (running Actors, reading datasets, managing storages) rather than building Actors, use the [Apify API client for Python](https://docs.apify.com/api/client/python) instead. It comes bundled with this SDK.
20
+
21
+
## Table of contents
22
+
23
+
-[Installation](#installation)
24
+
-[Quick start](#quick-start)
25
+
-[What are Actors?](#what-are-actors)
26
+
-[Features](#features)
27
+
-[What you can build](#what-you-can-build)
28
+
-[Usage examples](#usage-examples)
29
+
-[Documentation](#documentation)
30
+
-[Related projects](#related-projects)
31
+
-[Support and community](#support-and-community)
32
+
-[Contributing](#contributing)
33
+
-[License](#license)
17
34
18
35
## Installation
19
36
20
-
The Apify SDK for Python is available on PyPI as the `apify` package.
21
-
For default installation, using Pip, run the following:
37
+
The Apify SDK for Python requires **Python 3.11 or higher**. It is published on [PyPI](https://pypi.org/project/apify/) as the `apify` package and can be installed with [pip](https://pip.pypa.io/):
22
38
23
39
```bash
24
40
pip install apify
25
41
```
26
42
27
-
For users interested in integrating Apify with Scrapy, we provide a package extra called `scrapy`.
28
-
To install Apify with the `scrapy` extra, use the following command:
43
+
or with [uv](https://docs.astral.sh/uv/):
29
44
30
45
```bash
31
-
pip install apify[scrapy]
46
+
uv add apify
32
47
```
33
48
34
-
## Documentation
49
+
To use the Scrapy integration, install the `scrapy` extra:
50
+
51
+
```bash
52
+
pip install 'apify[scrapy]'
53
+
```
54
+
55
+
## Quick start
56
+
57
+
An Actor is a Python program that runs inside the `async with Actor:` context. The context initializes the Actor when it starts and tears it down when it finishes. Here's a minimal Actor that reads its input and stores a result:
58
+
59
+
```python
60
+
from apify import Actor
61
+
62
+
63
+
asyncdefmain() -> None:
64
+
asyncwith Actor:
65
+
actor_input =await Actor.get_input()
66
+
Actor.log.info('Actor input: %s', actor_input)
67
+
await Actor.set_value('OUTPUT', 'Hello, world!')
68
+
```
69
+
70
+
The quickest way to scaffold a full Actor project, with the `.actor` configuration, input schema, and Dockerfile already in place, is the [Apify CLI](https://docs.apify.com/cli):
71
+
72
+
1. Install the CLI:
73
+
74
+
```bash
75
+
npm install -g apify-cli
76
+
```
77
+
78
+
2. Create a new Actor from the Python "getting started" template:
79
+
80
+
```bash
81
+
apify create my-actor --template python-start
82
+
```
35
83
36
-
For usage instructions, check the documentation on [Apify Docs](https://docs.apify.com/sdk/python/).
84
+
3. Run it locally:
37
85
38
-
## Examples
86
+
```bash
87
+
cd my-actor
88
+
apify run
89
+
```
39
90
40
-
Below are few examples demonstrating how to use the Apify SDK with some web scraping-related libraries.
91
+
To create, run, and deploy your first Actor step by step, see the [Quick start guide](https://docs.apify.com/sdk/python/docs/quick-start).
41
92
42
-
### Apify SDK with HTTPX and BeautifulSoup
93
+
## What are Actors?
94
+
95
+
Actors are serverless cloud programs that can do almost anything a human can doin a web browser. They range from small tasks, such as filling in forms or unsubscribing from online services, all the way up to scraping and processing vast numbers of web pages.
96
+
97
+
They run either locally or on the [Apify platform](https://docs.apify.com/platform/), where you can run them at scale, monitor them, schedule them, or publish and monetize them. If you're new to Apify, learn [what Apify is](https://docs.apify.com/platform/about) in the platform documentation.
98
+
99
+
## Features
100
+
101
+
- Run the full Actor lifecycle inside `async with Actor:`, covering init, exit, failures, status messages, and reboots ([Actor lifecycle](https://docs.apify.com/sdk/python/docs/concepts/actor-lifecycle)).
102
+
- Read Actor input validated against your input schema with `Actor.get_input()` ([Actor input](https://docs.apify.com/sdk/python/docs/concepts/actor-input)).
103
+
- Read and write datasets, key-value stores, and request queues, locally or on the platform ([Working with storages](https://docs.apify.com/sdk/python/docs/concepts/storages)).
104
+
- React to platform events such as system info, migration, and abort ([Actor events](https://docs.apify.com/sdk/python/docs/concepts/actor-events)).
105
+
- Route requests through Apify Proxy with group selection, country targeting, and rotation ([Proxy management](https://docs.apify.com/sdk/python/docs/concepts/proxy-management)).
106
+
- Start, call, abort, and metamorph other Actors and tasks, and attach webhooks to run events ([Interacting with other Actors](https://docs.apify.com/sdk/python/docs/concepts/interacting-with-other-actors), [Webhooks](https://docs.apify.com/sdk/python/docs/concepts/webhooks)).
107
+
- Monetize your Actor with pay-per-event charging ([Pay-per-event](https://docs.apify.com/sdk/python/docs/concepts/pay-per-event)).
108
+
- Reach the full [Apify API](https://docs.apify.com/api/v2) through a preconfigured `ApifyClient` ([Accessing the Apify API](https://docs.apify.com/sdk/python/docs/concepts/access-apify-api)).
109
+
110
+
## What you can build
111
+
112
+
Almost any Python project can become an Actor, including projects for:
113
+
114
+
- **Web scraping and crawling** — The SDK is fully compatible with [Crawlee](https://crawlee.dev/python), which makes Apify a natural place to deploy and scale your crawlers (see the [Crawlee guide](https://docs.apify.com/sdk/python/docs/guides/crawlee)). It also works with other popular scraping libraries, such as [Scrapy](https://docs.apify.com/sdk/python/docs/guides/scrapy), [Scrapling](https://docs.apify.com/sdk/python/docs/guides/scrapling), or [Crawl4AI](https://docs.apify.com/sdk/python/docs/guides/crawl4ai).
115
+
- **Browser automation** — Drive a real browser with [Playwright](https://docs.apify.com/sdk/python/docs/guides/playwright) or [Selenium](https://docs.apify.com/sdk/python/docs/guides/selenium), or with higher-level tools such as [Browser Use](https://docs.apify.com/sdk/python/docs/guides/browser-use).
116
+
- **Web servers and APIs** — Run a [web server](https://docs.apify.com/sdk/python/docs/guides/running-webserver) inside an Actor to serve HTTP requests, for example to expose your scraper as a live API.
117
+
- **AI agents** — Host agents built with your framework of choice. Ready-made Actor templates cover [PydanticAI](https://apify.com/templates/python-pydanticai), [CrewAI](https://apify.com/templates/python-crewai), [LangGraph](https://apify.com/templates/python-langgraph), [LlamaIndex](https://apify.com/templates/python-llamaindex-agent), and [Smolagents](https://apify.com/templates/python-smolagents).
118
+
- **MCP servers** — Deploy a Python MCP server as an Actor and make its tools available to any MCP client. See [MCP server](https://apify.com/templates/python-mcp-empty) and [MCP proxy](https://apify.com/templates/python-mcp-proxy) templates
119
+
120
+
Whatever you build, the Apify SDK doesn't lock you into a particular framework. Bring the libraries you already use, and let Apify run your project in the cloud.
121
+
122
+
## Usage examples
123
+
124
+
The examples below show two common setups, but the same `async with Actor:` pattern works with any stack. For more, see the [guides](https://docs.apify.com/sdk/python/docs/guides/beautifulsoup-httpx).
125
+
126
+
### HTTPX with BeautifulSoup
43
127
44
-
This example illustrates how to integrate the Apify SDK with [HTTPX](https://www.python-httpx.org/) and [BeautifulSoup](https://pypi.org/project/beautifulsoup4/) to scrape data from web pages.
128
+
Scrape pages with [HTTPX](https://www.python-httpx.org/) and [BeautifulSoup](https://pypi.org/project/beautifulsoup4/), using the Actor's request queue to track URLs:
45
129
46
130
```python
47
131
from bs4 import BeautifulSoup
@@ -52,45 +136,31 @@ from apify import Actor
52
136
53
137
async def main() -> None:
54
138
async with Actor:
55
-
# Retrieve the Actor input, and use default values if not provided.
'h1s': [await h1.text_content() for h1 inawait context.page.locator('h1').all()],
130
-
'h2s': [await h2.text_content() for h2 inawait context.page.locator('h2').all()],
131
-
'h3s': [await h3.text_content() for h3 inawait context.page.locator('h3').all()],
132
-
}
133
-
134
-
# Store the extracted data to the default dataset.
135
-
await context.push_data(data)
136
-
137
-
# Enqueue additional links found on the current page.
184
+
})
185
+
# Follow links found on the page.
138
186
await context.enqueue_links()
139
187
140
-
# Run the crawler with the starting URLs.
141
188
await crawler.run(start_urls)
142
189
```
143
190
144
-
## What are Actors?
191
+
## Documentation
145
192
146
-
Actors are serverless cloud programs that can do almost anything a human can do in a web browser.
147
-
They can do anything from small tasks such as filling in forms or unsubscribing from online services,
148
-
all the way up to scraping and processing vast numbers of web pages.
193
+
The full SDK documentation lives at **[docs.apify.com/sdk/python](https://docs.apify.com/sdk/python)**. For the Apify platform itself, see the [Apify documentation](https://docs.apify.com/).
149
194
150
-
They can be run either locally, or on the [Apify platform](https://docs.apify.com/platform/),
151
-
where you can run them at scale, monitor them, schedule them, or publish and monetize them.
195
+
| Section | What you'll find |
196
+
|---|---|
197
+
| [Overview](https://docs.apify.com/sdk/python/docs/overview) | What the SDK is, what Actors are, and how the pieces fit together. |
198
+
| [Quick start](https://docs.apify.com/sdk/python/docs/quick-start) | Create, run, and deploy your first Python Actor. |
199
+
| [Concepts](https://docs.apify.com/sdk/python/docs/concepts/actor-lifecycle) | Actor lifecycle, input, storages, events, proxy management, interacting with other Actors, webhooks, accessing the Apify API, logging, configuration, and pay-per-event. |
200
+
| [Guides](https://docs.apify.com/sdk/python/docs/guides/beautifulsoup-httpx) | Integrations with BeautifulSoup, Parsel, Playwright, Selenium, Crawlee, Scrapy, Crawl4AI, and Browser Use, plus running a web server and using uv. |
201
+
| [Upgrading](https://docs.apify.com/sdk/python/docs/upgrading/upgrading-to-v4) | Migrating between major versions. |
202
+
| [API reference](https://docs.apify.com/sdk/python/reference) | Generated reference for every class and method. |
203
+
| [Changelog](https://docs.apify.com/sdk/python/docs/changelog) | Release history and breaking changes. |
152
204
153
-
If you're new to Apify, learn [what is Apify](https://docs.apify.com/platform/about)
154
-
in the Apify platform documentation.
205
+
## Related projects
155
206
156
-
## Creating Actors
207
+
- **[Apify API client for Python](https://docs.apify.com/api/client/python)** — talk to the Apify API directly from Python (bundled with this SDK).
208
+
- **[Crawlee for Python](https://crawlee.dev/python)** — web scraping and browser automation framework; fully compatible with this SDK.
209
+
- **[Apify SDK for JavaScript / TypeScript](https://docs.apify.com/sdk/js)** — the equivalent SDK for Node.js.
210
+
- **[Apify API client for JavaScript / TypeScript](https://docs.apify.com/api/client/js)** — the equivalent API client for Node.js.
211
+
- **[Crawlee for JavaScript / TypeScript](https://crawlee.dev)** — the original Node.js implementation of Crawlee.
212
+
- **[Apify CLI](https://docs.apify.com/cli)** — command-line tool for creating, running, and deploying Actors locally and on the platform.
157
213
158
-
To create and run Actors through Apify Console,
159
-
see the [Console documentation](https://docs.apify.com/academy/getting-started/creating-actors#choose-your-template).
214
+
## Support and community
160
215
161
-
To create and run Python Actors locally, check the documentation for
162
-
[how to create and run Python Actors locally](https://docs.apify.com/sdk/python/docs/quick-start).
216
+
- **Discord** — chat with the team and other users on the [Apify Discord server](https://discord.gg/jyEM2PRvMU).
217
+
- **GitHub issues** — report a bug or request a feature in the [issue tracker](https://github.com/apify/apify-sdk-python/issues).
163
218
164
-
## Guides
219
+
## Contributing
165
220
166
-
To see how you can use the Apify SDK with other popular libraries used for web scraping,
167
-
check out our guides for using
168
-
[BeautifulSoup with HTTPX](https://docs.apify.com/sdk/python/docs/guides/beautifulsoup-httpx),
169
-
[Parsel with Impit](https://docs.apify.com/sdk/python/docs/guides/parsel-impit),
or [Scrapy](https://docs.apify.com/sdk/python/docs/guides/scrapy).
221
+
Bug reports, fixes, and improvements are welcome! See [CONTRIBUTING.md](./CONTRIBUTING.md) for the development setup, coding standards, testing, and release process. The project uses [uv](https://docs.astral.sh/uv/) for project management and [Poe the Poet](https://poethepoet.natn.io/) as a task runner; the typical loop is:
222
+
223
+
```bash
224
+
uv run poe install-dev # install dev dependencies and git hooks
225
+
uv run poe check-code # lint, type-check, and unit tests
226
+
```
174
227
175
-
## Usage concepts
228
+
## License
176
229
177
-
To learn more about the features of the Apify SDK and how to use them,
178
-
check out the Usage Concepts section in the sidebar,
179
-
particularly the guides for the [Actor lifecycle](https://docs.apify.com/sdk/python/docs/concepts/actor-lifecycle),
180
-
[working with storages](https://docs.apify.com/sdk/python/docs/concepts/storages),
181
-
[handling Actor events](https://docs.apify.com/sdk/python/docs/concepts/actor-events)
182
-
or [how to use proxies](https://docs.apify.com/sdk/python/docs/concepts/proxy-management).
230
+
Released under the [Apache License 2.0](./LICENSE).
0 commit comments