feat: implement crawlee v4 RequestQueueClient over apify-client by B4nan · Pull Request #643 · apify/apify-sdk-js

B4nan · 2026-06-16T12:34:14Z

Crawlee v4 redesigned the request queue storage layer into a stateful, pull-based RequestQueueBackend interface, modeled on the Python SDK. The SDK's old name-remapping proxy can't satisfy it, so this PR implements it for the Apify platform, with two access modes. The Python SDK's request queue clients were the starting point, but the design follows the JS API: the crawlee JS frontend already keeps large dedup caches, so the backends only manage what the frontend can't.

Request queue access modes

single (default, ApifyRequestQueueSingleBackend) assumes the run is the only consumer of the queue. There is no request locking; the queue head is estimated locally, and requests added by this client are served from a local cache, so a typical request costs no read API calls. On the first add, up to 10k existing requests are prefetched, letting resurrected runs deduplicate re-added requests locally instead of paying a platform write per request. Multiple producers may still add requests concurrently.

shared (ApifyRequestQueueSharedBackend) is safe for any number of concurrent consumers. Fetched requests are locked server-side (listAndLockHead), and the lock duration follows the crawler's expected request processing time (setExpectedRequestProcessingTimeSecs, i.e. handler timeout plus padding). Reclaims release the lock so other consumers can pick the request up immediately, and isFinished consults queueHasLockedRequests so no consumer exits while another still holds work.

The mode is selected per storage backend, as in the Python SDK: Actor.init({ requestQueueAccess: 'shared' }) or new ApifyStorageBackend(client, { requestQueueAccess: 'shared' }). Opening the same queue in both modes at once is not supported; whichever backend opens it first wins (documented on getStorageBackendCacheKey).

Request queue clients now send a stable per-run clientKey (the run id, same as the Python SDK), so hadMultipleClients stays meaningful and a migrated or resurrected run re-acquires the locks of its previous incarnation.

Both backends honor the beta.105 contract: markRequestAsHandled/reclaimRequest of a request that does not exist return undefined without upserting it, and purge() throws, since the platform has no truncate endpoint (Python behaves the same).

Review feedback addressed

ApifyStorageClient renamed to ApifyStorageBackend (class and file), OpenStorageContext.client renamed to backend, stray comment removed.
The single/shared toggle from the review is implemented here rather than tracked in an issue.

Crawlee beta.105 adaptation (was beta.71)

create*Backend now receives crawlee's StorageIdentifier ({id} | {name} | {alias}). The backend resolves aliases itself: __default__ maps to the run's default storage, other aliases resolve via ACTOR_STORAGES_JSON, undeclared aliases on the platform get a clear error, and anywhere else an unnamed storage is created per alias and process. A KVS-persisted alias mapping like Python's AliasResolver, which would let undeclared aliases survive migrations, is left as a follow-up.
The key-value store backend is now a byte transport: getValue reads buffers and leaves parsing to crawlee's frontend (previously the value got parsed twice), listKeys returns the new paginated shape, and recordExists passes through.
getStorageBackendCacheKey partitions crawlee's storage cache by API base URL and token.
Mechanical drift: getGlobalConfig renamed to getGlobalConfiguration; QueueOperationInfo/Constructor/Dictionary moved to @crawlee/types; snakeCaseToCamelCase inlined (removed upstream); ProxyConfiguration internals were privatized upstream (own log, no more base-class field reads); KeyValueStore.config/client fields replaced by serviceLocator/backend.

Validation

Unit tests cover both backends against a mocked apify-client (head estimation, dedup, prefetch, in-progress tracking, forefront ordering, lock duration raising, queueHasLockedRequests, the no-upsert contract), plus the storage backend's alias resolution and access mode wiring. Build, lint, and the full suite (148 tests) pass. The single mode is the successor of the flow validated end-to-end on the platform earlier in this PR; re-validating both modes on the platform against beta.105 is the remaining step.

janbuchar

The description seems outdated, e.g.:

ApifyStorageClient.setStatusMessage — Crawlee v4 calls this on the storage client to surface crawl progress (it was warning setStatusMessage is not a function).

is no longer true since apify/crawlee#3818

janbuchar · 2026-07-21T09:36:37Z

 export interface OpenStorageContext {
    config: Configuration;
-    client?: StorageClient;
+    client?: StorageBackend;


I guess the field deserves a rename too?

janbuchar · 2026-07-21T09:39:44Z

+        // Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that
+        // can't be satisfied by name-remapping; it's implemented on apify-client.


Suggested change

// Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that

// can't be satisfied by name-remapping; it's implemented on apify-client.

ok thx bye

janbuchar · 2026-07-21T09:42:24Z

 * ```
 */
-export class ApifyStorageClient implements StorageClient {
+export class ApifyStorageClient implements StorageBackend {


Rename the class and file please

janbuchar · 2026-07-21T09:44:02Z

    }

-    async createRequestQueueClient(options?: CreateRequestQueueClientOptions): Promise<RequestQueueClient> {
+    async createRequestQueueBackend(options?: CreateRequestQueueBackendOptions): Promise<RequestQueueBackend> {


The options don't make it possible to toggle between single and shared modes (no locking vs locking) — is this tracked in an issue? Or shall we fix it right here right now?

janbuchar

No sense in blocking this, let's try it out in the wild

janbuchar · 2026-08-07T08:13:48Z

+    async createKeyValueStoreBackend(options?: StorageIdentifier): Promise<KeyValueStoreBackend> {
        const id = await this.resolveId(options, 'KeyValueStore');
        const client = this.client.keyValueStore(id);
        return adapt(


I don't know, it would be easier to understand if we just made two dummy classes, DatasetBackend and KeyValueStoreBackend, like for the request queue

Good point, done via d1b71f6

Adapts to the changed upstream APIs: storage backend factory methods now take a StorageIdentifier (create*Backend), setStatusMessage moved off the storage backend to the run API (apify/crawlee#3818), moved types (QueueOperationInfo, Constructor, Dictionary), removed helpers (snakeCaseToCamelCase is now inlined), privatized ProxyConfiguration internals, and the KeyValueStore frontend field changes. Also adds @apify/datastructures, used by the upcoming request queue backends.

Configuration.getGlobalConfig() is now getGlobalConfiguration(), and the public config properties are now called configuration: Actor.configuration (static getter and instance property), ProxyConfiguration.configuration and PlatformEventManager.configuration. Follows the same renames in Crawlee v4.

janbuchar · 2026-08-07T10:28:25Z

+ * dataset client (pay-per-event on `Actor.pushData()`), provided a charging
+ * manager is supplied and a default-dataset-item price is configured.
+ *
+ * `storageExists()` lets `Dataset.open(idOrName)` resolve a string to an id


Implementation details leaking in comment?

janbuchar · 2026-08-07T10:31:34Z

+
+describe('ApifyStorageBackend', () => {
+    test('creates a single-consumer request queue backend by default', async () => {
+        const client = createMockApifyClient();


This doesn't seem to test anything typescript wouldn't catch

Follows crawlee v4's StorageClient -> StorageBackend interface rename. The constructor now takes an options object ({ configuration }) instead of positional arguments, and OpenStorageContext.client is now OpenStorageContext.backend. (ApifyStorageBackend is new in v4, so this is not a breaking change against v3.)

Crawlee v4 passes StorageIdentifier aliases down to the storage backend (including the reserved __default__ alias used for every default storage open). The backend now resolves them itself: __default__ maps to the run's default storage id, other aliases resolve via the Actor schema storages (ACTOR_STORAGES_JSON), undeclared aliases on the platform get a clear error, and outside the platform an unnamed storage is created per alias and process. Also implements getStorageBackendCacheKey, which partitions crawlee's storage-instance cache by API base URL and token.

Implements crawlee v4's stateful, pull-based RequestQueueBackend for the Apify platform, modeled on the Python SDK's request queue clients: - single (default): assumes one consumer; no request locking, local head estimation and full-request caching, so most requests are processed without per-request read API calls. A one-time prefetch of existing queue contents lets resurrected runs deduplicate re-added requests locally instead of paying for platform writes. - shared (requestQueueAccess: 'shared'): safe for any number of concurrent consumers via server-side locking (listAndLockHead); the lock duration follows the crawler's expected request processing time. The mode is selected via Actor.init({ requestQueueAccess }) or the ApifyStorageBackend constructor option. Request queue clients send a stable per-run clientKey (the run id), so a migrated or resurrected run re-acquires the locks of its previous incarnation.

Crawlee v4's storage backends are byte transports: the KeyValueStore frontend parses record values according to their content type. The backend was handing over values already parsed by apify-client, so JSON records got parsed twice. Records are now fetched with buffer: true and returned unparsed. Also returns the paginated listKeys result shape the interface expects instead of a bare items array.

…classes Adds ApifyDatasetBackend and ApifyKeyValueStoreBackend, mirroring the request queue backends, instead of wrapping apify-client resource clients in a generic renaming Proxy. The pay-per-event marker now lives on the dataset backend instance, and the Apify-store detection in KeyValueStore.getPublicUrl() is a plain instanceof check.

B4nan added the adhoc Ad-hoc unplanned task added during the sprint. label Jun 16, 2026

B4nan requested a review from janbuchar June 16, 2026 13:21

janbuchar requested changes Jul 21, 2026

View reviewed changes

B4nan requested a review from szaganek as a code owner August 6, 2026 15:00

B4nan requested a review from janbuchar August 6, 2026 15:10

janbuchar approved these changes Aug 7, 2026

View reviewed changes

B4nan added 2 commits August 7, 2026 12:27

janbuchar approved these changes Aug 7, 2026

View reviewed changes

B4nan force-pushed the feat/crawlee-v4-request-queue branch from d1b71f6 to 82af3d3 Compare August 7, 2026 10:32

B4nan added 3 commits August 7, 2026 12:35

B4nan force-pushed the feat/crawlee-v4-request-queue branch from 82af3d3 to 5e83735 Compare August 7, 2026 10:35

B4nan added 2 commits August 7, 2026 12:44

B4nan force-pushed the feat/crawlee-v4-request-queue branch from 5e83735 to 16cb202 Compare August 7, 2026 10:44

B4nan merged commit 8bc2388 into v4 Aug 7, 2026
7 checks passed

B4nan deleted the feat/crawlee-v4-request-queue branch August 7, 2026 10:47

This was referenced Aug 7, 2026

Update ProxyConfiguration interface in v4 #567

Closed

Implement new apify-client-js methods in Crawlee storages apify/crawlee#3162

Closed

Navigation Menu

Uh oh!

feat: implement crawlee v4 RequestQueueClient over apify-client - #643

feat: implement crawlee v4 RequestQueueClient over apify-client#643
B4nan merged 7 commits into
v4from
feat/crawlee-v4-request-queue

B4nan commented Jun 16, 2026 •

edited

Loading

Uh oh!

janbuchar left a comment

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar Jul 21, 2026

Uh oh!

janbuchar left a comment

Uh oh!

janbuchar Aug 7, 2026

Uh oh!

B4nan Aug 7, 2026

Uh oh!

janbuchar Aug 7, 2026

Uh oh!

janbuchar Aug 7, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

		// Crawlee v4's RequestQueueBackend is a stateful, pull-based interface that
		// can't be satisfied by name-remapping; it's implemented on apify-client.

Uh oh!

Conversation

B4nan commented Jun 16, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Request queue access modes

Review feedback addressed

Crawlee beta.105 adaptation (was beta.71)

Validation

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

janbuchar left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

B4nan commented Jun 16, 2026 •

edited

Loading