Actor Run Queue

A manager Actor that starts one run of an existing Actor per item of a list, and keeps doing so for as long as the Apify account has memory to spare. Instead of one big run that works through a list sequentially, you get as many runs in parallel as your account's memory limit allows, and the memory is kept busy until the whole list is processed.

How it works

1. The manager walks through dynamicInputList and starts a target Actor run per item. The input of each run is inputTemplate with every {{}} placeholder filled in with the current item.
2. Runs are started as fast as the API accepts them. There is no concurrency setting - the account memory limit is what paces the manager.
3. When the API refuses to start a run because the account is out of memory (actor-memory-limit-exceeded) or already running as many Actors as it may (concurrent-runs-limit-exceeded), the manager waits limitRetryWaitSecs (5 seconds by default) and offers the same item again. Nothing is skipped, and a slot is taken as soon as one frees up.
4. Every started run is waited for. When repushDatasetItems is enabled, the default dataset of each finished run is loaded and pushed to the manager's own dataset, so all results end up in one place.
5. When the whole list has been processed, a summary is stored to the OUTPUT record of the default key-value store.

Input

Field	Type	Required	Description
targetActorId	string	yes	ID or full name of the Actor to run, e.g. apify/web-scraper.
dynamicInputList	string[]	yes	One target Actor run is started per item.
inputTemplate	object	yes	Input of the target Actor, with {{}} where the current item belongs. See below.
memoryMbytes	integer	no	Memory per target Actor run. Lower memory means more runs fit into the account limit. Defaults to the Actor's.
timeoutSecs	integer	no	Timeout per target Actor run, 0 means no timeout. Defaults to the target Actor's own default.
build	string	no	Build tag or number of the target Actor, e.g. latest. Defaults to the target Actor's default build.
repushDatasetItems	boolean	no	Load the dataset of every finished run and push its items to the manager's dataset. Default false.
limitRetryWaitSecs	integer	no	How long to wait before retrying the same item after hitting an account limit. Default 5.

The input template

inputTemplate is the target Actor's input, with a {{}} placeholder wherever the current item of dynamicInputList belongs:

{
    "targetActorId": "apify/web-scraper",
    "dynamicInputList": ["https://example.com", "https://example.org", "https://example.net"],
    "inputTemplate": {
        "startUrls": [{ "url": "{{}}" }],
        "maxPagesPerCrawl": 10
    },
    "memoryMbytes": 1024,
    "timeoutSecs": 3600,
    "repushDatasetItems": true
}

That starts apify/web-scraper three times, each with 1024 MB and the input { "startUrls": [{ "url": "<one of the three URLs>" }], "maxPagesPerCrawl": 10 }.

• Anything between the braces is ignored, so {{}}, {{url}} and {{ the url }} are all the same placeholder. Use whatever reads best.
• Every placeholder is filled in, however deeply nested it is - inside arrays, inside nested objects, in a key, or in several places at once.
• A placeholder does not have to be the whole value. "url": "https://example.com/{{}}?page=1" fills the item into the middle of the string and leaves the rest of it alone.
• Items are filled in as strings, and any item is safe to use. A value containing a quote, a backslash or a newline reaches the target Actor exactly as it appears in the list; nothing needs escaping.

The manager renders the template by substituting into its serialized JSON and parsing the result back, which is how a placeholder is found no matter where it sits in the template.

Output

The OUTPUT record of the default key-value store holds a summary:

{
    "targetActorId": "apify/web-scraper",
    "totalItems": 3,
    "startedRuns": 3,
    "succeededRuns": 2,
    "unsuccessfulRuns": 1,
    "startFailures": 0,
    "repushedItems": 128,
    "runsByStatus": { "SUCCEEDED": 2, "FAILED": 1 },
    "failedItems": [{ "itemValue": "https://example.net", "runId": "abc123", "status": "FAILED" }]
}

With repushDatasetItems enabled, the manager's default dataset contains the items of all the runs it started.

Restarts and migrations

The manager keeps its state in the MANAGER_STATE record of the default key-value store: how far it got in the list, the ID and status of every run it started, and how much of each run's dataset it has already re-pushed. A run ID is persisted the moment the run exists, so no run can be orphaned.

When the manager migrates to another server or is restarted (resurrected), it:

• picks up the runs that were still in progress and waits for them again, rather than starting them twice,
• continues re-pushing datasets of runs that finished while it was down, from the item it stopped at,
• continues dispatching from the first item it had not dispatched yet.

The state belongs to a specific job, identified by targetActorId, inputTemplate and dynamicInputList. If any of those change, the old state does not apply and the manager starts over. Changing the run options does not invalidate the state, so a resurrected run keeps its progress even if you give the runs more memory or a longer timeout.

Why Actor.start() and not Actor.call()

Actor.call() only resolves once the run has finished, which means the run ID is not available while the run is in progress. A manager that migrated would have no way to find its own runs again. So the manager uses Actor.start() and waits for the run separately - the two together are equivalent to Actor.call(), including the account limit errors, which come from the very same API call that starts the run.

A note on re-pushed items

Re-push progress is persisted after every batch of 1000 items, so a migration in the middle of a re-push can duplicate at most the batch that was in flight. Dataset pushes cannot be deduplicated on the platform side, so treat repushDatasetItems as at-least-once delivery.

Failures

• A run that ends as FAILED, ABORTED or TIMED-OUT is reported in the summary; the manager does not restart it, so the list is processed exactly once.
• An item whose run cannot be started for a reason other than an account limit is retried three times with a backoff and then reported in failedItems so the rest of the queue is not blocked.
• An item that cannot be turned into a run input is reported in failedItems right away - retrying cannot change the outcome - and the rest of the list is processed as usual.
• Account limit errors are never counted as failures - they only pace the manager. If they keep coming while the manager has nothing running, it warns that the requested memoryMbytes may not fit into the account limit at all, and keeps retrying.

Development

npm install
npm run build        # type-check and compile to dist/
npm test             # unit tests (vitest)
npm run lint         # eslint
npm run format       # prettier
apify run            # run locally (requires apify-cli and `apify login`)

The Actor calls the Apify API, so a local run needs a token - log in with apify login, or set APIFY_TOKEN.