Skip to content

Improve Tron restart times by adding parallelization to job_collection.restore_state() - #1111

Draft
CharlesPeacock wants to merge 1 commit into
masterfrom
TRON-2651_restart_improvements_2
Draft

Improve Tron restart times by adding parallelization to job_collection.restore_state()#1111
CharlesPeacock wants to merge 1 commit into
masterfrom
TRON-2651_restart_improvements_2

Conversation

@CharlesPeacock

Copy link
Copy Markdown
Contributor

Follow-up to the series of Tron restart improvements.

As of recently, Tron has been seeing restart times of over 6 minutes in pnw-prod. Optimizing these restart times allows for less downtime and more opportunities to experiment with new changes.

This change adds a ThreadPoolExecutor to parallelize job_collection.restore_state(). By setting max_workers to 5 we can see a significant improvement in state restoration times:

Original performance (tested with copy of tron-pnw-devc dynamodb table):

Run: linear run (Tue Jul 28 10:32:01 AM PDT 2026)
Total startup: 93.59s
Schedulers built: 3.786149263381958s
DynamoDB retrieval: 10:27:25 -> 10:27:55 (30.000s)
Apply state: 10:27:55 -> 10:28:53 (58.000s)
Unprocessed key retries: 5
Log: /tmp/output_log_linear_attempt1.log
Restore state: 57.425s <----- 

Performance after parallelization (same environment as original test, 5 workers):

Run: parallel run (Tue Jul 28 10:19:16 AM PDT 2026)
Total startup: 66.10s
Schedulers built: 3.781592607498169s
DynamoDB retrieval: 10:14:21 -> 10:14:52 (31.000s)
Apply state: 10:14:52 -> 10:15:22 (30.000s)
Unprocessed key retries: 7
Log: /tmp/output_log_parallel_attempt2.log
Restore state: 29.658s <-----

Tests passed

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant