What is speculative task execution: https://kb.databricks.com/scala/understanding-speculative-execution
In order to implement speculative task execution, we will have to introduce a source of non-determinism into the task runtimes. Without that, the static runtime we have today ( stats.avg ) would either never trigger a speculative run or lead to an infinite loop.
There are several ways to introduce non-determinism
- Randomize task runtimes – let users provide distribution as an input?
- Periodically slow-down tasks running on a random executor – this should be possible using interrupts
- Model IO
What is speculative task execution: https://kb.databricks.com/scala/understanding-speculative-execution
In order to implement speculative task execution, we will have to introduce a source of non-determinism into the task runtimes. Without that, the static runtime we have today (
stats.avg) would either never trigger a speculative run or lead to an infinite loop.There are several ways to introduce non-determinism