Performance tuning for NodeODM

[JamesSample]

This is not really an issue - more a set of notes to be used later to guide performance tuning of NodeODM.

During May 2023, Sindre flew several hundred seabird missions ranging in size from about 20 to more than 1000 images. These missions provided a nice opportunity to test the orthomosaicing performance and scaling of NodeODM on Sigma2. We had a lot of discussion about this on Teams, but I'm gathering the key points here for reference later.

• Processing time scales roughly linearly with the number of images.

  $T \approx 0.6 \times N$, where $T$ is expected time in minutes and $N$ is the number of raw images.

  

  This relationship generally applies for CPU-bound processing i.e. where memory is not a limiting factor and all CPUs are fully utilised. It is possible to process single missions faster by giving them excess memory and CPUs, but the best overall performance and resource usage seems to be achieved by running several missions simultaneously, where the aim is to use all available CPUs while avoiding out-of-memory errors.

  As an example, running a single mission with excess memory and CPU, we were able to process 800 images in 285 minutes. This likely represents the best-case performance for NodeODM running on Sigma, although further testing would be needed to explore variability. Note that this is considerably faster than the expected time of $0.6 \times 800 = 480$ minutes based on the (predominantly CPU-bound) regression data. However, running four missions simultaneously and saturating all available CPUs, we were able to process four missions at once:

  $N$	$T$
  757	480
  461	291
  734	543
  491	381

  These runtimes are broadly consistent with the regression relationship above i.e. each individual mission is processed slower than if it was running by itself. However, the total overall runtime to complete all four is 543 minutes (because they all started processing together at 15:00 in the afternoon and the last one finished just after midnight). The total number of images processed during this time was $757 + 461 + 734 + 491 = 2443$. If we assume linear scaling, which seems reasonable given the plot above, running these same missions sequentially with excess CPU and memory would take around $285 \times 2443 / 800 = 870$ minutes. In other words, even though the individual missions are processed more slowly, the overall performance running missions in parallel is about 1.6 times better than running them sequentially, as long as the pod doesn't run out of memory. This is because NodeODM is able to make full use of the available CPUs for a greater proportion of the time.

• NodeODM memory usage is "spiky".

  When processing a single mission with NodeODM, the memory footprint is usually small, but with some large spikes towards the end of the workflow. With our initial configuration these spikes caused problems, because NodeODM would run for a long time, then crash with an out-of-memory error right at the end.

  In our original setup, NodeODM had access to 96 GB of memory and it allowed two jobs to run simultaneously. This worked well for processing missions with up to about 450 images, but for larger missions we repeatedly experienced crashes if both tasks reached memory spikes at roughly the same time. This caused significant downtime and led to a backlog of "large" missions (> 450 images) needing to be processed.

  By setting the number of parallel tasks to 1, we were able to successfully process a single mission with 800 images (see above). This tentatively implies peak memory usage for an 800-image mission is less than 96 GB (although there might be considerable variability in this between missions).

Current configuration

To help solve the memory issue, Sigma increased our namespace memory allocation to 500 GB, of which we gave 400 GB to NodeODM. We are currently running with max_parallel_tasks = 4 on the basis that, using the data above, we should be able to run four 800-image missions simultaneously and have them all spike at the same time without crashing due to lack of memory (just). It remains to be seen whether this works robustly, but it seems OK so far.

For larger missions (> 1000 images?) it is probably worth reducing the number of parallel tasks on NodeODM to avoid memory errors.

Anything you'd like to add, @knl88, @awigeon ?

[knl88]

Nice, I think one thing we can try if we are lucky enough to have more tasks that are killed is to just restart them, but we should figure out how to adjust the nodeodm clean interval. When a task fails nodeodm usually writes the cause so we can just have a snippet to queue one mission at a time from the failed list when the cause is oom.