Resume tmp file after interrupt

[lh-astro]

It happens that the converter gets an interrupt through slurm timeouts or other causes. It would be great if the converter can detect where it has stopped at the tmp file and continue from there.

[confluence]

This issue came up again in a discussion with @lh-astro today, and I think that this would be a valuable feature.

At the moment the slow converter is very slow, and jobs converting large files easily hit the time limit and get killed. Even when we have completed the smart converter implementation, under optimal conditions very large files may still hit the limit. The work that goes into producing a partial file is currently completely wasted.

It would be very useful at least to be able to resume from a checkpoint that was determined up-front (with the process exiting cleanly), but it would be ideal (and probably easier) to resume from any partial file.

The difficulty is in the statistics calculation: some aggregate statistics span the whole file and are computed from partial values that are stored only in memory and never written out. To implement this feature we would need to restore these values after resuming. We could do this by storing them after each iteration in temporary fields which are removed at the end of conversion. We could also increment a counter value when each iteration is completed, so that we can resume from a known "good" point even if the process is killed, and lose at most one iteration's worth of work. The temporary values can vary depending on the type of converter (fast or slow) and the stage of the conversion (e.g. the slow converter does multiple distinct passes).

For the smart converter, which is expected to run in parallel, we could store multiple sets of checkpoint values (one per thread) at the point where we complete one chunk (when we synchronise threads anyway before writing data).