Checkpointing `workflow.json` incrementally for nat eval

[alessiodevoto]

Is this a new feature, an improvement, or a change to existing functionality?

Improvement

How would you describe the priority of this feature request

Critical (currently preventing usage)

Please provide a clear description of problem this feature solves

When running nat eval on a dataset, workflow_output.json is only written/visible after the entire dataset finishes. If evaluation hangs/fails or we have to interrupt it, we lose access to results from tasks that already completed, and the run isn’t recoverable without re-running. It would be very helpful to persist per-task intermediate outputs to disk as soon as each dataset item completes (before eval) so partial results are always recoverable.

Describe your ideal solution

Add incremental output writing for nat eval, e.g. write one JSONL record per completed item (dataset_item_..json) and optionally generate the aggregated workflow_output.json at the end.

Additional context

No response

Code of Conduct

• I agree to follow this project's Code of Conduct
• I have searched the open feature requests and have found no duplicates for this feature request

[Akshat8510]

Hi @alessiodevoto, I see this is a critical issue. I've been looking into the nat eval output logic recently and would love to work on implementing this incremental checkpointing. My plan would be to introduce a JSONL-style writer so that each task is persisted as it completes. Should I start looking into the eval loop implementation, or is someone from the core team already tackling this due to its priority?

[alessiodevoto]

Hi @Akshat8510, thanks for your message! Not sure, maybe @AnuradhaKaruppiah has info about that! 🙂

[Akshat8510]

Thanks @alessiodevoto! Hi @AnuradhaKaruppiah, if you haven't started on the implementation yet, I’d be happy to prepare a draft PR today. My plan is to modify the evaluation loop to write results incrementally to a .jsonl or temporary directory, ensuring data is saved even if the process is interrupted. Let me know if you'd like me to take a shot at it!

[AnuradhaKaruppiah]

Hey @Akshat8510, Sure go ahead with a PR

[Akshat8510]

Thanks @AnuradhaKaruppiah! I'm on it. I’ll prepare the PR and link it to this issue shortly.

[Akshat8510]

Hi @AnuradhaKaruppiah, I have just opened PR #1652 with the implementation for incremental checkpointing. I've ensured the logic handles both local and remote runs and flushes to disk immediately. Please let me know if you have any feedback or if there are any specific refinements or additional changes you'd like me to make to match the project's standards. Thanks!