Currently DeltaTableWriter, which is used for writing data to disk, can fail in the write-commit process which means data is not saved. We currently delete the files that we tried (but failed) to commit which means this data is lost. This should be changed so DeltaTableWriter is more robust to transient errors.
A discussion on this was started here: #382 (comment)
Currently
DeltaTableWriter, which is used for writing data to disk, can fail in the write-commit process which means data is not saved. We currently delete the files that we tried (but failed) to commit which means this data is lost. This should be changed soDeltaTableWriteris more robust to transient errors.A discussion on this was started here: #382 (comment)