When monitoring MWA ASVO jobs we regularly see bad Legacy MWA observations with corruption in at least one FITS file. Most of the time the remaining data is fine, but the current behaviour of Birli is to fail/abort irrecoverably.
E.g.
.../1207996720_20180417103824_gpubox01_00.fits HDU 86: Fits error: FitsError { status: 107, message: "tried to move past end of file"
It is possible in the above example that all other HDUs are fine- so can we maybe just skip the bad data and move on?
Suggested behaviour:
Maybe Birli could have a new command-line flag (e.g. --flag-bad-data that catches this sort of error and instead of aborting, will flag the relevant HDUs timestep for that coarse channel and just continue. It would also be good to show a summary of where corrupted data was detected and flagged. If the new cmd line arg is absent, the existing behaviour occurs (abort).
When monitoring MWA ASVO jobs we regularly see bad Legacy MWA observations with corruption in at least one FITS file. Most of the time the remaining data is fine, but the current behaviour of Birli is to fail/abort irrecoverably.
E.g.
It is possible in the above example that all other HDUs are fine- so can we maybe just skip the bad data and move on?
Suggested behaviour:
Maybe Birli could have a new command-line flag (e.g.
--flag-bad-datathat catches this sort of error and instead of aborting, will flag the relevant HDUs timestep for that coarse channel and just continue. It would also be good to show a summary of where corrupted data was detected and flagged. If the new cmd line arg is absent, the existing behaviour occurs (abort).