Problematic EK80 Environment Datagrams for Replayed Files and Large Files

[ctuguinay]

From conversation with @leewujung,

Currently, the Echopype parsing code in parse_base.py expects the EK80 Environment datagram to be unique, i.e., we only get one XML datagram with subtype environment. However, when parsing replayed files and large files (~2GiB) with EK80 application_version='23.6.2.0' and file_format_version='1.32' (the latest versions), we encounter multiple XML datagrams with subtype environment where only the first datagram contains all the environment variables we need for calibration (temp, salinity, pressure, etc.) and the latter datagrams contain only drop_keel_offset and drop_keel_offset_is_manual. The latter datagrams override the first datagram and we end up with an EchoData object that does not contain the environment parameters we need for calibration.

As a temporary fix to this, we will drop XML datagrams with subtype environment where the only data available is drop_keel_offset and drop_keel_offset_is_manual.

@leewjung update: A more thorough fix is to take in all environment datagrams and set the non-existing variables to NaN, and then ensure the downstream compute_Sv function can handle these variations.

[leewujung]

@ctuguinay : Apparently I did documented this. I think last year when I ran into this I was using files from Lasker that had the 18+ADCP and 5-frequency set up and also variable data collection range depending on bottom depth (so saving 2 files at any given time). This year I was testing replayed files using Shimada data which does not have the complication. Not sure if that was the difference. I know this year Echopype successfully convert 2GB files without problem.

I think I'll keep this open for now as a reminder.