Distinguishing the instrumental sources of .raw files (EK60 v.s. EK80)

[oftfrfbf]

Update: Changing the issue to focus on distinguishing the instrumental source of ".raw" files when processing data. If a user specifies a "sonar_model" of "EK60" when it is "EK80" (or vice versa), either a message or a meaningful exception should be returned.

---

I am encountering a lot of 'transceivers' exceptions as I spot check EK60 data. From 286 different cruises I have found ~60 cruises with this specific exception. I am not sure how similar each is to one another but below is an example of one error.

Code:

import echopype as ep
print(ep.__version__) # 0.5.5.dev49+g8180da1

raw_file_s3path = f"s3://noaa-wcsd-pds/data/raw/Falkor/FK180415/EK60/D20180416-T073530.raw"
try:
	ed = ep.open_raw(raw_file_s3path, sonar_model='EK60', storage_options={'anon': True})
	ds_Sv = ep.calibrate.compute_Sv(ed)
except Exception as e:
	print(f"Problem with: {raw_file_s3path}, {e}")

Output:

23:48:59  parsing file D20180416-T073530.raw, time of first ping: 2018-Apr-16 07:35:30
Problem with: s3://noaa-wcsd-pds/data/raw/Falkor/FK180415/EK60/D20180416-T073530.raw, 'transceivers'

echopype/echopype/convert/set_groups_ek60.py

Line 20 in 1548f46

for ch in self.parser_obj.config_datagram["transceivers"].keys():

In the "set_groups_ek60.py" the parser_obj.config_datagram has no "transceivers" key to reference. The config_datagram:

{
	'type': 'XML0',
	'low_date': 2127786896,
	'high_date': 30659925,
	'timestamp': numpy.datetime64('2018-04-16T07:35:30.059'),
	'bytes_read': 11104,
	'subtype': 'configuration',
	'configuration': {
		'GPT 0090720674d4-1 ES18': {
			'transceiver_name': 'GPT 0090720674d4',
			'ethernet_address': '0090720674d4',
			'ip_address': '157.237.15.3',
			'transceiver_version': 'GPT-Q18(2)-S 1.0 0090720674d4',
			'transceiver_software_version': '070413',
			'transceiver_number': 1,
			'market_segment': 'Scientific',
			'transceiver_type': 'GPT',
			'serial_number': '0',
			'impedance': 1000,
			'channel_id': 'GPT 0090720674d4-1 ES18',
			'channel_id_short': 'Falkor ES18',
			'max_tx_power_transceiver': 2000,
			'pulse_duration': [0.000512, 0.001024, 0.002048, 0.004096, 0.008192],
			'sample_interval': [0.000128, 0.000256, 0.000512, 0.001024, 0.002048],
			'hw_channel_configuration': '0',
			'transducer_name': 'ES38B',
			'transducer_serial_number': '0',
			'transducer_frequency': 18000.0,
			'transducer_frequency_minimum': 18000.0,
			'transducer_frequency_maximum': 18000.0,
			'transducer_beam_type': 1,
			'equivalent_beam_angle': -17.0,
			'gain': [20.3, 22.4, 22.9, 23.0, 23.0],
			'sa_correction': [0.0, 0.0, 0.0, 0.0, 0.0],
			'max_tx_power_transducer': 2000.0,
			'beam_width_alongship': 11.0,
			'beam_width_athwartship': 11.0,
			'angle_sensitivity_alongship': 15.5,
			'angle_sensitivity_athwartship': 15.5,
			'angle_offset_alongship': 0.0,
			'angle_offset_athwartship': 0.0,
			'directivity_drop_at_2x_beam_width': 0.0,
			'transducer_mounting': 'HullMounted',
			'transducer_custom_name': 'Falkor ES38B',
			'transducer_orientation': 'Vertical',
			'transducer_offset_x': 18.48,
			'transducer_offset_y': -0.5,
			'transducer_offset_z': 6.1,
			'transducer_alpha_x': 0.0,
			'transducer_alpha_y': 0.0,
			'transducer_alpha_z': 0.0,
			'copyright': 'Copyright(c) Kongsberg Maritime AS, Norway',
			'application_name': 'EK80',
			'application_version': '1.10.3.0',
			'file_format_version': '1.20',
			'time_bias': '0'
		},
		'GPT 009072069088-1 ES38B': {
			'transceiver_name': 'GPT 009072069088',
			'ethernet_address': '009072069088',
			'ip_address': '157.237.15.5',
			'transceiver_version': 'GPT-Q38(2)-S 1.0 009072069088',
			'transceiver_software_version': '070413',
			'transceiver_number': 2,
			'market_segment': 'Scientific',
			'transceiver_type': 'GPT',
			'serial_number': '0',
			'impedance': 1000,
			'channel_id': 'GPT 009072069088-1 ES38B',
			'channel_id_short': 'Falkor ES38B',
			'max_tx_power_transceiver': 2000,
			'pulse_duration': [0.000256, 0.000512, 0.001024, 0.002048, 0.004096],
			'sample_interval': [6.4e-05, 0.000128, 0.000256, 0.000512, 0.001024],
			'hw_channel_configuration': '0',
			'transducer_name': 'ES38B',
			'transducer_serial_number': '0',
			'transducer_frequency': 38000.0,
			'transducer_frequency_minimum': 38000.0,
			'transducer_frequency_maximum': 38000.0,
			'transducer_beam_type': 1,
			'equivalent_beam_angle': -20.7,
			'gain': [23.0, 25.0, 25.5, 25.5, 25.0],
			'sa_correction': [0.0, 0.0, 0.0, 0.0, 0.0],
			'max_tx_power_transducer': 2000.0,
			'beam_width_alongship': 7.0,
			'beam_width_athwartship': 7.0,
			'angle_sensitivity_alongship': 23.0,
			'angle_sensitivity_athwartship': 23.0,
			'angle_offset_alongship': 0.0,
			'angle_offset_athwartship': 0.0,
			'directivity_drop_at_2x_beam_width': 0.0,
			'transducer_mounting': 'HullMounted',
			'transducer_custom_name': 'Falkor ES38B',
			'transducer_orientation': 'Vertical',
			'transducer_offset_x': 18.48,
			'transducer_offset_y': -0.5,
			'transducer_offset_z': 6.1,
			'transducer_alpha_x': 0.0,
			'transducer_alpha_y': 0.0,
			'transducer_alpha_z': 0.0,
			'copyright': 'Copyright(c) Kongsberg Maritime AS, Norway',
			'application_name': 'EK80',
			'application_version': '1.10.3.0',
			'file_format_version': '1.20',
			'time_bias': '0'
		},
		'GPT 0090720694fc-1 ES70-7C': {
			'transceiver_name': 'GPT 0090720694fc',
			'ethernet_address': '0090720694fc',
			'ip_address': '157.237.15.7',
			'transceiver_version': 'GPT-Q70(1)-S 1.0 0090720694fc',
			'transceiver_software_version': '070413',
			'transceiver_number': 3,
			'market_segment': 'Scientific',
			'transceiver_type': 'GPT',
			'serial_number': '0',
			'impedance': 1000,
			'channel_id': 'GPT 0090720694fc-1 ES70-7C',
			'channel_id_short': 'Falkor ES70-7C',
			'max_tx_power_transceiver': 1000,
			'pulse_duration': [0.000128, 0.000256, 0.000512, 0.001024, 0.002048],
			'sample_interval': [3.2e-05, 6.4e-05, 0.000128, 0.000256, 0.000512],
			'hw_channel_configuration': '0',
			'transducer_name': 'ES38B',
			'transducer_serial_number': '0',
			'transducer_frequency': 70000.0,
			'transducer_frequency_minimum': 70000.0,
			'transducer_frequency_maximum': 70000.0,
			'transducer_beam_type': 1,
			'equivalent_beam_angle': -20.7,
			'gain': [27.0, 27.0, 27.0, 27.0, 27.0],
			'sa_correction': [0.0, 0.0, 0.0, 0.0, 0.0],
			'max_tx_power_transducer': 750.0,
			'beam_width_alongship': 7.0,
			'beam_width_athwartship': 7.0,
			'angle_sensitivity_alongship': 23.0,
			'angle_sensitivity_athwartship': 23.0,
			'angle_offset_alongship': 0.0,
			'angle_offset_athwartship': 0.0,
			'directivity_drop_at_2x_beam_width': 0.0,
			'transducer_mounting': 'HullMounted',
			'transducer_custom_name': 'Falkor ES38B',
			'transducer_orientation': 'Vertical',
			'transducer_offset_x': 18.48,
			'transducer_offset_y': -0.5,
			'transducer_offset_z': 6.1,
			'transducer_alpha_x': 0.0,
			'transducer_alpha_y': 0.0,
			'transducer_alpha_z': 0.0,
			'copyright': 'Copyright(c) Kongsberg Maritime AS, Norway',
			'application_name': 'EK80',
			'application_version': '1.10.3.0',
			'file_format_version': '1.20',
			'time_bias': '0'
		},
		'GPT 009072068b02-1 ES120-7C': {
			'transceiver_name': 'GPT 009072068b02',
			'ethernet_address': '009072068b02',
			'ip_address': '157.237.15.8',
			'transceiver_version': 'GPT-Q120(1)-S 1.0 009072068b02',
			'transceiver_software_version': '070413',
			'transceiver_number': 4,
			'market_segment': 'Scientific',
			'transceiver_type': 'GPT',
			'serial_number': '0',
			'impedance': 1000,
			'channel_id': 'GPT 009072068b02-1 ES120-7C',
			'channel_id_short': 'Falkor ES120-7C',
			'max_tx_power_transceiver': 1000,
			'pulse_duration': [6.4e-05, 0.000128, 0.000256, 0.000512, 0.001024],
			'sample_interval': [1.6e-05, 3.2e-05, 6.4e-05, 0.000128, 0.000256],
			'hw_channel_configuration': '0',
			'transducer_name': 'ES38B',
			'transducer_serial_number': '0',
			'transducer_frequency': 120000.0,
			'transducer_frequency_minimum': 120000.0,
			'transducer_frequency_maximum': 120000.0,
			'transducer_beam_type': 1,
			'equivalent_beam_angle': -20.7,
			'gain': [27.0, 27.0, 27.0, 27.0, 27.0],
			'sa_correction': [0.0, 0.0, 0.0, 0.0, 0.0],
			'max_tx_power_transducer': 250.0,
			'beam_width_alongship': 7.0,
			'beam_width_athwartship': 7.0,
			'angle_sensitivity_alongship': 23.0,
			'angle_sensitivity_athwartship': 23.0,
			'angle_offset_alongship': 0.0,
			'angle_offset_athwartship': 0.0,
			'directivity_drop_at_2x_beam_width': 0.0,
			'transducer_mounting': 'HullMounted',
			'transducer_custom_name': 'Falkor ES38B',
			'transducer_orientation': 'Vertical',
			'transducer_offset_x': 18.48,
			'transducer_offset_y': -0.5,
			'transducer_offset_z': 6.1,
			'transducer_alpha_x': 0.0,
			'transducer_alpha_y': 0.0,
			'transducer_alpha_z': 0.0,
			'copyright': 'Copyright(c) Kongsberg Maritime AS, Norway',
			'application_name': 'EK80',
			'application_version': '1.10.3.0',
			'file_format_version': '1.20',
			'time_bias': '0'
		},
		'GPT 009072068af7-1 ES200-7C': {
			'transceiver_name': 'GPT 009072068af7',
			'ethernet_address': '009072068af7',
			'ip_address': '157.237.15.9',
			'transceiver_version': 'GPT-Q200(1)-S 1.0 009072068af7',
			'transceiver_software_version': '070413',
			'transceiver_number': 5,
			'market_segment': 'Scientific',
			'transceiver_type': 'GPT',
			'serial_number': '0',
			'impedance': 1000,
			'channel_id': 'GPT 009072068af7-1 ES200-7C',
			'channel_id_short': 'Falkor ES200-7C',
			'max_tx_power_transceiver': 1000,
			'pulse_duration': [6.4e-05, 0.000128, 0.000256, 0.000512, 0.001024],
			'sample_interval': [1.6e-05, 3.2e-05, 6.4e-05, 0.000128, 0.000256],
			'hw_channel_configuration': '0',
			'transducer_name': 'ES38B',
			'transducer_serial_number': '0',
			'transducer_frequency': 200000.0,
			'transducer_frequency_minimum': 200000.0,
			'transducer_frequency_maximum': 200000.0,
			'transducer_beam_type': 1,
			'equivalent_beam_angle': -20.7,
			'gain': [27.0, 27.0, 27.0, 27.0, 27.0],
			'sa_correction': [0.0, 0.0, 0.0, 0.0, 0.0],
			'max_tx_power_transducer': 150.0,
			'beam_width_alongship': 7.0,
			'beam_width_athwartship': 7.0,
			'angle_sensitivity_alongship': 23.0,
			'angle_sensitivity_athwartship': 23.0,
			'angle_offset_alongship': 0.0,
			'angle_offset_athwartship': 0.0,
			'directivity_drop_at_2x_beam_width': 0.0,
			'transducer_mounting': 'HullMounted',
			'transducer_custom_name': 'Falkor ES38B',
			'transducer_orientation': 'Vertical',
			'transducer_offset_x': 18.48,
			'transducer_offset_y': -0.5,
			'transducer_offset_z': 6.1,
			'transducer_alpha_x': 0.0,
			'transducer_alpha_y': 0.0,
			'transducer_alpha_z': 0.0,
			'copyright': 'Copyright(c) Kongsberg Maritime AS, Norway',
			'application_name': 'EK80',
			'application_version': '1.10.3.0',
			'file_format_version': '1.20',
			'time_bias': '0'
		},
		'GPT 00907206909b-1 710-30': {
			'transceiver_name': 'GPT 00907206909b',
			'ethernet_address': '00907206909b',
			'ip_address': '157.237.15.11',
			'transceiver_version': 'GPT-S710(0.3)-S 1.0 00907206909b',
			'transceiver_software_version': '070413',
			'transceiver_number': 6,
			'market_segment': 'Scientific',
			'transceiver_type': 'GPT',
			'serial_number': '0',
			'impedance': 1000,
			'channel_id': 'GPT 00907206909b-1 710-30',
			'channel_id_short': 'Falkor 710-30',
			'max_tx_power_transceiver': 300,
			'pulse_duration': [6.4e-05, 0.000128, 0.000256, 0.000512, 0.001024],
			'sample_interval': [1.6e-05, 3.2e-05, 6.4e-05, 0.000128, 0.000256],
			'hw_channel_configuration': '0',
			'transducer_name': 'ES38B',
			'transducer_serial_number': '0',
			'transducer_frequency': 710000.0,
			'transducer_frequency_minimum': 710000.0,
			'transducer_frequency_maximum': 710000.0,
			'transducer_beam_type': 0,
			'equivalent_beam_angle': -23.0,
			'gain': [27.1, 27.5, 27.5, 27.5, 27.5],
			'sa_correction': [0.0, 0.0, 0.0, 0.0, 0.0],
			'max_tx_power_transducer': 100.0,
			'beam_width_alongship': 5.4,
			'beam_width_athwartship': 5.4,
			'angle_sensitivity_alongship': 0.0,
			'angle_sensitivity_athwartship': 0.0,
			'angle_offset_alongship': 0.0,
			'angle_offset_athwartship': 0.0,
			'directivity_drop_at_2x_beam_width': 0.0,
			'transducer_mounting': 'HullMounted',
			'transducer_custom_name': 'Falkor ES38B',
			'transducer_orientation': 'Vertical',
			'transducer_offset_x': 18.48,
			'transducer_offset_y': -0.5,
			'transducer_offset_z': 6.1,
			'transducer_alpha_x': 0.0,
			'transducer_alpha_y': 0.0,
			'transducer_alpha_z': 0.0,
			'copyright': 'Copyright(c) Kongsberg Maritime AS, Norway',
			'application_name': 'EK80',
			'application_version': '1.10.3.0',
			'file_format_version': '1.20',
			'time_bias': '0'
		}
	},
	'xml': '<?xml version="1.0" encoding="utf-8"?>\r\n<Configuration>\r\n  <Header Copyright="Copyright(c) Kongsberg Maritime AS, Norway" ApplicationName="EK80" Version="1.10.3.0" FileFormatVersion="1.20" TimeBias="0" />\r\n  <Transceivers MergeOperation="AddNodeTree">\r\n    <Transceiver TransceiverName="GPT 0090720674d4" EthernetAddress="0090720674d4" IPAddress="157.237.15.3" Version="GPT-Q18(2)-S 1.0 0090720674d4" TransceiverSoftwareVersion="070413" TransceiverNumber="1" MarketSegment="Scientific" TransceiverType="GPT" SerialNumber="0" Impedance="1000">\r\n      <Channels>\r\n        <Channel ChannelID="GPT 0090720674d4-1 ES18" ChannelIdShort="Falkor ES18" MaxTxPowerTransceiver="2000" PulseDuration="0.000512;0.001024;0.002048;0.004096;0.008192" SampleInterval="0.000128;0.000256;0.000512;0.001024;0.002048" HWChannelConfiguration="0">\r\n          <Transducer TransducerName="ES18" SerialNumber="0" Frequency="18000" FrequencyMinimum="18000" FrequencyMaximum="18000" BeamType="1" EquivalentBeamAngle="-17" Gain="20.3;22.4;22.9;23;23" SaCorrection="0;0;0;0;0" MaxTxPowerTransducer="2000" BeamWidthAlongship="11" BeamWidthAthwartship="11" AngleSensitivityAlongship="15.5" AngleSensitivityAthwartship="15.5" AngleOffsetAlongship="0" AngleOffsetAthwartship="0" DirectivityDropAt2XBeamWidth="0" />\r\n        </Channel>\r\n      </Channels>\r\n    </Transceiver>\r\n    <Transceiver TransceiverName="GPT 009072069088" EthernetAddress="009072069088" IPAddress="157.237.15.5" Version="GPT-Q38(2)-S 1.0 009072069088" TransceiverSoftwareVersion="070413" TransceiverNumber="2" MarketSegment="Scientific" TransceiverType="GPT" SerialNumber="0" Impedance="1000">\r\n      <Channels>\r\n        <Channel ChannelID="GPT 009072069088-1 ES38B" ChannelIdShort="Falkor ES38B" MaxTxPowerTransceiver="2000" PulseDuration="0.000256;0.000512;0.001024;0.002048;0.004096" SampleInterval="6.4E-05;0.000128;0.000256;0.000512;0.001024" HWChannelConfiguration="0">\r\n          <Transducer TransducerName="ES38B" SerialNumber="0" Frequency="38000" FrequencyMinimum="38000" FrequencyMaximum="38000" BeamType="1" EquivalentBeamAngle="-20.7" Gain="23;25;25.5;25.5;25" SaCorrection="0;0;0;0;0" MaxTxPowerTransducer="2000" BeamWidthAlongship="7" BeamWidthAthwartship="7" AngleSensitivityAlongship="23" AngleSensitivityAthwartship="23" AngleOffsetAlongship="0" AngleOffsetAthwartship="0" DirectivityDropAt2XBeamWidth="0" />\r\n        </Channel>\r\n      </Channels>\r\n    </Transceiver>\r\n    <Transceiver TransceiverName="GPT 0090720694fc" EthernetAddress="0090720694fc" IPAddress="157.237.15.7" Version="GPT-Q70(1)-S 1.0 0090720694fc" TransceiverSoftwareVersion="070413" TransceiverNumber="3" MarketSegment="Scientific" TransceiverType="GPT" SerialNumber="0" Impedance="1000">\r\n      <Channels>\r\n        <Channel ChannelID="GPT 0090720694fc-1 ES70-7C" ChannelIdShort="Falkor ES70-7C" MaxTxPowerTransceiver="1000" PulseDuration="0.000128;0.000256;0.000512;0.001024;0.002048" SampleInterval="3.2E-05;6.4E-05;0.000128;0.000256;0.000512" HWChannelConfiguration="0">\r\n          <Transducer TransducerName="ES70-7C" SerialNumber="0" Frequency="70000" FrequencyMinimum="70000" FrequencyMaximum="70000" BeamType="1" EquivalentBeamAngle="-20.7" Gain="27;27;27;27;27" SaCorrection="0;0;0;0;0" MaxTxPowerTransducer="750" BeamWidthAlongship="7" BeamWidthAthwartship="7" AngleSensitivityAlongship="23" AngleSensitivityAthwartship="23" AngleOffsetAlongship="0" AngleOffsetAthwartship="0" DirectivityDropAt2XBeamWidth="0" />\r\n        </Channel>\r\n      </Channels>\r\n    </Transceiver>\r\n    <Transceiver TransceiverName="GPT 009072068b02" EthernetAddress="009072068b02" IPAddress="157.237.15.8" Version="GPT-Q120(1)-S 1.0 009072068b02" TransceiverSoftwareVersion="070413" TransceiverNumber="4" MarketSegment="Scientific" TransceiverType="GPT" SerialNumber="0" Impedance="1000">\r\n      <Channels>\r\n        <Channel ChannelID="GPT 009072068b02-1 ES120-7C" ChannelIdShort="Falkor ES120-7C" MaxTxPowerTransceiver="1000" PulseDuration="6.4E-05;0.000128;0.000256;0.000512;0.001024" SampleInterval="1.6E-05;3.2E-05;6.4E-05;0.000128;0.000256" HWChannelConfiguration="0">\r\n          <Transducer TransducerName="ES120-7C" SerialNumber="0" Frequency="120000" FrequencyMinimum="120000" FrequencyMaximum="120000" BeamType="1" EquivalentBeamAngle="-20.7" Gain="27;27;27;27;27" SaCorrection="0;0;0;0;0" MaxTxPowerTransducer="250" BeamWidthAlongship="7" BeamWidthAthwartship="7" AngleSensitivityAlongship="23" AngleSensitivityAthwartship="23" AngleOffsetAlongship="0" AngleOffsetAthwartship="0" DirectivityDropAt2XBeamWidth="0" />\r\n        </Channel>\r\n      </Channels>\r\n    </Transceiver>\r\n    <Transceiver TransceiverName="GPT 009072068af7" EthernetAddress="009072068af7" IPAddress="157.237.15.9" Version="GPT-Q200(1)-S 1.0 009072068af7" TransceiverSoftwareVersion="070413" TransceiverNumber="5" MarketSegment="Scientific" TransceiverType="GPT" SerialNumber="0" Impedance="1000">\r\n      <Channels>\r\n        <Channel ChannelID="GPT 009072068af7-1 ES200-7C" ChannelIdShort="Falkor ES200-7C" MaxTxPowerTransceiver="1000" PulseDuration="6.4E-05;0.000128;0.000256;0.000512;0.001024" SampleInterval="1.6E-05;3.2E-05;6.4E-05;0.000128;0.000256" HWChannelConfiguration="0">\r\n          <Transducer TransducerName="ES200-7C" SerialNumber="0" Frequency="200000" FrequencyMinimum="200000" FrequencyMaximum="200000" BeamType="1" EquivalentBeamAngle="-20.7" Gain="27;27;27;27;27" SaCorrection="0;0;0;0;0" MaxTxPowerTransducer="150" BeamWidthAlongship="7" BeamWidthAthwartship="7" AngleSensitivityAlongship="23" AngleSensitivityAthwartship="23" AngleOffsetAlongship="0" AngleOffsetAthwartship="0" DirectivityDropAt2XBeamWidth="0" />\r\n        </Channel>\r\n      </Channels>\r\n    </Transceiver>\r\n    <Transceiver TransceiverName="GPT 00907206909b" EthernetAddress="00907206909b" IPAddress="157.237.15.11" Version="GPT-S710(0.3)-S 1.0 00907206909b" TransceiverSoftwareVersion="070413" TransceiverNumber="6" MarketSegment="Scientific" TransceiverType="GPT" SerialNumber="0" Impedance="1000">\r\n      <Channels>\r\n        <Channel ChannelID="GPT 00907206909b-1 710-30" ChannelIdShort="Falkor 710-30" MaxTxPowerTransceiver="300" PulseDuration="6.4E-05;0.000128;0.000256;0.000512;0.001024" SampleInterval="1.6E-05;3.2E-05;6.4E-05;0.000128;0.000256" HWChannelConfiguration="0">\r\n          <Transducer TransducerName="710-30" SerialNumber="0" Frequency="710000" FrequencyMinimum="710000" FrequencyMaximum="710000" BeamType="0" EquivalentBeamAngle="-23" Gain="27.1;27.5;27.5;27.5;27.5" SaCorrection="0;0;0;0;0" MaxTxPowerTransducer="100" BeamWidthAlongship="5.4" BeamWidthAthwartship="5.4" AngleSensitivityAlongship="0" AngleSensitivityAthwartship="0" AngleOffsetAlongship="0" AngleOffsetAthwartship="0" DirectivityDropAt2XBeamWidth="0" />\r\n        </Channel>\r\n      </Channels>\r\n    </Transceiver>\r\n  </Transceivers>\r\n  <Transducers MergeOperation="AddNodeTree">\r\n    <Transducer TransducerName="ES38B" TransducerMounting="HullMounted" TransducerCustomName="Falkor ES38B" TransducerSerialNumber="0" TransducerOrientation="Vertical" TransducerOffsetX="18.48" TransducerOffsetY="-0.5" TransducerOffsetZ="6.1" TransducerAlphaX="0" TransducerAlphaY="0" TransducerAlphaZ="0" />\r\n    <Transducer TransducerName="ES70-7C" TransducerMounting="HullMounted" TransducerCustomName="Falkor ES70-7C" TransducerSerialNumber="0" TransducerOrientation="Vertical" TransducerOffsetX="18.07" TransducerOffsetY="-0.73" TransducerOffsetZ="6.1" TransducerAlphaX="0" TransducerAlphaY="0" TransducerAlphaZ="0" />\r\n    <Transducer TransducerName="ES120-7C" TransducerMounting="HullMounted" TransducerCustomName="Falkor ES120-7C" TransducerSerialNumber="0" TransducerOrientation="Vertical" TransducerOffsetX="18.07" TransducerOffsetY="-0.4" TransducerOffsetZ="6.11" TransducerAlphaX="0" TransducerAlphaY="0" TransducerAlphaZ="0" />\r\n    <Transducer TransducerName="ES200-7C" TransducerMounting="HullMounted" TransducerCustomName="Falkor ES200-7C" TransducerSerialNumber="0" TransducerOrientation="Vertical" TransducerOffsetX="18.23" TransducerOffsetY="-0.22" TransducerOffsetZ="6.11" TransducerAlphaX="0" TransducerAlphaY="0" TransducerAlphaZ="0" />\r\n    <Transducer TransducerName="710-30" TransducerMounting="HullMounted" TransducerCustomName="Falkor 710-30" TransducerSerialNumber="0" TransducerOrientation="Vertical" TransducerOffsetX="17.94" TransducerOffsetY="-0.28" TransducerOffsetZ="6.11" TransducerAlphaX="0" TransducerAlphaY="0" TransducerAlphaZ="0" />\r\n    <Transducer TransducerName="ES18" TransducerMounting="HullMounted" TransducerCustomName="Falkor ES18" TransducerSerialNumber="0" TransducerOrientation="Vertical" TransducerOffsetX="17.6" TransducerOffsetY="-0.5" TransducerOffsetZ="6.11" TransducerAlphaX="0" TransducerAlphaY="0" TransducerAlphaZ="0" />\r\n  </Transducers>\r\n  <ConfiguredSensors MergeOperation="AddNodeTree">\r\n    <Sensor Name="MotionBinary From Serial Port 2" Type="MotionBinary" Port="Serial Port 2" TalkerID="" X="0" Y="0" Z="0" AngleX="0" AngleY="0" AngleZ="0" Unique="0" Timeout="20">\r\n      <Telegram Name="MRU from MotionBinary From Serial Port 2" SensorType="MotionBinary" Type="MRU" SubscriptionPath="MotionBinary From Serial Port 2@MotionBinary.MotionData" Enabled="1">\r\n        <Value Name="Roll" Priority="1" />\r\n        <Value Name="Pitch" Priority="1" />\r\n        <Value Name="Heading" Priority="1" />\r\n        <Value Name="Heave" Priority="1" />\r\n      </Telegram>\r\n    </Sensor>\r\n    <Sensor Name="GPS From Serial Port 1" Type="GPS" Port="Serial Port 1" TalkerID="" X="0" Y="0" Z="0" AngleX="0" AngleY="0" AngleZ="0" Unique="0" Timeout="20">\r\n      <Telegram Name="GLL from GPS From Serial Port 1" SensorType="GPS" Type="GLL" SubscriptionPath="GPS From Serial Port 1@GPS.Geographical.Position" Enabled="1">\r\n        <Value Name="Latitude" Priority="1" />\r\n        <Value Name="Longitude" Priority="1" />\r\n      </Telegram>\r\n      <Telegram Name="GGA from GPS From Serial Port 1" SensorType="GPS" Type="GGA" SubscriptionPath="GPS From Serial Port 1@GPS.Global.Position" Enabled="1">\r\n        <Value Name="Latitude" Priority="2" />\r\n        <Value Name="Longitude" Priority="2" />\r\n      </Telegram>\r\n      <Telegram Name="RMC from GPS From Serial Port 1" SensorType="GPS" Type="RMC" SubscriptionPath="GPS From Serial Port 1@GPS.Specific.PositionSpeedCourse" Enabled="1">\r\n        <Value Name="Latitude" Priority="3" />\r\n        <Value Name="Longitude" Priority="3" />\r\n        <Value Name="Course" Priority="1" />\r\n        <Value Name="Speed" Priority="1" />\r\n      </Telegram>\r\n      <Telegram Name="VTG from GPS From Serial Port 1" SensorType="GPS" Type="VTG" SubscriptionPath="GPS From Serial Port 1@GPS.Ground" Enabled="1">\r\n        <Value Name="Course" Priority="2" />\r\n        <Value Name="CourseNotUsedMagnetic" Priority="1" />\r\n        <Value Name="Speed" Priority="2" />\r\n      </Telegram>\r\n      <Telegram Name="ZDA from GPS From Serial Port 1" SensorType="GPS" Type="ZDA" SubscriptionPath="GPS From Serial Port 1@GPS.TimeInfo" Enabled="1">\r\n        <Value Name="TimeInfo" Priority="1" />\r\n      </Telegram>\r\n    </Sensor>\r\n  </ConfiguredSensors>\r\n</Configuration>'
}

Under "self.parser_obj.config_datagram['configuration']" there is a dictionary of individual transceivers. Should that be the object that is parsed?

---

Note: These exceptions were with both the latest tagged release as well as with the latest version of the dev branch (ep.version of '0.5.5.dev49+g8180da1').

files_with_exceptions.txt A list of files from all cruises with different exceptions.

[gavinmacaulay]

That file looks to be from an EK80, not EK60. That may be the cause of the error.

Which raises the question of how to tell if a .raw file comes from an ek60 or ek80... I have a simple bit of code that does that and can include it here tomorrow (need to find it first....)

[oftfrfbf]

Nice catch @gavinmacaulay!

Somewhat obvious in retrospect:

'application_name': 'EK80'

Some sort of pragmatic method for deriving if a ".raw" files is EK60 or EK80 would be very useful for bulk processing. Please share if you find it.

I am guessing that a lot of data is mislabeled then. I'll follow up with the NOAA data managers to fix that. We aren't immediately concerned with EK80 data so unless I find something contrary this can be closed for now.

[leewujung]

The presence of the XML0 datagram is very telling for distinguishing EK80 from EK60.

@gavinmacaulay : It'll be nice to add in the snippet of check that you have to either warn user of the file type error and proceed to conversion, or raise an error with this info so that users will try again with the right sonar_model parameter.

@oftfrfbf : How about keeping this issue open but changing the title of the PR (and update the first comment) to something along the line of distinguishing the instrumental sources of .raw file?

[gavinmacaulay]

Distinguishing between a .raw file created by the ER60/EK60/ES70 software and the ES80/EK80 software is readily done by looking at the type of the first datagram in the .raw file. If it is 'CON0', it is from the ER60/EK60/ES70 software and if it is 'XML0', it is from the ES80/EK80 software:

import echopype as ep
rawFile = 'abc.raw'

with open(rawFile, "rb") as file:
    headers = file.read(8)

first_datagram = headers[4:].decode()

if first_datagram == 'CON0': # EK/ES60 and ES70 have a CON0 datagram first
    sonar_model = 'EK60'
elif first_datagram == 'XML0': # ES80/EK80 file have a XML0 datagram first
    sonar_model = 'EK80'
else:
    print(f'Unknown first datagram type: {first_datagram}')
    sonar_model = 'unknown'
            
ed = ep.open_raw(rawFile, sonar_model=sonar_model)

This does make me wonder if something like this can be done automatically for all file formats that echopype accepts and then the user wouldn't need to specify the sonar_model...

[gavinmacaulay]

@oftfrfbf: The Falkor Transit data isn't necessarily mislabeled. I might be telling you things you already know, but here goes...

An EK60 echosounder is made up of a transceiver (called a GPT) and software (called the ER60). But the newer EK80 software will happily work with GPT transceivers and that is what it looks like the Falkor had. So the system was an EK60 in hardware, but used the EK80 software to record the data. The EK80 transceiver is called a WBT and EK80 files can contain data from both GPT and WBT transceivers (but see #459).

@leewujung: This rather fluid distinction between Simrad data in EK60 format and EK80 format is perhaps a reason to have echopype automatically work out the file format (e.g., sonar_model='Simrad' perhaps).

[oftfrfbf]

Thanks for the script. And the explanation is very helpful, I am new to a lot of this.