Add functions for broadband TS spectrum by LOCEANlloydizard · Pull Request #1664 · echostack-org/echopype

LOCEANlloydizard · 2026-05-17T05:08:03Z

Implement a CRIMAC-equivalent broadband TS(f) processing pipeline, including pulse compression, matched-filter autocorrelation, normalized spectra, frequency-dependent calibration, and beam compensation (Test broadband processing against CRIMAC outputs #1033)
Add public compute_TS_spectrum() and compute_Sp() APIs, and extend the calibration framework to support frequency-dependent TS(f) products (Add broadband Sv and TS code #1034)
Update compute_TS() for EK80 complex data to return a beam-compensated band-averaged TS product, while keeping compute_TS_spectrum() as the frequency-dependent TS(f) implementation (Add broadband Sv and TS code #1034, discussion from Add variable Sv attributes to output dataset from compute_Sv #582)
Return TS(f) as an xarray Dataset indexed by (channel, target_id, frequency) (Add broadband Sv and TS code #1034)
Deprecate the public "BB" waveform mode alias in favor of "FM" while preserving backward compatibility (Change waveform_mode default vocabulary to "FM" for broadband EK80 data #1651)

- Add public `compute_Sv_f()` and `compute_TS_f()` for the API - Add `_cal_complex_samples_f()` prototype - Expose `frequency_resolution` and `range_step` parameters for future spectral calibration - Return prototype `Sv_f` xarray output structure: `(channel, ping_time, svf_range, frequency)`

Use lazy method getattr in calibration dispatcher to avoid access to unsupported compute_Sv_f / compute_TS_f methods on non-FM calibrators

codecov-commenter · 2026-05-17T13:55:45Z

Codecov Report

❌ Patch coverage is 55.00000% with 18 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.56%. Comparing base (6cf6cee) to head (c6886e3).
⚠️ Report is 45 commits behind head on main.

Files with missing lines	Patch %	Lines
echopype/calibrate/calibrate_ek.py	35.29%	11 Missing ⚠️
echopype/calibrate/api.py	66.66%	7 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #1664      +/-   ##
==========================================
- Coverage   85.58%   85.56%   -0.02%     
==========================================
  Files          79       78       -1     
  Lines        6998     7074      +76     
==========================================
+ Hits         5989     6053      +64     
- Misses       1009     1021      +12

Flag	Coverage Δ
integration	`80.81% <55.00%> (+0.17%)`	⬆️
unit	`60.22% <55.00%> (-0.20%)`	⬇️
unittests	`85.46% <55.00%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

leewujung · 2026-05-24T22:14:57Z

Hey @LOCEANlloydizard, I thought of a few other params after our discussion the other day so thought I'd lay them out and see what you think. Feel free to ping for more discussions!

def compute_Sv_spectrum(
    self,
    NFFT: int,
    range_step: float = 0.5,
    window: str = "hann",  # TODO: should this be default to None?
    frequency_resolution: Optional[float] = None,
) -> xr.Dataset:
    """Compute broadband EK80 Sv spectrum, or Sv(f).

    Parameters
    ----------
    NFFT : int
        Number of points in the FFT window for computing spectrum.
    range_step : float, default 0.5
        Range spacing in meters for the output Sv(f) window centres.
    window : str, default "hann"
        Type of window function to apply to each FFT segment.
        Options include "hann", "hamming", etc.
    frequency_resolution : optional[float], default None
        Frequency spacing in Hz for the output spectral grid.
        When this is provided, the output Sv(f) will be interpolated to this frequency grid.
        If None, the output Sv(f) will be on the original FFT frequency grid.

    Returns
    -------
    xr.Dataset
        A Dataset containing frequency-dependent volume backscattering strength.
    """


def compute_TS_spectrum(
    self,
    point_locations: xr.DataSet,
    NFFT: int,
    split_front: float = 0.25,
    window: str = "hann",
    frequency_resolution: Optional[float] = None,
) -> xr.Dataset:
    """Compute broadband EK80 Sv spectrum, or Sv(f).

    Parameters
    ----------
    point_locations : xr.DataSet
        Locations of the points for which to compute the spectrum.
        This is required to be an xarray.DataSet with dimensions ("target_id") and
        data variables "ping_time" and "depth" for each ``target_id``.
    NFFT : int
        Number of points in the FFT window for computing spectrum.
    split_front : float, default 0.25
        Each echo spectrum is computed from a segment of the complex echo signal.
        This parameter specifies how to identify the segment around the specified point location.
        For example, if split_front=0.25, then 25% the NFFT length will be before the point location,
        and the remaining 75% will be after the point location.
    window : str, default "hann"
        Type of window function to apply to each FFT segment.
        Options include "hann", "hamming", etc.
    frequency_resolution : optional[float], default None
        Frequency spacing in Hz for the output spectral grid.
        When this is provided, the output Sv(f) will be interpolated to this frequency grid.
        If None, the output Sv(f) will be on the original FFT frequency grid.

    Returns
    -------
    xr.Dataset
        A Dataset containing frequency-dependent point backscattering strength.
    """

- Add compute_TS_spectrum() for EK80 FM complex data - Add broadband TS spectrum processing pipeline and calibration helpers - Add CRIMAC reference dataset and validation tests for TS(f) - Add frequency-dependent absorption validation against CRIMAC - Add broadband Sp and Sv(f) integration tests - Add pooch support for ts_spectrum_example_data bundle - Refactor TS/Sp calibration workflow and API exports - Add cached test-data handling to avoid unnecessary downloads

adding tqdm, most likely temporarily

try to fix error occuring only in ubuntu job

change the test to a NotImplementedError test

- Remove tqdm import and use - Deprecate BB waveform_mode alias (keep backward compatibility) - Change test_fm_equals_bb test to mark "BB" it as deprecated (others would need it too)

LOCEANlloydizard · 2026-07-18T18:15:56Z

also look if we can close #629

leewujung · 2026-07-31T03:34:22Z

+            stacklevel=2,
+        )
+
+    waveform_mode = "BB" if waveform_mode in ("FM", "BB") else waveform_mode


This basically makes BB the default instead of FM. I think it'll be better to just wire everything to be based on FM right now, so that when we think it's time to remove support for the BB syntax, we just need to remove this line and the warning.

leewujung · 2026-07-31T03:45:47Z

+
+        if compute_method is None:
+            raise ValueError(
+                f"{cal_type} calibration is not supported for " f"{echodata.sonar_model} data."


Suggested change

f"{cal_type} calibration is not supported for " f"{echodata.sonar_model} data."

f"{cal_type} computation is not supported for " f"{echodata.sonar_model} data."

leewujung · 2026-07-31T03:46:23Z

+            "Sp": "compute_Sp",
+            "TS": "compute_TS",
+            "Sv": "compute_Sv",
+            # add Sp_spectrum??


Yes I think there should be compute_Sp_spectrum and compute_Sv_spectrum.

leewujung · 2026-07-31T04:00:03Z

+                        if "filter_time" in cal_ds_iteration:
+                            cal_ds_iteration = cal_ds_iteration.drop_vars("filter_time")


What are the cases that cal_ds_iteration would not have filter_time? I have forgotten since I review this section last time, but is it from dataset where there is only 1 filter_time?

Maybe this is related to #1653?

leewujung · 2026-07-31T04:04:43Z

+        if "range_sample" in ds:
+            ds["range_sample"].attrs = {"long_name": "Along-range sample number, base 0"}
+
+        if "echo_range" in ds:
+            ds["echo_range"].attrs = {
+                "long_name": "Range distance",
+                "units": "m",
+            }
+
+        if "frequency" in ds:
+            ds["frequency"].attrs = {
+                "long_name": "Frequency",
+                "units": "Hz",
+            }
+
        ds[cal_type].attrs = {
            "long_name": {
-                "Sv": "Volume backscattering strength (Sv re 1 m-1)",
+                "Sp": "Point scattering strength (Sp re 1 m^2)",
                "TS": "Target strength (TS re 1 m^2)",
+                "Sv": "Volume backscattering strength (Sv re 1 m-1)",
+                "TS_spectrum": "Frequency-dependent target strength spectrum (TS(f) re 1 m^2)",
            }[cal_type],
            "units": "dB",
        }


I think it'll be nice for us to have a consolidated place to specify all the attributes for variables. Right now we define these in an ad-hoc way and missed many of them. It seems would be a good idea to just have a YAML or similar to list the corresponding long_name, units, and other attributes for specific variables, and in functions that generates organized datasets, always pass the dataset through a function that calls the YAML to add attributes -- so like a decorator? I'll add an issue for this.

tracked in #1747.

leewujung · 2026-07-31T04:14:02Z

+    For CW data, Sp is computed from received power samples on the range grid.
+    For EK80 broadband/FM complex data, Sp is computed after pulse compression
+    and represents a band-averaged point-scattering-strength echogram.


I am not exactly sure what you mean between "on the range grid" vs "point-scattering-strength echogram". Aren't both still on the range grid, either it's the original one or the one after pulse compression?

LOCEANlloydizard added 2 commits May 16, 2026 22:06

Fix calibration dispatcher for unsupported FM methods

c6886e3

Use lazy method getattr in calibration dispatcher to avoid access to unsupported compute_Sv_f / compute_TS_f methods on non-FM calibrators

Merge remote-tracking branch 'upstream/main' into broadband_processing

38a012e

LOCEANlloydizard changed the title ~~Add prototype API for FM EK80~~ Add functions for broadband Sv and TS spectrum Jun 6, 2026

LOCEANlloydizard added 5 commits June 6, 2026 18:58

add tqdm

5f05da1

adding tqdm, most likely temporarily

Update conftest.py

fe4adec

change org name

47ad17d

Update conftest.py

b2ff5a2

try to fix error occuring only in ubuntu job

LOCEANlloydizard changed the title ~~Add functions for broadband Sv and TS spectrum~~ Add functions for broadband TS spectrum Jun 8, 2026

LOCEANlloydizard added 16 commits June 8, 2026 20:45

Fix formatting in test_calibrate_ek80.py

31d401b

removing _get_splitbeam_angles and _get_transducer_halves

c8c657a

Merge remote-tracking branch 'upstream/main' into broadband_processing

263f78b

remove Sv(f) API and refactor TS(f) methds

2b9db6b

removing Sv_f tests

7e845bc

append cal_ds_iteration

7fe30d4

Update api.py

68af1bb

Update test_calibrate_ek80_broadband_crimac.py

c905d3e

change the test to a NotImplementedError test

Merge branch 'main' into broadband_processing

c51e936

restore conftest.py flattening, after fixing github asset zip file

78926cc

Update conftest.py

8bc448b

add beam correction to band-averaged TS

f386382

Update calibrate_ek.py

33ae025

Remove tqdm and deprecate BB waveform_mode alias

526f36f

- Remove tqdm import and use - Deprecate BB waveform_mode alias (keep backward compatibility) - Change test_fm_equals_bb test to mark "BB" it as deprecated (others would need it too)

add test for different windows from scipy

7f88c4c

add changes and test for split front

5b18b2c

LOCEANlloydizard closed this Jun 19, 2026

LOCEANlloydizard reopened this Jun 19, 2026

LOCEANlloydizard closed this Jun 19, 2026

LOCEANlloydizard reopened this Jun 19, 2026

LOCEANlloydizard added 2 commits June 19, 2026 13:08

Merge branch 'main' into broadband_processing

b058ac3

adjust the TS tests

78da373

LOCEANlloydizard mentioned this pull request Jun 19, 2026

Add FM single target detection #1688

Open

2 tasks

LOCEANlloydizard added 5 commits June 19, 2026 17:04

make an internal helper for the extractor

884567c

Update test_calibrate_ek80_broadband_crimac.py

509e709

Merge remote-tracking branch 'upstream/main' into broadband_processing

be3ed86

add beam compensated gain against crimac

cb6c9e3

Merge branch 'main' into broadband_processing

d5a71bb

leewujung reviewed Jul 31, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Add functions for broadband TS spectrum - #1664

Add functions for broadband TS spectrum#1664
LOCEANlloydizard wants to merge 31 commits into
echostack-org:mainfrom
LOCEANlloydizard:broadband_processing

LOCEANlloydizard commented May 17, 2026 •

edited

Loading

Uh oh!

codecov-commenter commented May 17, 2026 •

edited

Loading

Uh oh!

leewujung commented May 24, 2026

Uh oh!

LOCEANlloydizard commented Jul 18, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

leewujung Jul 31, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

	f"{cal_type} calibration is not supported for " f"{echodata.sonar_model} data."
	f"{cal_type} computation is not supported for " f"{echodata.sonar_model} data."

		if "filter_time" in cal_ds_iteration:
		cal_ds_iteration = cal_ds_iteration.drop_vars("filter_time")

Uh oh!

Conversation

LOCEANlloydizard commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented May 17, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

leewujung commented May 24, 2026

Uh oh!

LOCEANlloydizard commented Jul 18, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

LOCEANlloydizard commented May 17, 2026 •

edited

Loading

codecov-commenter commented May 17, 2026 •

edited

Loading