QC data filtering and non-standard product loading

[thomasteisberg]

This PR is intended to address two feature requests:

1. Add xopr QC module after loading echograms and picks #76 - Adding an easy way to filter data by various "QC" criterion such as minimum ice thickness
2. xopr compatability with CSARP_qlook_coh data product #75 - Adding the ability to load data products that are publicly available from the OPR servers but are not (yet) indexed by the STAC catalog

This is a work in progress, but I wanted to share something so that we could get feedback from everyone on it.

The best thing is to look at the two new docs notebooks added. They are called qc_demo.ipynb and nonstandard_data_products.ipynb and should be available in the rendered docs preview shortly.

A few specific questions:

1. What should the "QC" module be called? @espg points out that "QC" doesn't feel quite right because the data that's being filtered out isn't necessarily "bad" data, it just isn't suitable for whatever the users' current purpose. "Filtering" is too overloaded in this context, though, so I also don't want to call it that.
2. What do you (any of you) think of the runner approach for the QC checks? Is this the right balance on modularity and making it easy on the user?
3. Some checks may require more data loaded than others. Right now, we force the full dataset to be loaded before running any QC checks, but you could imagine allowing some checks to run with only layers loaded and potentially avoiding loading the radar data at all if a threshold of "good" data was not met.

And the most important question: For @elizadawson, does this meet the needs of your workflow?

[github-actions]

📚 Documentation Preview Ready!

🌐 Live Preview

URL: https://xopr-pr-79.surge.sh/xopr/

Preview updates automatically with new commits while the preview-docs label is present.

📦 Download Artifact

You can also download the docs for local viewing.

Commit: b4bea15

---

To trigger a preview, add the preview-docs label or use the /preview command.

[thomasteisberg]

Direct links to relevant notebooks to look at:

• https://xopr-pr-79.surge.sh/xopr/qc-demo/
• https://xopr-pr-79.surge.sh/xopr/nonstandard-data-products/

[elizadawson]

The qc_heading_change doesn't seem to be working correctly (at least not for 2008_Antarctica_ BaslerJKB). Heading is in radians (and is for a lot of seasons).... might be the problem

Edit: this actually looks like it's coming from a problem with the heading field in a lot of the data

[thomasteisberg]

@elizadawson I added a notebook (https://xopr-pr-79.surge.sh/xopr/repicking/) showing a couple of ways of doing surface and bed repicking. After looking at this a bit, I'm not sure we need to actually add anything to xOPR. I think it can be expressed pretty cleanly with just a helper function expressing the logic you want (local_max and find_peaks_local as examples in the notebook) and a call to apply_ufunc. Let us know if this matches your use case or not.

[thomasteisberg]

I rebased this off main since there have been a lot of changes there.

@elizadawson How has the QC workflow been going for you? Should we get this merged into main?

[elizadawson]

Yes, merging this branch into main would be great!

[espg]

/preview

[github-actions]

✅ Documentation Preview Ready!

🌐 Live Preview

URL: https://xopr-pr-79.surge.sh/xopr/

Commit: db41c4f

---

Preview deployed via /preview command by @espg

[espg]

This looks good-- some minor comments / suggestions on a few lines and notebook outputs.

Out of scope for this PR, but the repicking notebook could be made cooler if we were to fold in the tool we did for Michael as a way to manually repick the bed...

[espg]

The :::{info} block directive isn't valid and doesn't render properly -- we need either:

tip, warning, important, caution, danger, error, hint, attention, seealso, or the generic admonition (with a user-supplied title). Guessing either {note} or {tip}...

[espg]

The output from the flight track map would be a bit easier to read if we were filling in the ocean like we do for the background map here.

[espg]

claude seems to think that we're calculating amplitude here rather than bed power-- no idea if this is correct, but worth double checking...

snr_bed_pick dB scaling (line 225): data is np.abs(Data) → amplitude, not power. For amplitude the convention is 20·log10(...); for power it's 10·log10(...). Current code labels the intermediate bed_power but feeds amplitude through the 10× formula, so reported SNRs are half of what users probably expect. Worth reconciling (either square first, or change the factor).

[thomasteisberg]

Data is an "Nt by Nx single or double array containing the radar echogram in linear power units" (Echogram File Guide), so I think Claude is wrong here. It's assuming that Data is amplitude but it's actually power.

[espg]

Could be vectorized:

idx = np.clip(np.searchsorted(twtt, bottom_twtt), 0, data.shape[1] - 1)
bed_power = np.where(np.isnan(bottom_twtt), np.nan, data[np.arange(n_traces), idx])

[thomasteisberg]

Good call. Fixed