DAP4 Serialization Patch

[ndp-opendap]

We need to fix the fix for the dap4 serialization change.

There will be several sub tickets to this issue as we need to make changes in:

• libdap4 - code and baselines
• bes - code and baselines
• hyrax_regression_tests - baselines

The Change

• Revert/rewind the current value of the xml attribute dmrVersion from 2.0 to 1.0. Since this value is set and produced by software in libdap and the bes we must make changes there.
• Add a new attribute to the Dataset element in the DMR: dap:serialization;="4.0" which indicates the server will serialize the DAP4 data response according to the DAP4 serialization scheme.
• Fix a lot of baselines. Since this is a change in the DMR document transmitted both as metadata and in the data response we can expect a widespread testing impacts.

Now that we have discovered and fixed an order of serialization bug we need to provide a mechanism for clients to identify the servers that are "patched".

Examples

XML attribute and value

dap:serialization="4.0" 

Dataset element

<Dataset 
    xmlns="http://xml.opendap.org/ns/DAP/4.0#" 
    xml:base="http://test.opendap.org:8080/opendap/NSIDC/ATL01_20201221111151_04882305_002_01.h5.dmr" 
    dapVersion="4.0" 
    dmrVersion="1.0" 
    dap:serialization="4.0" 
    name="ATL01_20201221111151_04882305_002_01.h5">

Questions

• Should we include the dap:serialization attribute with every DMR, or only with the DMR in the DAP4 data response?

@Mikejmnez , @jgallagher59701 , @dh-opendap , @hannahilea

[ndp-opendap]

OK, as I make this change I am coming to regret my choice of names for the serialization flag

Proposed:

• Add dap:serialization="4.0" to only the DAP4 Data response's DMR section (aka chunk 0).

Issues:

• Using a namespace dap requires that we define it. And I have come to learn that we don't (I thought we did). So that means we have to add xmlns:dap="http://xml.opendap.org/ns/DAP/4.0#" as well. Which is sort of fine, but feels wrong.
• I think that the semantics of the name dap:serialization seem mismatched with the issue, which manifested as a hyrax issue (caused by libdap). Maybe the right choice here is to drop dap:serialization and move to a non-namespace attribute named hyrax_dap="4.0" or something to that effect.

What say ye?

@jgallagher59701 @Mikejmnez @hannahilea @dh-opendap

[Mikejmnez]

With pydap, correctly deserializing a dap4 response from Hyrax (when accessing data using a new API approach I am trying to push people to use) relies on

a) identifying that the response comes from a Hyrax Server
b) checking the version of the remote server

And so correctly deserializing a dap response from Hyrax depends less on the flag/attribute itself, and more on the identification of an attribute that is unique to Hyrax. That is because this is a Hyrax bug. (It does not depend on the attribute itself because we found some issues with stale dmr++ with old attribute values that were being inherited into the dap response )

Currently, there is no "unique" attribute to Hyrax, and pydap instead identifies a pair of attributes uniquely included by the TDS that are not included in the Hyrax responses. That is a fragile situation but works for now since TDS and Hyrax are the only dap4 servers (for now).

Summary

Having an attribute that unmistakably identifies a dap response (in the wild) from a Hyrax data server is a must, and it does not matter what this is. It could be as simple as including the server version/details similar to what a DMR++ does

[ndp-opendap]

I claim that this change to the response headers solves this issue and does so without borking the DAP4 response content.

[ndp-opendap]

☝️ @hannahilea ☝️ @jgallagher59701 ☝️ @Mikejmnez ☝️ @dh-opendap ☝️

[Mikejmnez]

I also agree that it solves the issues with the client, except in one scenario that may not be relevant. See my comment OPENDAP/libdap4#353 (comment)

[ndp-opendap]

Good point @Mikejmnez - seems like more debate is in order. I think that adding the server version as a DAP4 Attribute to the DMR would work. And I think it's much better than some quirky non-dap4 xml attribute getting added to the Dataset element.