Compatibility with DAS standards/groups

[d-chambers]

We received some good feedback from a recent DAS workshop at Image2022. The presentation can be accessed here.

One theme was that we need to be aware of other DAS standards/projects and not just doing our own thing. Of course, I agree. However, each standard/effort needs to be evaluated in light of DASDAE goals. Some may not be relevant, or best supported through implementing additional format readers or data conversion layers.

OSDU

One comment was that we need to be aware of the Open Group's Open Subsurface Data Universe. I spent about an hour going through their website. I was disappointed to see that their library of standards and some gitlab content require registration to view (not as open as one might hope), but alas registration is free (I think?) so I signed up for an account. I am still working on wrapping my head around all they have going on, but from what I can tell from the vision statement they are focused on building an open-platform/market place for energy sector code/data to be run in the cloud.

I then went poking through the OSDU gitlab repos and it looks like they are mostly focused on creating and interacting with their platform. The platform is composed of several microservices, each containerized and communicating through the specified protocols.

In summary, it looks like OSDU has some very innovative work going on, but I didn't find any applicable DAS standards or libraries. I could see DASCore being used in a OSDU microservice in the future, but apart from this I am not sure how DASCore and OSDU can "synergize" as they seem to be fundamentally different things. Perhaps someone more familiar with OSDU might have better ideas on this?

IRIS

IRIS' DAS Research Coordination Network has been working on some guidelines/recommendations for DAS metadata and data structures. Although I have seen a preliminary copy of this report, I am not sure if a final, or at least public and versioned, form exists yet. From what I can tell, we are in a strong position to make DASCore's data-structure, metadata, etc. mostly conform with this standard but we do need a public document to reference (pinging @jinwar and @eileenrmartin).

Others??

Does anyone else know of other DAS standards committees or major efforts we should discus here?

[eileenrmartin]

Andreas Wuestefeld just emailed me to let me know he's put the DAS RCN White Paper's format in a simple form here: https://github.com/andreas-wuestefeld/IRIS_DASformat It's possible he'll move this over to the DAS-RCN organization and the current link will break.

He's on board with us reusing any of his code to create a reader/writer in DASDAE/DASCore. We'll do this as version 1, assuming there will likely be changes following the global DAS recording month in February.

[d-chambers]

So what are the next steps for implementing the format? Did you find anything about DASCore's current data structures incompatible?

I will try to look through the White Paper when I have more time (probably next week).

[eileenrmartin]

I think everything shoudl be compatible once we add an inventory. Also, new links for reference on this format: https://github.com/DAS-RCN/DAS_metadata and https://github.com/DAS-RCN/IRIS_DASformat

[d-chambers]

I am looking over this and there are potentially a few issues.

The main one is that the document states "only strain or strainrate data are acceptable". This is problematic for storing Terra15 data in velocity format, or other types of non-DAS fiber measurements. Its not a big deal to raise an exception if the data don't comply when we call the write method for this format but I would hate to see IRIS not support storing velocity data as a rule.

[eileenrmartin]

I think we should continue supporting the velocity format to maintain flexibility. I'm okay with DASCore being capable of supporting a wider variety of data than the standards since it will ensure more people are able to get started with their DAS data quickly.

[eileenrmartin]

Here's a quick overview of the plan for cable inventories (being pursued on the addinventories branch).

We'll have an inventory of key/value pairs that will contain experiment objects and their IDs, cable_type objects and their IDs (would refer to manufacturer properties of cables, not our installation), instrument objects and their IDs, cable_segment objects and their IDs (specific to installations), splice objects and their IDs, tap_test objects and their IDs.

Each patch of data will have an attribute that is an experiment_id (which you can use in the inventory to look up the associated experiment object).

The experiment, cable_segment, splice, cable_type, instrument, and tap_test objects will all be subclasses of the general_component class. The general_component class will have two attributes: a version (string, will allow versioning of these classes) and a notes (string).

Each experiment object will have an instrument_id (which you can use in the inventory to look up the associated instrument object)), a cable_list that is a list in sequential order of item ID's starting with the interrogator (which is a list of cable_segment_id and splice_id keys that can point back to the associated cable_segment and splice objects through the inventory), a tap_test_id (which can use the inventory to look up associated tap_test object), a start_time (very first start time of the campaign) and an end_time (very last end time of the campaign).

Each instrument object will have an instrument_id, and its attributes will include a manufacturer_name (strring), serial_number (string).

Each cable_segment object will have a cable_segment_id, and it will have attributes: coupling (string, special keywords), length (float), a cable_type_id (which can be used to look up a cable_type in the inventory).

Each splice object (for splices and end points) will have a splice_id, and it will have attributes: type (string, special keywords), db_loss (float, optional).

Each cable_type object will have a cable_type_id, and it will have attributes: manufacturer_name (string), serial_number (string), fiber_type (string, e.g. singlemode or multimode).

Each tap_test object will have a tap_test_id, and it will have a list of labels (e.g. ['t', 'x', 'y', 'z']), a tests array (columns corresponding to the labels in order with rows each representing one tap tests spatial and temporal locations), and a distances array (same size as tests but with NaN values until these are filled in). It's expected these objects would be only filled in with labels and tests initially, then used for visualizing and picking locations, then the distances array would be filled in piece-by-piece in many use cases.