Harmonize schema for monitoring pipeline outputs

[dlebauer]

From discussion last Friday Apr 10.

• there are distinct parquet styles being produced by different pipelines.
  • monitoring outputs from @sarahkanee
  • irrigation from @ashiklom
  • nitrogen (not yet created) by @divine7022
• we agreed that parquet should be a common format, including an ensemble number even if there is only one member
• N and organic matter application should be per-site and consistent format, despite having minimal spatial structure
• Each output will have distinct covariates, but the basic structure should be consistent to facilitate downstream use (e.g. event.json generation)

Tasks

• Define common structure
• Convert existing outputs to this structure
• Update other pipelines to generate this structure (or this can be deferred and written as a separate ticket)
• Document the structure (in PEcAn docs)

[dlebauer]

a minimal shared schema definition might be to have the set of columns site_id, date, event_type, ensemble_id, [magnitude of covariates 1..n] , where the [] cols are event-type specific. event_type isn't strictly necessary if that can be inferred from the filename or metadata. An alternative would be to put the covariates in a json type column.

[ashiklom]

After thinking about this some, I think I would like to propose the following design:

• The outputs of the monitoring pipeline (CC: @sarahkanee @divine7022) should produce tabular data that matches the PEcAn events.json schema, and should also include an event-specific ensemble ID (e.g., irrigation_member_id, tillage_member_id), even if only one "ensemble member" is produced. So, for example, irrigation.parquet will have columns: site_id, irrigation_member_id, date, amount_mm, method; tillage will have site_id, tillage_member_id, date, tillage_eff_0to1 (and other optional values); etc.

• PEcAn receives an arbitrary number of events.json files as its events ensemble. These might just be named like events_1.json, events_2.json, events_3.json, etc. They are passed to settings.xml via the events$paths$source tag.

• A file events_ensemble.csv captures the relationship between different events_X.json files and their parents in the monitoring pipeline, similar to how the sens_design$X data frame relates PEcAn ensemble IDs to specific members of the IC/met/events/parameter ensembles. It might look something like:

ensemble_d	irrigation_member_id	tillage_member_id	planting_member_id	harvest_member_id
1	"irr_ens_12"	"tillage_ens_1"	"plant_harvest_ens_3"	"plant_harvest_ens_3"
2	"irr_ens_5"	"tillage_ens_2"	"plant_harvest_ens_1"	"plant_harvest_ens_1"
3	"irr_ens_3"	"tillage_ens_2"	"plant_harvest_ens_2"	"plant_harvest_ens_2"

Note that in this example, planting and harvest uncertainties are coordinated through shared member_id values. This need not be the case in practice; it's just for demonstration. Alternatively, it may be the case that irrigation ensembles and planting/harvest ensembles originate from the same parent phenology ensembles. That can be accommodated by having upstream code that links irrigation_member_id, planting_member_id, and harvest_member_id together. How you get to this data frame becomes irrelevant --- the point is that this data frame is capable of implicitly representing all kinds of complex covariance / dependency structures. You can even link ensemble members manually if you want, by creating the CSV file by hand!

[infotroph]

A couple requests that I composed earlier and apparently never posted:

• I request we use the same ID name for all event types: Some current files use parcel_id and others site_id. I like parcel_id because once they're pulled into the PEcAn pipeline site_id can mean something different (e.g. during validation we might simulate multiple plots from the same parcel).
• Weak vote against an event_type column when the info is already present in the filename.

[dlebauer]

@ashiklom
I think that this is flexible enough to allow separation of timing and event attribute uncertainty, which is not necessary in the near term but I wanted to make sure this design could support it. Since it is a separate topic, I wrote it up in a discussion. https://github.com/orgs/ccmmf/discussions/233

@infotroph

I like parcel_id because once they're pulled into the PEcAn pipeline site_id can mean something different

If the goal is to have a way to distinguish subunits within a larger site or field with its own identifier, then plot_id seems more intuitive and specific than parcel_id.

In BETYdb, we've just used one table for all geospatial attributes and call it site. Hierarchy can be represented explicitly via sitegroups and implicitly via polygon containment, so I think a new term is only worth introducing if we can define it clearly and show what additional value it provides.

[divine7022]

catching up on this thread since #3977 just landed. one thing worth surfacing for the architecture discussion when @ashiklom wrote

it may be the case that irrigation ensembles and planting/harvest ensembles originate from the same parent phenology ensembles. That can be accommodated by having upstream code that links irrigation_member_id, planting_member_id, and harvest_member_id together

just wanted to flag a piece of existing pecan machinery for future direction discussion, generate_joint_ensemble_design reads parent field from settings$ensemble$samplingspace and threads the parent's sample ids through input.ens.gen.It already handles "input B follows input A's sample ordering" declaratively in pecan.xml

so in pecan.xml you can already write something like:

<samplingspace>
  <phenology><method>sampling</method></phenology>
  <irrigation><method>sampling</method><parent>phenology</parent></irrigation>
  <planting><method>sampling</method><parent>phenology</parent></planting>
</samplingspace>

where I am less sure is how that mechanism would compose with new manifest csv. pecan's parent declarations operate per input tag, and right now <events> is one input tag with per event type structure (irrigation, planting, tillage, ...) bundled inside each events.json; so parent mechanism could coordinate events with met or poolinitcond (already separate tags), but expressing "irrigation_member_id depends on phenology_member_id" via <parent> would mean splitting events into per even type tags, which is a bigger restructure than manifest CSV approach in #3977

could be worth thinking about long term whether to lift per event type structure up to the <input> level so parent declarations apply natively, or whether manifest CSV stays the right home for those cross event correlations.
probably its own follow up issue rather than near term thing