Test presence of a metadata SID in publish_update

[dmullen17]

When a package has an SID attached to a metadata object publish_update might fail.

This seems to happen whether the user specifies the SID or the version (pid) as the metadata_pid argument. It's possible this isn't actually an issue and was due to slow indexing. But it's worth creating a dummy SID to test this, and determining the correct workflow for updating a package with an SID if there is no bug.

[amoeba]

Interesting. SIDs are less well-tested territory.

publish_update might fail.

How does it fail?

[jagoldstein]

That is the question... @dmullen17 will try to reproduce this failure using a UUID as a SID on test.arcticdata.io.

It may fail by not aggregating the EML with the rest of the package.
OR it may appear to fail silently on the front end by simply not updating.
OR it may take longer than usual to update (or index) a package with a SID (4 to 16 hours) and we just think it failed after the first few hours of waiting and refreshing the landing pages.
OR something else...

[dmullen17]

It disassociates the metadata from the resource map + data objects. The way Sharis and I tried it was calling publish_update on a parent with an SID and using the version pid for the metadata_pid argument. It seems like this failed in both cases, although we haven't ruled out slow indexing (it took 4+ hours for both of ours).

Let me see if I can come up with an MRE on test to rule out slow indexing

[dmullen17]

@amoeba @jagoldstein
Here's the first version of the package with a SID: https://test.arcticdata.io/#view/urn:uuid:385710b2-4cbe-43b4-b83a-9d1e6c21d7bd

Second version after publish_update (not working as of this comment - no resource map, data pids, or children present): https://test.arcticdata.io/#view/urn:uuid:8c96da2e-b2c5-42f7-a4ae-a85eefd0b2bb

Hopefully this will resolve by tomorrow and we can chalk it up to an indexing issue.

Here's the MRE:

## MRE publish_update SID Issue 

## Set test token

## Load nodes
cn <- CNode('PROD')
mn_prod <- getMNode(cn,'urn:node:ARCTIC')
cnTest <- CNode('STAGING')
mn_test <- getMNode(cnTest,'urn:node:mnTestARCTIC')

## Load working version of clone_package
devtools::install_github("dmullen17/datamgmt@clone_package")
library(datamgmt)

pkg <- clone_package("resource_map_urn:uuid:8c4cb5f6-9b11-4975-9668-c875dc4bbc2a",
                     from = mn_prod,
                     to = mn_test,
                     clone_child_packages = TRUE)

## Add SID
sid <-generateIdentifier(mn_test)
s1 <- getSystemMetadata(mn_test, pkg$metadata)
s1@seriesId <- sid
updateSystemMetadata(mn_test, pkg$metadata, s1)

## Check that no pids changed 
pkg_with_sid <- get_package(mn_test, pkg$resource_map)
expect_true(all(unlist(pkg) %in% unlist(pkg_with_sid)))

## Update metadata 
eml <- read_eml(rawToChar(getObject(mn_test, pkg$metadata)))
eml@dataset@title <- c(new("title", .Data = "Updated package with SID"))
eml_path <- file.path(tempdir(), "science_metadata.xml")
write_eml(eml, eml_path)

update <- publish_update(mn_test,
                         pkg$metadata,
                         pkg$resource_map,
                         metadata_path = eml_path,
                         child_pids = pkg$child_packages)

[amoeba]

Harumph, I didn't update this Issue when I did some research a while back. We do have a bug related to SIDs somewhere in here. I'll look for and link in relevant tickets/notes.

[amoeba]

Found some SID-related bugs:

https://redmine.dataone.org/issues/8520: Causes metadata objects with SIDs to not get indexed sometimes. A dev can fix this after-the-fact but it does require an intervention.
https://redmine.dataone.org/issues/8536
https://redmine.dataone.org/issues/8537

It looks like SIDs are pretty touchy right now.

[jagoldstein]

Yes, the SIDs have been pretty touchy and we are reluctant to suggest or apply them as is.

We have a publishing work-around for SIDs right now, but using it does not instill a lot of confidence in me, thus we are only using SIDs sparingly (when contributors really push us for one or if it seem like it's the only good solution). That is why I have applied my new label "blocker" here.

[amoeba]

I'll bring this up on this weeks dev call to see where we need to go to get SIDs working.

[jagoldstein]

thanks @amoeba

[amoeba]

We talked about this today on our weekly dev call, which some DataONE staff joined for another reason. Dave at DataONE will take a look at this and other SID issues and we'll work from there.