Data Gitops in the Product Metadata Repo

[alexrichey]

Sketching out the flow, from our chat with GIS on April 1. Still mentally churning how to make this the easiest for all involved.

Overview

How can we operationalize the product-metadata repo to initiate distribution, in a more gitops-like approach.

We should record the versions (including the current version) in the product metadata repo. PR sketch here.

Flow:

1. GIS user opens a PR with a change to a version (e.g. PLUTO 25v4 -> 26v1). The PR should kick off a dry-run / draft of distribution and post the revision to the Pull Request. Maybe it should also kick off packaging, which would create all assets like pdfs and OTI xlsx's for the modified products/datasets.
2. The PR should kick off a comparison between main and the PR branch to calculate a diff in expected current versions for product.dataset.destinations and posts it to the PR. It also generates all metadata assets (e.g. OTI xlsx's), maybe as a pre-release?
3. They merge the PR to main, which kicks off a diff to determine changed versions.
4. This kicks off distribution.
5. Maybe we run a nightly action to scrape versions from all destinations and compare to our expected version. This is the product-metadata version of nightly QA

E.g. you might open a Pull request to change the current_version in lion/metadata.yml. This calculates a diff for each dataset and destination under LION:

lion.atomic_polygons.socrata:v1 -> lion.atomic_polygons.socrata:v2
lion.rpl.socrata:v1 -> lion.rpl.socrata:v2
...

We kick off a separate distribution job for each of these. And now instead of generating the assets (specifically the OTI xlsx) on the fly, we use what's generated by the release.

This would also make use of the existing override behavior, so if you wanted to pin a version of a dataset or destination, you could do that easily.

[alexrichey]

cc'ing @damonmcc @croswell81 @jackrosacker @caseysmithpgh @UshashiP

[alexrichey]

@damonmcc @croswell81 @jackrosacker @caseysmithpgh @UshashiP this is flow I'm thinking.

TLDR: you'd edit any field in the product-metadata repo, it would generate all assets, create a release noting the bumped versions (e.g. lion.atomic_polygons 26a -> 26b). Then in the DE repo, you could point distribution at the release, or just wait for the nightly sync to update Socrata.

In the event of distribution failures, it would create a GH issue, and just run again the following night to try again.

┌─────────────────────────────────────────────────────────┐
│ User Changes Metadata (version bump, etc.)              │
└──────────────────┬──────────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│ Product-Metadata Repo: Pull Request                     │
├─────────────────────────────────────────────────────────┤
│ • Generate version diff CSV                             │
│ • Generate ALL destination assets (PDF, XLSX, XML)      │
│ • Run validation checks                                 │
│ • Attach artifacts to PR                                │
└──────────────────┬──────────────────────────────────────┘
                   │ (Review + Approve)
                   ▼
┌─────────────────────────────────────────────────────────┐
│ Product-Metadata Repo: Merge to Main                    │
├─────────────────────────────────────────────────────────┤
│ • Create GitHub Release (tag: YYYY-MM-DD-HHmmss)        │
│ • Package ALL assets into metadata-assets.zip           │
│ • Attach to release                                     │
└──────────────────┬──────────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│ Data-Engineering Repo: Distribution                     │
├─────────────────────────────────────────────────────────┤
│ • Download specified release (default: latest)          │
│ • Extract pre-generated assets                          │
│ • Distribute to destinations (Socrata, for now)         │
└──────────────────┬──────────────────────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────────────────────┐
│ Data-Engineering Repo: Nightly Version Sync             │
├─────────────────────────────────────────────────────────┤
│ • Check Socrata destinations vs metadata versions       │
│ • Run distribution                                      │
│ • Create issues for failures                            │
└─────────────────────────────────────────────────────────┘

[damonmcc]

thanks for this! definitely have ideas and feedback on the details and diagram but the first big one that comes to mind is thinking about how the edm-publishing bucket could be useful to

• indicate if a version is even ready for distribution (aka passed QA)
• be one of the triggers/inputs to that first step "User Changes Metadata (version bump, etc.)"
  • but maybe it's better to start with metadata update as the trigger for the entire data update!

another one is related to cadence. definitely seems good to plan for ad-hoc/daily distribution, but I don't know if we want to do a full repo releases at that same frequency. so maybe

1. the pre-release reflects what's on main, so every merged PR updates it. distribution is driven by the pre-release (if it exists)
2. we do a monthly release as a snapshot

[damonmcc]

whiteboarding pic