Skip to content

docs: spec the omnigraph 0.13 upgrade and format-14 cutover - #461

Open
feoh wants to merge 3 commits into
mainfrom
docs/omnigraph-0-13-upgrade-spec
Open

feoh wants to merge 3 commits into
mainfrom
docs/omnigraph-0-13-upgrade-spec

Conversation

@feoh

@feoh feoh commented Oct 8, 2026

Copy link
Copy Markdown
Contributor

What are the relevant tickets?

N/A. Upstream context: ModernRelay/omnigraph#882, fixed by ModernRelay/omnigraph#878.

Description (What does it do?)

Adds the spec for moving witan's omnigraph from 0.11.0 to 0.13.0 (storage format 14): docs/internals/design/omnigraph-0-13-upgrade-spec.md.

  • Retargets the project from 0.12.0 to 0.13.0. 0.12.0 can't change policy group membership once graphs exist, which our per-boot membership rendering relies on. 0.13.0 fixes that.
  • Records the data-tier operating-model decisions: automatic lock handoff on restart, direct apply only in the entrypoint with a read-only cluster plan gate, a stop-window maintenance CronJob, and --require-all-graphs left unset.
  • Records the consumer changes (header, DateTime precision, commit ids, error classification, remote schema apply) and the migration-Job changes.
Implementation details
  • Lock handoff (D3): 0.12+ keeps a persisted cluster lock even after a clean shutdown. Recreate only orders pods during a rollout, so the entrypoint takes a Kubernetes Lease, waits until no other live pod matches the selector (Failed/Succeeded pods count as gone, Unknown as live), then force-unlocks. Every unlocker takes the same Lease, because upstream's force-unlock has no conditional delete.
  • Apply (D4): the pre-deploy Job becomes a read-only cluster plan gate. The entrypoint also refuses a plan that removes a graph not on an allow-list, because 0.13 apply deletes a removed graph's storage and history, and the ConfigMap changes before the Job runs.
  • Maintenance (D5): one CronJob scales to 0, optimizes and cleans under the Lease, and restores the recorded replica count from a TERM/EXIT trap. It runs under its own ServiceAccount. The council probe schedule skips the 03:00 UTC hour.
  • Sequencing: the entrypoint rewrite merges with the pin bump, since it breaks on 0.11. The ol-infrastructure changes sit behind a per-environment stack setting, since one Pulumi program serves CI, QA and Production.

How can this be tested?

Docs only. prek passes on the file. Claims marked "verified" were run against the released v0.13.0 binary (linux-x86_64, sha256 0123daa64f0f2db9860284200bd71c79a22a6ad2c6f5501b6fca24cac86a7a30) on local file:// scratch clusters built from mcp/servers/witan/policy. A reviewer can re-check any of them the same way. Nothing was run against S3 or the deployed clusters.

Additional Context

  • Two independent review passes found problems with the lock handoff, maintenance and sequencing designs, and the spec was revised after each. The last round of fixes (live-pod rule, Lease, plan in the entrypoint, probe schedule, per-environment gating) has not had its own independent review, so it's worth the closest look.
  • Still open, and flagged unverified in the spec: which cluster status field reports an outstanding deployment, and how long a schema migration takes against the startup-probe budget.
  • Don't merge the Renovate 0.12.0 PR (chore(deps): update dependency modernrelay/omnigraph to v0.12.0 #459).

Retargets the upgrade from 0.12.0 to 0.13.0, which fixes the policy-membership
freeze (ModernRelay/omnigraph#882). Records the data-tier operating model
decisions (automatic lock handoff in the entrypoint behind a Kubernetes Lease
and a live-pod check, entrypoint-only direct apply with a read-only plan gate,
a stop-window maintenance CronJob, --require-all-graphs left unset) and the
consumer and migration-Job changes. Claims marked verified were run against the
0.13.0 binary on local scratch clusters; the rest cite the 0.12 discovery or
upstream docs and source.
@feoh
feoh requested review from blarghmatey and a balanced review from Copilot October 8, 2026 13:15

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟡 Changes recommended

The boot sequence mishandles the post-apply lock, and coordination races remain unresolved.

5 open findings
What changed in this PR

Adds the implementation specification for upgrading Witan to OmniGraph 0.13 and storage format 14.

Changes:

  • Defines lock handoff, deployment, and maintenance procedures.
  • Documents consumer compatibility and migration changes.
  • Establishes rollout sequencing across environments.
File Description
docs/​internals/​design/​omnigraph-0-13-upgrade-spec.md Specifies the OmniGraph 0.13 cutover and operating model.

🧠 Review effort: Balanced


Give feedback about Copilot approvals in this survey to enter a drawing for a $150 gift card.

Comment thread docs/internals/design/omnigraph-0-13-upgrade-spec.md Outdated
Comment thread docs/internals/design/omnigraph-0-13-upgrade-spec.md Outdated
Comment thread docs/internals/design/omnigraph-0-13-upgrade-spec.md Outdated
Comment thread docs/internals/design/omnigraph-0-13-upgrade-spec.md Outdated
Comment thread docs/internals/design/omnigraph-0-13-upgrade-spec.md Outdated
feoh added 2 commits October 8, 2026 10:11
- D3: Lease holders fail closed on lost renewal and re-confirm the Lease
  before every force-unlock; Pulumi pre-creates the Lease because RBAC
  cannot restrict create by resourceNames.
- D3: the boot sequence re-reads status after apply before releasing the
  new deployment lock.
- D5: maintenance restore is conditional on no armed migration, using a
  resourceVersion-guarded PUT on deployments/scale.
- D6: /readyz is also 200 for an empty applied inventory.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants