feat(omnigraph): hand the cluster lock over in the server entrypoint

.github/workflows/witan-tests.yml

@@ -21,6 +21,11 @@ on:
       # agreement — so a Dockerfile-only bump has to trigger this workflow.
       - "docker/omnigraph-server.Dockerfile"
       - "docker/witan.Dockerfile"
+      # The data-tier entrypoint, maintenance window and plan gate live
+      # beside their Dockerfile, and their tests
+      # (tests/test_omnigraph_server_entrypoint.py, test_omnigraph_maintenance.py)
+      # run in test-witan.
+      - "docker/omnigraph_*.py"
       - "justfile"
       - "bin/**"
   pull_request:
@@ -37,6 +42,11 @@ on:
       # agreement — so a Dockerfile-only bump has to trigger this workflow.
       - "docker/omnigraph-server.Dockerfile"
       - "docker/witan.Dockerfile"
+      # The data-tier entrypoint, maintenance window and plan gate live
+      # beside their Dockerfile, and their tests
+      # (tests/test_omnigraph_server_entrypoint.py, test_omnigraph_maintenance.py)
+      # run in test-witan.
+      - "docker/omnigraph_*.py"
       - "justfile"
       - "bin/**"
 

.gitignore

@@ -26,7 +26,7 @@ build/
 *.egg-info/
 # omnigraph binary downloaded by hatch_build.py — not committed
 _bin/
-# omnigraph cluster state (populated by `omnigraph cluster import`)
+# omnigraph cluster state (populated by `omnigraph cluster apply`)
 __cluster/
 # The witan UI bundle, written by `npm run build` in mcp/servers/witan/ui/.
 # A build output, so not committed — hatch's `artifacts` setting in that

docker/README.md

@@ -119,22 +119,89 @@ stub in tests). See `witan_code/github_app.py`.
   stack mounts its generated `cluster.yaml` alongside it via a ConfigMap
   `subPath` (single-file overlay) so it does not shadow the baked-in schema.
 - **Auto-bootstraps the cluster on start**
-  ([`omnigraph-server-entrypoint.sh`](./omnigraph-server-entrypoint.sh)):
-  `omnigraph cluster import` (first boot) or `cluster refresh` (state already
-  exists in the storage backend) followed by `cluster apply`, then `exec`s the
-  server. This is the remote analogue of witan's local `_ensure_graph`
-  auto-bootstrap — it is what creates the cluster's declared graph(s) under the
-  S3 storage root and applies their schemas, so `Pulumi up` alone produces a
-  *queryable* graph with no manual runbook step. (Graph names are set in the
-  deployment's `cluster.yaml`, not in this image; the convention is to name
-  them after the owning package — `council`, `code` — overridable per env.) The sequence is idempotent: `apply` is a
-  no-op once converged, and the import→refresh split keeps pod restarts (whose
-  local `__cluster/` working state is ephemeral) reconciling with the existing
-  storage-backed ledger instead of erroring.
+  ([`omnigraph_server_entrypoint.py`](./omnigraph_server_entrypoint.py)):
+  renders policy group membership, runs `omnigraph cluster apply`, then
+  `exec`s the server. This is the remote analogue of witan's local
+  `_ensure_graph` auto-bootstrap — it is what creates the cluster's declared
+  graph(s) under the S3 storage root and applies their schemas, so `Pulumi up`
+  alone produces a *queryable* graph with no manual runbook step. (Graph names
+  are set in the deployment's `cluster.yaml`, not in this image; the
+  convention is to name them after the owning package — `council`, `code` —
+  overridable per env.) A direct apply on an empty root creates the ledger,
+  and on a converged root it is a no-op.
+- **Hands the cluster lock over on every restart.** From omnigraph 0.12 on,
+  the server, `cluster apply`, `optimize` and `cleanup` each leave an
+  exclusive lock in the storage root when they exit, and the next writer is
+  refused until it is force-unlocked. The entrypoint unlocks the previous
+  server's `serve` lock and its own apply's `deployment` lock, and only after
+  it holds the unlock Lease and the Kubernetes API shows no other live server
+  pod. It refuses to start (exit 1) on a `graph_operation` lock, on an
+  interrupted apply, and on a `cluster.yaml` that drops a graph (0.13 deletes
+  a dropped graph's storage). The module docstring has the sequence;
+  [the 0.13 upgrade spec](../docs/internals/design/omnigraph-0-13-upgrade-spec.md)
+  (D3, D4) has the reasoning.
+- **Carries the two commands that run beside the server**, for the
+  `omnigraph` stack to use as a Job's `command`:
+  - [`omnigraph_plan_gate.py`](./omnigraph_plan_gate.py) runs the
+    entrypoint's plan check and nothing else. As a pre-deploy Job it fails a
+    deploy on a schema or policy change the next boot would refuse, while the
+    old pod is still serving. It reads the entrypoint's
+    `OMNIGRAPH_SERVER_BEARER_TOKENS_FILE`, `OMNIGRAPH_APPLY_ACTOR` and
+    `OMNIGRAPH_GRAPH_REMOVAL_ALLOWLIST`, and needs no Kubernetes access.
+  - [`omnigraph_maintenance.py`](./omnigraph_maintenance.py) is the nightly
+    maintenance window. `optimize` and `cleanup` are refused while the server
+    holds its lock, so it scales the server Deployment to 0, takes over the
+    `serve` lock, maintains each graph in `cluster.yaml` (every `optimize`
+    before any `cleanup`, whose pass starts at a different graph each week),
+    and restores the
+    replica count. It also restores on SIGTERM. The module docstring has the
+    cases where it leaves the server down on purpose.
 - `ENTRYPOINT` is the bootstrap wrapper; the default `CMD`
   (`--cluster /etc/omnigraph/cluster --bind 0.0.0.0:8080`) matches the stack's
   `args`.
 
+The entrypoint reads these from the environment:
+
+| Variable | Required | Purpose |
+|---|---|---|
+| `OMNIGRAPH_SERVER_BEARER_TOKENS_FILE` | with policy bundles | Actor-token map the group membership is rendered from (also read by the server). |
+| `OMNIGRAPH_APPLY_ACTOR` | with policy bundles | Actor passed to `cluster plan`/`apply` as `--as`. With policy installed, omnigraph refuses a schema change that names no actor. |
+| `OMNIGRAPH_UNLOCK_LEASE` | in Kubernetes | Name of the pre-created `coordination.k8s.io` Lease every force-unlocker takes. The ServiceAccount needs `get` and `update` on it. |
+| `OMNIGRAPH_POD_SELECTOR` | in Kubernetes | Label selector of the server Deployment's pods. The ServiceAccount needs `list` on pods. |
+| `POD_NAME` | no | This pod's name. Defaults to `HOSTNAME`. |
+| `OMNIGRAPH_PEER_WAIT_SECONDS` | no | How long to wait for another server pod to exit before giving up (default 50). |
+| `OMNIGRAPH_GRAPH_REMOVAL_ALLOWLIST` | no | Comma-separated graph ids that `cluster.yaml` is allowed to drop, which deletes them. |
+| `OMNIGRAPH_UNLOCK_WITHOUT_PEERS` | outside Kubernetes | `1` allows force-unlock with no Lease or pod check. Ignored in Kubernetes. |
+
+The maintenance window reads `OMNIGRAPH_UNLOCK_LEASE` and `POD_NAME` as above,
+and these. It takes the pod selector from the Deployment itself, and its
+ServiceAccount needs `list` on pods too.
+
+| Variable | Required | Purpose |
+|---|---|---|
+| `OMNIGRAPH_SERVER_DEPLOYMENT` | yes | Name of the server Deployment. The ServiceAccount needs `get` on it and `get` and `update` on its `scale`. |
+| `OMNIGRAPH_WINDOW_SECONDS` | yes | How long after the start a new command may still begin. The Job's `activeDeadlineSeconds` has to be longer by one command's timeout and the restore. |
+| `OMNIGRAPH_CLEANUP_OLDER_THAN` | yes | Retention passed to `cleanup --older-than`, e.g. `30d`. |
+| `OMNIGRAPH_CLEANUP_ISO_WEEKDAY` | no | UTC day `cleanup` also runs on (1 is Monday, 7 is Sunday). Unset, it never runs. |
+| `OMNIGRAPH_COMMAND_TIMEOUT_SECONDS` | no | Longest one `optimize` or `cleanup` may run before it is stopped (default 600). |
+| `OMNIGRAPH_SERVER_STOP_SECONDS` | no | How long to wait for the server pods to exit after the scale-down (default 90). |
+
+Its pod needs `terminationGracePeriodSeconds: 120`. After SIGTERM it stops the
+running command (up to 30s), removes that command's lock and scales the server
+back up, and on the default 30s it is killed before the scale-up. It exits 0
+only when every graph was maintained and the server is back (or left down for
+a migration).
+
+The Deployment carries the annotation `ol.mit.edu/migration-armed` while a
+storage-format migration is armed. The window skips its run when it sees it,
+and does not scale the server back up if it appears during the window.
+
+Outside Kubernetes (no `KUBERNETES_SERVICE_HOST`) the Lease and the pod check
+are skipped, and the entrypoint refuses to force-unlock unless
+`OMNIGRAPH_UNLOCK_WITHOUT_PEERS=1` is set. `cluster apply` always leaves a lock,
+so a local `docker run` needs it even on an empty root. Set it only when this
+container is the only writer of the cluster root.
+
 ## Base image / glibc
 
 Both images use **Debian trixie** bases (`debian:trixie-slim`,

docker/omnigraph-server.Dockerfile

@@ -30,15 +30,15 @@
 # exists because this file went a full release cycle claiming the manager
 # covered it when it did not, and a partial bump is silent until deploy.
 
-ARG OMNIGRAPH_VERSION=0.11.0
+ARG OMNIGRAPH_VERSION=0.13.0
 # Upstream tag to fetch from. A real release is `v${OMNIGRAPH_VERSION}`; `edge`
 # is the rolling build of upstream main, republished on every push there.
 # Kept separate because on a moving tag the two differ — see
 # witan_core/omnigraph_install.py :: _OMNIGRAPH_RELEASE_TAG, which also records
 # why this is a release tag again and what Lance 11 asks of an existing graph.
-ARG OMNIGRAPH_RELEASE_TAG=v0.11.0
-ARG OMNIGRAPH_SHA256_X86_64=da192e1a050875a93ee642b9df485203a00d8c0d44ca39204439463ad41a766d
-ARG OMNIGRAPH_SHA256_ARM64=af6be5f1069d7591985285871450bc0d68d4dc363bf7c2c0a7cca33915a8f4bb
+ARG OMNIGRAPH_RELEASE_TAG=v0.13.0
+ARG OMNIGRAPH_SHA256_X86_64=0123daa64f0f2db9860284200bd71c79a22a6ad2c6f5501b6fca24cac86a7a30
+ARG OMNIGRAPH_SHA256_ARM64=e61e7379131a05056bfeace206c8c5bcdf185d169d1ce86bc192d52f13693f96
 # The on-disk storage format the pinned binary reads, mirrored from witan_core's
 # declaration (packages/witan-core/witan_core/omnigraph_install.py ::
 # _OMNIGRAPH_INTERNAL_SCHEMA) and stamped onto the image below as
@@ -59,7 +59,7 @@ ARG OMNIGRAPH_SHA256_ARM64=af6be5f1069d7591985285871450bc0d68d4dc363bf7c2c0a7cca
 # runtime stage's re-declaration of it, and the LABEL line that emits it — an
 # ARG not re-declared in the stage expands to empty, which builds a clean image
 # advertising no format at all.
-ARG OMNIGRAPH_INTERNAL_SCHEMA=9
+ARG OMNIGRAPH_INTERNAL_SCHEMA=14
 
 # ── Fetch + checksum-verify the release, extract both binaries ────────────────
 FROM debian:trixie-slim@sha256:a99cfc517144bc59b1978475ec53b46ecabec7e43635402ee5b77cc54cd1b20a AS fetch
@@ -117,30 +117,39 @@ LABEL org.opencontainers.image.title="omnigraph-server" \
       edu.mit.ol.omnigraph.internal-schema="${OMNIGRAPH_INTERNAL_SCHEMA}"
 
 # ca-certificates: the omnigraph binaries talk to S3 over TLS.
-# python3-minimal + python3-yaml: the entrypoint renders the Cedar bundles'
-# group membership from the mounted actor-token map before converging (see
-# policy/render_groups.py). Deliberately a real YAML parser rather than a
-# sed/awk rewrite of a file that decides who can write the graph — a silent
-# mis-render here denies every user at once.
+# python3: the entrypoint (omnigraph_server_entrypoint.py) is Python and uses
+# the standard library's urllib.request (for the Kubernetes API) and secrets,
+# neither of which imports under python3-minimal.
+# python3-yaml: the entrypoint renders the Cedar bundles' group membership from
+# the mounted actor-token map before converging (see policy/render_groups.py).
+# Deliberately a real YAML parser rather than a sed/awk rewrite of a file that
+# decides who can write the graph — a silent mis-render here denies every user
+# at once.
 RUN apt-get update \
     && apt-get install -y --no-install-recommends \
-        ca-certificates python3-minimal python3-yaml \
+        ca-certificates python3 python3-yaml \
     && rm -rf /var/lib/apt/lists/* \
     && useradd --uid 1000 --user-group --create-home omnigraph
 
 COPY --from=fetch /out/omnigraph /usr/local/bin/omnigraph
 COPY --from=fetch /out/omnigraph-server /usr/local/bin/omnigraph-server
-COPY docker/omnigraph-server-entrypoint.sh /usr/local/bin/omnigraph-server-entrypoint.sh
+COPY docker/omnigraph_server_entrypoint.py /usr/local/bin/omnigraph-server-entrypoint.py
+# The maintenance window and the pre-deploy plan gate, which the `omnigraph`
+# stack's CronJob and Job run as `command`. Both import the entrypoint for its
+# Lease, pod check and CLI wrappers, hence the importable name beside them.
+COPY docker/omnigraph_maintenance.py docker/omnigraph_plan_gate.py /usr/local/bin/
 COPY mcp/servers/witan/policy/render_groups.py /usr/local/bin/render-policy-groups.py
 
 # The cluster config dir holds the baked-in schema.pg, the ConfigMap-mounted
-# cluster.yaml (added at deploy time), and the ephemeral `__cluster/` state the
-# entrypoint's `cluster import`/`apply` writes — so it must be owned by the
-# non-root runtime user. Keep this path in lockstep with CLUSTER_CONFIG_DIR in
+# cluster.yaml (added at deploy time), and the policy bundles the entrypoint
+# re-renders in place on every boot — so it must be owned by the non-root
+# runtime user. Keep this path in lockstep with CLUSTER_CONFIG_DIR in
 # ol-infrastructure src/ol_infrastructure/applications/omnigraph/data_tier.py.
 ENV OMNIGRAPH_CLUSTER_DIR=/etc/omnigraph/cluster
 RUN mkdir -p "${OMNIGRAPH_CLUSTER_DIR}" \
-    && chmod +x /usr/local/bin/omnigraph-server-entrypoint.sh \
+    && chmod +x /usr/local/bin/omnigraph-server-entrypoint.py \
+    && ln -s omnigraph-server-entrypoint.py /usr/local/bin/omnigraph_server_entrypoint.py \
+    && chmod +x /usr/local/bin/omnigraph_maintenance.py /usr/local/bin/omnigraph_plan_gate.py \
     && chmod +x /usr/local/bin/render-policy-groups.py
 # All three cluster schemas are baked in: schema.pg (witan memory/work graph →
 # the `council` graph), code-schema.pg (per-repo `code-<repo>` graphs), and
@@ -165,7 +174,7 @@ RUN chown -R omnigraph:omnigraph /etc/omnigraph
 
 USER omnigraph
 EXPOSE 8080
-ENTRYPOINT ["/usr/local/bin/omnigraph-server-entrypoint.sh"]
+ENTRYPOINT ["/usr/local/bin/omnigraph-server-entrypoint.py"]
 # Overridden by the `omnigraph` stack's Deployment `args`
 # (--cluster /etc/omnigraph/cluster --bind 0.0.0.0:8080); this default keeps the
 # image runnable standalone and documents the expected invocation.