feat(omnigraph): add the maintenance window and the pre-deploy plan gate

[blarghmatey]

What are the relevant tickets?

N/A (witan tasks tk-ol-infrastructure-omnigraph-stack-for-0-12-clust-2dee94 and tk-cover-the-omnigraph-server-entrypoint-s-untested-f43c59, part of the omnigraph 0.13 upgrade, spec in #461 D4 and D5)

Description (What does it do?)

Stacked on #479. On omnigraph 0.13 optimize and cleanup are refused with state_lock_held while the server holds its serve lock, so the nightly sweeps need the server stopped. The entrypoint in #479 also applies schema and policy changes only after Recreate has stopped the old pod, so a change the apply refuses is first seen by a pod that cannot start.

This adds two commands to the omnigraph-server image, for the omnigraph stack in ol-infrastructure to run as a Job's command. They are here and not in ol-infrastructure because both need the entrypoint's Lease, pod check and plan check, and this repo's CI has the real binary.

docker/omnigraph_maintenance.py is the stop window from D5:

1. Exit 0 if the Deployment carries ol.mit.edu/migration-armed or is already at 0 replicas.
2. Take the unlock Lease (exit 1 if someone else holds it, which includes a server pod that restarted in the last minute).
3. Scale the Deployment to 0, conditional on the resourceVersion it read, and wait until no live pod matches the selector.
4. Force-unlock the serve lock. Any other lock stops the window.
5. Per graph in cluster.yaml: optimize, force-unlock. On OMNIGRAPH_CLEANUP_ISO_WEEKDAY, a second pass: cleanup --older-than … --confirm --yes, force-unlock. A failed graph does not stop the rest. The cleanup pass starts one graph further along each ISO week, because Production's cleanup sweep took up to 1552s in the last 30 days (beside a live 0.11 server) and a window that runs out must not skip the same graphs every week.
6. Restore the replica count and release the Lease.

Step 6 also runs on SIGTERM, on a failed step and when the window's own deadline passes. The restore is skipped when the Deployment is no longer at 0 or a migration was armed in the meantime (the write is conditional, so an arm between the read and the write is seen on the retry). If the window's own graph_operation lock could not be removed (Lease lost, or force-unlock failed), the Deployment stays at 0, as D5 describes.

docker/omnigraph_plan_gate.py renders group membership and runs cluster plan through the entrypoint's check_plan, and nothing else. It takes no Lease and never unlocks.

Changes to the entrypoint from #479:

• run_omnigraph stops its child on any exception and takes a timeout. A SIGTERM handler raising in the main thread used to leave Popen.__exit__ waiting for the child to finish on its own.
• KubernetesApi gains get_deployment and replace_scale, and UnlockLease gains release().
• The plan half of converge is now checked_plan, which the gate calls.

Two differences from D5 as written:

• It is Python in the image, not /bin/sh -c with traps in the CronJob.
• The window identifies its own lock from cluster status (lock_operation == "graph_operation"), not from the lock_id= the CLI prints on stderr. The root is unlocked before each command, the Lease excludes other unlockers and no server pod is live, so a graph_operation lock found afterwards is that command's. Root-addressed cluster status --cluster <uri> does not report the operation, so the CronJob has to mount cluster.yaml and the window reads the graph list and storage root from it.

The second commit is the coverage Copilot asked for on #479: the real KubernetesApi against a local TLS server, main, a Lease lost between the two unlocks, the non-deployment branch of release_apply_lock, and confirm() racing the renew thread.

Like #479, this only works on 0.13 and merges with the pin bump.

How can this be tested?

• Against the released 0.13.0 binary: test_omnigraph_maintenance.py, test_omnigraph_server_entrypoint.py and test_omnigraph_kubernetes_api.py, 99 passed, none skipped. Two of the new cases use the real binary on a file:// root: a whole window after a real server has left its serve lock (optimize and cleanup on three graphs, no lock left, the server boots again), and the gate reading a served root without touching its lock and failing on a schema change that adds a required field.
• On the pinned 0.11.0 the same three files: 92 passed, 7 skipped (the real-binary cases).
• I built the image with the 0.13.0 build args and ran it with docker run on a local volume: the gate passes on an empty root and beside a running server, the server boots, stops and boots again, and omnigraph_maintenance.py imports the entrypoint through the symlink (it gets as far as the missing OMNIGRAPH_SERVER_DEPLOYMENT).
• Not tested: the window against a real Kubernetes API (the Deployment and scale calls run against a fake, and that a deployments/scale PUT honours the Deployment's resourceVersion is from the API conventions, not observed), a real SIGTERM delivered to the process (the test raises the handler's exception), and anything on S3. These get observed in the CI cutover.
• I did not run the full just test-witan-council suite.

https://claude.ai/code/session_01W4TjT25fD1W4TS1uRUY1hC