scaling: bound the estate's autoscaling, and turn off the five HPAs that split state - #1116
Open
mdheller wants to merge 1 commit into
Open
scaling: bound the estate's autoscaling, and turn off the five HPAs that split state#1116mdheller wants to merge 1 commit into
mdheller wants to merge 1 commit into
Conversation
…plit state
Recon found the opposite of "no autoscaling": 18 HPAs were already live, all
CPU-based and all reading real utilisation. The defect was not absence, it was
that nobody had asked what a second replica would split. Five of the eighteen
were on services that hold request-spanning state in process memory:
memoryd MEMORYMESH_STORE_URI=memory:// selects InMemoryStore, so
`_memories` AND the `_revoked` tombstone set are per-pod.
store.py:96 calls this "read-enforced revocation"; at two
replicas a revoked memory keeps being served by the other
pod. A governance contract failing silently.
algo-engine _LEDGER (server.py:243) — cash/positions/orders per pod,
each booting with the full $1,000,000.
grl-mesh MeshAggregator (server.py:30) — the community prior is the
state; two replicas each serve half the evidence as if whole.
regis-acr-api _PROOFS/_EVENT_LOG (er_spine.py:68) — a proof minted on one
pod 404s on the follow-up GET routed to the other.
search-orchestrator academy_repository falls through to the in-memory repo
(repositories.py:200); this one ran at maxReplicas: 4.
None had ever fired — idle CPU sits near 6% against a 75% target — so the split
had not happened in production yet. Each is now disabled with the reason and the
RE-ENABLEMENT CONDITION written beside it, and every condition is a shared-store
change, never "wait for more load".
Chart:
- hpa.yaml gains an explicit behavior block: 300s scale-down stabilisation (the
k8s default, but on the record so an API default cannot move it), +2 pods/min
up, -1 pod/min down.
- hpa.yaml FAILS RENDERING when autoscaling is on and resources.requests.cpu is
unset. averageUtilization is a percentage OF THE REQUEST; with no denominator
the HPA reports <unknown> and sits at min forever while looking installed.
- new pdb.yaml, rendered ONLY at a guaranteed replica floor >= 2 and always as
maxUnavailable: 1. A minAvailable PDB on a singleton can never be satisfied and
wedges Autopilot node upgrades, repairs and consolidation indefinitely — worse
than no PDB. Six workloads qualify; the estate had zero PDBs before this.
- the statefulness test that disqualifies a workload is written into values.yaml
where the next person will read it.
Thirteen verified-stateless services gain a bounded HPA. Twelve are min 1 (no
baseline cost, burst only). Only `api` moves its floor to 2: it is the tritRPC
core, gateway proxies every call to it and health-checks it with a real
HealthPing, so one replica meant an api restart took every gateway replica
unhealthy at once. socbase-auth/rest were already at replicaCount 2, so their
floor is unchanged. Baseline delta is one small replica, ~$0.09/day.
Bounds are budgets, not round numbers: max 3 for default-sized services, max 2
for embeddings because it requests 250m/1Gi (~$0.50/day/replica) and loads its
model before binding, so a scale-up is capacity for sustained load, not spike
relief.
Alerting: a new socioprophet.autoscaling group built on kube-state-metrics —
deliberately NOT on the app metrics plane, whose http_server_request_duration_
seconds has zero emitters estate-wide. HPAAtMaxReplicas makes the ceiling audible
instead of silent; HPACannotComputeMetric catches the decorative-autoscaler case
the chart guard cannot. Both series were confirmed present in Prometheus before
the rules were written. The existing request-slo rules are untouched — that
metric decision belongs to another lane.
Proven on the live cluster in an isolated namespace, not asserted: 1 -> 3 under
load ("SuccessfulRescale ... above target"), held at the ceiling with
ScalingLimited=True/TooManyReplicas while the alert expression returned exactly
one firing series, then back to 1 after the stabilisation window.
Left deliberately alone, each verified stateful: hellgraph-service (the graph is
a per-process AtomSpace on an ephemeral pod-local dir), health-twin (in-memory
consult AND grant ledgers — a revoke on one pod leaves the grant live on others),
compute-gateway (SQLite on RWO PVC + in-process grants), clickhouse and
arcticdb-gateway (PVCs), and the six single-writer loops (market-replay,
prophet-materializer-clickhouse, lifecycle-warden, device-service,
nugget-extractor, reporting-watcher).
Contributor
There was a problem hiding this comment.
Pull request overview
Note
Copilot couldn't run its full agentic review because it didn't start before the timeout. Make sure your repository has a runner available, or add a copilot-code-review.yml file specifying one with the runs-on attribute. See the docs for more details.
Adds bounded, safer autoscaling across the estate by introducing explicit HPA behavior controls, disabling HPAs for workloads that split in-process state, and adding guardrails/alerting to make scaling failures visible.
Changes:
- Add HPA
behaviordefaults + a fail-closed Helm guard for missing CPU requests. - Introduce a guarded PodDisruptionBudget template (only when replica floor ≥ 2).
- Add Prometheus alerts for HPA ceiling, inert HPAs, and disruption-blocking PDBs; update many service values to enable/disable autoscaling with documented rationale.
Reviewed changes
Copilot reviewed 23 out of 23 changed files in this pull request and generated 5 comments.
Show a summary per file
| File | Description |
|---|---|
| infra/k8s/observability/base/prometheusrule-slos.yaml | Adds kube-state-metrics-based autoscaling/PDB alert rules. |
| deploy/values/synapse-bridge.yaml | Enables autoscaling with documented statelessness rationale. |
| deploy/values/socbase-rest.yaml | Enables autoscaling with minReplicas=2 to preserve existing floor. |
| deploy/values/socbase-auth.yaml | Enables autoscaling with minReplicas=2 to preserve existing floor. |
| deploy/values/search-orchestrator.yaml | Disables autoscaling due to in-memory runtime-ingested state. |
| deploy/values/regis-acr-api.yaml | Disables autoscaling due to in-process proofs/event log. |
| deploy/values/portfolio-agent.yaml | Enables autoscaling for CPU-bound, request-local work. |
| deploy/values/osm-map-api.yaml | Enables autoscaling for read-only GET-only workload. |
| deploy/values/memoryd.yaml | Disables autoscaling due to in-memory store + revocation tombstones. |
| deploy/values/holmes.yaml | Enables autoscaling for stateless proxy behavior. |
| deploy/values/grl-mesh.yaml | Disables autoscaling due to in-memory aggregation state. |
| deploy/values/evidence-receipts.yaml | Enables autoscaling with explicit “disable if writer enabled” condition. |
| deploy/values/evidence-console.yaml | Enables autoscaling for stateless proxy behavior. |
| deploy/values/eval-fabric-api.yaml | Enables autoscaling with explicit “disable if receipts emission enabled” condition. |
| deploy/values/embeddings.yaml | Enables autoscaling with tight maxReplicas due to cost/startup constraints. |
| deploy/values/api.yaml | Enables autoscaling with minReplicas=2 for availability + PDB eligibility. |
| deploy/values/algo-engine.yaml | Disables autoscaling due to process-local execution ledger. |
| deploy/values/agentic-os-api.yaml | Enables autoscaling for constant-returning stateless API. |
| deploy/values/academy-board.yaml | Enables autoscaling for request-pure grading behavior. |
| charts/socioprophet-service/values.yaml | Documents autoscaling statefulness test; adds behavior + memory target + PDB defaults. |
| charts/socioprophet-service/templates/pdb.yaml | New guarded PDB template rendered only when replica floor ≥ 2. |
| charts/socioprophet-service/templates/hpa.yaml | Adds CPU-request guard, optional memory metric, and HPA behavior policies. |
| charts/socioprophet-service/templates/_helpers.tpl | Adds svc.replicaFloor helper used for guarded PDB rendering. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Comment on lines
+58
to
+66
| scaleDown: | ||
| # THE ANTI-THRASH CONTROL. The HPA holds the highest recommendation seen in this | ||
| # window before it will shrink, so a workload that dips between bursts does not | ||
| # flap. On Autopilot a scale-down also has a cost tail: the pod stops billing | ||
| # immediately, but the node it vacated only bills out once the cluster autoscaler | ||
| # consolidates — so rapid up/down churn pays for capacity it never used. | ||
| stabilizationWindowSeconds: {{ .Values.autoscaling.behavior.scaleDown.stabilizationWindowSeconds }} | ||
| selectPolicy: Max | ||
| policies: |
Comment on lines
+100
to
+102
| - alert: PDBBlocksAllDisruption | ||
| expr: kube_poddisruptionbudget_status_expected_pods > 0 and kube_poddisruptionbudget_status_pod_disruptions_allowed == 0 | ||
| for: 30m |
Comment on lines
+66
to
+71
| - alert: HPAAtMaxReplicas | ||
| expr: | | ||
| kube_horizontalpodautoscaler_status_current_replicas | ||
| >= on (namespace, horizontalpodautoscaler) | ||
| kube_horizontalpodautoscaler_spec_max_replicas | ||
| for: 15m |
| target: | ||
| type: Utilization | ||
| averageUtilization: {{ .Values.autoscaling.targetCPUUtilizationPercentage }} | ||
| {{- with .Values.autoscaling.targetMemoryUtilizationPercentage }} |
Comment on lines
+41
to
+47
| - type: Resource | ||
| resource: | ||
| name: memory | ||
| target: | ||
| type: Utilization | ||
| averageUtilization: {{ . }} | ||
| {{- end }} |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this is
Basic, bounded scaling automation for the estate — plus the removal of five autoscalers that were already live and already wrong.
DO NOT MERGE without reading "What is still live" at the bottom. The five unsafe HPAs described here are running in production right now; this PR is the only thing that turns them off.
Recon first: the premise was wrong, and the truth is worse
The brief expected "no HPAs, or one that can't fire." Actual state of
deploy/argocd/platform-services.yaml(44 services):cpu: N%/75%, never<unknown>)50m/128Mi→1/512Mi)Two things worth correcting up front:
50m/128Mi), limits sit above them. The feared "billed at the limit" cost blow-up is not happening.http_server_request_duration_secondssituation — that series has zero emitters, which is why nothing here depends on it.The actual defect: nobody asked what a second replica would split
Five of the eighteen live HPAs are on services holding request-spanning state in process memory. All five verified by reading the store and handler code, not inferred from names.
InMemoryStore—_memoriesand the_revokedtombstone set (store.py:38,96,122), becauseMEMORYMESH_STORE_URI=memory://store.py:96calls this "read-enforced revocation"; revocation only tombstones on the pod that received it_LEDGER— cash/positions/orders (server.py:243)MeshAggregator._stats/_contributors(server.py:30)_PROOFS/_EVENT_LOG(er_spine.py:68)academy_repositoryfalls through to the in-memory repo (repositories.py:200)None had ever fired. Idle CPU sits near 6% against a 75% target, so the split has not happened in production — it was one traffic spike away. That is the whole point: an autoscaler nobody has watched scale is not "working," it is untested.
Each is now disabled with the reason and the re-enablement condition written beside it. Every condition is a shared-store change (
memoryd→ the Postgres backend that already exists inpostgres_store.py;search-orchestrator→SEARCH_ORCHESTRATOR_ACADEMY_STORE). None is "wait for more load."Chart changes
hpa.yamlgains abehaviorblock — 300s scale-down stabilisation, +2 pods/min up, −1 pod/min down. 300s matches the k8s default but is now on the record, so a future API default can't move it silently.hpa.yamlfails rendering whenautoscaling.enabledis true andresources.requests.cpuis unset.averageUtilizationis a percentage of the request; with no denominator the HPA reports<unknown>and sits at min forever while looking installed. Proven to fail:--set resources.requests.cpu=null→ template error.pdb.yaml, rendered only at a guaranteed replica floor ≥ 2, always asmaxUnavailable: 1. AminAvailablePDB on a singleton can never be satisfied and wedges Autopilot node upgrades, repairs and consolidation indefinitely — strictly worse than no PDB. Six workloads qualify; the estate had zero PDBs before this.values.yamlwhere the next person adding a service will read it.What gained an autoscaler
Thirteen services, each verified stateless by reading its handlers and stores.
minReplicas: 1— no baseline cost, burst headroom only.apimoves its floor to 2. It is the tritRPC core;gateway(2-6) proxies every call and health-checks it with a realHealthPinground-trip, so at one replica anapirestart took every gateway replica unhealthy simultaneously.socbase-auth/socbase-restwere alreadyreplicaCount: 2, so their floor is unchanged — pure burst headroom, no new spend.Bounds are budgets, not round numbers.
max: 3for default-sized services;max: 2forembeddings, which requests250m/1Gi(~$0.50/day/replica, ~5x a normal pod) and loads its model before binding — a scale-up there is capacity for sustained load, not spike relief.Two services carry a documented condition rather than a clean bill of health:
evidence-receiptsis safe only because its local-disk writer (telemetry_runtime.py) is not imported bymain.py, andeval-fabric-apiis safe only whileEVAL_FABRIC_EMIT_RECEIPTSstays unset. Both notes say to disable autoscaling in the same change that flips those.Alerting
New
socioprophet.autoscalinggroup, built entirely on kube-state-metrics — deliberately not on the app metrics plane. Both series were confirmed present in Prometheus before the rules were written (22 current-replica series, 198 condition series).HPAAtMaxReplicas— the ceiling becomes audible instead of silent.HPACannotComputeMetric(ScalingActive=false) — the decorative-autoscaler detector; catches the case the chart guard can't.PDBBlocksAllDisruption— catches an unsatisfiable budget arriving by any other path.The existing
request-slorules are untouched — that metric decision belongs to another lane.Proof it fires, both directions
Run on the live cluster in an isolated
scaling-proofnamespace (Argo self-healssocioprophet, so proving there would have fought the controller). Real service, real values file, real chart.Scale-down began 343s after load was removed — the 300s stabilisation window plus a sync interval — and stepped down one pod per minute exactly as the
scaleDownpolicy specifies. It did not thrash: CPU sat at 4-8% for five straight minutes without the autoscaler flinching.At-max is a signal, not a silent cap. The exact
HPAAtMaxReplicasexpression, evaluated against live Prometheus while capped, returned exactly one firing series —scaling-proof/academy-board current=3— with no false positives across the other 21 HPAs in the estate.Left deliberately alone — every one verified stateful
Not scaled, and each will stay unscaled until its state moves somewhere shared:
hellgraph-service— the graph is a per-process AtomSpace on an ephemeral pod-local dir (server.ts:42; no PVC in its values, andserver.ts:486says so outright).GET /api/graph/logreads that in-process atom set, so a load-balanced materializer tailing?since=would skip or replay depending on which pod answered.health-twin— in-memory consult ledger (consult.ts:49) and a grant ledger (server.ts:66) where revocation only applies to the pod that received it — an authorization failure, same shape as memoryd.compute-gateway— SQLite on an RWO PVC + in-process grants. Already correctly pinned; its own source comment admits "a 2-replica gateway already returns different chains per pod."clickhouse,arcticdb-gateway— PVCs.market-replay,prophet-materializer-clickhouse,lifecycle-warden,device-service,nugget-extractor,reporting-watcher. These were already correct, with the reasoning already in their values files.sherlock-engine— carries a bareautoscaling: { enabled: false }with no stated reason. My audit says it would be safe (and its serving loop is single-threaded, which argues for replicas), but I am not flipping someone else's undocumented decision. Owner should confirm.Cost
Autopilot bills on requests. A default pod (
50m/128Mi/1Gi eph) is ~$0.089/day.apireplica. Nothing else raises the floor.$3.00/day ($90/month). Realistically unreachable — these services idle at 4-8% CPU.min: 1running one replica) — it is a correctness fix, not a cost fix.Verification status
tools/preflight_deploy_contract.pypasses (exit 0). Its warnings are pre-existinglatest-tag ratchet entries, untouched here.What is still live (needs a merge, not a follow-up)
Argo tracks
mainwithselfHeal: true. Until this merges, all five unsafe HPAs remain active in production. Deleting them by hand would just be recreated by Argo, so the merge is the fix. Current risk is bounded — all five sit at one replica under light load — but it is a latent split waiting on a traffic spike.search-orchestrator,eval-fabric-apiandhellgraph-servicealso deploy throughdeploy/argocd/fogstack/fogstack-appset.yaml, which reads the same values files. Thesearch-orchestratorfix therefore reaches all three namespaces (it carried the identical max-4 HPA in each).