Autoscaler internal error

[sebastianpawlak]

What version of Knative?

1.22.0

Expected Behavior

A deployment is scaled up or down by autoscaler.

Actual Behavior

There are internal errors such as:

{
  "insertId": "ep4uyzcmfxz4tqfa",
  "jsonPayload": {
    "logger": "autoscaler.event-broadcaster",
    "commit": "370ad5a",
    "timestamp": "2026-05-21T07:35:33.930551477Z",
    "caller": "record/event.go:377",
    "message": "Event(v1.ObjectReference{Kind:\"PodAutoscaler\", Namespace:\"headless-customer\", Name:\"hqbht9iojq2hf8njzu18pcsnz-00008\", UID:\"a431f570-bfd3-4349-b699-c27ce693c175\", APIVersion:\"autoscaling.internal.knative.dev/v1alpha1\", ResourceVersion:\"1779341502140639018\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' error scaling target: failed to apply scale 0 to scale target hqbht9iojq2hf8njzu18pcsnz-00008-deployment: Internal error occurred: add operation does not apply: doc is missing path: \"/spec/template/metadata/labels\": missing value"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "container_name": "autoscaler",
      "project_id": "wp-engine-headless-dev",
      "namespace_name": "knative-serving",
      "cluster_name": "dev-us-central1-data-1",
      "location": "us-central1",
      "pod_name": "autoscaler-6ff64864bc-6crfm"
    }
  },
  "timestamp": "2026-05-21T07:35:33.930664297Z",
  "severity": "INFO",
  "labels": {
    "k8s-pod/repo": "headless-infra",
    "k8s-pod/pod-template-hash": "6ff64864bc",
    "k8s-pod/app": "autoscaler",
    "k8s-pod/app_kubernetes_io/name": "knative-serving",
    "k8s-pod/jira": "NEB",
    "k8s-pod/app_kubernetes_io/component": "autoscaler",
    "k8s-pod/slack": "atlas-infrastructure-monitoring",
    "compute.googleapis.com/resource_name": "gke-dev-us-central1--dev-us-central1--7fcf9d9a-t2yh",
    "k8s-pod/app_kubernetes_io/version": "1.22.0",
    "k8s-pod/owner": "pe-atlas-nebula",
    "k8s-pod/topology_kubernetes_io/region": "us-central1",
    "k8s-pod/topology_kubernetes_io/zone": "us-central1-c",
    "k8s-pod/team": "nebula"
  },
  "logName": "projects/wp-engine-headless-dev/logs/stderr",
  "receiveTimestamp": "2026-05-21T07:35:38.628480138Z"
}

Another example:

{
  "insertId": "4ev65mtc3qqcki46",
  "jsonPayload": {
    "commit": "370ad5a",
    "timestamp": "2026-05-21T07:19:07.692521554Z",
    "caller": "record/event.go:377",
    "message": "Event(v1.ObjectReference{Kind:\"PodAutoscaler\", Namespace:\"headless-customer\", Name:\"h8qhg50z882e25jm6l90hyqom-00006\", UID:\"c6b08ad2-f0c2-47ab-a6b9-54147320ae2d\", APIVersion:\"autoscaling.internal.knative.dev/v1alpha1\", ResourceVersion:\"1779175295378063018\", FieldPath:\"\"}): type: 'Warning' reason: 'InternalError' error scaling target: failed to apply scale 1 to scale target h8qhg50z882e25jm6l90hyqom-00006-deployment: Internal error occurred: add operation does not apply: doc is missing path: \"/spec/template/metadata/labels\": missing value",
    "logger": "autoscaler.event-broadcaster"
  },
  "resource": {
    "type": "k8s_container",
    "labels": {
      "pod_name": "autoscaler-6ff64864bc-6crfm",
      "project_id": "wp-engine-headless-dev",
      "namespace_name": "knative-serving",
      "cluster_name": "dev-us-central1-data-1",
      "location": "us-central1",
      "container_name": "autoscaler"
    }
  },
  "timestamp": "2026-05-21T07:19:07.692780384Z",
  "severity": "INFO",
  "labels": {
    "k8s-pod/pod-template-hash": "6ff64864bc",
    "k8s-pod/owner": "pe-atlas-nebula",
    "k8s-pod/app_kubernetes_io/version": "1.22.0",
    "k8s-pod/slack": "atlas-infrastructure-monitoring",
    "k8s-pod/team": "nebula",
    "k8s-pod/app_kubernetes_io/name": "knative-serving",
    "k8s-pod/app": "autoscaler",
    "k8s-pod/jira": "NEB",
    "k8s-pod/repo": "headless-infra",
    "k8s-pod/topology_kubernetes_io/region": "us-central1",
    "k8s-pod/topology_kubernetes_io/zone": "us-central1-c",
    "compute.googleapis.com/resource_name": "gke-dev-us-central1--dev-us-central1--7fcf9d9a-t2yh",
    "k8s-pod/app_kubernetes_io/component": "autoscaler"
  },
  "logName": "projects/wp-engine-headless-dev/logs/stderr",
  "receiveTimestamp": "2026-05-21T07:19:08.706395958Z"
}

Steps to Reproduce the Problem

The problems started to happen after upgrading knative to 1.22.0

[linkvt]

Hi, thanks for the bug report!

Can you give us more information, because it seems really strange that a deployment would not have /spec/template/metadata/labels set. I guess it would be helpful to know how the kservice and the deployment look right now and if you have kyverno/OPA/... installed in the cluster.
Edit: Which kubernetes version are you using?

[sebastianpawlak]

Hi @linkvt !

Here's a deployment spec:

k get deployment/hqbht9iojq2hf8njzu18pcsnz-00008-deployment -n headless-customer -o yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
    headless.wpengine.com/appName: atlas-simple-express-app-2026-03-17-1
    headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
    headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
    headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
    headless.wpengine.com/commitSHA: 707f367cb223d039c9b4a198830e16e17f806f83
    headless.wpengine.com/createdOn: "2026-05-19T09:29:36Z"
    headless.wpengine.com/default-domain: hqbht9iojq2hf8njzu18pcsnz....nip.io
    headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
    headless.wpengine.com/envBranch: main
    headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
    headless.wpengine.com/envName: main
    headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
    headless.wpengine.com/region: US-C
    serving.knative.dev/creator: headless-api-service@wp-engine-headless-dev.iam.gserviceaccount.com
  creationTimestamp: "2026-05-19T09:29:37Z"
  generation: 1
  labels:
    CE_MITM_RESOURCE_MUTATED_ENVVARS: "true"
    app: hqbht9iojq2hf8njzu18pcsnz-00008
    headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
    headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
    headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
    headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
    headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
    headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
    headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
    headless.wpengine.com/environment-category: headless
    headless.wpengine.com/ksvc-version: 2.0.0
    headless.wpengine.com/releaseChannel: regular
    headless.wpengine.com/serviceType: headless
    jira: NEB
    owner: sec-headless
    repo: headless-infra
    service.istio.io/canonical-name: hqbht9iojq2hf8njzu18pcsnz
    service.istio.io/canonical-revision: hqbht9iojq2hf8njzu18pcsnz-00008
    serving.knative.dev/configuration: hqbht9iojq2hf8njzu18pcsnz
    serving.knative.dev/configurationGeneration: "8"
    serving.knative.dev/configurationUID: 368a390d-65fd-4864-9509-2387dcb370c3
    serving.knative.dev/deployment-hash: 1qzbkm8
    serving.knative.dev/revision: hqbht9iojq2hf8njzu18pcsnz-00008
    serving.knative.dev/revisionUID: 6c81d2d7-5d36-44be-9c4c-40a2a8551ab2
    serving.knative.dev/service: hqbht9iojq2hf8njzu18pcsnz
    serving.knative.dev/serviceUID: e428b53c-82e2-4bae-ba88-99c752317daa
    slack: atlas-infrastructure-monitoring
  name: hqbht9iojq2hf8njzu18pcsnz-00008-deployment
  namespace: headless-customer
  ownerReferences:
  - apiVersion: serving.knative.dev/v1
    blockOwnerDeletion: true
    controller: true
    kind: Revision
    name: hqbht9iojq2hf8njzu18pcsnz-00008
    uid: 6c81d2d7-5d36-44be-9c4c-40a2a8551ab2
  resourceVersion: "1779182986340751018"
  uid: 782eb4fd-caf2-40a2-9d16-8612f684553a
spec:
  progressDeadlineSeconds: 300
  replicas: 1
  revisionHistoryLimit: 0
  selector:
    matchLabels:
      serving.knative.dev/revisionUID: 6c81d2d7-5d36-44be-9c4c-40a2a8551ab2
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 0
    type: RollingUpdate
  template:
    metadata:
      annotations:
        cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
        headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
        headless.wpengine.com/appName: atlas-simple-express-app-2026-03-17-1
        headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
        headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
        headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
        headless.wpengine.com/commitSHA: 707f367cb223d039c9b4a198830e16e17f806f83
        headless.wpengine.com/createdOn: "2026-05-19T09:29:36Z"
        headless.wpengine.com/default-domain: hqbht9iojq2hf8njzu18pcsnz.....nip.io
        headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
        headless.wpengine.com/envBranch: main
        headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
        headless.wpengine.com/envName: main
        headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
        headless.wpengine.com/region: US-C
        kubectl.kubernetes.io/default-container: nginx
        serving.knative.dev/creator: headless-api-service@wp-engine-headless-dev.iam.gserviceaccount.com
      labels:
        CE_MITM_RESOURCE_MUTATED_ENVVARS: "true"
        app: hqbht9iojq2hf8njzu18pcsnz-00008
        headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
        headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
        headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
        headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
        headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
        headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
        headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
        headless.wpengine.com/environment-category: headless
        headless.wpengine.com/ksvc-version: 2.0.0
        headless.wpengine.com/releaseChannel: regular
        headless.wpengine.com/serviceType: headless
        jira: NEB
        owner: sec-headless
        repo: headless-infra
        service.istio.io/canonical-name: hqbht9iojq2hf8njzu18pcsnz
        service.istio.io/canonical-revision: hqbht9iojq2hf8njzu18pcsnz-00008
        serving.knative.dev/configuration: hqbht9iojq2hf8njzu18pcsnz
        serving.knative.dev/configurationGeneration: "8"
        serving.knative.dev/configurationUID: 368a390d-65fd-4864-9509-2387dcb370c3
        serving.knative.dev/revision: hqbht9iojq2hf8njzu18pcsnz-00008
        serving.knative.dev/revisionUID: 6c81d2d7-5d36-44be-9c4c-40a2a8551ab2
        serving.knative.dev/service: hqbht9iojq2hf8njzu18pcsnz
        serving.knative.dev/serviceUID: e428b53c-82e2-4bae-ba88-99c752317daa
        slack: atlas-infrastructure-monitoring
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  serving.knative.dev/revision: hqbht9iojq2hf8njzu18pcsnz-00008
              topologyKey: kubernetes.io/hostname
            weight: 100
      containers:
      - env:
        - (some deleted envvars)
        image: us-central1-docker.pkg.dev/wp-engine-headless-dev/customer-images/qbht9iojq2hf8njzu18pcsnz@sha256:4470ed23003ddc58d341211a63d9ddf4bbd292a59a274da8932d1ddfa97d7a9e
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            httpGet:
              path: /wait-for-drain
              port: 8022
              scheme: HTTP
        name: user-container
        resources:
          limits:
            cpu: "4"
            ephemeral-storage: 20Gi
            memory: 5Gi
          requests:
            cpu: 300m
            ephemeral-storage: 50Mi
            memory: 1200Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/log
          name: knative-var-log
          subPathExpr: $(K_INTERNAL_POD_NAMESPACE)_$(K_INTERNAL_POD_NAME)_user-container
      - env:
        - (some deleted envvars)
        image: us-docker.pkg.dev/wpe-art/headless-images/headless-infra/nginx-sidecar:3.40.3
        imagePullPolicy: IfNotPresent
        lifecycle:
          preStop:
            httpGet:
              path: /wait-for-drain
              port: 8022
              scheme: HTTP
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz
            port: 8083
            scheme: HTTP
          initialDelaySeconds: 10
          periodSeconds: 1
          successThreshold: 1
          timeoutSeconds: 1
        name: nginx
        ports:
        - containerPort: 8081
          name: user-port
          protocol: TCP
        - containerPort: 9113
          name: nginx-metrics
          protocol: TCP
        - containerPort: 9090
          name: nginx-healthz
          protocol: TCP
        resources:
          limits:
            cpu: 250m
            ephemeral-storage: 1Gi
            memory: 200Mi
          requests:
            cpu: 50m
            ephemeral-storage: 100Mi
            memory: 200Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: FallbackToLogsOnError
        volumeMounts:
        - mountPath: /var/log
          name: knative-var-log
          subPathExpr: $(K_INTERNAL_POD_NAMESPACE)_$(K_INTERNAL_POD_NAME)_nginx
      - env:
        - name: SERVING_NAMESPACE
          value: headless-customer
        - name: SERVING_SERVICE
          value: hqbht9iojq2hf8njzu18pcsnz
        - name: SERVING_CONFIGURATION
          value: hqbht9iojq2hf8njzu18pcsnz
        - name: SERVING_REVISION
          value: hqbht9iojq2hf8njzu18pcsnz-00008
        - name: QUEUE_SERVING_PORT
          value: "8012"
        - name: QUEUE_SERVING_TLS_PORT
          value: "8112"
        - name: CONTAINER_CONCURRENCY
          value: "0"
        - name: REVISION_TIMEOUT_SECONDS
          value: "35"
        - name: REVISION_RESPONSE_START_TIMEOUT_SECONDS
          value: "0"
        - name: REVISION_IDLE_TIMEOUT_SECONDS
          value: "0"
        - name: SERVING_POD
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: metadata.name
        - name: SERVING_POD_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.podIP
        - name: SERVING_LOGGING_CONFIG
        - name: SERVING_LOGGING_LEVEL
        - name: USER_PORT
          value: "8081"
        - name: SYSTEM_NAMESPACE
          value: knative-serving
        - name: SERVING_READINESS_PROBE
          value: '{"httpGet":{"path":"/wp-engine-health-check","port":8081,"host":"127.0.0.1","scheme":"HTTP"},"timeoutSeconds":1,"periodSeconds":1,"successThreshold":1,"failureThreshold":3}'
        - name: HOST_IP
          valueFrom:
            fieldRef:
              apiVersion: v1
              fieldPath: status.hostIP
        - name: ENABLE_HTTP2_AUTO_DETECTION
          value: "false"
        - name: ENABLE_HTTP_FULL_DUPLEX
          value: "false"
        - name: ROOT_CA
        - name: QUEUE_PROXY_TLS_MIN_VERSION
        - name: QUEUE_PROXY_TLS_MAX_VERSION
        - name: QUEUE_PROXY_TLS_CIPHER_SUITES
        - name: QUEUE_PROXY_TLS_CURVE_PREFERENCES
        - name: ENABLE_MULTI_CONTAINER_PROBES
          value: "false"
        - name: OBSERVABILITY_CONFIG
          value: '{"tracing":{"protocol":"none"},"metrics":{"protocol":"prometheus"},"runtime":{"profiling":"disabled","exportInterval":15000000000},"requestMetrics":{"protocol":"prometheus"},"EnableVarLogCollection":true,"requestLogTemplate":"{\"httpRequest\":
            {\"requestMethod\": \"{{.Request.Method}}\", \"requestUrl\": \"{{js .Request.RequestURI}}\",
            \"requestSize\": \"{{.Request.ContentLength}}\", \"status\": {{.Response.Code}},
            \"responseSize\": \"{{.Response.Size}}\", \"userAgent\": \"{{js .Request.UserAgent}}\",
            \"remoteIp\": \"{{js .Request.RemoteAddr}}\", \"serverIp\": \"{{.Revision.PodIP}}\",
            \"referer\": \"{{js .Request.Referer}}\", \"latency\": \"{{.Response.Latency}}s\",
            \"protocol\": \"{{.Request.Proto}}\", \"cfRay\": \"{{index .Request.Header
            \"Cf-Ray\"}}\"}, \"traceId\": \"{{index .Request.Header \"X-B3-Traceid\"}}\"}"}'
        image: gcr.io/knative-releases/knative.dev/serving/cmd/queue@sha256:371c23206b7bd924474b07e51435198579133eb98fba27a5cb3db9e6ba63b06b
        imagePullPolicy: IfNotPresent
        name: queue-proxy
        ports:
        - containerPort: 9090
          name: http-autometric
          protocol: TCP
        - containerPort: 9091
          name: http-usermetric
          protocol: TCP
        - containerPort: 8012
          name: queue-port
          protocol: TCP
        - containerPort: 8112
          name: https-port
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            httpHeaders:
            - name: K-Network-Probe
              value: queue
            path: /
            port: 8012
            scheme: HTTP
          periodSeconds: 1
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 250m
            ephemeral-storage: 1Gi
            memory: 100Mi
          requests:
            cpu: 25m
            ephemeral-storage: 100Mi
            memory: 100Mi
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          readOnlyRootFilesystem: true
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
      dnsConfig:
        options:
        - name: ndots
          value: "1"
      dnsPolicy: ClusterFirst
      enableServiceLinks: false
      hostAliases:
      - hostnames:
        - redis
        - mysql
        ip: 127.0.0.1
      nodeSelector:
        sandbox.gke.io/runtime: gvisor
      restartPolicy: Always
      runtimeClassName: gvisor
      schedulerName: default-scheduler
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      serviceAccount: headless-customer-workload
      serviceAccountName: headless-customer-workload
      terminationGracePeriodSeconds: 35
      tolerations:
      - effect: NoSchedule
        key: sandbox.gke.io/runtime
        operator: Equal
        value: gvisor
      topologySpreadConstraints:
      - labelSelector:
          matchLabels:
            serving.knative.dev/revision: hqbht9iojq2hf8njzu18pcsnz-00008
        maxSkew: 1
        nodeTaintsPolicy: Honor
        topologyKey: kubernetes.io/hostname
        whenUnsatisfiable: ScheduleAnyway
      - labelSelector:
          matchLabels:
            serving.knative.dev/revision: hqbht9iojq2hf8njzu18pcsnz-00008
        maxSkew: 1
        nodeTaintsPolicy: Honor
        topologyKey: topology.kubernetes.io/zone
        whenUnsatisfiable: ScheduleAnyway
      volumes:
      - emptyDir: {}
        name: knative-var-log
status:
  availableReplicas: 1
  conditions:
  - lastTransitionTime: "2026-05-19T09:29:46Z"
    lastUpdateTime: "2026-05-19T09:29:46Z"
    message: Deployment has minimum availability.
    reason: MinimumReplicasAvailable
    status: "True"
    type: Available
  - lastTransitionTime: "2026-05-19T09:29:37Z"
    lastUpdateTime: "2026-05-19T09:29:46Z"
    message: ReplicaSet "hqbht9iojq2hf8njzu18pcsnz-00008-deployment-57484dc6ff" has
      successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  observedGeneration: 1
  readyReplicas: 1
  replicas: 1
  terminatingReplicas: 0
  updatedReplicas: 1

Here's ksvc:

k get ksvc/hqbht9iojq2hf8njzu18pcsnz -n headless-customer -o yaml
apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  annotations:
    headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
    headless.wpengine.com/appName: atlas-simple-express-app-2026-03-17-1
    headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
    headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
    headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
    headless.wpengine.com/commitSHA: 707f367cb223d039c9b4a198830e16e17f806f83
    headless.wpengine.com/createdOn: "2026-05-19T09:29:36Z"
    headless.wpengine.com/default-domain: hqbht9iojq2hf8njzu18pcsnz.....nip.io
    headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
    headless.wpengine.com/envBranch: main
    headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
    headless.wpengine.com/envName: main
    headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
    headless.wpengine.com/region: US-C
    serving.knative.dev/creator: headless-api-service@wp-engine-headless-dev.iam.gserviceaccount.com
    serving.knative.dev/lastModifier: headless-api-service@wp-engine-headless-dev.iam.gserviceaccount.com
  creationTimestamp: "2026-03-17T10:42:57Z"
  generation: 8
  labels:
    CE_MITM_RESOURCE_MUTATED_ENVVARS: "true"
    headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
    headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
    headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
    headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
    headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
    headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
    headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
    headless.wpengine.com/environment-category: headless
    headless.wpengine.com/ksvc-version: 2.0.0
    headless.wpengine.com/releaseChannel: regular
    headless.wpengine.com/serviceType: headless
    jira: NEB
    owner: sec-headless
    repo: headless-infra
    slack: atlas-infrastructure-monitoring
  name: hqbht9iojq2hf8njzu18pcsnz
  namespace: headless-customer
  resourceVersion: "1779182994402623022"
  uid: e428b53c-82e2-4bae-ba88-99c752317daa
spec:
  template:
    metadata:
      annotations:
        headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
        headless.wpengine.com/appName: atlas-simple-express-app-2026-03-17-1
        headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
        headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
        headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
        headless.wpengine.com/commitSHA: 707f367cb223d039c9b4a198830e16e17f806f83
        headless.wpengine.com/createdOn: "2026-05-19T09:29:36Z"
        headless.wpengine.com/default-domain: hqbht9iojq2hf8njzu18pcsnz.....nip.io
        headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
        headless.wpengine.com/envBranch: main
        headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
        headless.wpengine.com/envName: main
        headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
        headless.wpengine.com/region: US-C
      labels:
        CE_MITM_RESOURCE_MUTATED_ENVVARS: "true"
        headless.wpengine.com/appID: ckkfi83u4xodv7ijsdg8xu12
        headless.wpengine.com/appUUID: 08599066-8cbf-2fab-27fe-89a1698511c3
        headless.wpengine.com/buildID: pjyto6ieiiyk1chln3f65i3h
        headless.wpengine.com/buildUUID: 37c7f12f-301d-b1ea-2f9a-9698afcb3af7
        headless.wpengine.com/deploymentUUID: b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
        headless.wpengine.com/envID: qbht9iojq2hf8njzu18pcsnz
        headless.wpengine.com/envUUID: 3a9315cd-2b87-0eb1-3cf3-a24536e2fc8d
        headless.wpengine.com/environment-category: headless
        headless.wpengine.com/ksvc-version: 2.0.0
        headless.wpengine.com/releaseChannel: regular
        headless.wpengine.com/serviceType: headless
        jira: NEB
        owner: sec-headless
        repo: headless-infra
        slack: atlas-infrastructure-monitoring
    spec:
      containerConcurrency: 0
      containers:
      - env:
        - (some deleted envvars)
        image: us-central1-docker.pkg.dev/wp-engine-headless-dev/customer-images/qbht9iojq2hf8njzu18pcsnz:pjyto6ieiiyk1chln3f65i3h
        imagePullPolicy: IfNotPresent
        name: user-container
        resources: {}
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
      - env:
        - (some deleted envvars)
        image: us-docker.pkg.dev/wpe-art/headless-images/headless-infra/nginx-sidecar
        imagePullPolicy: IfNotPresent
        name: nginx
        ports:
        - containerPort: 8081
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: /wp-engine-health-check
            port: 0
          periodSeconds: 1
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 250m
            memory: 500M
          requests:
            cpu: 50m
            memory: 250M
        securityContext:
          allowPrivilegeEscalation: false
          capabilities:
            drop:
            - ALL
          runAsNonRoot: true
          seccompProfile:
            type: RuntimeDefault
      securityContext:
        seccompProfile:
          type: RuntimeDefault
      serviceAccountName: headless-customer-workload
      timeoutSeconds: 35
  traffic:
  - latestRevision: true
    percent: 100
status:
  address:
    url: http://hqbht9iojq2hf8njzu18pcsnz.headless-customer.svc.cluster.local
  annotations:
    headless.wpengine.com/lastReconcileKey: 1-b4196bbb-d9d4-4a9b-a9fc-a05bc08ca94a
    headless.wpengine.com/lastReconciledRevision: hqbht9iojq2hf8njzu18pcsnz-00008
    headless.wpengine.com/latest-ready-reconciled: 1-hqbht9iojq2hf8njzu18pcsnz-00008
  conditions:
  - lastTransitionTime: "2026-05-19T09:29:46Z"
    status: "True"
    type: ConfigurationsReady
  - lastTransitionTime: "2026-05-19T09:29:48Z"
    status: "True"
    type: Ready
  - lastTransitionTime: "2026-05-19T09:29:48Z"
    status: "True"
    type: RoutesReady
  latestCreatedRevisionName: hqbht9iojq2hf8njzu18pcsnz-00008
  latestReadyRevisionName: hqbht9iojq2hf8njzu18pcsnz-00008
  observedGeneration: 8
  traffic:
  - latestRevision: true
    percent: 100
    revisionName: hqbht9iojq2hf8njzu18pcsnz-00008
  url: http://hqbht9iojq2hf8njzu18pcsnz.34.68.143.237.nip.io

Edit: here's kubernetes version:

kubectl version
Client Version: v1.35.3-dispatcher
Kustomize Version: v5.7.1
Server Version: v1.35.3-gke.1943000

Edit 2: We don't use kyverno/OPA

[linkvt]

Hi, thanks for these resources, that gave me a hint. Not sure though if any of these IPs etc. are confidential in case you want to scratch them from the comment and its history.

It seems you have something mutating the resources based on the label I see: CE_MITM_RESOURCE_MUTATED_ENVVARS: "true"
We changed how scaling happens in 1.22, see #16540 . We dont modify the whole deployment anymore but just the scale subresource.
I assume your mutating webhook can't handle this payload (which has no labels field) but expects the deployment when modifying labels causing the error you see.

Can you verify that?

[dprotaso]

I assume your mutating webhook can't handle this payload (which has no labels field) but expects the deployment when modifying labels causing the error you see.

We only PATCH the scale subresource's replica count. What's your webhook config?

[sebastianpawlak]

It seems that the problem was our admission controller, which wanted to modify spec based on specific labels/annotations. And the scale subresource doesn't have such metadata. We modified the admission controller and there are no more scaling errors in activator.

We're going to double check that tomorrow and we'll let you know. Thanks!

[dprotaso]

Thanks @sebastianpawlak

Can you confirm a few things for posterity:

1. The webhook rule you had that was intercepting the update to the /scale subresource
2. What's the payload of the admissionreview? I'm guessing your webhook thought it would always be a deployment but was it autoscalingv1.Scale object?

[sebastianpawlak]

@dprotaso

1. Here's our rule before fixing the webhook:

  rules:
  - apiGroups:
    - apps
    apiVersions:
    - v1
    operations:
    - CREATE
    - UPDATE
    resources:
    - deployments/*

2. We were unmarshalling the payload to Deployment object.

In the fix we changed 1. to accept only deployments and not subresources:

    resources:
    - deployments

also we changed 2. to not process subresources if admission review request's SubResource != ""

[dprotaso]

Thanks for circling back - going to close this out