maxConcurrency with Websocket connections

[sjaakiejj]

Hi all,

We've got a system where we run a fairly CPU-heavy process as part of incoming websocket requests. To ensure the system both protects itself from running out of resources and can handle sudden high loads, we're looking into Knative Serving as a potential solution, given that the containerConcurrency property seems to do exactly what we need.

Effectively we're looking to limit each container to 10 active websocket connections and force the auto scaler to add new instances if we need more. We've setup our deployment yaml as follows:

apiVersion: serving.knative.dev/v1
kind: Service
metadata:
  name: socket-service
  namespace: sockets

spec:
  template:
    metadata:
      annotations:
        autoscaling.knative.dev/metric: "concurrency"
        autoscaling.knative.dev/target-utilization-percentage: "80"
        autoscaling.knative.dev/min-scale: '1'
        autoscaling.knative.dev/max-scale: '10'
        autoscaling.knative.dev/target: '1'

      labels:
        app: socket-service
    spec:
      containerConcurrency: 10
      timeoutSeconds: 600
      containers:
      - name: socket-service
        image: my-socket-image:latest
        imagePullPolicy: Always

        ports:
        - containerPort: 8080

However, when applying a load of 40 active socket connections to this service, all of them go to the same pod. I checked netstat to get the number of active connections in the pod, and it returned 210 ESTABLISHED records on port 8080.

Displaying the list of pods, I get this:

NAME                                                  READY   STATUS    RESTARTS   AGE
pod/socket-service-00001-deployment-cf64c4bd5-npdcj   2/2     Running   0          9h

It seems no matter how much load I throw at it, it refuses to scale. There are no errors in the autoscaler logs but also no indications that it attempts to scale. I feel like I'm missing something obvious, but I can't figure it out based on the documentation.

Is this expected to work the way I think it should? What's the best way to debug issues like this?

[BennyG93]

@sjaakiejj - Did you find a solution to this? I have a websocket application and I need to guarantee a 1:1 ratio, 1 client to 1 pod.

I was hoping the Knative with containerConcurrency was the answer.

[dprotaso]

@BennyG93 containerConcurrency should be the answer.

But you'll want our activator component to always be in the request path - it performs better load balancing for lower values of concurrency

You can configure that using target burst capacity
https://knative.dev/docs/serving/load-balancing/target-burst-capacity/#setting-the-target-burst-capacity