Skip to content

Knative operator is active while knative-serving installation failed #278

Open
Open
Knative operator is active while knative-serving installation failed#278
Labels
bugSomething isn't working

Description

@Deezzir
Deezzir
opened on Jan 28, 2025

Bug Description

The knative-serving charm is in active state while the logs indicate there is an unsupported version of K8s in the cluster. It might be a bad UX design, because nothing is deployed and the charm does not complain about it.

To Reproduce

Deploy Kubeflow charm 1.9 in EKS cluster with K8s version 1.25

Environment

Deploy Kubeflow charm 1.9 in EKS cluster with K8s version 1.25

Relevant Log Output

2025-01-15T11:10:37.013Z [knative-operator] {"severity":"EMERGENCY","timestamp":"2025-01-15T11:10:37.013411678Z","logger":"knative-operator","caller":"sharedmain/main.go:386","message":"Version c
heck failed","commit":"fbc6e15","knative.dev/pod":"knative-operator","error":"kubernetes version \"1.25.16-eks-2d5f260\" is not compatible, need at least \"1.26.0-0\" (this can be overridden with
 the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/share
dmain/main.go:386\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:256\nknative.dev/pkg/injection/sharedmain.
MainWithContext\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:210\nknative.dev/pkg/injection/sharedmain.Main\n\t/root/parts/operator/build/vendor/knative.dev/p
kg/injection/sharedmain/main.go:140\nmain.main\n\t/root/parts/operator/build/cmd/operator/main.go:26\nruntime.main\n\t/snap/go/10335/src/runtime/proc.go:250"}

Additional Context

No response

Activity

  1. Deezzir
    added
    bugSomething isn't working
    on Jan 28, 2025

  2. syncronize-issues-to-jira commented on Jan 28, 2025

    @syncronize-issues-to-jira
    syncronize-issues-to-jirabot
    on Jan 28, 2025 – with Syncronize issues to Jira

    Thank you for reporting your feedback to us!

    The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-6794.

    This message was autogenerated

  3. orfeas-k commented on Jan 29, 2025

    @orfeas-k
    orfeas-k
    on Jan 29, 2025
    Contributor

    Discussed with the team, we should evaluate what is the expected behaviour of the upstream controller when it logs a SEVERITY message? In this case, we 'll be able to know better if this is something that we need pebble or charm to handle.

  4. kimwnasptd commented on Jan 30, 2025

    @kimwnasptd
    kimwnasptd
    on Jan 30, 2025
    Contributor

    I tried to look into the behaviour of the upstream images.

    The following are the case in the upstream image:

    1. We see the same logs on the upstream knative-operator Deployment, when we use an incompatible version of K8s
    2. The container crashes and the Pod goes to Error state
    kubectl get pods -n knative-operator knative-operator-7b5d6b545f-4hk5c
NAME                                READY   STATUS             RESTARTS      AGE
knative-operator-7b5d6b545f-4hk5c   0/1     CrashLoopBackOff   6 (94s ago)   7m28s

    And the logs are:

    kubectl logs -n knative-operator knative-operator-7b5d6b545f-4hk5c
2025/01/30 10:45:48 Registering 2 clients
2025/01/30 10:45:48 Registering 3 informer factories
2025/01/30 10:45:48 Registering 3 informers
2025/01/30 10:45:48 Registering 2 controllers
{"severity":"INFO","timestamp":"2025-01-30T10:45:48.884580007Z","logger":"knative-operator","caller":"profiling/server.go:65","message":"Profiling enabled: false","commit":"4959713-dirty","knative.dev/pod":"knative-operator-7b5d6b545f-4hk5c"}
{"severity":"EMERGENCY","timestamp":"2025-01-30T10:45:48.885154491Z","logger":"knative-operator","caller":"sharedmain/main.go:390","message":"Version check failed","commit":"4959713-dirty","knative.dev/pod":"knative-operator-7b5d6b545f-4hk5c","error":"kubernetes version \"1.25.16\" is not compatible, need at least \"1.30.0-0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/injection/sharedmain/main.go:390\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/injection/sharedmain/main.go:255\nknative.dev/pkg/injection/sharedmain.MainWithContext\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/injection/sharedmain/main.go:209\nknative.dev/pkg/injection/sharedmain.Main\n\tknative.dev/pkg@v0.0.0-20250117084104-c43477f0052b/injection/sharedmain/main.go:140\nmain.main\n\tknative.dev/operator/cmd/operator/main.go:26\nruntime.main\n\truntime/proc.go:272"}

  5. kimwnasptd commented on Jan 30, 2025

    @kimwnasptd
    kimwnasptd
    on Jan 30, 2025
    Contributor

    We can reproduce the issue with the following:

    sudo snap install microk8s --classic --channel=1.25/stable
sudo microk8s kubectl config view --raw > $HOME/.kube/config
kubectl apply -f https://github.com/knative/operator/releases/download/knative-v1.17.0/operator.yaml

  6. kimwnasptd commented on Jan 30, 2025

    @kimwnasptd
    kimwnasptd
    on Jan 30, 2025
    Contributor

    In the charm, I see that the process fails and Pebble never starts it again. Most probably because it failed in less that 1 second

    2025-01-30T11:10:38.278Z [pebble] HTTP API server listening on ":38813".
2025-01-30T11:10:38.278Z [pebble] Started daemon.
2025-01-30T11:10:50.854Z [pebble] GET /v1/plan?format=yaml 372.961µs 200
2025-01-30T11:10:50.855Z [pebble] POST /v1/layers 174.951µs 200
2025-01-30T11:10:50.865Z [pebble] POST /v1/services 5.71997ms 202
2025-01-30T11:10:50.872Z [pebble] Service "knative-operator" starting: /ko-app/operator
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 2 clients
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 3 informer factories
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 3 informers
2025-01-30T11:10:50.889Z [knative-operator] 2025/01/30 11:10:50 Registering 2 controllers
2025-01-30T11:10:50.895Z [knative-operator] {"severity":"INFO","timestamp":"2025-01-30T11:10:50.895659082Z","logger":"knative-operator","caller":"profiling/server.go:65","message":"Profiling enabled: false","commit":"fbc6e15","knative.dev/pod":"knative-operator"}
2025-01-30T11:10:50.896Z [knative-operator] {"severity":"EMERGENCY","timestamp":"2025-01-30T11:10:50.896234204Z","logger":"knative-operator","caller":"sharedmain/main.go:386","message":"Version check failed","commit":"fbc6e15","knative.dev/pod":"knative-operator","error":"kubernetes version \"1.25.16\" is not compatible, need at least \"1.26.0-0\" (this can be overridden with the env var \"KUBERNETES_MIN_VERSION\")","stacktrace":"knative.dev/pkg/injection/sharedmain.CheckK8sClientMinimumVersionOrDie\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:386\nknative.dev/pkg/injection/sharedmain.MainWithConfig\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:256\nknative.dev/pkg/injection/sharedmain.MainWithContext\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:210\nknative.dev/pkg/injection/sharedmain.Main\n\t/root/parts/operator/build/vendor/knative.dev/pkg/injection/sharedmain/main.go:140\nmain.main\n\t/root/parts/operator/build/cmd/operator/main.go:26\nruntime.main\n\t/snap/go/10335/src/runtime/proc.go:250"}
2025-01-30T11:10:50.916Z [pebble] GET /v1/changes/1/wait?timeout=4.000s 50.048053ms 200
2025-01-30T11:10:59.298Z [pebble] GET /v1/plan?format=yaml 128.13µs 200
2025-01-30T11:11:08.315Z [pebble] GET /v1/notices?timeout=30s 30.008120623s 200
2025-01-30T11:11:12.713Z [pebble] GET /v1/plan?format=yaml 114.713µs 200
2025-01-30T11:11:14.508Z [pebble] GET /v1/plan?format=yaml 115.518µs 200
2025-01-30T11:11:18.235Z [pebble] GET /v1/plan?format=yaml 106.775µs 200
2025-01-30T11:11:21.270Z [pebble] GET /v1/plan?format=yaml 117.282µs 200
2025-01-30T11:11:24.295Z [pebble] GET /v1/plan?format=yaml 135.279µs 200

  7. orfeas-k commented on Feb 13, 2025

    @orfeas-k
    orfeas-k
    on Feb 13, 2025
    Contributor

    Tested with 3.6 and it looks like the service keeps restarting every 30 seconds (the restart issue was fixed in 3.6.1). What I 'd suggest here would be to add a health check to ensure the service is up and running and leverage PebbleCheckFailedEvent to put the charm in Error state.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions