dynu with cert-manager 1.9.1 on arm: RBACs problem

[rbaumgar]

I have installed cert-manager 1.9.1
dynu latest version on arm.

dynu-webhook log

I0829 19:50:00.984703       1 requestheader_controller.go:169] Starting RequestHeaderAuthRequestController
I0829 19:50:00.984990       1 shared_informer.go:255] Waiting for caches to sync for RequestHeaderAuthRequestController
I0829 19:50:00.984761       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file"
I0829 19:50:00.985142       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0829 19:50:00.984774       1 configmap_cafile_content.go:202] "Starting controller" name="client-ca::kube-system::extension-apiserver-authentication::client-ca-file"
I0829 19:50:00.986033       1 shared_informer.go:255] Waiting for caches to sync for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
I0829 19:50:00.986042       1 dynamic_serving_content.go:132] "Starting controller" name="serving-cert::/tls/tls.crt::/tls/tls.key"
I0829 19:50:00.985992       1 tlsconfig.go:240] "Starting DynamicServingCertificateController"
I0829 19:50:00.985932       1 secure_serving.go:210] Serving securely on [::]:10250
I0829 19:50:00.987461       1 apf_controller.go:317] Starting API Priority and Fairness config controller
W0829 19:50:00.997963       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0829 19:50:00.998303       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: flowschemas.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "flowschemas" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
W0829 19:50:01.004129       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0829 19:50:01.004464       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
I0829 19:50:01.086246       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::requestheader-client-ca-file
I0829 19:50:01.086270       1 shared_informer.go:262] Caches are synced for RequestHeaderAuthRequestController
I0829 19:50:01.086342       1 shared_informer.go:262] Caches are synced for client-ca::kube-system::extension-apiserver-authentication::client-ca-file
W0829 19:50:02.154477       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope
E0829 19:50:02.154663       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: prioritylevelconfigurations.flowcontrol.apiserver.k8s.io is forbidden: User "system:serviceaccount:cert-manager:cert-manager-dynu-webhook" cannot list resource "prioritylevelconfigurations" in API group "flowcontrol.apiserver.k8s.io" at the cluster scope

SA system:serviceaccount:cert-manager:cert-manager-dynu-webhook has not sufficient rights for flowcontrol.apiserver.k8s.io

[rbaumgar]

found similar issue puzzle/cert-manager-webhook-dnsimple#8

[anon-software]

I believe I saw these messages in my installation too. I shall be back home to confirm that in a couple of days. However, they seem inconsequential as my system was able to get the certificate signed successfully and I did not have time to look into them further. Did it work for you otherwise?

[rbaumgar]

I was able to fix the RBAC problem by applying the following clusterrole and clusterrolebinding

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  annotations:
    meta.helm.sh/release-name: cert-manager-dynu-webhook
    meta.helm.sh/release-namespace: cert-manager
  labels:
    app: dynu-webhook
    app.kubernetes.io/managed-by: Helm
    chart: dynu-webhook-0.1.1
    heritage: Helm
    release: cert-manager-dynu-webhook
  name: cert-manager-dynu-webhook:flowcontrol-solver
rules:
  - apiGroups:
      - "flowcontrol.apiserver.k8s.io"
    resources:
      - 'prioritylevelconfigurations'
      - 'flowschemas'
    verbs:
      - 'list'
      - 'watch'
---      
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  annotations:
    meta.helm.sh/release-name: cert-manager-dynu-webhook
    meta.helm.sh/release-namespace: cert-manager
  labels:
    app: dynu-webhook
    app.kubernetes.io/managed-by: Helm
    chart: dynu-webhook-0.1.1
    heritage: Helm
    release: cert-manager-dynu-webhook
  name: cert-manager-dynu-webhook:flowcontrol-solver
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cert-manager-dynu-webhook:flowcontrol-solver
subjects:
- kind: ServiceAccount
  name: cert-manager-dynu-webhook
  namespace: cert-manager

now I have the following error messages

W0829 21:24:32.439246       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.PriorityLevelConfiguration: the server could not find the requested resource
E0829 21:24:32.439742       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.PriorityLevelConfiguration: failed to list *v1beta2.PriorityLevelConfiguration: the server could not find the requested resource
W0829 21:25:00.737869       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: failed to list *v1beta2.FlowSchema: the server could not find the requested resource
E0829 21:25:00.737959       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.2/tools/cache/reflector.go:167: Failed to watch *v1beta2.FlowSchema: failed to list *v1beta2.FlowSchema: the server could not find the requested resource

[rbaumgar]

I found now that I am using dynu version 0.1.1 and 0.1.2. Installed 2 hours ago.
Any idea why?

NAME                     	NAMESPACE   	REVISION	UPDATED                                	STATUS  	CHART             	APP VERSION
cert-manager-dynu-webhook	cert-manager	1       	2022-08-29 19:49:30.930351429 +0000 UTC	deployed	dynu-webhook-0.1.1	1.0        

[anon-software]

The only difference between these two versions is the readme file. I cannot comment on the new errors you see now.

[rbaumgar]

both objects exist, but I don't know which name the webhook is looking for and who should create it...

# kubectl get prioritylevelconfiguration
NAME              TYPE      ASSUREDCONCURRENCYSHARES   QUEUES   HANDSIZE   QUEUELENGTHLIMIT   AGE
catch-all         Limited   5                          <none>   <none>     <none>             2d14h
exempt            Exempt    <none>                     <none>   <none>     <none>             2d14h
global-default    Limited   20                         128      6          50                 2d14h
leader-election   Limited   10                         16       4          50                 2d14h
system            Limited   30                         64       6          50                 2d14h
workload-high     Limited   40                         128      6          50                 2d14h
workload-low      Limited   100                        128      6          50                 2d14h
[root@microshift ~]# kubectl get flowschema
NAME                           PRIORITYLEVEL     MATCHINGPRECEDENCE   DISTINGUISHERMETHOD   AGE     MISSINGPL
exempt                         exempt            1                    <none>                2d14h   False
system-leader-election         leader-election   100                  ByUser                2d14h   False
workload-leader-election       leader-election   200                  ByUser                2d14h   False
system-nodes                   system            500                  ByUser                2d14h   False
kube-controller-manager        workload-high     800                  ByNamespace           2d14h   False
kube-scheduler                 workload-high     800                  ByNamespace           2d14h   False
kube-system-service-accounts   workload-high     900                  ByNamespace           2d14h   False
service-accounts               workload-low      9000                 ByUser                2d14h   False
global-default                 global-default    9900                 ByUser                2d14h   False
catch-all                      catch-all         10000                ByUser                2d14h   False

[rbaumgar]

very interesting! Audit log shows following error

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"df17037b-902f-4779-8dae-d49b4e8699c0","stage":"ResponseComplete","requestURI":"/apis/flowcontrol.apiserver.k8s.io/v1beta2/flowschemas?limit=500\u0026resourceVersion=0","verb":"list","user":{"username":"system:serviceaccount:cert-manager:cert-manager-dynu-webhook","uid":"890b445d-a09a-4037-a443-65081814a78b","groups":["system:serviceaccounts","system:serviceaccounts:cert-manager","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["cert-manager-dynu-webhook-79f445dfd4-dr285"],"authentication.kubernetes.io/pod-uid":["751180b3-ba97-4f8d-abfb-dc81b103791b"]}},"sourceIPs":["10.42.0.10"],"userAgent":"webhook/v0.0.0 (linux/arm64) kubernetes/$Format","objectRef":{"resource":"flowschemas","apiGroup":"flowcontrol.apiserver.k8s.io","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2022-08-30T13:12:58.533325Z","stageTimestamp":"2022-08-30T13:12:58.577099Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "cert-manager-dynu-webhook:flowcontrol-solver" of ClusterRole "cert-manager-dynu-webhook:flowcontrol-solver" to ServiceAccount "cert-manager-dynu-webhook/cert-manager""}}

{"kind":"Event","apiVersion":"audit.k8s.io/v1","level":"Metadata","auditID":"a2332bde-38d9-4fca-8beb-58a9accb70be","stage":"ResponseComplete","requestURI":"/apis/flowcontrol.apiserver.k8s.io/v1beta2/prioritylevelconfigurations?limit=500\u0026resourceVersion=0","verb":"list","user":{"username":"system:serviceaccount:cert-manager:cert-manager-dynu-webhook","uid":"890b445d-a09a-4037-a443-65081814a78b","groups":["system:serviceaccounts","system:serviceaccounts:cert-manager","system:authenticated"],"extra":{"authentication.kubernetes.io/pod-name":["cert-manager-dynu-webhook-79f445dfd4-dr285"],"authentication.kubernetes.io/pod-uid":["751180b3-ba97-4f8d-abfb-dc81b103791b"]}},"sourceIPs":["10.42.0.10"],"userAgent":"webhook/v0.0.0 (linux/arm64) kubernetes/$Format","objectRef":{"resource":"prioritylevelconfigurations","apiGroup":"flowcontrol.apiserver.k8s.io","apiVersion":"v1beta2"},"responseStatus":{"metadata":{},"status":"Failure","reason":"NotFound","code":404},"requestReceivedTimestamp":"2022-08-30T13:13:50.036863Z","stageTimestamp":"2022-08-30T13:13:50.145252Z","annotations":{"authorization.k8s.io/decision":"allow","authorization.k8s.io/reason":"RBAC: allowed by ClusterRoleBinding "cert-manager-dynu-webhook:flowcontrol-solver" of ClusterRole "cert-manager-dynu-webhook:flowcontrol-solver" to ServiceAccount "cert-manager-dynu-webhook/cert-manager""}}

[rbaumgar]

oh I found the reason for the errors.
I have only version v1beta1 of flowcontrol, but the pod is looking for version v1beta2

# kubectl api-resources --api-group=flowcontrol.apiserver.k8s.io
NAME                          SHORTNAMES   APIVERSION                             NAMESPACED   KIND
flowschemas                                flowcontrol.apiserver.k8s.io/v1beta1   false        FlowSchema
prioritylevelconfigurations                flowcontrol.apiserver.k8s.io/v1beta1   false        PriorityLevelConfiguration

[davlloyd]

This has been fixed in latest update so can be closed