SASL/OAUTHBEARER Fails for GKE Workload Identity to Managed Kafka

[solarpunkin]

ERROR: The connection consistently fails during the SASL handshake, resulting in either a client-side KafkaError{code=_MSG_TIMED_OUT} or a broker-side SaslAuthenticationException: Authentication failed due to invalid credentials.

WIF is enabled on the cluster (workload_pool is set) and the node pool (mode: GKE_METADATA). A ServiceAccount is annotated with iam.gke.io/return-principal-id-as-email: “true” and is used by the application pods. A Google Service Account (GSA) has the roles/managedkafka.client and roles/iam.serviceAccountTokenCreator roles. The Kubernetes Service Account is bound to this GSA with the roles/iam.workloadIdentityUser role. The Python client is configured for SASL_SSL with sasl.mechanism: OAUTHBEARER and uses google.auth.default() within an oauth_cb callback to provide the authentication token.

I have validated network connectivity (via socket tests), all IAM/GKE/K8s policies and annotations, and Kafka ACLs (including User:*)—every component is correctly configured as per the docs.

the authentication fails with both the direct federation (return-principal-id-as-email annotation) and the KSA-to-GSA impersonation (gcp-service-account annotation) methods of Workload Identity.

[eshanp32]

Hi @solarpunkin,
currently facing the same issue while integrating in go, did you find a workaround for this?
or is using SASL/PLAIN the only way forward

[solarpunkin]

I had to compromise with Pub/Sub. SASL/PLAIN is okay for development purpose. Seems like GCP has given up on maintaining its resources.

[eshanp32]

Yeah, seems like it; No maintenance of this repository from the gcp team.
Well thanks for the response - I’ll look into alternative solutions on my side.

[kir-titievsky-google]

@solarpunkin sorry for leaving you without a response. I suspect that you ran into the issue because you were using a Google.auth.default() as the auth callback. Kafka requires a particular format for its token. You can see a python example here.

@eshanp32 the same goes for Go. We don't have a full public implementation of the token provider in Go, but Gemini is quite good at translating python & Java for this. Please reach out through a support case or kafka-hotline@google.com (best effort) if you need help getting this token provider implemented.