Skip to content

feat(bigquery): support GCP workload identity - #4607

Draft
jkaflik wants to merge 3 commits into
mainfrom
kk/clp-962-bigquery-workload-identity
Draft

feat(bigquery): support GCP workload identity#4607
jkaflik wants to merge 3 commits into
mainfrom
kk/clp-962-bigquery-workload-identity

Conversation

@jkaflik

@jkaflik jkaflik commented Jul 21, 2026

Copy link
Copy Markdown
Contributor

Problem

BigQuery peers currently require long-lived service-account keys. ClickPipes needs a keyless mode where a single-tenant PeerDB deployment uses its projected Kubernetes token to impersonate its deployment-scoped tenant GSA.

Solution

  • recognize service_account_workload_identity through the existing BigQuery auth_type field
  • build external-account credentials from trusted deployment configuration and fixed Google STS/IAM endpoints
  • preserve ambient ADC for internal platform access and preserve the existing service-account-key path
  • refresh projected tokens and impersonated credentials without restarting workers
  • reject missing or unknown authentication modes with actionable errors

Target service account and cluster context remain deployment configuration; peers only select the authentication mode.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Deprecated destination connector

This issue or pull request relates to a deprecated destination connector (Snowflake, BigQuery, ElasticSearch, Kafka, Redpanda, Confluent, Azure Event Hubs, Google Pub/Sub, or S3).

These destinations are no longer actively maintained, but remain functional. We are unlikely to prioritize new work here.

Note: BigQuery is deprecated only as a destination — it remains a supported source.

If you depend on one of these connectors, we recommend:

  • Pin to a known-good PeerDB version so behavior stays stable.
  • Fork the repository if you need to carry your own changes.

See the deprecated connectors documentation for details and migration guidance.

@jkaflik jkaflik left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review follow-ups addressed.

Comment thread .env.example Outdated
Comment thread flow/connectors/bigquery/auth.go
@jkaflik

jkaflik commented Jul 29, 2026

Copy link
Copy Markdown
Contributor Author

Published the current PR head (d57cba6e) to customer-clp-962 to produce testable GHCR images.

Customer Docker workflow completed successfully. Images are available with the tags latest-clp-962 and clp-962-d57cba6e, for example ghcr.io/peerdb-io/flow-worker:clp-962-d57cba6e.

workloadIdentityTokenFileEnv: "/token",
}
config, err := resolveWorkloadIdentityDeploymentConfig(t.Context(), workloadIdentityConfigSource{
lookupEnv: func(name string) (string, bool) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks a bit strange to thread a stdlib function as a callback. Can the same be achieved with t.Setenv()?

Comment on lines +55 to +56
projectID func(context.Context) (string, error)
instanceAttributeValue func(context.Context, string) (string, error)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Have you explored GCE_METADATA_HOST? Callbacks are a bit hard on the brain

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants