ClientID cannot be specified by user with Hydra v2.0.1 (incompatible with v2)

[nbrns]

Preflight checklist

• I could not find a solution in the existing issues, docs, nor discussions.
• I agree to follow this project's Code of Conduct.
• I have read and am following this repository's Contribution Guidelines.
• This issue affects my Ory Network project.
• I have joined the Ory Community Slack.
• I am signed up to the Ory Security Patch Newsletter.

Describe the bug

I deploy Hydra v2.0.1 to k8s via Helm and I really love the idea to create OAuth2 clients via a CR.
However, with Hydra v2 it is no longer possible to change

ERROR	controllers.OAuth2Client	error processing client authentication-broker-oauth2-client/taal-evolution 	{"oauth2client": "register", "error": "POST http://authentication-broker-hydra-admin:4445/admin/clients http request returned unexpected status code: 400 Bad Request"}

on Hydra side:

level=info msg=An error occurred while handling a request audience=application error=map[debug: message:The request was malformed or contained invalid parameters reason:It is no longer possible to set an OAuth2 Client ID as a user. The system will generate a unique ID for you. status:Bad Request status_code:400] http_request=map[headers:map[accept:application/json accept-encoding:gzip content-length:407 content-type:application/json user-agent:Go-http-client/1.1] host:authentication-broker-hydra-admin:4445 method:POST path:/admin/clients query:<nil> remote:10.244.5.33:44274 scheme:http] http_response=map[status_code:400] service_name=Ory Hydra service_version=v2.0.1

Reproducing the bug

Deploy Hydra > v.2.0.0 and create a OAuth2Client CR for reconciliation by Hydra-Maester

Relevant log output

No response

Relevant configuration

No response

Version

v0.0.26

On which operating system are you observing this issue?

No response

In which environment are you deploying?

No response

Additional Context

No response

[Demonsthere]

Hello there,
Thanks for the report. Did you verify if the proper endpoint is being used by the controller? We had a similar case reported here ory/k8s#543

Hello there, Thanks for the report. Did you verify if the proper endpoint is being used by the controller? We had a similar case reported here ory/k8s#543

I did very to hit correct end point. I started suddenly getting this error probably after getting a hydra chart update.
If you run a sample below which includes client_id the request will fail with the reported error. Remove client_id & it works.

My logs clearly show same error

Sample of a failing request is below

curl -X POST \
  'http://localhost:4203/admin/clients' \
  --header 'Accept: */*' \
  --header 'Content-Type: application/json' \
  --data-raw '{
  "client_id": "asdasdasdasd",
  "audience": [
    "toggleblock-user",
    "toggleblock-admin"
  ],
  "backchannel_logout_session_required": true,
  "client_secret_expires_at": 0,
  "frontchannel_logout_session_required": true,
  "grant_types": [
    "client_credentials",
    "implicit",
    "authorization_code",
    "refresh_token"
  ],
  "post_logout_redirect_uris": [
    "https://www.toggleblock.local:4200"
  ],
  "redirect_uris": [
    "https://www.toggleblock.local:4200"
  ],
  "response_types": [
    "id_token",
    "code",
    "token"
  ],
  "scope": "openid offline_access",
  "token_endpoint_auth_method": "none"
}'

[yasha-spare]

I'm seeing this error too. My Hydra Maester logs look like:

2022-12-07T18:57:34.134Z	INFO	controllers.OAuth2Client	using default client
2022-12-07T18:57:34.138Z	INFO	controllers.OAuth2Client	using default client
2022-12-07T18:57:34.146Z	INFO	controllers.OAuth2Client	using default client
2022-12-07T18:57:34.148Z	ERROR	controllers.OAuth2Client	error processing client oauth2-client-matching-service/main 	{"oauth2client": "register", "error": "POST http://hydra-admin:4445/admin/clients http request returned unexpected status code: 400 Bad Request"}
github.com/go-logr/zapr.(*zapLogger).Error
	/go/pkg/mod/github.com/go-logr/zapr@v0.2.0/zapr.go:132
github.com/ory/hydra-maester/controllers.(*OAuth2ClientReconciler).updateReconciliationStatusError
	/go/src/github.com/ory/hydra-maester/controllers/oauth2client_controller.go:367
github.com/ory/hydra-maester/controllers.(*OAuth2ClientReconciler).registerOAuth2Client
	/go/src/github.com/ory/hydra-maester/controllers/oauth2client_controller.go:271
github.com/ory/hydra-maester/controllers.(*OAuth2ClientReconciler).Reconcile
	/go/src/github.com/ory/hydra-maester/controllers/oauth2client_controller.go:238
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:298
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:253
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func1.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.8.3/pkg/internal/controller/controller.go:216
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.BackoffUntil.func1
	/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:155
k8s.io/apimachinery/pkg/util/wait.BackoffUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:156
k8s.io/apimachinery/pkg/util/wait.JitterUntil
	/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:133
k8s.io/apimachinery/pkg/util/wait.JitterUntilWithContext
	/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:185
k8s.io/apimachinery/pkg/util/wait.UntilWithContext
	/go/pkg/mod/k8s.io/apimachinery@v0.20.2/pkg/util/wait/wait.go:99

And the corresponding Hydra logs look like:

time=2022-12-07T18:50:02Z level=info msg=An error occurred while handling a request audience=application error=map[debug: message:The request was malformed or contained invalid parameters reason:It is no longer possible to set an OAuth2 Client ID as a user. The system will generate a unique ID for you. status:Bad Request status_code:400] http_request=map[headers:map[accept:application/json accept-encoding:gzip content-length:256 content-type:application/json user-agent:Go-http-client/1.1] host:hydra-admin:4445 method:POST path:/admin/clients query:<nil> remote:<ip_redacted>:44702 scheme:http] http_response=map[status_code:400] service_name=Ory Hydra service_version=v2.0.1

Seems like Hydra Maester is trying to POST path:/admin/clients while providing a client id, which is not valid in Hydra 2.

I have Hydra Maester and Hydra both installed with 0.26.4 of the Hydra Helm chart, which installs Hydra v2.0.1 and Hydra Maester v0.0.25. 0.26.4 is the latest version of the Helm charts, but I was getting the same errors on 0.26.3.
My K8s config for the OAuth2Clients looks like:

apiVersion: hydra.ory.sh/v1alpha1
kind: OAuth2Client
metadata:
  name: oauth2-client-matching-service
spec:
  grantTypes:
    - client_credentials
  scope: serviceToService:*
  metadata:
    serviceName: matching-service
  secretName: oauth2-client-matching-service

@yasha-spare Do not use Hydra Maester for now. it is clearly out of sync with the Hydra APIs in the latest Helm. Just register your clients manually by running the curl I posted

[yasha-spare]

Ah, is it actually the case that Hydra Maester just doesn't work with Hydra 2? Can any maintainers comment ... @Demonsthere or @aeneasr maybe?

Ah, is it actually the case that Hydra Maester just doesn't work with Hydra 2? Can any maintainers comment ... @Demonsthere maybe?

My logs indicated it is pretty clearly the case ... actually had a log saying client_id cannot be set manually

[yasha-spare]

using default client

Mine suggest that too (see logs above), but I also noticed this log using default client, which is maybe suspicious/indicates misconfiguration or something?

FWIW, I'm happy to hop on a video call to debug together, if that would help for any maintainers looking at this.

[nbrns]

Yes, it is no longer supported to set the client ID.
See here: https://www.ory.sh/oauth2-server-openid-ory-hydra-v2#primary-keys

[yasha-spare]

Yes, it is no longer supported to set the client ID. See here: https://www.ory.sh/oauth2-server-openid-ory-hydra-v2#primary-keys

Yeah, I know with Hydra v2 you can't provide your own client ids, and it does seem that's what Hydra Maester is still trying to do. What's not clear to me is whether or not there's some way to make Hydra Maester work with Hydra v2? Or does it only work with Hydra v1?

[Demonsthere]

Hello there,
Yes, the maester controller is outdated and we were not able to patch it to support the changes for hydra v2.0 😞. The support for the project is not dropped, but we simply lack the man-power to work on it right now, as the controller needs to be rebuild with the new kubebuilder and for the new api changes.
Sadly, for now I would not recommend using it for hydra v2.0.
My apologies 😞

[aeneasr]

We always accept contritbutions though :) :)

[yasha-spare]

Cool, ty for confirming, and totally get that there's always a lack of dev resources :)

Spoke with my team today, we considered taking this on and pushing a contribution upstream, but ultimately decided against it - we don't have the dev resources either! Would take us awhile to ramp up as we aren't a Go shop, have no experience with writing K8s CRDs/controllers, no experience with kubebuilder, etc.

If anyone else is curious about our workaround/Hydra Maester alternative:

• We're using Hydra for secure service-to-service communication
  • i.e. service A gets a Hydra token (client credentials flow) proving it's service A, attaches that token when calling service B, and service B validates it against Hydra
• We have a lot of services, and a lot of different environments, so manual management of Hydra clients isn't a great option
• However, we already use ArgoCD, which has these PreSync hooks that run before a service is deployed. So we're just going to write a PreSync job to create Hydra clients and store the id/secret in a K8s secret, if the K8s secret doesn't already exist
  • Ends up similar in practice to the Maester workflow, but should be simpler for our use case

[dan2kx]

As long as the k8 secret doesnt exist already, when you specify the secretName in the CR, hydra will create you a random client_id and client_secret which will then be saved to the k8 secret (named whatever you specified) so it is still possible to use the maestor with v2 as long as you dont mind what the client_id/secret are