Task: Decide on how to implement dynamic ClusterConfig references

[Diaphteiros]

Understand the Task

Description

This is basically a follow-up for the discussion from #437. See there for further details.
We need to decide on how we get dynamic (= different from each cluster which are created from the same template) references to ClusterConfig resources.

The goal of this task is to have a discussion in which we decide on how to go forward here.

Option 1: Mutating Webhook

Comes with the usual webhook-related problems (complexity for setup and debugging, requires networking, etc.). Also, injecting fields into the spec causes problems with gitops.

Option 2: Templating in Scheduler templates

We allow some kind of templating in the scheduler config where the Cluster templates are defined. Comes with the usual drawbacks of templating: unintuitive, hard to debug, and (depending on the templating language) probably not yaml-compliant.

Option 3: Magic Prefixes

While foo refers to a ClusterConfig named foo, something like id:foo could refer to a ClusterConfig whose name is a hash made from foo and the Cluster's name, allowing static templates which still refer to different resources in the cluster.
Drawback is that we would need some helper method which is used by every controller that resolves these references or creates ClusterConfigs that should be targeted by it, and the connection is rather implicit.

Option 4: IPAM logic in Gardener ClusterProvider

We could avoid this topic (in fact, the whole PlatformService) completely if we moved the IPAM logic into the Gardener ClusterProvider. There are several reasons why I initially wanted to avoid this:

• The Gardener ClusterProvider is already very complex and this IPAM topic is also not trivial, so it would add another large, complex chunk to the repo which makes maintenance more difficult.
• Most environments support VPCs, so we would add a complex functionality to a provider which is only needed in very few situations. If the logic is in a separate PlatformService, we have a little bit more effort setting this up in these few situations, but a less complicated ClusterProvider in all other situations.
• We either need to introduce cloudprovider-specific code into the otherwise completely cloudprovider-agnostic ClusterProvider (and then the solution works only for this CloudProvider), or we need to work with reflection to inject the CIDR into the right place in the shoot manifest. The current implementation for ClusterConfig patches avoids this (still somewhat ugly, though) by converting the generated shoot manifest into json, using an external library to apply the json patches, and then converting the new json back into a shoot struct.

Any further valuable resources.

No response

What is required to accept the Task as done.

Done Criteria

• We decided on a way to solve this problem.

[Diaphteiros]

There is another option:

Option 5: Let the controller do the reference injection

Instead of having a webhook or something similar, the controller which creates the referenced ClusterConfig resources also takes care of injecting the references into the Cluster resource.

Pro:

• Likely the easiest solution.

Contra:

• Same problem as with the webhook: Modifying a resource's spec after its creation clashes with gitops tools like flux.
• Opposed to a webhook, the controller would only react to the Cluster resource after it has already been created. At this time, the Gardener ClusterProvider also reacts to it and tries to create the corresponding shoot, which won't work due to the missing CIDRs. Now there are two possibilities:
  • Usually, Gardener uses webhooks to reject invalid shoot manifests. If they do this, the shoot manifest will not be created until the references have been injected and everything will work as desired afterwards.
  • If, for some reason, the shoot manifest is not rejected without the injected CIDRs, then the shoot will be created and it is very likely that injecting the CIDRs afterwards will not work due to the respective fields being immutable. In this case, we would have a 'wrong' shoot and immediately after, the Cluster would be broken because the references were injected and it tries to update the shoot manifest, which is then rejected. There is no way to recover from this broken state, except for manually modifying the Cluster resource to not match the IPAM controller's selector anymore (in which case the Cluster will again be healthy, but the shoot will still be missing the desired CIDRs).

This is basically the 'easy path', which heavily depends on Gardener rejecting the shoot manifest without the injected CIDRs (which is likely). Still causes problems with gitops, though.