How can I ensure that containerd cannot create a container if the NRI plugin is unavailable?

[t33m]

Is there a way to configure containerd so that starting containers is impossible if the NRI plugin is not working or hasn't registered yet?

[kad]

It is impossible now, and not sure that this is going to be good idea. Some of NRI plugins are going to be deployed as containers, and infrastructure components are also in containers (e.g. kubelet's static pod manifests), so not starting until NRI plugin registers will lead to non-functional node. Another thing is the crashes of the plugins and re-connects: we shouldn't have scenarios where crash of the plugin will render node even temporary to non-usable state.

For all of those the sync-on-connect calls were implemented: the plugin during start can inspect and adjust running containers which were started before NRI plugin registers. As well, if there is something that can't be modified via Update cals, there is always possibility for NRI plugin to trigger stop of the existing container, and eventual re-creation of it by orchestration layer above.

[mikebrow]

Some discussion here .. wip #43 (comment)

[mikebrow]

@kad good insight

Should probably develop some use cases .. an integration bucket and a e2e bucket that exercises the supported use cases.

[klihub]

It is impossible now, and not sure that this is going to be good idea. Some of NRI plugins are going to be deployed as containers, and infrastructure components are also in containers (e.g. kubelet's static pod manifests), so not starting until NRI plugin registers will lead to non-functional node. Another thing is the crashes of the plugins and re-connects: we shouldn't have scenarios where crash of the plugin will render node even temporary to non-usable state.

For all of those the sync-on-connect calls were implemented: the plugin during start can inspect and adjust running containers which were started before NRI plugin registers. As well, if there is something that can't be modified via Update cals, there is always possibility for NRI plugin to trigger stop of the existing container, and eventual re-creation of it by orchestration layer above.

I agree with @kad that it's probably not a good idea to bake such logic into the runtime itself.

If I had to do something to this effect, my first idea would probably be to roll some extra tooling for it. Run the critical plugin(s) as DaemonSets, monitor whether they are ready/live (or have them refresh a CRD or a label on their node periodically and monitor that), then taint and if necessary drain/cordon/uncordon the nodes as necessary by extra tooling. If useful, maybe also label workloads that need the critical plugins as such, and have a mutating webhook to inject tolerations for unlabeled workloads to tolerate nodes without the critical plugins being up and running.

[zhaodiaoer]

In the scenarios where I work, i also encountered similar problems... I'm not sure if it's a good idea for us to provide the ability to categorize all plugins by role and make corresponding distinctions during the plugin registration/deregistration process. such as give a flag used to identify what the critical level for plugin to notify NRI when Initiate registration, and when deal with different level plugin's deregistration use different policy, like only when plugin explicitly initiates one rpc request can the deregistration be completed. (just an example and hasn't been carefully considered)

[samuelkarp]

@t33m I think #171 will satisfy your use case. This introduces a new builtin validator plugin that can:

• Reject new containers that have an annotation requiring a specific plugin to execute if the plugin is not running
• Reject new containers if a globally-required plugin is not executing
• Allow globally-required plugins to be bypassed via configurable annotation (to allow for the case where your globally-required plugin is running as a pod itself)

This issue is older, but there's some more discussion on #142 for the same goal. I'm going to close this as a duplicate just to consolidate the discussion in one place. Thank you for opening it.