Add ARTIFACTS_PROXY_ENABLED config option #973

Context

ARTIFACTS_PROXY_ENABLED is a configuration option that has been added in the pipeline-install-conifg ConfigMap. It was introduced in the PR that replaced minio with seaweedfs. According to the comments above the definition:

Controls whether the artifact proxy is enabled to support accessing out-of-kubeflow scope buckets. The artifact proxy has known security and architectural flaws, so it should only be enabled if you specifically need to access unique pipeline root buckets per namespace that are outside the kubeflow namespace's access valid values are 'true' and 'false'. Defaults to 'false'.

In practice, this config option affects the creation of ml-pipeline-ui deployment in the profile namespaces.

Handling of ml-pipeline-ui deployments in the user namespace

• sync.py was updated in chore: Seaweedfs as Minio replacement kubeflow/pipelines#11965 to conditionally create the ml-pipeline-ui deployments in the user namespace as well as the respective service, see https://github.com/kubeflow/pipelines/blob/bc89227f26afb8c5601581c2b7b5634ce017728d/manifests/kustomize/base/installs/multi-user/pipelines-profile-controller/sync.py#L230.
• This condition is based on the value of the environment variable ARTIFACTS_PROXY_ENABLED.
• At the same time, the function getArtifactsProxyHandler used for the frontend image in the kfp-ui conditionally uses this environment variable to redirect requests to the ml-pipeline-ui service in the user deployment.
• The default value of ARTIFACTS_PROXY_ENABLED was changed to false in chore: Seaweedfs as Minio replacement kubeflow/pipelines#11965. Comments in https://github.com/kubeflow/manifests/blob/1a12e1be42f69ad4a3d374dc3e9cb2da6eb26706/applications/pipeline/upstream/base/installs/generic/pipeline-install-config.yaml#L97-L104 explain that this change was made for security reasons, the reasoning behind the change being still unclear.

Advantages

• The main advantage of this config option is to improve resource scaling of a Kubeflow cluster with a lot of Kubeflow profiles. In a standard Kubeflow cluster with 100 profiles:
• With ARTIFACTS_PROXY_ENABLED=true, there will be one ml-pipeline-ui pod in the control plane namespace, and 100 pods in the profile namespaces, totaling 101 pods
• With ARTIFACTS_PROXY_ENABLED=false, there will only be one ml-pipeline-ui pod in the control plane namespace

What needs to get done

• Add config option in the kfp-api charm named artifacts-proxy-enabled
• Discuss what the default value should be
• Add unit tests
• Add integration tests. The integration tests should check that with ARTIFACTS_PROXY_ENABLED=false, a ml-pipeline-ui pod is not created in the profile namespace

Definition of Done

.