Default artifact root charm config is not propagated to the notebook servers

[toaksoy]

Bug Description

Pipelines are failing due to the log message shared below. The root cause, as identified with @misohu, is that the notebook server does not have the MLFLOW_ARTIFACT_URI environment variable set because the mlflow-server-minio PodDefault inside the user namespace doesn't set this value. Although, the charm config default_artifact_root for mlflow-server is all set.

root@mlflow-server-0:/# pebble plan
services:
    mlflow-server:
        summary: Entrypoint of mlflow-server image
        startup: enabled
        override: replace
        command: mlflow server --host 0.0.0.0 --port 5000 --backend-store-uri XXXX --default-artifact-root s3://<custom-bucket-name>/ --expose-prometheus /metrics

The code snippet below runs just fine with the artifact correctly being uploaded to the custom bucket:

import mlflow
import os
os.environ['MLFLOW_ARTIFACT_URI'] = "s3://<custom-bucket-name>/"
mlflow.set_experiment('test')
with mlflow.start_run():
    mlflow.log_artifact("requirements.txt")

However, if we use the autologger from MLFlow, it fails.

To Reproduce

1. Deploy CKF 1.10
2. Create a custom bucket
3. Set MinIO to run in Gateway mode
4. Run a notebook using MLFlow

Environment

• EKS 1.32
• Juju 3.6.9
• CKF 1.10

Relevant Log Output

File "/usr/local/lib/python3.11/site-packages/boto3/s3/transfer.py", line 378, in upload_file
    raise S3UploadFailedError(
boto3.exceptions.S3UploadFailedError: Failed to upload /tmp/tmpslhzbz8k/model/MLmodel to mlflow/0/e4376d4b52cb4c27866082715027ec5b/artifacts/model/MLmodel: An error occurred (InternalError) when calling the PutObject operation (reached max retries: 4): We encountered an internal error, please try again.: cause(Put "
http://mlflow.s3.dualstack.eu-west-1.amazonaws.com/0/e4376d4b52cb4c27866082715027ec5b/artifacts/model/MLmodel
": 301 response missing Location header)

Additional Context

No response

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-7953.

This message was autogenerated

[toaksoy]

As an update, when a notebook server gets launched, it has an artifact location already set by the default experiment, although it has no active runs:

import mlflow
print(mlflow.active_run()) # <ActiveRun: >
print(mlflow.get_artifact_uri()) # s3://mlflow/0/20b84f7cf99949ceb32eab8a2edf5dee/artifacts

This is because the experiment has a different artifact location stored in the backend store:

I run a new notebook with a new experiment for MLflow, and I can see in the DB that it has the artifact_location set to what I configured in the PodDefault manually.

[toaksoy]

Update: the default_artifact_root charm config actually sets the correct value for the artifact_location of the Default experiment entry in the experiments table of the mlflow database, BUT only if the config option is already set before the relation between mlflow-server and mlflow-mysql is established.

[mvlassis]

We should ensure that the default_artifact_root for the default experiment in the relation database is set after the configuration value is updated.