kfp-persistence 2.15 fails to create token directory during CKF 1.10 to 1.11 upgrade

[avinash-fde]

Bug Description

Summary

During an upgrade from CKF 1.10 to CKF 1.11, kfp-persistence remains in waiting after being refreshed to 2.15/stable.

All other CKF 1.11 applications are healthy.

The same issue remains after testing 2.15/edge.

Environment

• CKF upgrade: 1.10 to 1.11
• Juju: 3.6.27
• Kubernetes platform: Tanzu Kubernetes
• Base: Ubuntu 24.04
• Application: kfp-persistence
• Stable revision tested: 2568
• Channels tested: 2.15/stable and 2.15/edge

Error

The charm successfully creates the ServiceAccount token:

Executing component: 'sa-token:persistenceagent'
Token for kfp-persistence ServiceAccount created and persisted.
Execution for component 'sa-token:persistenceagent' complete.

Then it fails while pushing the token into the workload container:

Executing component: 'container:persistenceagent'
ops.pebble.PathError: permission-denied - cannot create directory:
mkdir /var/run/secrets/kubeflow.mkdir-new: permission denied

Juju Status:

App	Version	Status	Scale	Charm	Channel	Rev	Address	Exposed	Message
kfp-persistence		waiting	1	kfp-persistence	2.15/edge	2801	xxx.xx.x.xxx	no	[container:persistenceagent] Waiting for Pebble services (persistenceagent). If this persists, it could be a blockin...

Pebble status:

Service           Startup  Current
persistenceagent  enabled  inactive

Validation performed

• The Pod uses the kfp-persistence ServiceAccount.
• The ServiceAccount exists.
• kubectl auth can-i create serviceaccounts/token returns yes.
• The charm container can authenticate to the Kubernetes API and receives HTTP 200.
• Recreating the Pod does not fix the issue.
• The directory /var/run/secrets/kubeflow/tokens does not exist in the workload container.
• The issue also occurs with 2.15/edge.

Expected behavior

The charm should create the required directory, write the token to:

/var/run/secrets/kubeflow/tokens/persistenceagent-sa-token

and start the persistenceagent service.

Could you confirm whether this is a known issue and provide a fix?

To Reproduce

Follow this documentation
https://documentation.ubuntu.com/charmed-kubeflow/latest/how-to/manage/upgrade/upgrade-1.10-1.11/

at step 4, do the following as per the documentation.
juju refresh kfp-persistence --channel 2.15/stable

Environment

• CKF upgrade: 1.10 to 1.11
• Juju: 3.6.27
• Kubernetes platform: Tanzu Kubernetes
• Base: Ubuntu 24.04
• Application: kfp-persistence
• Stable revision tested: 2568
• Channels tested: 2.15/stable and 2.15/edge

Relevant Log Output

juju debug-log \
  --include kfp-persistence/0 \
  --level INFO \
  --tail


unit-kfp-persistence-0: 13:00:53 INFO unit.kfp-persistence/0.juju-log Executing component: 'relation:kfp-api-grpc'
unit-kfp-persistence-0: 13:00:53 INFO unit.kfp-persistence/0.juju-log Execution for component 'relation:kfp-api-grpc' complete.  Component now has status 'ActiveStatus('')'
unit-kfp-persistence-0: 13:00:53 INFO unit.kfp-persistence/0.juju-log Executing component: 'sa-token:persistenceagent'
unit-kfp-persistence-0: 13:00:53 WARNING unit.kfp-persistence/0.juju-log Token file for kfp-persistence ServiceAccount already exists, will be overridden.
unit-kfp-persistence-0: 13:00:53 INFO unit.kfp-persistence/0.juju-log Token for kfp-persistence ServiceAccount created and persisted.
unit-kfp-persistence-0: 13:00:53 INFO unit.kfp-persistence/0.juju-log Execution for component 'sa-token:persistenceagent' complete.  Component now has status 'ActiveStatus('')'
unit-kfp-persistence-0: 13:00:53 INFO unit.kfp-persistence/0.juju-log Executing component: 'container:persistenceagent'
unit-kfp-persistence-0: 13:00:53 ERROR unit.kfp-persistence/0.juju-log execute_components caught unhandled exception when executing configure_charm for container:persistenceagent
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/charm_reconciler.py", line 93, in reconcile
    component_item.component.configure_charm(event)
  File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/component.py", line 51, in configure_charm
    self._configure_unit(event)
  File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/pebble_component.py", line 275, in _configure_unit
    self._push_files_to_container()
  File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/lib/python3.12/site-packages/charmed_kubeflow_chisme/components/pebble_component.py", line 246, in _push_files_to_container
    container.push(**container_file_template.get_inputs_for_push())
  File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 2853, in push
    self._pebble.push(
  File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2731, in push
    self._raise_on_path_error(typing.cast('_FilesResponse', resp), path)
  File "/var/lib/juju/agents/unit-kfp-persistence-0/charm/venv/lib/python3.12/site-packages/ops/pebble.py", line 2670, in _raise_on_path_error
    raise PathError(error['kind'], error['message'])
ops.pebble.PathError: permission-denied - cannot create directory: mkdir /var/run/secrets/kubeflow.mkdir-new: permission denied
unit-kfp-persistence-0: 13:00:53 INFO unit.kfp-persistence/0.juju-log execute_components execution loop complete.

Additional Context

No response