`kfp-persistence` relation reconciliation reports "versions not found for apps"

[wctaylor]

Bug Description

Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.
Filed by @canonical/solutions-qa

kfp-persistence units repeatedly get stuck in waiting status with [relation:<endpoint>] List of <ops.model.Relation ...> versions not found for apps: <remote-app>, meaning the relation's remote-application databag "versions" key is never observed as populated from the kfp-persistence side.

This is not confined to one specific relation, interface, or remote charm. Across 784 failed test results for kfp-persistence (2026-01-01 onward), 56 show this exact "versions not found for apps" signature, spanning:

• 3 different interfaces: k8s-service (endpoints kfp-api, kfp-api-grpc), loki_push_api (logging endpoint), cross_model_mesh (require-cmr-mesh endpoint)
• 7+ different remote/provider charms: kfp-api, loki-k8s, grafana-agent-k8s, kfp-ui, katib-db-manager, opentelemetry-collector-k8s/opentelemetry-collector, kfp-viz, kubeflow-profiles
• Both in-model and cross-model (CMR) relation topologies (32 in-model vs 24 CMR occurrences)

Because the failure recurs identically regardless of which remote charm, interface, or relation topology is involved, the one common factor across all 56 failures is kfp-persistence itself, pointing to its own relation-reconciliation/readiness handling (or a shared library/base it uses across these relations) rather than to any specific remote/provider charm.

To Reproduce

1. Deploy kfp-persistence alongside any one of the affected remote charms/interfaces, e.g.:
   • katib-db-manager:k8s-service-info ↔ kfp-persistence:kfp-api (in-model or CMR)
   • loki-k8s:logging ↔ kfp-persistence:logging (loki_push_api interface)
   • opentelemetry-collector-k8s:provide-cmr-mesh ↔ kfp-persistence:require-cmr-mesh
2. Relate the applications and wait for all apps/units to reach active (test harness timeout: 15 minutes).
3. Observe juju status on the model containing kfp-persistence.

Environment

Component	Value
Juju versions observed	3.6.x (3/stable), 4.x (4/stable)
Bases observed	ubuntu:20.04, ubuntu:24.04
Topologies observed	single-model kubernetes, cmr/kubernetes/kubernetes, cmr/kubernetes/openstack
kfp-persistence revision (example)	2166 (channel: 2.4/stable)
Affected interfaces	k8s-service, loki_push_api, cross_model_mesh
Affected remote charms	kfp-api, loki-k8s, grafana-agent-k8s, kfp-ui, katib-db-manager, opentelemetry-collector(-k8s), kfp-viz, kubeflow-profiles

Relevant Log Output

2026-06-04T01:11:35.206Z [container-agent] ERROR juju-log Failed to handle <PebbleReadyEvent via KatibDBManagerOperator/on/katib_db_manager_pebble_ready[16]> with error: Waiting for relational-db data
2026-06-04T01:11:37.464Z [container-agent] ERROR juju-log Failed to handle <ConfigChangedEvent via KatibDBManagerOperator/on/config_changed[21]> with error: Waiting for relational-db data
2026-06-04T01:11:41.104Z [container-agent] ERROR juju-log relational-db:3: Failed to handle <RelationJoinedEvent ...> with error: Waiting for relational-db data
2026-06-04T01:14:40.037Z [container-agent] ERROR juju-log relational-db:3: Missing attribute 'username' in relational-db relation data
2026-06-04T01:14:40.093Z [container-agent] ERROR juju-log relational-db:3: Failed to handle <RelationChangedEvent ...> with error: Incorrect/incomplete data found in relation relational-db. See logs


statusinfo: '[relation:kfp-api] Expected data from exactly 1 related applications'
statusinfo: running kfp-api-relation-created hook
statusinfo: 'Reconciling charm: executing component relation:kfp-api'
statusinfo: '[relation:kfp-api] List of <ops.model.Relation kfp-api:0> versions not found for apps: neighbor-offer'
statusinfo: running kfp-api-relation-joined hook for neighbor-offer/0
statusinfo: running kfp-api-relation-changed hook for neighbor-offer/0
statusinfo: 'Reconciling charm: executing component relation:kfp-api'
statusinfo: '[relation:kfp-api] List of <ops.model.Relation kfp-api:0> versions not found for apps: neighbor-offer'

Additional Context

Example Juju status (neighbor model):

App       Version  Status   Scale  Charm            Channel      Rev  Address         Exposed  Message
neighbor           waiting      1  kfp-persistence  2.4/stable  2166  10.152.183.200  no       [relation:kfp-api] List of <ops.model.Relation kfp-api:0> versions not found for apps: neighbor-offer

Unit         Workload  Agent  Address     Ports  Message
neighbor/0*  waiting   idle   10.1.1.118         [relation:kfp-api] List of <ops.model.Relation kfp-api:0> versions not found for apps: neighbor-offer

Integration provider             Requirer          Interface    Type     Message
neighbor-offer:k8s-service-info  neighbor:kfp-api  k8s-service  regular

Example Juju status (target model):

App        Version  Status  Scale  Charm             Channel            Rev  Address         Exposed  Message
mysql-k8s  8.0.44   active      1  mysql-k8s         8.0/stable         400  10.152.183.165  no
target              active      1  katib-db-manager  latest/candidate  1039  10.152.183.245  no

Unit          Workload  Agent  Address     Ports  Message
mysql-k8s/0*  active    idle   10.1.1.169         Primary
target/0*     active    idle   10.1.2.195

Offer           Application  Charm             Rev   Connected  Endpoint          Interface    Role
neighbor-offer  target       katib-db-manager  1039  1/1        k8s-service-info  k8s-service  provider

Example Test Observer Link:
https://test-observer.canonical.com/#/charms/136669?testExecutionId=540773&testResultId=11406014

• Aggregated failure counts by remote charm/interface across 56 matching results (2026-01-01–present):

  Remote charm	Interface	Count
  kfp-api	k8s-service	17
  loki-k8s	loki_push_api	11
  grafana-agent-k8s	loki_push_api	9
  kfp-ui	k8s-service	9
  katib-db-manager	k8s-service-info	4
  opentelemetry-collector(-k8s)	cross_model_mesh	4
  kfp-viz / kubeflow-profiles	k8s-service	2

• Aggregated failure counts by environment topology:

  Environment	Count
  ubuntu:24.04, single-model kubernetes	24
  ubuntu:20.04, cmr/kubernetes/kubernetes	17
  ubuntu:20.04, single-model kubernetes	8
  ubuntu:24.04, cmr/kubernetes/kubernetes	3
  ubuntu:20.04, cmr/kubernetes/openstack	2
  juju 4/stable, cmr/kubernetes/kubernetes	2

• Sample affected Test Observer test executions (non-exhaustive): 540769, 540773, 548542, 548546, 548628, 548632.

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/KF-8896.

This message was autogenerated

[wctaylor]

This applies to kfp-viz as well, not just kfp-persistence