Issues with juju 3.6.19 and secrets on postgresql (also valid for pgbouncer, data-integrator charms)

[dparv]

Deployed on Canonical k8s 1.32.2 with juju model version 3.6.19 results in charms being in error and can't access secrets. Re-deploying with earlier agent-versions works.

juju debug-log --replay --include=postgresql/0
unit-postgresql-0: 14:59:02 INFO juju.cmd running containerAgent [3.6.19 5a261e58fcd3bd366b36229bfd1c46e6b3b61402 gc go1.25.8]
unit-postgresql-0: 14:59:02 INFO juju.cmd.containeragent.unit start "unit"
unit-postgresql-0: 14:59:02 INFO juju.worker.upgradesteps upgrade steps for 3.6.19 have already been run.
unit-postgresql-0: 14:59:02 INFO juju.worker.probehttpserver starting http server on 127.0.0.1:65301
unit-postgresql-0: 14:59:02 INFO juju.api connection established to "wss://192.168.2.177:17070/model/4c6ea613-bcb6-43fe-80f9-cd76accaa069/api"
unit-postgresql-0: 14:59:02 INFO juju.worker.apicaller [4c6ea6] "unit-postgresql-0" successfully connected to "wss://192.168.2.177:17070"
unit-postgresql-0: 14:59:02 INFO juju.api connection established to "wss://10.100.2.48:17070/model/4c6ea613-bcb6-43fe-80f9-cd76accaa069/api"
unit-postgresql-0: 14:59:02 INFO juju.worker.apicaller [4c6ea6] "unit-postgresql-0" successfully connected to "wss://10.100.2.48:17070"
unit-postgresql-0: 14:59:02 INFO juju.worker.migrationminion migration migration phase is now: NONE
unit-postgresql-0: 14:59:02 INFO juju.worker.logger logger worker started
unit-postgresql-0: 14:59:03 INFO juju.worker.leadership postgresql/0 promoted to leadership of postgresql
unit-postgresql-0: 14:59:03 WARNING juju.worker.proxyupdater unable to set snap core settings [proxy.http= proxy.https= proxy.store=]: exec: "snap": executable file not found in $PATH, output: ""
unit-postgresql-0: 14:59:03 INFO juju.worker.caasupgrader abort check blocked until version event received
unit-postgresql-0: 14:59:03 INFO juju.worker.caasupgrader unblocking abort check
unit-postgresql-0: 14:59:03 INFO juju.agent.tools ensure jujuc symlinks in /var/lib/juju/tools/unit-postgresql-0
unit-postgresql-0: 14:59:03 INFO juju.worker.uniter unit "postgresql/0" started
unit-postgresql-0: 14:59:03 INFO juju.worker.uniter resuming charm install
unit-postgresql-0: 14:59:03 INFO juju.worker.uniter.charm downloading ch:amd64/postgresql-k8s-495 from API server
unit-postgresql-0: 14:59:03 INFO juju.downloader downloading from ch:amd64/postgresql-k8s-495
unit-postgresql-0: 14:59:03 WARNING juju.worker.s3caller Response has no supported checksum. Not validating response payload.%!(EXTRA []interface {}=[])
unit-postgresql-0: 14:59:03 INFO juju.downloader download complete ("ch:amd64/postgresql-k8s-495")
unit-postgresql-0: 14:59:04 INFO juju.downloader download verified ("ch:amd64/postgresql-k8s-495")
unit-postgresql-0: 14:59:40 INFO juju.worker.uniter hooks are retried true
unit-postgresql-0: 14:59:40 INFO juju.worker.uniter found queued "install" hook
unit-postgresql-0: 14:59:47 INFO juju.worker.uniter.operation ran "install" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 14:59:51 INFO juju.worker.uniter.operation ran "restart-relation-created" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 14:59:54 INFO juju.worker.uniter.operation ran "database-peers-relation-created" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 14:59:57 INFO juju.worker.uniter.operation ran "upgrade-relation-created" hook (via hook dispatching script: dispatch)
unit-postgresql-0: 14:59:57 INFO juju.worker.uniter found queued "leader-elected" hook
unit-postgresql-0: 15:00:03 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: saving content for secret "0vfht8vi50g71rqijpig": secrets "0vfht8vi50g71rqijpig-1" is forbidden: User "system:serviceaccount:canonical:juju-secret-consumer-f885f4e2-5753-4400-acb1-d54166244a9f" cannot patch resource "secrets" in API group "" in the namespace "canonical"
unit-postgresql-0: 15:00:03 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:00:08 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:00:13 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: saving content for secret "udlq8fc2qvhtv7863dn0": secrets "udlq8fc2qvhtv7863dn0-1" is forbidden: User "system:serviceaccount:canonical:juju-secret-consumer-b255ecb5-ad61-4583-b474-0bdcb31abd5d" cannot patch resource "secrets" in API group "" in the namespace "canonical"
unit-postgresql-0: 15:00:13 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:00:23 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:00:28 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: saving content for secret "h28nbahuapil63ed1bhg": secrets "h28nbahuapil63ed1bhg-1" is forbidden: User "system:serviceaccount:canonical:juju-secret-consumer-6fa644f7-4b92-4429-a646-9e462f6553f0" cannot patch resource "secrets" in API group "" in the namespace "canonical"
unit-postgresql-0: 15:00:28 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:00:49 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:00:54 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: saving content for secret "4au0efaij2652k192s9g": secrets "4au0efaij2652k192s9g-1" is forbidden: User "system:serviceaccount:canonical:juju-secret-consumer-7feb4a35-ea4f-4f24-83a4-a4c3ee5a13d3" cannot patch resource "secrets" in API group "" in the namespace "canonical"
unit-postgresql-0: 15:00:54 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:01:36 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:01:40 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: saving content for secret "icdj77tfso4c19vnt1ig": secrets "icdj77tfso4c19vnt1ig-1" is forbidden: User "system:serviceaccount:canonical:juju-secret-consumer-1f77da11-e21e-45cb-96df-af00184fa54e" cannot patch resource "secrets" in API group "" in the namespace "canonical"
unit-postgresql-0: 15:01:40 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:02:18 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:02:18 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:02:19 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:02:21 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:02:41 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:02:43 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:02:45 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:03:05 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:03:10 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: saving content for secret "d8q9a2mr8onj2548ojcg": secrets "d8q9a2mr8onj2548ojcg-1" is forbidden: User "system:serviceaccount:canonical:juju-secret-consumer-fd1db784-a05f-4429-8772-56e28227bc08" cannot patch resource "secrets" in API group "" in the namespace "canonical"
unit-postgresql-0: 15:03:10 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:03:42 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:06:05 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook
unit-postgresql-0: 15:06:10 ERROR juju.worker.uniter.operation hook "leader-elected" (via hook dispatching script: dispatch) failed: saving content for secret "rd1gfdjng32mes2rp8tg": secrets "rd1gfdjng32mes2rp8tg-1" is forbidden: User "system:serviceaccount:canonical:juju-secret-consumer-a93a8665-9c67-4395-bfb8-63e15154690c" cannot patch resource "secrets" in API group "" in the namespace "canonical"
unit-postgresql-0: 15:06:10 INFO juju.worker.uniter awaiting error resolution for "leader-elected" hook

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-9665.

This message was autogenerated

[dragomirp]

Hi, @dparv, I was unable to optimistically replicate in my dev env (single node, k8s snap revision 5092 (v1.32.11)). What I did notice however, is that the juju-secret-consumer-* service accounts are created on demand for hook executions, so I would guess some kind of race is happening between the secret consumer's service account, role and role binding. Can you try to trace that everything is set up your cluster when the juju hook is executing? I think the name of all the resources is the same.

[hpidcock]

This has been reproduced in a HA k8s controller node cluster. It is an eventual consistency problem with RBAC.
We're mitigating against this in Juju 3.6.20.

[taurus-forever]

@dparv Juju 3.6.21 is out (and 3.6.20 suppose to have a fix), can you please confirm it is no longer reproducible?