[postgresql-k8s] Unit enters blocked "Insufficient permissions" after pod deletion despite trust: true

[moisesbenzan]

⚠️ This issue was generated with AI assistance (GitHub Copilot) as part of automated test triage.

Summary

postgresql-k8s (rev 845, channel 14/edge) incorrectly enters blocked: Insufficient permissions after pod deletion, even when deployed with trust: true. In this broken state, it cannot handle subsequent relation events (e.g., pg_hba.conf is not updated), causing dependent charms to fail.

Observed Failure

• Test: test_remove_and_restore_integration
• Test Execution: 451804
• Test Result ID: 10244066
• Test Observer: https://test-observer.canonical.com/#/charms/406319?testExecutionId=451804&testResultId=10244066
• CI Run: https://github.com/canonical/charm-integration-testing/actions/runs/23904297719

Error

target/0*  blocked  idle  10.1.2.208  Insufficient permissions, try: `juju trust target --scope=cluster`

After test_pod_deletion, the pod is recreated and the charm re-runs upgrade to ch:amd64/postgresql-k8s-845. Despite the bundle deploying with trust: true, the unit transitions to blocked state:

To fix this issue, run `juju trust target --scope=cluster` (or remove & re-deploy target with --trust)

Failure Chain

test_pod_deletion (PASSES but leaves broken state)
    │
    └──► postgresql-k8s (target/0) enters "blocked: Insufficient permissions"
         despite bundle deploying with trust: true
              │
              └──► test_remove_and_restore_integration:
                   Integration removed → re-added
                        │
                        └──► postgresql-k8s CANNOT update pg_hba.conf
                             (blocked state prevents handling relation events)
                                  │
                                  └──► livepatch-schema-tool: FATAL:
                                       no pg_hba.conf entry for host "10.1.1.93",
                                       user "relation_id_5"
                                            │
                                            └──► database-relation-changed HOOK FAILED
                                                 → JujuWaitTimeoutError after 15 min

Juju Debug Log Evidence

unit.target/0.juju-log hook dispatching script: dispatch
  "To fix this issue, run `juju trust target --scope=cluster`"  (lines 17958, 17999, 18383)

unit-neighbor-0: database:5: Failed to determine if schema upgrade required:
  non-zero exit code 1 executing ['/usr/local/bin/livepatch-schema-tool', 'check', '--db', ...],
  stderr='Error: failed to connect ... server error (FATAL: no pg_hba.conf entry for host
  "10.1.1.93", user "relation_id_5", database "livepatch-server", no encryption (SQLSTATE 28000))'

{"level":"fatal","ts":"2026-04-02T14:47:26.501Z","caller":"srv/server.go:137",
"msg":"failed to configure server: failed to connect ...
FATAL: no pg_hba.conf entry for host \"10.1.1.93\", user \"relation_id_5\",
database \"livepatch-server\", no encryption (SQLSTATE 28000)"}

Final State

neighbor/0*  error    idle  10.1.1.93   hook failed: "database-relation-changed" for target:database
target/0*    blocked  idle  10.1.2.208  Insufficient permissions, try: `juju trust target --scope=cluster`

Notes

• canonical-livepatch-server-k8s hook failure is a downstream effect — not a livepatch bug
• Previous execution 451801 (same rev 845, same test plan) PASSED all tests — confirms intermittent/race condition on trust/RBAC handling during pod recreation
• Workaround: running juju trust target --scope=cluster after pod deletion clears the blocked state

Charms in Bundle

• postgresql-k8s rev 845 (14/edge) ← failing charm
• canonical-livepatch-server-k8s (neighbor)

Environment

• Juju: 3.6.20
• Cloud: k8s-production (kubernetes)
• Ubuntu: 22.04
• Date: 2026-04-02

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/DPE-9733.

This message was autogenerated