Spread CI prepare: /home/ubuntu/.local owned by root breaks charmcraft for ubuntu user

[javierdelapuente]

Problem

When using integration-test.yml with a charm that builds tester charms during tests (e.g. traefik-k8s-operator), the tester charm build fails with:

FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.local/state/charmcraft/log'

During handling of the above exception, another exception occurred:

PermissionError: [Errno 13] Permission denied: '/home/ubuntu/.local/state/charmcraft'

Root cause analysis

The CI prepare script (_CI_PREPARE_BEFORE_USER in src/opcli/core/spread.py) runs entirely as root with export HOME=/home/ubuntu (line 605). It then calls:

1. opcli env provision -c "$CONCIERGE" (line 606)

Concierge sees SUDO_USER=ubuntu and bootstraps juju/microk8s writing to /home/ubuntu/.local/share/juju/. The intermediate directory /home/ubuntu/.local/ (and /home/ubuntu/.local/state/) gets created with root ownership because the running process is root.

Additionally, if concierge.yaml includes host.snaps.charmcraft, the snap installation as root may also create /home/ubuntu/.local/state/ with root ownership.

The final chown -R ubuntu:ubuntu "${SPREAD_PATH}" in _CI_PREPARE_AFTER_USER (line 617) only fixes the project directory (/home/ubuntu/proj), not /home/ubuntu/.local/.

When the test subsequently runs as ubuntu (via runuser -l ubuntu) and tries to run charmcraft pack for in-test helper charms, charmcraft can't create its log directory at /home/ubuntu/.local/state/charmcraft/log because /home/ubuntu/.local/state/ is owned by root.

Reproduction

Observed in: canonical/traefik-k8s-operator#687 (run 26392788986)

CI log showing the full error chain:

E  File "/snap/charmcraft/current/usr/lib/python3.12/pathlib.py", line 1313, in mkdir
E    os.mkdir(self, mode)
E  FileNotFoundError: [Errno 2] No such file or directory: '/home/ubuntu/.local/state/charmcraft/log'
E  
E  During handling of the above exception, another exception occurred:
E  ...
E    os.mkdir(self, mode)
E  PermissionError: [Errno 13] Permission denied: '/home/ubuntu/.local/state/charmcraft'

The traefik-k8s-operator builds several tester charms (forward-auth, ipa, ipu, route, tcp, health) inside the integration tests via ops_test.build_charm(). All 14 test jobs fail with this same permission error.

Suggested fix

Add a chown for /home/ubuntu/.local to _CI_PREPARE_AFTER_USER (or at the end of _CI_PREPARE_BEFORE_USER after opcli env provision):

# Fix ownership of .local created by root-run tools (juju bootstrap, snap install)
chown -R ubuntu:ubuntu /home/ubuntu/.local 2>/dev/null || true

Or more conservatively, fix the entire home directory:

chown -R ubuntu:ubuntu /home/ubuntu

Additional context

• The main charm path (CHARM_PATH bridging) works correctly — artifacts are fetched and localized
• Only in-test charmcraft pack calls (for tester charms via ops_test.build_charm()) hit this issue
• The CI adhoc backend runs spread directly on the GHA runner (ADDRESS localhost), so all prepare runs as root
• Concierge's realUser() function respects SUDO_USER for writing juju/kubeconfig, but intermediate XDG directories still get root ownership