feat: Add script for updating the base of all charms to Ubuntu 24.04
#6
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,3 +1,4 @@ | ||
| .idea | ||
| *.pyc | ||
| credentials.json | ||
| venv |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,163 @@ | ||
| - branch: main | ||
| charms: | ||
| - name: admission_webhook | ||
| path: . | ||
| url: https://github.com/canonical/admission-webhook-operator | ||
| - branch: main | ||
| charms: | ||
| - name: argo_controller | ||
| path: charms/argo-controller | ||
| url: https://github.com/canonical/argo-operators | ||
| - branch: main | ||
| charms: | ||
| - name: dex_auth | ||
| path: . | ||
| url: https://github.com/canonical/dex-auth-operator | ||
| - branch: main | ||
| charms: | ||
| - name: envoy | ||
| path: . | ||
| url: https://github.com/canonical/envoy-operator | ||
| - branch: main | ||
| charms: | ||
| - name: github_profiles_automator | ||
| path: . | ||
| url: https://github.com/canonical/github-profiles-automator | ||
| - branch: main | ||
| charms: | ||
| - name: istio_gateway | ||
| path: charms/istio-gateway | ||
| - name: istio_pilot | ||
| path: charms/istio-pilot | ||
| url: https://github.com/canonical/istio-operators | ||
| - branch: main | ||
| charms: | ||
| - name: katib_controller | ||
| path: charms/katib-controller | ||
| - name: katib_db_manager | ||
| path: charms/katib-db-manager | ||
| - name: katib_ui | ||
| path: charms/katib-ui | ||
| url: https://github.com/canonical/katib-operators | ||
| - branch: main | ||
| charms: | ||
| - name: kfp_api | ||
| path: charms/kfp-api | ||
| - name: kfp_metadata_writer | ||
| path: charms/kfp-metadata-writer | ||
| - name: kfp_persistence | ||
| path: charms/kfp-persistence | ||
| - name: kfp_profile_controller | ||
| path: charms/kfp-profile-controller | ||
| - name: kfp_schedwf | ||
| path: charms/kfp-schedwf | ||
| - name: kfp_ui | ||
| path: charms/kfp-ui | ||
| - name: kfp_viewer | ||
| path: charms/kfp-viewer | ||
| - name: kfp_viz | ||
| path: charms/kfp-viz | ||
| url: https://github.com/canonical/kfp-operators | ||
| - branch: main | ||
| charms: | ||
| - name: knative_eventing | ||
| path: charms/knative-eventing | ||
| - name: knative_operator | ||
| path: charms/knative-operator | ||
| - name: knative_serving | ||
| path: charms/knative-serving | ||
| url: https://github.com/canonical/knative-operators | ||
| - branch: main | ||
| charms: | ||
| - name: kserve_controller | ||
| path: charms/kserve-controller | ||
| url: https://github.com/canonical/kserve-operators | ||
| - branch: main | ||
| charms: | ||
| - name: kubeflow_dashboard | ||
| path: . | ||
| url: https://github.com/canonical/kubeflow-dashboard-operator | ||
| - branch: main | ||
| charms: | ||
| - name: kubeflow_profiles | ||
| path: . | ||
| url: https://github.com/canonical/kubeflow-profiles-operator | ||
| - branch: main | ||
| charms: | ||
| - name: kubeflow_roles | ||
| path: . | ||
| url: https://github.com/canonical/kubeflow-roles-operator | ||
| - branch: main | ||
| charms: | ||
| - name: tensorboard_controller | ||
| path: charms/tensorboard-controller | ||
| - name: tensorboards_web_app | ||
| path: charms/tensorboards-web-app | ||
| url: https://github.com/canonical/kubeflow-tensorboards-operator | ||
| - branch: main | ||
| charms: | ||
| - name: kubeflow_volumes | ||
| path: . | ||
| url: https://github.com/canonical/kubeflow-volumes-operator | ||
| - branch: main | ||
| charms: | ||
| - name: metacontroller_operator | ||
| path: . | ||
| url: https://github.com/canonical/metacontroller-operator | ||
| - branch: main | ||
| charms: | ||
| - name: minio | ||
| path: . | ||
| url: https://github.com/canonical/minio-operator | ||
| - branch: main | ||
| charms: | ||
| - name: mlflow_server | ||
| path: . | ||
| url: https://github.com/canonical/mlflow-operator | ||
| - branch: main | ||
| charms: | ||
| - name: mlmd | ||
| path: . | ||
| url: https://github.com/canonical/mlmd-operator | ||
| - branch: main | ||
| charms: | ||
| - name: mlops-libs | ||
| path: . | ||
| url: https://github.com/canonical/mlops-libs | ||
| - branch: main | ||
| charms: | ||
| - name: namespace_node_affinity | ||
| path: . | ||
| url: https://github.com/canonical/namespace-node-affinity-operator | ||
| - branch: main | ||
| charms: | ||
| - name: ngc_integrator_operator | ||
| path: . | ||
| url: https://github.com/canonical/ngc-integrator-operator | ||
| - branch: main | ||
| charms: | ||
| - name: jupyter_controller | ||
| path: charms/jupyter-controller | ||
| - name: jupyter_ui | ||
| path: charms/jupyter-ui | ||
| url: https://github.com/canonical/notebook-operators | ||
| - branch: main | ||
| charms: | ||
| - name: oidc_gatekeeper | ||
| path: . | ||
| url: https://github.com/canonical/oidc-gatekeeper-operator | ||
| - branch: main | ||
| charms: | ||
| - name: pvcviewer_operator | ||
| path: . | ||
| url: https://github.com/canonical/pvcviewer-operator | ||
| - branch: main | ||
| charms: | ||
| - name: resource_dispatcher | ||
| path: . | ||
| url: https://github.com/canonical/resource-dispatcher | ||
| - branch: main | ||
| charms: | ||
| - name: training_operator | ||
| path: . | ||
| url: https://github.com/canonical/training-operator |
|
Owner
There was a problem hiding this comment. I would skip these honestly. I suppose it is easy (and custom to the user case) select from the full list depending on the confidence we have on the modification and which modifications. I would nudge against having too many files in the If we want to say that the process we have ALWAYS goes through the same repo first, and then roll out to the rest, that's fine (but at least for the rest I would use the |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,58 @@ | ||
| - branch: main | ||
| charms: | ||
| - name: istio_gateway | ||
| path: charms/istio-gateway | ||
| - name: istio_pilot | ||
| path: charms/istio-pilot | ||
| url: https://github.com/canonical/istio-operators | ||
| - branch: main | ||
| charms: | ||
| - name: katib_controller | ||
| path: charms/katib-controller | ||
| - name: katib_db_manager | ||
| path: charms/katib-db-manager | ||
| - name: katib_ui | ||
| path: charms/katib-ui | ||
| url: https://github.com/canonical/katib-operators | ||
| - branch: main | ||
| charms: | ||
| - name: kfp_api | ||
| path: charms/kfp-api | ||
| - name: kfp_metadata_writer | ||
| path: charms/kfp-metadata-writer | ||
| - name: kfp_persistence | ||
| path: charms/kfp-persistence | ||
| - name: kfp_profile_controller | ||
| path: charms/kfp-profile-controller | ||
| - name: kfp_schedwf | ||
| path: charms/kfp-schedwf | ||
| - name: kfp_ui | ||
| path: charms/kfp-ui | ||
| - name: kfp_viewer | ||
| path: charms/kfp-viewer | ||
| - name: kfp_viz | ||
| path: charms/kfp-viz | ||
| url: https://github.com/canonical/kfp-operators | ||
| - branch: main | ||
| charms: | ||
| - name: knative_eventing | ||
| path: charms/knative-eventing | ||
| - name: knative_operator | ||
| path: charms/knative-operator | ||
| - name: knative_serving | ||
| path: charms/knative-serving | ||
| url: https://github.com/canonical/knative-operators | ||
| - branch: main | ||
| charms: | ||
| - name: tensorboard_controller | ||
| path: charms/tensorboard-controller | ||
| - name: tensorboards_web_app | ||
| path: charms/tensorboards-web-app | ||
| url: https://github.com/canonical/kubeflow-tensorboards-operator | ||
| - branch: main | ||
| charms: | ||
| - name: jupyter_controller | ||
| path: charms/jupyter-controller | ||
| - name: jupyter_ui | ||
| path: charms/jupyter-ui | ||
| url: https://github.com/canonical/notebook-operators |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -4,4 +4,5 @@ PyYAML<=6.0.2 | |
| python-hcl2==7.0.1 | ||
| prettytable==3.16.0 | ||
| envyaml==1.10.211231 | ||
| pandas==2.2.3 | ||
| oyaml==1.0 | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,3 @@ | ||
| Ref: https://github.com/canonical/bundle-kubeflow/issues/1290 | ||
|
|
||
| This PR re-enables the usage of cache in all `multi-charm` repos. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,66 @@ | ||
| import os | ||
| import shutil | ||
| import subprocess | ||
| import sys | ||
|
|
||
| import oyaml as yaml | ||
|
|
||
| sys.path.append(os.path.abspath(os.path.join(os.path.dirname(__file__), "../../"))) | ||
|
|
||
| from kfcicli.main import * | ||
| from kfcicli.utils import setup_logging | ||
| import json | ||
|
|
||
| logger = setup_logging(log_level="INFO", logger_name=__name__) | ||
|
|
||
| with open("./credentials.json", "r") as fid: | ||
| credentials = GitCredentials(**json.loads(fid.read())) | ||
|
|
||
| tmp_folder = f"/home/ubuntu/tmp/kfcicli/charm_repos" | ||
| logger.info(f"Using temporary directory: {tmp_folder}") | ||
|
|
||
| filename=Path("../../presets/kubeflow-multi-charm.yaml") | ||
|
|
||
| client = KubeflowCI.read( | ||
| filename=filename, | ||
| base_path=Path(f"{tmp_folder}"), | ||
| credentials=credentials | ||
| ) | ||
|
|
||
| CURRENT_FOLDER = Path(__file__).parent | ||
|
|
||
| def call_script(script: Path, path: Path) -> bool: | ||
|
|
||
| shutil.copy(script, path / script.name) | ||
|
|
||
| try: | ||
| subprocess.check_call(["/bin/bash", script.name], | ||
| cwd=path, | ||
| stdout=subprocess.DEVNULL, | ||
| stderr=subprocess.DEVNULL) | ||
| return True | ||
| except subprocess.CalledProcessError: | ||
| return False | ||
| finally: | ||
| os.remove(path / script.name) | ||
|
|
||
|
|
||
| def process_repo(repo: Client, charms: list[LocalCharmRepo], dry_run: bool): | ||
| success = call_script(CURRENT_FOLDER / "replace-cache.sh", repo.base_path) | ||
|
|
||
| if repo.is_dirty(): | ||
| repo.update_branch( | ||
| commit_msg=f"Updating GitHub action file", directory=".", | ||
| push=not dry_run, force=True | ||
| ) | ||
|
|
||
| with open("body.md", "r") as f: | ||
| pr_body = f.read() | ||
|
|
||
| client.canon_run( | ||
| wrapper_func=process_repo, | ||
| branch_name="kf-7561-enable-cache", | ||
| title="chore: Re-enable usage of cache while bulding charms", | ||
| body=pr_body, | ||
| dry_run=False | ||
| ) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,4 @@ | ||
| #!/usr/bin/env bash | ||
|
|
||
| # Replace cache: false to cache: false, and ignore all comments on that line | ||
| sed -E '/^[[:space:]]*cache:[[:space:]]*false/ s/cache:[[:space:]]*false.*/cache: true/' .github/workflows/ci.yaml |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| Ref: https://github.com/canonical/bundle-kubeflow/issues/1277 | ||
|
|
||
| This PR: | ||
| - Updates the dependencies using `pip-compile` with Python 3.12. | ||
| - Updates `charmcraft.yaml` to use Ubuntu `24.04` as a base. Note that we're pinning the version of `pip` to avoid issues with different versions. | ||
| - Removes the `setup-python` action in all CI workflows, since we will be using the default Python version in 24.04. | ||
| - Updates all references from the `20.04` base to the new `24.04` base. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,75 @@ | ||
| # Copyright 2025 Canonical Ltd. | ||
| # See LICENSE file for licensing details. | ||
|
|
||
| type: charm | ||
| platforms: | ||
| ubuntu@24.04:amd64: | ||
|
|
||
| # Files implicitly created by charmcraft without a part: | ||
| # - dispatch (https://github.com/canonical/charmcraft/pull/1898) | ||
| # - manifest.yaml | ||
| # (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/services/package.py#L259) | ||
| # Files implicitly copied/"staged" by charmcraft without a part: | ||
| # - actions.yaml, config.yaml, metadata.yaml | ||
| # (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/services/package.py#L290-L293 | ||
| # https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/services/package.py#L156-L157) | ||
| parts: | ||
| # "python-deps" part name is arbitrary; use for consistency | ||
| # (but could become a magic constant in the future, similar to "poetry-deps" | ||
| # https://github.com/canonical/craft-parts/pull/901) | ||
| python-deps: | ||
| plugin: nil | ||
| override-build: | | ||
| # Use environment variable instead of `--break-system-packages` to avoid failing on older | ||
| # versions of pip that do not recognize `--break-system-packages` | ||
| # `--user` needed (in addition to `--break-system-packages`) for Ubuntu >=24.04 | ||
| PIP_BREAK_SYSTEM_PACKAGES=true python3 -m pip install --user --upgrade pip==24.3.1 # renovate: charmcraft-pip-latest | ||
|
|
||
| # "charm-python" part name is arbitrary; use for consistency | ||
| # Avoid using "charm" part name since that has special meaning to charmcraft | ||
| charm-python: | ||
| # By default, the `python` plugin creates/stages these directories: | ||
| # - lib, src | ||
| # (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/parts/plugins/_python.py#L79-L81) | ||
| # - venv | ||
| # (https://github.com/canonical/charmcraft/blob/9ff19c328e23b50cc06f04e8a5ad4835740badf4/charmcraft/parts/plugins/_python.py#L100 | ||
| # https://github.com/canonical/craft-parts/blob/afb0d652eb330b6aaad4f40fbd6e5357d358de47/craft_parts/plugins/base.py#L270) | ||
| plugin: python | ||
| source: . | ||
| after: | ||
| - python-deps | ||
| python-requirements: [requirements.txt] | ||
| build-packages: | ||
| - libffi-dev # Needed to build Python dependencies with Rust from source | ||
| - libssl-dev # Needed to build Python dependencies with Rust from source | ||
| - pkg-config # Needed to build Python dependencies with Rust from source | ||
| override-build: | | ||
| # Workaround for https://github.com/canonical/charmcraft/issues/2068 | ||
| # rustup used to install rustc and cargo, which are needed to build Python dependencies with Rust from source | ||
| if [[ "$CRAFT_PLATFORM" == ubuntu@20.04:* || "$CRAFT_PLATFORM" == ubuntu@22.04:* ]] | ||
| then | ||
| snap install rustup --classic | ||
| else | ||
| apt-get install rustup -y | ||
| fi | ||
|
|
||
| # If Ubuntu version < 24.04, rustup was installed from snap instead of from the Ubuntu | ||
| # archive—which means the rustup version could be updated at any time. Print rustup version | ||
| # to build log to make changes to the snap's rustup version easier to track | ||
| rustup --version | ||
|
|
||
| # rpds-py (Python package) >=0.19.0 requires rustc >=1.76, which is not available in the | ||
| # Ubuntu 22.04 archive. Install rustc and cargo using rustup instead of the Ubuntu archive | ||
| rustup set profile minimal | ||
| rustup default 1.83.0 # renovate: charmcraft-rust-latest | ||
|
|
||
| craftctl default | ||
| # Include requirements.txt in *.charm artifact for easier debugging | ||
| cp requirements.txt "$CRAFT_PART_INSTALL/requirements.txt" | ||
|
|
||
| # "files" part name is arbitrary; use for consistency | ||
| files: | ||
| plugin: dump | ||
| source: . | ||
| stage: | ||
| - LICENSE |
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why not changing the
kubeflow-repos.yaml? My idea was to have one file that we could re-use with all of the repositories