Skip to content

APT caching revisited #214

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
lucaCigarini merged 2 commits into from
Mar 17, 2025
Merged

APT caching revisited #214

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
38032c5
APT caching revisited
lucaCigarini Mar 14, 2025
cf32401
Merge branch 'develop' into revisiting_apt_requirements_caching
lucaCigarini Mar 17, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
53 changes: 41 additions & 12 deletions .github/actions/apt_requirements/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,54 @@
name: Composite action install apt requirements
name: Composite action install apt requirements
description: Composite action install apt requirements
inputs:
working_directory:
description: Working directory
required: true
requirements_file:
description: Requirements file
required: true

runs:
using: "composite"
steps:
- name: Export apt requirements
id: export-apt-requirements
- name: Compute APT requirements file SHA256 hash
id: compute_apt_requirements_file_sha256_hash
run: |
COMPUTED_HASH=$(sha256sum ${{ inputs.requirements_file }} | cut -d ' ' -f 1)
echo "cache_key=$COMPUTED_HASH" >> $GITHUB_OUTPUT
shell: bash

# Vital to be able to restore cache
# If write permission is not set, a permissions error will be raised
- name: Modification to /var/cache/apt/archives permissions
run: |
PKG=$(cat ${{ inputs.requirements_file }})
echo apt_packages=$PKG | awk '{print}' ORS=' ' >> $GITHUB_OUTPUT
sudo chmod a+w /var/cache/apt/archives
shell: bash

- name: Cache apt packages
id: cache-apt-packages
uses: awalsh128/cache-apt-pkgs-action@latest
- uses: actions/cache/restore@v4
id: restore_cache_from_parent_branch
with:
path: /var/cache/apt/archives/*.deb
key: ${{ github.base_ref }}-${{ steps.compute_apt_requirements_file_sha256_hash.outputs.cache_key }}

- uses: actions/cache/restore@v4
id: restore_cache_from_current_branch
if: steps.restore_cache_from_parent_branch.outputs.cache-hit != 'true'
with:
path: /var/cache/apt/archives/*.deb
key: ${{ github.ref_name }}-${{ steps.compute_apt_requirements_file_sha256_hash.outputs.cache_key }}

- name: Refresh repositories
if: steps.restore_cache_from_parent_branch.outputs.cache-hit != 'true' && steps.restore_cache_from_current_branch.outputs.cache-hit != 'true'
run: |
sudo apt-get update
shell: bash

- name: Install requirements
run: |
sudo apt-get install -y --no-install-recommends $(tr '\n' ' ' < ${{ inputs.requirements_file }})
shell: bash

- uses: actions/cache/save@v4
id: cache_apt_requirements_for_current_branch
if: steps.restore_cache_from_parent_branch.outputs.cache-hit != 'true' && steps.restore_cache_from_current_branch.outputs.cache-hit != 'true'
with:
packages: ${{ steps.export-apt-requirements.outputs.apt_packages }}
path: /var/cache/apt/archives/*.deb
key: ${{ github.ref_name }}-${{ steps.compute_apt_requirements_file_sha256_hash.outputs.cache_key }}
39 changes: 39 additions & 0 deletions .github/workflows/create_apt_cache.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Create APT cache

# GitHub will remove any cache entries that have not been accessed in over 7 days.

on:
push:
branches:
- main
- master
- develop
- dev
paths:
# Path to APT requirements file
- '.github/test/python_test/packages.txt'


jobs:
create-cache:
name: Create cache for dependencies
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

# Remember to set the same APT requirements file path set before!
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get -y install --no-install-recommends $(tr '\n' ' ' < apt_packages.txt)

- name: Compute apt_packages.txt SHA256 hash
id: compute_apt_packages_sha256_hash
run: |
COMPUTED_HASH=$(sha256sum apt_packages.txt | cut -d ' ' -f 1)
echo "cache_key=$COMPUTED_HASH" >> $GITHUB_OUTPUT

- uses: actions/cache/save@v4
with:
path: /var/cache/apt/archives/*.deb
key: ${{ github.ref_name }}-${{ steps.compute_apt_packages_sha256_hash.outputs.cache_key }}
4 changes: 3 additions & 1 deletion CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
## 1.6.0
### Features

* Added both frontend and backend exclusions on _detect_changes.yaml (paths that won't be considered by git diff)
* Added *create_apt_cache.yaml* workflow to cache APT requirements each time a commit is pushed on selected branch and **when the requirements file has changed**.

### Changes

Expand All @@ -13,3 +13,5 @@
* isort 5.13.2 -> 6.0.1
* pylint-django 2.5.5 -> 2.6.1
* pylint 3.2.6 -> 3.3.5
* Removed `awalsh128/cache-apt-pkgs-action@latest` action and rewrote APT caching using GitHub's `actions/cache/restore@v4` and `actions/cache/save@v4`.
* Added both frontend and backend exclusions on _detect_changes.yaml (paths that won't be considered by git diff)
53 changes: 41 additions & 12 deletions actions/apt_requirements/action.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,25 +1,54 @@
name: Composite action install apt requirements
name: Composite action install apt requirements
description: Composite action install apt requirements
inputs:
working_directory:
description: Working directory
required: true
requirements_file:
description: Requirements file
required: true

runs:
using: "composite"
steps:
- name: Export apt requirements
id: export-apt-requirements
- name: Compute APT requirements file SHA256 hash
id: compute_apt_requirements_file_sha256_hash
run: |
COMPUTED_HASH=$(sha256sum ${{ inputs.requirements_file }} | cut -d ' ' -f 1)
echo "cache_key=$COMPUTED_HASH" >> $GITHUB_OUTPUT
shell: bash

# Vital to be able to restore cache
# If write permission is not set, a permissions error will be raised
- name: Modification to /var/cache/apt/archives permissions
run: |
PKG=$(cat ${{ inputs.requirements_file }})
echo apt_packages=$PKG | awk '{print}' ORS=' ' >> $GITHUB_OUTPUT
sudo chmod a+w /var/cache/apt/archives
shell: bash

- name: Cache apt packages
id: cache-apt-packages
uses: awalsh128/cache-apt-pkgs-action@latest
- uses: actions/cache/restore@v4
id: restore_cache_from_parent_branch
with:
path: /var/cache/apt/archives/*.deb
key: ${{ github.base_ref }}-${{ steps.compute_apt_requirements_file_sha256_hash.outputs.cache_key }}

- uses: actions/cache/restore@v4
id: restore_cache_from_current_branch
if: steps.restore_cache_from_parent_branch.outputs.cache-hit != 'true'
with:
path: /var/cache/apt/archives/*.deb
key: ${{ github.ref_name }}-${{ steps.compute_apt_requirements_file_sha256_hash.outputs.cache_key }}

- name: Refresh repositories
if: steps.restore_cache_from_parent_branch.outputs.cache-hit != 'true' && steps.restore_cache_from_current_branch.outputs.cache-hit != 'true'
run: |
sudo apt-get update
shell: bash

- name: Install requirements
run: |
sudo apt-get install -y --no-install-recommends $(tr '\n' ' ' < ${{ inputs.requirements_file }})
shell: bash

- uses: actions/cache/save@v4
id: cache_apt_requirements_for_current_branch
if: steps.restore_cache_from_parent_branch.outputs.cache-hit != 'true' && steps.restore_cache_from_current_branch.outputs.cache-hit != 'true'
with:
packages: ${{ steps.export-apt-requirements.outputs.apt_packages }}
path: /var/cache/apt/archives/*.deb
key: ${{ github.ref_name }}-${{ steps.compute_apt_requirements_file_sha256_hash.outputs.cache_key }}
39 changes: 39 additions & 0 deletions workflows/create_apt_cache.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
name: Create APT cache

# GitHub will remove any cache entries that have not been accessed in over 7 days.

on:
push:
branches:
- main
- master
- develop
- dev
paths:
# Path to APT requirements file
- '.github/test/python_test/packages.txt'


jobs:
create-cache:
name: Create cache for dependencies
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4

# Remember to set the same APT requirements file path set before!
- name: Install dependencies
run: |
sudo apt-get update
sudo apt-get -y install --no-install-recommends $(tr '\n' ' ' < apt_packages.txt)

- name: Compute apt_packages.txt SHA256 hash
id: compute_apt_packages_sha256_hash
run: |
COMPUTED_HASH=$(sha256sum apt_packages.txt | cut -d ' ' -f 1)
echo "cache_key=$COMPUTED_HASH" >> $GITHUB_OUTPUT

- uses: actions/cache/save@v4
with:
path: /var/cache/apt/archives/*.deb
key: ${{ github.ref_name }}-${{ steps.compute_apt_packages_sha256_hash.outputs.cache_key }}
Loading