Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
182 changes: 110 additions & 72 deletions .github/workflows/build-and-release.yml
Original file line number Diff line number Diff line change
Expand Up @@ -24,187 +24,225 @@
- 'ALPACA_README.md'

jobs:
build:
runs-on: self-hosted
prepare:
name: Prepare Build Metadata
runs-on: ubuntu-latest
if: github.event_name != 'pull_request'
outputs:
should_release: ${{ steps.check_release.outputs.should_release }}
tag: ${{ steps.version.outputs.tag }}
branch: ${{ steps.version.outputs.branch }}
release_type: ${{ steps.version.outputs.release_type }}
docker_tag: ${{ steps.docker_tags.outputs.tag }}
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Cache pip packages
# Skip cache for self-hosted runners (files persist locally)
if: ${{ runner.name != 'git01' }}
uses: actions/cache@v4
with:
path: |
~/.cache/pip
/root/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}
restore-keys: |
${{ runner.os }}-pip-

- name: Determine Docker tags and cache
id: docker_tags
run: |
BRANCH_NAME="${GITHUB_REF#refs/heads/}"

if [ "$BRANCH_NAME" = "main" ]; then
echo "tag=latest" >> $GITHUB_OUTPUT
echo "cache_key=${{ runner.os }}-buildx-main-${{ github.sha }}" >> $GITHUB_OUTPUT
echo "cache_restore=${{ runner.os }}-buildx-main-" >> $GITHUB_OUTPUT
else
echo "tag=dev" >> $GITHUB_OUTPUT
echo "cache_key=${{ runner.os }}-buildx-dev-${{ github.sha }}" >> $GITHUB_OUTPUT
echo "cache_restore=${{ runner.os }}-buildx-dev-" >> $GITHUB_OUTPUT
fi

- name: Cache Docker layers
# Skip GitHub Actions cache for self-hosted runners (files persist locally)
if: ${{ runner.name != 'git01' }}
uses: actions/cache@v4
with:
path: /tmp/.buildx-cache
key: ${{ steps.docker_tags.outputs.cache_key }}
restore-keys: |
${{ steps.docker_tags.outputs.cache_restore }}
${{ runner.os }}-buildx-

- name: Determine if release should be created
id: check_release
run: |
# Create release on push to main or dev branch
if [ "${{ github.event_name }}" = "push" ]; then
echo "should_release=true" >> $GITHUB_OUTPUT
BRANCH_NAME="${GITHUB_REF#refs/heads/}"
if [ "$BRANCH_NAME" = "main" ]; then
echo "📦 Release will be created (push to main)"
else
echo "📦 Test release will be created (push to $BRANCH_NAME)"
fi
else
echo "should_release=false" >> $GITHUB_OUTPUT
echo "🔨 Build only (no release)"
fi

- name: Get version tag for release
id: version
if: steps.check_release.outputs.should_release == 'true'
run: |
# Fetch all tags to ensure we have complete tag history
git fetch --tags --force

# Determine branch-specific tag prefix
BRANCH_NAME="${GITHUB_REF#refs/heads/}"
echo "Branch: $BRANCH_NAME"

if [ "$BRANCH_NAME" = "main" ]; then
TAG_PREFIX="v"
TAG_PATTERN="v[0-9]*.[0-9]*"
RELEASE_TYPE="stable"
else
TAG_PREFIX="v-${BRANCH_NAME}-"
TAG_PATTERN="v-${BRANCH_NAME}-[0-9]*.[0-9]*"
RELEASE_TYPE="test"
fi

# Get all matching tags and find the highest version
echo "Looking for tags matching pattern: ${TAG_PATTERN}"
LATEST_TAG=$(git tag -l "${TAG_PATTERN}" | sort -V | tail -n1)

if [ -z "$LATEST_TAG" ]; then
# No existing tags, start at 0.1
MAJOR=0
MINOR=0
echo "No existing tags found, starting at ${TAG_PREFIX}0.1"
else
echo "Latest tag: $LATEST_TAG"
# Extract version numbers (remove prefix first)
VERSION="${LATEST_TAG#${TAG_PREFIX}}"
IFS='.' read -ra PARTS <<< "$VERSION"
MAJOR=${PARTS[0]:-0}
MINOR=${PARTS[1]:-0}
echo "Current version: $MAJOR.$MINOR"
fi

# Increment minor version by 1 (0.1 increments)
MINOR=$((MINOR + 1))
NEW_TAG="${TAG_PREFIX}${MAJOR}.${MINOR}"

# Ensure the new tag doesn't already exist (keep incrementing if it does)
ATTEMPTS=0
while git rev-parse "$NEW_TAG" >/dev/null 2>&1; do
echo "⚠️ Tag $NEW_TAG already exists, incrementing..."
MINOR=$((MINOR + 1))
NEW_TAG="${TAG_PREFIX}${MAJOR}.${MINOR}"
ATTEMPTS=$((ATTEMPTS + 1))
if [ $ATTEMPTS -gt 100 ]; then
echo "❌ Error: Too many version increment attempts"
exit 1
fi
done

echo "✅ New tag: $NEW_TAG"
echo "tag=$NEW_TAG" >> $GITHUB_OUTPUT
echo "release_type=$RELEASE_TYPE" >> $GITHUB_OUTPUT
echo "branch=$BRANCH_NAME" >> $GITHUB_OUTPUT

build:
Comment thread Dismissed
name: Build ${{ matrix.platform_tag }}
needs: prepare
runs-on: ${{ matrix.runner }}
strategy:
fail-fast: false
matrix:
include:
- runner: git01
platform: linux/amd64
platform_tag: amd64
- runner: gitpi01
platform: linux/arm64
platform_tag: arm64
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
fetch-depth: 0

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Set cache paths
run: |
echo "CACHE_PATH=${HOME}/.cache/buildx" >> $GITHUB_ENV
echo "CACHE_PATH_NEW=${HOME}/.cache/buildx-new" >> $GITHUB_ENV

- name: Cache Docker layers
# Skip GitHub Actions cache for self-hosted runners (files persist locally)
if: ${{ runner.name != 'git01' && runner.name != 'gitpi01' }}
uses: actions/cache@v4
with:
# CHANGE: Use a path in the home directory, not /tmp
path: ~/.cache/buildx
key: ${{ runner.os }}-buildx-${{ github.ref_name }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-${{ github.ref_name }}-
${{ runner.os }}-buildx-

- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ steps.docker_tags.outputs.tag }}
cache-from: type=local,src=/tmp/.buildx-cache
# CHANGED: mode=min speeds up export by only caching final layers, avoiding massive I/O
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=min
platforms: linux/amd64,linux/arm64
# Push to a temporary tag specific to the architecture
tags: ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ needs.prepare.outputs.docker_tag }}-${{ matrix.platform_tag }}
# Use persistent cache on self-hosted runner
cache-from: type=local,src=${{ env.CACHE_PATH }}
cache-to: type=local,dest=${{ env.CACHE_PATH_NEW }},mode=min
platforms: ${{ matrix.platform }}

- name: Move cache
if: always()
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
rm -rf ${{ env.CACHE_PATH }}
mv ${{ env.CACHE_PATH_NEW }} ${{ env.CACHE_PATH }}

merge:
Comment thread Dismissed
name: Merge Multi-Arch Image
needs: [prepare, build]
runs-on: ubuntu-latest
steps:
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Create and push manifest list
# Combines the amd64 and arm64 tags into the single tag (latest or dev)
run: |
docker buildx imagetools create -t ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ needs.prepare.outputs.docker_tag }} \
${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ needs.prepare.outputs.docker_tag }}-amd64 \
${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ needs.prepare.outputs.docker_tag }}-arm64

- name: Inspect image
run: |
docker buildx imagetools inspect ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ needs.prepare.outputs.docker_tag }}

- name: Get image size
if: github.ref == 'refs/heads/dev'
run: |
docker pull ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ needs.prepare.outputs.docker_tag }}
IMAGE_SIZE=$(docker images --format "{{.Size}}" ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ needs.prepare.outputs.docker_tag }} | head -n1)
echo "Docker image size: $IMAGE_SIZE"

- name: Checkout repository
if: github.ref == 'refs/heads/main'
uses: actions/checkout@v4

- name: Docker Hub Description
if: github.ref == 'refs/heads/main'
uses: peter-evans/dockerhub-description@v4
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}
repository: ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect
short-description: "ML-based cloud detection for AllSky cameras with MQTT and ASCOM Alpaca"
readme-filepath: ./readme.md

- name: Get image size
if: github.ref == 'refs/heads/dev'
run: |
docker pull ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ steps.docker_tags.outputs.tag }}
IMAGE_SIZE=$(docker images --format "{{.Size}}" ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:${{ steps.docker_tags.outputs.tag }} | head -n1)
echo "Docker image size: $IMAGE_SIZE"

release:

Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {contents: read}
needs: build
if: needs.build.outputs.should_release == 'true'
needs: [prepare, merge]
if: needs.prepare.outputs.should_release == 'true'
runs-on: ubuntu-latest
permissions:
contents: write
Expand All @@ -219,7 +257,7 @@
id: changelog
run: |
# Get the previous tag for this branch (excluding the tag we're about to create)
BRANCH_NAME="${{ needs.build.outputs.branch }}"
BRANCH_NAME="${{ needs.prepare.outputs.branch }}"

if [ "$BRANCH_NAME" = "main" ]; then
TAG_PREFIX="v"
Expand All @@ -229,7 +267,7 @@
TAG_PATTERN="${TAG_PREFIX}*"
fi

NEW_TAG="${{ needs.build.outputs.tag }}"
NEW_TAG="${{ needs.prepare.outputs.tag }}"

# Get all matching tags, exclude the new tag if it exists, and get the latest
# For main branch, also exclude dev tags (v-dev-*) to prevent incorrect comparisons
Expand Down Expand Up @@ -264,32 +302,32 @@
git config user.email "github-actions[bot]@users.noreply.github.com"

# Check if tag exists (should not happen due to version increment logic)
if git rev-parse "${{ needs.build.outputs.tag }}" >/dev/null 2>&1; then
echo "❌ Error: Tag ${{ needs.build.outputs.tag }} already exists"
if git rev-parse "${{ needs.prepare.outputs.tag }}" >/dev/null 2>&1; then
echo "❌ Error: Tag ${{ needs.prepare.outputs.tag }} already exists"
echo "This should not happen - version increment logic failed"
exit 1
else
git tag ${{ needs.build.outputs.tag }}
git push origin ${{ needs.build.outputs.tag }}
echo "✅ Created and pushed tag ${{ needs.build.outputs.tag }}"
git tag ${{ needs.prepare.outputs.tag }}
git push origin ${{ needs.prepare.outputs.tag }}
echo "✅ Created and pushed tag ${{ needs.prepare.outputs.tag }}"
fi

- name: Create Release
uses: softprops/action-gh-release@v2
with:
tag_name: ${{ needs.build.outputs.tag }}
name: ${{ needs.build.outputs.release_type == 'test' && format('🧪 Test Release {0} ({1} branch)', needs.build.outputs.tag, needs.build.outputs.branch) || format('Release {0}', needs.build.outputs.tag) }}
prerelease: ${{ needs.build.outputs.release_type == 'test' }}
tag_name: ${{ needs.prepare.outputs.tag }}
name: ${{ needs.prepare.outputs.release_type == 'test' && format('🧪 Test Release {0} ({1} branch)', needs.prepare.outputs.tag, needs.prepare.outputs.branch) || format('Release {0}', needs.prepare.outputs.tag) }}
prerelease: ${{ needs.prepare.outputs.release_type == 'test' }}
files: |
README.md
ALPACA_README.md
body: |
${{ needs.build.outputs.release_type == 'test' && '## 🧪 Test Release' || '## 📦 SimpleCloudDetect' }}
${{ needs.build.outputs.tag }}
${{ needs.prepare.outputs.release_type == 'test' && '## 🧪 Test Release' || '## 📦 SimpleCloudDetect' }}
${{ needs.prepare.outputs.tag }}

${{ needs.build.outputs.release_type == 'test' && format('⚠️ **This is a pre-release test build from the `{0}` branch.**', needs.build.outputs.branch) || '' }}
${{ needs.build.outputs.release_type == 'test' && '**Use for testing purposes only. For stable releases, use builds from the main branch.**' || '' }}
${{ needs.build.outputs.release_type == 'test' && '' || 'ML-based cloud detection for AllSky cameras with MQTT and ASCOM Alpaca SafetyMonitor support.' }}
${{ needs.prepare.outputs.release_type == 'test' && format('⚠️ **This is a pre-release test build from the `{0}` branch.**', needs.prepare.outputs.branch) || '' }}
${{ needs.prepare.outputs.release_type == 'test' && '**Use for testing purposes only. For stable releases, use builds from the main branch.**' || '' }}
${{ needs.prepare.outputs.release_type == 'test' && '' || 'ML-based cloud detection for AllSky cameras with MQTT and ASCOM Alpaca SafetyMonitor support.' }}

### Documentation
- [README.md](README.md) - Complete setup and features
Expand Down
86 changes: 58 additions & 28 deletions .github/workflows/snd.yml
Original file line number Diff line number Diff line change
Expand Up @@ -10,67 +10,97 @@

jobs:
build:
runs-on: self-hosted
name: Build ${{ matrix.platform_tag }}
runs-on: ${{ matrix.runner }}
strategy:
fail-fast: false
matrix:
include:
- runner: git01
platform: linux/amd64
platform_tag: amd64
- runner: gitpi01
platform: linux/arm64
platform_tag: arm64
steps:
- name: Checkout repository
uses: actions/checkout@v4
with:
ref: snd

- name: Set up QEMU
uses: docker/setup-qemu-action@v3

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Cache pip packages
# Skip cache for self-hosted runners (files persist locally)
if: ${{ runner.name != 'git01' }}
uses: actions/cache@v4
with:
path: |
~/.cache/pip
/root/.cache/pip
key: ${{ runner.os }}-pip-${{ hashFiles('requirements*.txt') }}
restore-keys: |
${{ runner.os }}-pip-

- name: Set cache paths
run: |
echo "CACHE_PATH=${HOME}/.cache/buildx" >> $GITHUB_ENV
echo "CACHE_PATH_NEW=${HOME}/.cache/buildx-new" >> $GITHUB_ENV

- name: Cache Docker layers
# Skip GitHub Actions cache for self-hosted runners (files persist locally)
if: ${{ runner.name != 'git01' }}
if: ${{ runner.name != 'git01' && runner.name != 'gitpi01' }}
uses: actions/cache@v4
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-snd-${{ github.sha }}
# CHANGE: Use a path in the home directory, not /tmp
path: ~/.cache/buildx
key: ${{ runner.os }}-buildx-${{ github.ref_name }}-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-snd-
${{ runner.os }}-buildx-${{ github.ref_name }}-
${{ runner.os }}-buildx-

- name: Build and push
uses: docker/build-push-action@v5
with:
context: .
push: true
tags: ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd
cache-from: type=local,src=/tmp/.buildx-cache
# OPTIMIZATION: Changed mode=max to mode=min to speed up export on self-hosted runners
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=min
platforms: linux/amd64,linux/arm64
# Push to a temporary tag specific to the architecture (e.g., :snd-arm64)
tags: ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd-${{ matrix.platform_tag }}
# Use persistent cache on self-hosted runner
cache-from: type=local,src=${{ env.CACHE_PATH }}
cache-to: type=local,dest=${{ env.CACHE_PATH_NEW }},mode=min
platforms: ${{ matrix.platform }}

- name: Move cache
if: always()
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
rm -rf ${{ env.CACHE_PATH }}
mv ${{ env.CACHE_PATH_NEW }} ${{ env.CACHE_PATH }}

merge:
Comment thread Dismissed
name: Merge Multi-Arch Image
needs: build
runs-on: ubuntu-latest
steps:
- name: Login to Docker Hub
uses: docker/login-action@v3
with:
username: ${{ vars.DOCKERHUB_USERNAME }}
password: ${{ secrets.DOCKERHUB_TOKEN }}

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3

- name: Create and push manifest list
# Combines the amd64 and arm64 tags into the single ':snd' tag
run: |
docker buildx imagetools create -t ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd \
${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd-amd64 \
${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd-arm64

- name: Inspect image
run: |
docker buildx imagetools inspect ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd

- name: Get image size
run: |
docker pull ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd
IMAGE_SIZE=$(docker images --format "{{.Size}}" ${{ vars.DOCKERHUB_USERNAME }}/simpleclouddetect:snd | head -n1)
echo "Docker image size: $IMAGE_SIZE"
echo "Docker image size: $IMAGE_SIZE"


Check warning

Code scanning / CodeQL

Workflow does not contain permissions Medium

Actions job or workflow does not limit the permissions of the GITHUB_TOKEN. Consider setting an explicit permissions block, using the following as a minimal starting point: {}

Copilot Autofix

AI 7 months ago

In general, fix this by explicitly specifying a minimal permissions: block for the workflow or for each job, reducing GITHUB_TOKEN to read‑only access (or disabling unused scopes) while still allowing the current steps to succeed.

The best targeted fix here is to add a permissions: block at the workflow root level (just under name: or on:), so it applies to both build and merge jobs. None of the steps require write access to the repository; they only need to read the code (handled by actions/checkout) and use external Docker credentials. Therefore, we can set contents: read and disable everything else by using permissions: read-all. For maximum clarity and least privilege, we can explicitly use permissions: read-all, which is a valid shortcut meaning “all scopes read‑only”. This will satisfy CodeQL by explicitly constraining GITHUB_TOKEN while not changing the existing behavior of the jobs.

Concretely:

  • Edit .github/workflows/snd.yml.
  • Insert a permissions: read-all block near the top, after the on: block (or directly after name:; both are valid, but we’ll put it after on: to keep triggers and permissions grouped).
  • No imports or extra methods are needed; this is pure YAML configuration.
Suggested changeset 1
.github/workflows/snd.yml

Autofix patch

Autofix patch
Run the following command in your local git repository to apply this patch
cat << 'EOF' | git apply
diff --git a/.github/workflows/snd.yml b/.github/workflows/snd.yml
--- a/.github/workflows/snd.yml
+++ b/.github/workflows/snd.yml
@@ -8,6 +8,8 @@
       - '*.md'
       - 'images/**'
 
+permissions: read-all
+
 jobs:
   build:
     name: Build ${{ matrix.platform_tag }}
EOF
@@ -8,6 +8,8 @@
- '*.md'
- 'images/**'

permissions: read-all

jobs:
build:
name: Build ${{ matrix.platform_tag }}
Copilot is powered by AI and may make mistakes. Always verify output.
Unable to commit as this autofix suggestion is now outdated
Loading