Skip to content

ci: squash gh-pages history on every release #15743

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
susnux merged 1 commit into from
Oct 8, 2026
Merged

ci: squash gh-pages history on every release #15743

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
136 changes: 136 additions & 0 deletions .github/workflows/squash-gh-pages.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
# SPDX-FileCopyrightText: 2026 Nextcloud GmbH and Nextcloud contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
#
# Squash the gh-pages history on every release.
#
# Every deploy commits a full set of rebuilt PDF and ePub files, so gh-pages
# grows by ~18 MiB a day. build/squash-gh-pages.sh rebuilds the branch as one
# snapshot commit per release day, keeps the deploys made since the last
# release, and stores out of support version folders only once. The published
# tree is unchanged; see the script for details.
#
# Required setup (repository admin):
# GH_PAGES_DEPLOY_KEY – private half of a deploy key with write access, added
# as a bypass actor to the "Pages" ruleset, which
# otherwise blocks force pushes to gh-pages.
name: "Squash gh-pages history"

on:
push:
tags:
- 'v*'
pull_request:
paths:
- '.github/workflows/squash-gh-pages.yml'
- 'build/squash-gh-pages.sh'
- 'build/tests/**'
workflow_dispatch:
inputs:
dry_run:
description: "Compute the rewrite without pushing it"
type: boolean
default: true

permissions:
contents: read

concurrency:
group: squash-gh-pages
cancel-in-progress: false

jobs:
test:
name: Test squash script
runs-on: ubuntu-latest

steps:
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
persist-credentials: false

- name: Run unit tests
run: bash build/tests/test-squash-gh-pages.sh

squash:
name: Squash gh-pages
needs: test
# Final releases only: rc, beta and alpha tags do not get a snapshot.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event_name == 'push' && !contains(github.ref_name, 'rc') &&
!contains(github.ref_name, 'beta') && !contains(github.ref_name, 'alpha'))
runs-on: ubuntu-latest

permissions:
contents: read

steps:
# Full history: the release dates are read from the tags.
- name: Checkout repository
uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
with:
ref: ${{ github.event.repository.default_branch }}
fetch-depth: 0
persist-credentials: false

- name: Setup PHP
uses: shivammathur/setup-php@f3e473d116dcccaddc5834248c87452386958240 # v2.37.2
with:
php-version: '8.3'

- name: Detect lowest supported version
id: versions
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
mapfile -t nums < <(git ls-remote --heads origin "heads/stable[0-9][0-9]" \
| sed -n 's?.*refs/heads/stable\([0-9]\{2\}\)$?\1?p' | sort -n)
eval "$(php build/detect-versions.php "${nums[@]}")"
echo "Folders below server/${lowest_stable} are frozen"
echo "lowest_stable=${lowest_stable}" >> "$GITHUB_OUTPUT"

- name: Fetch gh-pages history
run: git fetch --no-tags origin +refs/heads/gh-pages:refs/heads/gh-pages

- name: Rewrite gh-pages
id: rewrite
run: |
echo "old=$(git rev-parse gh-pages)" >> "$GITHUB_OUTPUT"
build/squash-gh-pages.sh gh-pages "${{ steps.versions.outputs.lowest_stable }}" \
refs/heads/gh-pages-squashed | tee -a "$GITHUB_OUTPUT"

# Deploy PRs are based on the old history: merging one after the rewrite
# would bring every squashed commit back. The next nightly deploy opens a
# fresh one on top of the new history.
- name: Close open deploy pull requests
if: steps.rewrite.outputs.changed == 'true' && inputs.dry_run != true
env:
GH_TOKEN: ${{ secrets.COMMAND_BOT_PAT }}
run: |
gh pr list --repo "${{ github.repository }}" --base gh-pages --state open \
--json number,headRefName \
--jq '.[] | select(.headRefName | startswith("automated/deploy/")) | .number' \
| while read -r number; do
gh pr close "${number}" --repo "${{ github.repository }}" --delete-branch \
--comment "Closed because gh-pages was squashed. The next scheduled deploy reopens it on the new history."
done

# The lease makes the push fail if anything landed on gh-pages since the
# fetch, rather than dropping it. Re-run the workflow in that case.
- name: Push rewritten gh-pages
if: steps.rewrite.outputs.changed == 'true' && inputs.dry_run != true
env:
DEPLOY_KEY: ${{ secrets.GH_PAGES_DEPLOY_KEY }}
run: |
key="${RUNNER_TEMP}/gh-pages-deploy-key"
hosts="${RUNNER_TEMP}/github-known-hosts"
install -m 600 /dev/null "${key}"
printf '%s\n' "${DEPLOY_KEY}" > "${key}"
# GitHub's published host keys, fetched over TLS rather than trusted on first use.
curl -fsSL https://api.github.com/meta | jq -r '.ssh_keys[] | "github.com " + .' > "${hosts}"
GIT_SSH_COMMAND="ssh -i ${key} -o IdentitiesOnly=yes -o UserKnownHostsFile=${hosts} -o StrictHostKeyChecking=yes" git push \
--force-with-lease="refs/heads/gh-pages:${{ steps.rewrite.outputs.old }}" \
"git@github.com:${{ github.repository }}.git" \
refs/heads/gh-pages-squashed:refs/heads/gh-pages
rm -f "${key}"
141 changes: 141 additions & 0 deletions build/squash-gh-pages.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
#!/usr/bin/env bash
# SPDX-FileCopyrightText: 2026 Nextcloud GmbH and Nextcloud contributors
# SPDX-License-Identifier: AGPL-3.0-or-later
#
# Rewrite the gh-pages history to keep it from growing without bounds.
#
# Every deploy commits a full set of rebuilt PDF and ePub files, so the branch
# grows by tens of MiB per deploy even when the docs barely change. The built
# output has no value as history, so this script rebuilds the branch as:
#
# - one snapshot commit per release day (final vX.Y.Z tags only), holding the
# tree of the last deploy made before the end of that day (UTC),
# - every deploy made after the last release, unchanged,
# - version folders below <frozen-below> (out of support, no more releases)
# replaced by their current content in every commit, so they are stored once.
#
# The tip tree is never modified, so the published site is identical. The
# rewrite is deterministic: running it again on its own output yields the same
# commits, which makes it safe to run on every release tag.
#
# Usage: squash-gh-pages.sh <branch> <frozen-below> <output-ref>
#
# Prints changed=true|false and tip=<sha> for $GITHUB_OUTPUT, and a commit
# count summary on stderr.

set -euo pipefail

# Group final release tags by UTC day.
# stdin: "<tag> <unix timestamp>" lines
# stdout: "<end of day timestamp> <tag>[,<tag>...]" lines, oldest day first
release_days() {
grep -E '^v[0-9]+\.[0-9]+\.[0-9]+ ' \
| awk '{ print int($2 / 86400) * 86400 + 86399, $1 }' \
| LC_ALL=C sort -k1,1n -k2,2 \
| awk '$1 == day { tags = tags "," $2; next }
day { print day, tags }
{ day = $1; tags = $2 }
END { if (day) print day, tags }'
}

# Pick the commits to keep.
# $1: file of "<sha> <committer timestamp>" lines, oldest first
# $2: file of release_days output
# stdout: "<sha> <tags>" for a snapshot, "<sha> -" for an unreleased commit
plan() {
awk 'NR == FNR { sha[NR] = $1; ts[NR] = $2; n = NR; next }
{
last = 0
for (i = 1; i <= n; i++) if (ts[i] <= $1) last = i
if (!last) next
if (last in tags) tags[last] = tags[last] "," $2
else { order[++m] = last; tags[last] = $2 }
}
END {
for (k = 1; k <= m; k++) print sha[order[k]], tags[order[k]]
for (i = (m ? order[m] + 1 : 1); i <= n; i++) print sha[i], "-"
}' "$1" "$2"
}

# ls-tree lines of server/<N> folders below $2 in tree-ish $1 (or the others with $3 = keep)
version_entries() {
git ls-tree "$1" 2>/dev/null | awk -F '\t' -v below="$2" -v keep="${3:-}" '
{ frozen = $2 ~ /^[0-9]+$/ && $2 + 0 < below }
frozen != (keep == "keep")'
}

# Tree of commit $1 with every frozen server/<N> folder replaced by its tip content.
frozen_tree() {
local server
server=$( { version_entries "$1:server" "$FROZEN_BELOW" keep; printf '%s' "$FROZEN_ENTRIES"; } \
| awk NF | git mktree)
{
git ls-tree "$1^{tree}" | awk -F '\t' '$2 != "server"'
if [ "$server" != "$(git mktree </dev/null)" ]; then
printf '040000 tree %s\tserver\n' "$server"
fi
} | git mktree
}

# Commit tree $2 on parent $3 (may be empty) with the metadata of commit $1.
# stdin: commit message
commit_like() {
local header author committer author_email committer_email
header=$(git cat-file commit "$1" | sed '/^$/q')
# "Name <email> <timestamp> <tz>"
author=$(sed -n 's/^author //p' <<<"$header")
committer=$(sed -n 's/^committer //p' <<<"$header")
author_email=${author##*<}
committer_email=${committer##*<}
GIT_AUTHOR_NAME=${author% <*} GIT_AUTHOR_EMAIL=${author_email%%>*} GIT_AUTHOR_DATE=${author##*> } \
GIT_COMMITTER_NAME=${committer% <*} GIT_COMMITTER_EMAIL=${committer_email%%>*} GIT_COMMITTER_DATE=${committer##*> } \
git commit-tree "$2" ${3:+-p "$3"}
}

# Rebuild branch $1 with folders below $2 frozen, print the new tip (the branch is not moved).
rewrite() {
local branch=$1 tmp parent='' sha tags tree
FROZEN_BELOW=$2
FROZEN_ENTRIES=$(version_entries "$branch:server" "$FROZEN_BELOW")
tmp=$(mktemp -d)

git log --first-parent --reverse --format='%H %ct' "$branch" >"$tmp/commits"
git for-each-ref --format='%(refname:short) %(creatordate:unix)' refs/tags \
| { release_days || true; } >"$tmp/days"

while read -r sha tags; do
tree=$(frozen_tree "$sha")
if [ "$tags" = - ]; then
parent=$(git cat-file commit "$sha" | sed '1,/^$/d' | commit_like "$sha" "$tree" "$parent")
else
parent=$(printf 'chore: documentation snapshot for %s\n\nSquashed gh-pages history up to this release.\n' \
"${tags//,/, }" | commit_like "$sha" "$tree" "$parent")
fi
done < <(plan "$tmp/commits" "$tmp/days")

rm -rf "$tmp"
echo "$parent"
}

main() {
if [ $# -ne 3 ]; then
echo "Usage: $0 <branch> <frozen-below> <output-ref>" >&2
exit 1
fi
local branch=$1 tip
tip=$(rewrite "$branch" "$2")

if [ "$(git rev-parse "$tip^{tree}")" != "$(git rev-parse "$branch^{tree}")" ]; then
echo "Rewritten tip tree differs from the current branch, aborting" >&2
exit 1
fi
git update-ref "$3" "$tip"

echo "$branch: $(git rev-list --count --first-parent "$branch") commits -> $(git rev-list --count "$tip") commits" >&2
if [ "$tip" = "$(git rev-parse "$branch")" ]; then echo changed=false; else echo changed=true; fi
echo "tip=$tip"
}

if [ "${BASH_SOURCE[0]}" = "$0" ]; then
main "$@"
fi
Loading
Loading