Skip to content

Commit b2c51cb

Browse files
claude[bot]claude
andauthored
ci(board-snapshot): mirror the board archive to Cloudflare R2 (#17998)
Fixes #17989 One step in `.github/workflows/board-snapshot.yml`, right after 「Commit and push the archive」: it mirrors the archive checkout to Cloudflare R2 with `--delete` and, once per UTC day, writes the tree as a tarball. `scripts/pm/board-snapshot.mjs` is untouched and so is what the snapshot reads and commits. Why: the `board-archive` branch survives an account suspension, because a branch belongs to the repository and not to a user — it does not survive an action against the repository itself. The maintainer's answer, verbatim and untranslated: 「那如果推 s3 呢?」「Cloudflare R2」. It lands before the bucket exists: with any of `R2_ACCOUNT_ID` / `R2_BUCKET` / `R2_ACCESS_KEY_ID` / `R2_SECRET_ACCESS_KEY` unset the step prints one notice and exits 0, while a CONFIGURED upload that fails goes red. `pull_request` runs never upload, the same guard the archive commit carries. No secret value is in the diff, the log or the summary, and no dependency is added — `aws` is the ubuntu-latest runner's own CLI. Two decisions to overrule if you disagree: the daily tarball goes to `objectstack/snapshots/`, OUTSIDE the mirrored `objectstack/board/` prefix, because inside it the next `--delete` sync deletes it and the history is one day long; and `.git` is excluded under two patterns, because in a linked worktree it is a FILE that `.git/*` does not match. The step's shell, exactly as it lands: ```bash set -euo pipefail if [ -z "${AWS_ACCESS_KEY_ID:-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY:-}" ] \ || [ -z "${R2_ACCOUNT_ID:-}" ] || [ -z "${R2_BUCKET:-}" ]; then echo "status=skipped — no R2 credentials configured in this repository" >> "$GITHUB_OUTPUT" echo "::notice::R2 upload skipped: R2_ACCOUNT_ID, R2_BUCKET, R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY are not all set as repository secrets; this run is archived on the board-archive branch only." exit 0 fi endpoint="https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com" mirror="s3://${R2_BUCKET}/objectstack/board/" # `archive/` is a linked worktree, so its `.git` is a FILE and the # pattern `.git/*` does not match it; both spellings are excluded. # No pipe between the command and `$?` — see the snapshot step. set +e aws s3 sync archive/ "$mirror" \ --endpoint-url "$endpoint" \ --delete --no-progress \ --exclude '.git' --exclude '.git/*' \ > "$RUNNER_TEMP/r2-sync.log" 2>&1 code=$? set -e if [ "$code" != "0" ]; then cat "$RUNNER_TEMP/r2-sync.log" >&2 echo "status=FAILED — aws s3 sync exited $code" >> "$GITHUB_OUTPUT" echo "::error::aws s3 sync to R2 exited $code. This run IS archived on board-archive; the out-of-GitHub copy is now behind and every later run will stay behind until this is fixed." exit 1 fi # One line per object moved. The first configured run moves the whole # board and every later run moves a handful, so the log is truncated # here and the counts below are taken over all of it. transfers=$(wc -l < "$RUNNER_TEMP/r2-sync.log") head -n 50 "$RUNNER_TEMP/r2-sync.log" if [ "$transfers" -gt 50 ]; then echo "... and $((transfers - 50)) further transfer line(s), not printed." fi uploaded=$(grep -c '^upload:' "$RUNNER_TEMP/r2-sync.log" || true) deleted=$(grep -c '^delete:' "$RUNNER_TEMP/r2-sync.log" || true) day=$(date -u +%Y-%m-%d) tarball="s3://${R2_BUCKET}/objectstack/snapshots/${day}.tar.gz" if aws s3 ls "$tarball" --endpoint-url "$endpoint" > /dev/null 2>&1; then tarred="already written for ${day}" else tar -czf "$RUNNER_TEMP/${day}.tar.gz" --exclude=.git -C archive . aws s3 cp "$RUNNER_TEMP/${day}.tar.gz" "$tarball" \ --endpoint-url "$endpoint" --no-progress tarred="written for ${day}" fi echo "status=synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}" >> "$GITHUB_OUTPUT" echo "R2: synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}" ``` Verified offline with a stand-in on PATH in place of `aws` — nothing in this branch ever contacted R2. Every case ran the shell above as a YAML parser extracted it, never as retyped text: | case | result | |---|---| | four secrets empty | exactly one `::notice::` line, exit 0, zero `aws` invocations | | three set, one empty | same | | configured, no tarball for today | sync, `s3 ls` miss, `tar`, `s3 cp`; `status=synced 2 object(s), deleted 1; daily tarball written for 2026-09-13` | | configured, today's tarball present | sync only; status says `already written` | | configured, sync exits 3 | log to stderr, one `::error::` line, exit 1 | | 60 objects moved | log truncated at 50 lines, counts taken over all 60 | The tarball built in the third case holds `./board/...` and no `.git`. Gates: `node scripts/pm/dispatch-gates.mjs --commands --repo objectstack-ai/objectstack` derived 35 families for this diff; all 35 ran, all 35 exited 0, reconciled through `--ran` (35 accounted, 0 NOT-MEASURED, 0 UNRUN). The nine workflow-roster gates the tool cannot place for a `.github/workflows` path were run as well: eight exited 0, and `check:single-claim-paths` exits 2 as NOT WIRED without a `PR_NUMBER` — its declared list holds only `.objectui-sha`, which this diff does not touch. `skip-changeset`: `.github/**` publishes nothing from any released package. ## Acceptance notes - The mirrored keys read `objectstack/board/board/...`, because the sync source is the archive checkout and that branch's tree root holds a `board/` directory. Keeping the ruling's source path is what makes the mirror and the daily tarball the same shape, both rooted at the branch tip. - `AWS_REQUEST_CHECKSUM_CALCULATION: when_required` is set for the S3-compatible endpoint and is NOT measured here — nothing in this run contacted R2. The first configured run is its measurement. --- _Generated by [Claude Code](https://claude.ai/code/session_01DAcomhvR9kKizeYgg89Vo8)_ Co-authored-by: Claude <noreply@anthropic.com>
1 parent 4f1a56b commit b2c51cb

1 file changed

Lines changed: 109 additions & 0 deletions

File tree

‎.github/workflows/board-snapshot.yml‎

Lines changed: 109 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -274,6 +274,113 @@ jobs:
274274
echo "committed=${files}" >> "$GITHUB_OUTPUT"
275275
echo "pushed ${files} changed file(s) to board-archive"
276276
277+
# ## The copy that outlives the platform, not just the account
278+
#
279+
# The archive branch above survives an ACCOUNT suspension, because a
280+
# branch belongs to the repository rather than to a user. It does not
281+
# survive an action against the organisation or the repository itself,
282+
# and that is the remaining hole. The maintainer's answer, verbatim and
283+
# untranslated, in chat on 2026-09-13:
284+
#
285+
# 「那如果推 s3 呢?」 「Cloudflare R2」
286+
#
287+
# So this step mirrors the archive CHECKOUT — the same tree the commit
288+
# above just pushed — into one R2 prefix. Two things follow from R2
289+
# having NO bucket versioning:
290+
#
291+
# - `--delete` makes the prefix equal to the branch tip and nothing
292+
# else, so a restore never reads a file the board no longer has;
293+
# - a mirror therefore keeps no HISTORY at all, so once per UTC day the
294+
# tree is also written as `snapshots/YYYY-MM-DD.tar.gz`. That key
295+
# lives OUTSIDE the mirrored prefix on purpose: inside it, the very
296+
# next `--delete` sync would remove it as an object with no local
297+
# counterpart, and the history would be exactly one day long.
298+
#
299+
# The daily tarball is decided by asking the bucket for today's key
300+
# rather than by the clock, so a missed 02:07 run, a re-run and a
301+
# `workflow_dispatch` all converge on one tarball per day.
302+
#
303+
# ## Unset secrets are a notice, not a failure
304+
#
305+
# The bucket and its scoped API token are the maintainer's to create.
306+
# Until all four secrets exist this step prints one notice and exits 0,
307+
# so this workflow keeps archiving to the branch and nothing here has to
308+
# be merged in the same hour as the bucket. A CONFIGURED upload that then
309+
# fails is the opposite case and goes red: a backup that silently stops
310+
# copying is the failure mode this whole file exists to prevent.
311+
#
312+
# ⛔ The four secrets are read, never echoed, and never written to the
313+
# summary; the run below also never touches the board, `main`, or the
314+
# archive branch — it only reads the checkout that is already on disk.
315+
- name: Upload the archive to R2
316+
id: r2
317+
if: github.event_name != 'pull_request'
318+
env:
319+
AWS_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
320+
AWS_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
321+
# R2 is S3-compatible with one fixed pseudo-region.
322+
AWS_DEFAULT_REGION: auto
323+
# S3-compatible endpoints other than S3 itself are the documented
324+
# case for this setting: the CLI then sends an integrity checksum
325+
# only where the API requires one, instead of on every request.
326+
AWS_REQUEST_CHECKSUM_CALCULATION: when_required
327+
R2_ACCOUNT_ID: ${{ secrets.R2_ACCOUNT_ID }}
328+
R2_BUCKET: ${{ secrets.R2_BUCKET }}
329+
run: |
330+
set -euo pipefail
331+
332+
if [ -z "${AWS_ACCESS_KEY_ID:-}" ] || [ -z "${AWS_SECRET_ACCESS_KEY:-}" ] \
333+
|| [ -z "${R2_ACCOUNT_ID:-}" ] || [ -z "${R2_BUCKET:-}" ]; then
334+
echo "status=skipped — no R2 credentials configured in this repository" >> "$GITHUB_OUTPUT"
335+
echo "::notice::R2 upload skipped: R2_ACCOUNT_ID, R2_BUCKET, R2_ACCESS_KEY_ID and R2_SECRET_ACCESS_KEY are not all set as repository secrets; this run is archived on the board-archive branch only."
336+
exit 0
337+
fi
338+
339+
endpoint="https://${R2_ACCOUNT_ID}.r2.cloudflarestorage.com"
340+
mirror="s3://${R2_BUCKET}/objectstack/board/"
341+
342+
# `archive/` is a linked worktree, so its `.git` is a FILE and the
343+
# pattern `.git/*` does not match it; both spellings are excluded.
344+
# No pipe between the command and `$?` — see the snapshot step.
345+
set +e
346+
aws s3 sync archive/ "$mirror" \
347+
--endpoint-url "$endpoint" \
348+
--delete --no-progress \
349+
--exclude '.git' --exclude '.git/*' \
350+
> "$RUNNER_TEMP/r2-sync.log" 2>&1
351+
code=$?
352+
set -e
353+
if [ "$code" != "0" ]; then
354+
cat "$RUNNER_TEMP/r2-sync.log" >&2
355+
echo "status=FAILED — aws s3 sync exited $code" >> "$GITHUB_OUTPUT"
356+
echo "::error::aws s3 sync to R2 exited $code. This run IS archived on board-archive; the out-of-GitHub copy is now behind and every later run will stay behind until this is fixed."
357+
exit 1
358+
fi
359+
# One line per object moved. The first configured run moves the whole
360+
# board and every later run moves a handful, so the log is truncated
361+
# here and the counts below are taken over all of it.
362+
transfers=$(wc -l < "$RUNNER_TEMP/r2-sync.log")
363+
head -n 50 "$RUNNER_TEMP/r2-sync.log"
364+
if [ "$transfers" -gt 50 ]; then
365+
echo "... and $((transfers - 50)) further transfer line(s), not printed."
366+
fi
367+
uploaded=$(grep -c '^upload:' "$RUNNER_TEMP/r2-sync.log" || true)
368+
deleted=$(grep -c '^delete:' "$RUNNER_TEMP/r2-sync.log" || true)
369+
370+
day=$(date -u +%Y-%m-%d)
371+
tarball="s3://${R2_BUCKET}/objectstack/snapshots/${day}.tar.gz"
372+
if aws s3 ls "$tarball" --endpoint-url "$endpoint" > /dev/null 2>&1; then
373+
tarred="already written for ${day}"
374+
else
375+
tar -czf "$RUNNER_TEMP/${day}.tar.gz" --exclude=.git -C archive .
376+
aws s3 cp "$RUNNER_TEMP/${day}.tar.gz" "$tarball" \
377+
--endpoint-url "$endpoint" --no-progress
378+
tarred="written for ${day}"
379+
fi
380+
381+
echo "status=synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}" >> "$GITHUB_OUTPUT"
382+
echo "R2: synced ${uploaded} object(s), deleted ${deleted}; daily tarball ${tarred}"
383+
277384
- name: Publish the run to the summary
278385
if: always()
279386
run: |
@@ -282,6 +389,8 @@ jobs:
282389
echo
283390
echo "Committed: ${{ steps.commit.outputs.committed || 'nothing (the run stopped before the commit step)' }}"
284391
echo
392+
echo "R2: ${{ steps.r2.outputs.status || 'not attempted (a pull_request run, or the job stopped before the upload step)' }}"
393+
echo
285394
echo '```'
286395
cat "$RUNNER_TEMP/snapshot.md" 2>/dev/null || echo '(no report produced)'
287396
echo '```'

0 commit comments

Comments
 (0)