Skip to content

Retry ports tree synchronisation on transient mirror failures - #4

Open
jolierabideau wants to merge 3 commits into
fix-version-assertionfrom
sync-retry
Open

jolierabideau wants to merge 3 commits into
fix-version-assertionfrom
sync-retry

Conversation

@jolierabideau

@jolierabideau jolierabideau commented Sep 10, 2026

Copy link
Copy Markdown

Problem

macOS CI jobs in paranext-core have been failing at the Update MacPorts ports tree step:

--->  Updating the ports tree
Error: Synchronization of the local ports tree failed doing rsync
port sync failed: Synchronization of 1 source failed

The default ports tree source, rsync://rsync.macports.org/macports/release/tarballs/ports.tar, is served by a rotating pool of volunteer mirrors. When a runner draws an unresponsive one, rsync hangs until the connection drops and then fails.

Evidence that this is a stalled mirror and not a configuration problem:

Run Time Sync
34475657170 12:14
34478788344 12:48 ✅ 13s
34480385432 13:04 ❌ 75s
34483764921 13:37
34483979256 13:39 ❌ 75s

A healthy sync takes ~13 seconds; the failures burned 75 seconds each, which is a stall rather than a rejection. The mirror answered normally from outside CI throughout.

Change

Synchronise the ports tree as part of the installation, with retries:

  • sync_ports in subr/macports.sh — up to 4 attempts, delays increasing 10s / 20s / 30s. The last attempt runs port -d sync so a persistent failure leaves debug output in the log without paying for an extra 142 MB download.
  • write_rsync_options — sets rsync_options in macports.conf with --timeout=60 --contimeout=15. This is what makes retrying worthwhile; without it, 4 attempts against a stalled mirror would be a 5-minute step. The MacPorts default flags -rtzvl --delete-after are carried over, so only the timeouts are added.
  • Wired into install_macports after the install (which is when macports.conf first exists) and before install_ports, which needs a populated tree anyway.

Consumers no longer need their own port sync step; paranext-core gets a follow-up PR to drop its step and bump the action ref.

Tests

New testsuite/modules/macports, covering the retry loop with a stubbed port and sleep so it needs no network:

  • a successful sync is not retried
  • a sync that fails twice then succeeds makes exactly 3 attempts
  • a sync that always fails gives up after the configured attempt count and exits non-zero
  • delays increase between attempts, and there is no delay after the final attempt
  • rsync_options is written when absent, keeps the MacPorts defaults, and replaces any existing setting without disturbing the rest of the file

Each test was watched failing before the implementation existed, and mutation-checked afterwards — forcing attempts=1, dropping the timeouts, and flattening the backoff each kill exactly the tests they should.

write_rsync_options was also run against the real macports.conf from macports-base to confirm it removes the commented #rsync_options line, appends ours, and leaves rsync_server / rsync_dir untouched.

Notes

  • Ports tree caching was considered and rejected. A healthy sync is 13 seconds; restoring a 142 MB tree under /opt/local with permission fixups would not pay for itself.
  • Mirror pinning / fallback lists were rejected in favour of staying on the official endpoint and letting DNS rotation do its job across retries.
  • The MacPorts version is not implicated. paranext-core pins MacPorts 2.11.5 while this repo defaults to 2.12.5, but the only sync-related change in 2.12.x is a tarball extraction optimisation. The rsync client is macOS's, and the failure is mirror-side.

🤖 Generated with Claude Code


This change is Reviewable

jolierabideau and others added 2 commits September 10, 2026 10:17
The default ports tree source, rsync://rsync.macports.org, is served by
a rotating pool of volunteer mirrors. When a runner draws an
unresponsive mirror, `port sync` hangs until the connection drops and
then fails with

    Error: Synchronization of the local ports tree failed doing rsync

A healthy synchronisation takes about 13 seconds, so the failure is a
stalled mirror rather than a configuration problem, and it clears on
its own. Consumers of this action were working around it with their own
`port sync` step, which had the same single-attempt behaviour.

Synchronise the ports tree as part of the installation instead, and
retry it a few times with an increasing delay. Set connection and
transfer timeouts in `rsync_options` so a stalled mirror fails promptly
rather than holding the step open, which is what makes retrying
worthwhile. Run the last attempt with debug output so a persistent
failure leaves something to triage in the workflow log.

The rsync_options default is carried over from macports.conf rather
than rewritten, so `-rtzvl --delete-after` is preserved and only the
timeouts are added.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The --contimeout option was added in rsync 3.1.0. macOS 14 runners ship
rsync 2.6.9, which rejects it outright, so configuring it made every
synchronisation attempt fail with

    rsync: --contimeout=15: unknown option
    rsync error: syntax or usage error (code 1) [client=2.6.9]

Recent macOS versions ship openrsync, which does accept it, so this
could not be caught on a single runner. Keep --timeout, which all of
them support and which covers the stalled transfer this is meant to
guard against; a connection that is never established fails through
the TCP timeout anyway.

Cover this with a test that runs the configured options through the
rsync the runner actually provides, so an unsupported option is caught
by the testsuite rather than by an installation four minutes later.
The implementations word the rejection differently: rsync 2.6.9 says
"unknown option" while openrsync says "unrecognized option".

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@jolierabideau

Copy link
Copy Markdown
Author

CI is red for a pre-existing reason, not because of this PR

The Install MacPorts action step in this PR succeeds — it syncs the ports tree and installs db48, gmp, dbus and expat with their dependencies (macos-14 job). The new testsuite passes on macos-14, macos-15, macos-26 and ubuntu-latest.

What fails is a later, unrelated step:

Run test "$(port version)" = 'Version: 2.12.5'
##[error]Process completed with exit code 1

This fails on unmodified main too — confirmed by dispatching the workflow there: run 34489699502, same step, same assertion.

Cause (recorded here since repository issues are disabled)

The MacPorts .pkg postflight runs port -v selfupdate (portmgr/dmg/postflight.in), which upgrades base to the newest release. We install the pinned MacPorts-2.12.5-*.pkg (verified 7,417,131 bytes, matching the v2.12.5 asset), and the postflight then upgrades base to 2.12.6, released 2026-08-25.

The assertion only ever passed because the pin happened to equal the latest release — true from 2026-04-23 until 2026-08-25. The last CI run on main was 2026-06-24, so this went unnoticed.

The broader point: the documented version: parameter does not currently pin MacPorts base at all.

Follow-up options, for a separate PR

  1. Suppress the postflight selfupdate so version: genuinely pins. This became practical because of this PR — the ports tree is now synced explicitly and with retries, so the postflight's own sync is redundant. Likely also cuts most of the ~4 minute install time, since the postflight rebuilds base from source. Deserves its own design discussion.
  2. Relax the assertion to treat the requested version as a floor. Smallest change, but contradicts the README.
  3. Bump the pin to 2.12.6. Green today, breaks on the next MacPorts release.

Also worth noting

The postflight's selfupdate failure is explicitly non-fatal — it prints a message and the install still reports success. So before this PR the ports tree sync was silently best-effort, which is why consumers such as paranext-core needed their own port sync step. This PR makes that sync verified and retried instead.

@jolierabideau
jolierabideau changed the base branch from main to fix-version-assertion September 10, 2026 15:10
@jolierabideau

Copy link
Copy Markdown
Author

Restacked onto #5 so CI runs with both fixes present.

The two PRs were blocking each other: this one failed on the stale port version assertion (#5's subject), while #5 failed on the very sync flakiness this PR fixes — run 34493037442 died with installer: The install was successful. immediately followed by Error: Port db48 not found, which is the postflight's silent sync failure this PR is about.

Base is now fix-version-assertion; GitHub will retarget it to main automatically when #5 merges. Review #5 first.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant